Compare commits

..
6 Commits
Author SHA1 Message Date
simon d4cfcc8f56 chore: 移除项目级 MCP/Serena 配置(.mcp.json、.serena/) 2026-08-05 20:51:45 +08:00
simon 6ec198687c feat(djapi): 新增日报查询 API(news/reports + news/events)及文档
- api/report/ 包:query(连库+SQL)/ views(2 视图)/ serializers(OpenAPI)/ tests(17 单测)
- urls.py 注册 news/reports/、news/events/;settings.py SPECTACULAR 加「日报」tag
- .env.example 补 NEWS_DB_* 占位配置;README/continuation.md 更新
- docs/news_report_api.md 使用手册;CLAUDE*.md 修正 CLI 路径为 finance/ 前缀
2026-08-05 20:51:42 +08:00
simonandSimon 22ce3a6aea security: 安全规范 + 敏感数据脱敏
- CLAUDE.md 新增安全规范章节(禁止提交 / 必须提供 .env.example)
- finance/config/settings.py 硬编码密码移除
- djapi/api/video/audioRead.py API Key 替换为占位符
- 新增 finance/.env.example 示例配置
- .gitignore 解除 .env.example 排除

Co-Authored-By: Simon <simon@doorcome.cn>
2026-06-17 21:20:07 +08:00
simonandSimon bb1a9c4470 fix: video 模块与 continuation 状态更新
Co-Authored-By: Simon <simon@doorcome.cn>
2026-06-17 20:48:42 +08:00
simonandClaude Opus 4.7 2f1d8b4d03 feat: djapi 数据源归一化 + bug 修复 + 废弃 getDivData_AK
- 新增 djapi/api/stock/data_source.py 统一数源入口 (Tushare 单例)
- 迁移 10 个模块至统一数据源入口
- 废弃 getDivData_AK.py
- 修复 getStockDiv2.py / smoothBrush.py 等模块
- indexDatas API 参数 tscode 类型修正 (股票→指数代码)
- views.py + urls.py 接口清理
- continuation.md 状态更新

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-16 15:01:04 +08:00
simonandClaude Opus 4.7 271a9343a5 Initial commit: cc-cursor 全链路量化研究平台
7 Sprints 全部完成:
  Sprint 0: 基础设施 (DataManager + MariaDB)
  Sprint 1: 因子引擎 (34因子/12分类)
  Sprint 2: VectorBT 回测 (5策略+截面)
  Sprint 3: Optuna 优化 (+Walk-Forward)
  Sprint 4: ML 模型 (LightGBM+CatBoost)
  Sprint 5: Qwen 情绪因子 (三源新闻+日期对齐)
  Sprint 6: Agent 系统 (4Agent+日报.md/.html)

生产加固 (15项): Tushare双源fallback, SSH自动恢复, pool_pre_ping,
  save_daily先删后插, load_dotenv绝对路径, 日报5d/20d修复,
  RiskAgent改上证指数, 昨日对比+数据截止, mac_report utf8mb4,
  CLAUDE-*.md 9条已知Bug, demo全参数化, djapi数据源归一化,
  indexDatas API修正

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-07 15:59:05 +08:00
301 changed files with 60592 additions and 236 deletions
+26
View File
@@ -0,0 +1,26 @@
# batch-sync skill
批量预热股票数据到 DB 缓存。
## 触发
用户说:预热缓存 / 同步数据 / warmup / batch sync / 补齐数据 / 全量同步
## 执行
```bash
cd finance && python cli/agent_cli.py warmup 50
```
## 说明
- 每批 50 只股票,依次执行 `dm.sync_daily()`
- 已缓存 + 最新的 → 0 条跳过(增量)
- 未缓存 → Tushare 优先 → AkShare fallback
- 范围由 `.env``SENTIMENT_SCOPE_TYPE` + `SENTIMENT_SCOPE_INDEXES` 决定(默认沪深300+中证500
- 可多次执行直到覆盖率 100%
## 参数
`python cli/agent_cli.py warmup [N]`
- N: 每批股票数,默认 50。网络稳定时可调大到 100
+62
View File
@@ -0,0 +1,62 @@
# Python
__pycache__/
*.pyc
*.pyo
*.egg-info/
dist/
build/
# Virtual environments
.venv/
venv/
env/
# IDE
.idea/
.vscode/
*.swp
*.swo
# OS
.DS_Store
Thumbs.db
# Cache
.cache/
*.parquet
# Sensitive
.env
*.token
credentials.*
# Env examples are safe to commit
!.env.example
# Database
*.sqlite3
*.db
# Data files
*.csv.gz
finance/factors/sentiment/.cache/
# MCP / Serena cache
.serena/cache/
# Reports output (keep reports/ directory but ignore generated files)
finance/reports/daily_*.md
finance/reports/daily_*.html
# Node (if any)
node_modules/
# Jinja2 templates cache
__jinja2_*
# IPython
.ipynb_checkpoints/
# Claude Code local
.claude/local/
mcp-servers/serena
+84
View File
@@ -0,0 +1,84 @@
# CLAUDE-agents.md — Agent 系统 + CLI
## Agent 架构
4 个 Agent,通过依赖注入复用已有引擎(不重建轮子)。
```python
from agents.orchestrator import AgentOrchestrator
orch = AgentOrchestrator(dm=dm, fe=fe, bt=bt, opt=opt, sent=sent)
orch.setup() # 注册 4 个 Agent
results = orch.run_daily() # 5 步流程
```
## 5 步每日流程
```
[Step 1/5] 增量同步 → 只更新已缓存股票 (0.04s/只)
[Step 2/5] 风险评估 → RiskAgent: high/medium/low + 仓位
[Step 3/5] 股票打分 → SelectionAgent: 多因子等权打分
[Step 4/5] 情绪因子 → SentimentEngine.compute()
[Step 5/5] 生成日报 → ReportAgent: .md + .html + 解读
```
## 各 Agent 职责
| Agent | 文件 | 职责 |
|-------|------|------|
| ResearchAgent | `research_agent.py` | 因子 IC/IC_IR 评估 |
| SelectionAgent | `selection_agent.py` | 多因子股票打分(等权) |
| RiskAgent | `risk_agent.py` | 波动率+回撤→仓位建议 |
| ReportAgent | `report_agent.py` | 市场+选股+情绪+风险→日报 |
## CLI (`finance/cli/agent_cli.py`)
```bash
python finance/cli/agent_cli.py daily # 5 步流程
python finance/cli/agent_cli.py picks 15 # 选股
python finance/cli/agent_cli.py risk # 风险评估
python finance/cli/agent_cli.py research # 因子研究
python finance/cli/agent_cli.py report 20260603 # 生成日报
python finance/cli/agent_cli.py warmup 50 # 首次预热缓存
```
## Demo 脚本 (`finance/cli/demo_*.py`)
全部支持 `--ts_code` `--date` 等参数。
| 脚本 | 用途 |
|------|------|
| `demo_data_manager.py` | Sprint 0: DB→建表→日线→同步 |
| `demo_factor_engine.py` | Sprint 1: 34因子→NaN检查→截面 |
| `demo_backtest.py` | Sprint 2: 5策略回测 |
| `demo_optimizer.py` | Sprint 3: Optuna寻优+Walk-Forward |
| `demo_ml.py` | Sprint 4: LightGBM+CatBoost+回测 |
| `demo_sentiment.py` | Sprint 5: 情绪因子快速验证 |
| `demo_sentiment_detail.py` | Sprint 5: 情绪因子详细演示(14参数) |
## 报告 (`finance/reports/`)
- `daily_YYYYMMDD.md` + `.html` — 每日双格式输出
- `storage.py``save_report()` / `query_reports()` → 存入 mac_report 表
- 日报包含"昨日对比"区块:标注数据是否与前一日相同
## ⚠️ 已知 Bug 速查(避免重复踩坑)
### B6: 5 日/20 日涨跌幅恒为 0
- **症状**: 日报中 `| +0.00% | +0.00% |`,实际数据应该有非零值
- **根因**: `idx = daily.index.get_loc(date) if date in daily.index else -1`,目标日期不在索引时 `-1 >= 5` → False,计算被短路
- **修复**: 用 `pos = len(daily) - 1` 替代 `idx = -1`,确保位置值非负
### B7: 两日报告完全一致
- **症状**: 20260604 和 20260605 报告的 TOP 15、风险评估完全一样
- **根因**: ① T+1 数据未产出,两份报告基于同一份 DB 快照 ② 多因子在 1 天内变化极小(非 bug,是特性)
- **缓解**: 日报增加"昨日对比"区块 + 数据截止标注
### B8: 风险回撤恒为 -52%
- **症状**: 不管哪天看,风险等级一直是 high,回撤一直是 -52%
- **根因**: RiskAgent 默认用 `market_index="000001.SZ"`(平安银行个股),其 2020 年历史高价导致永远极端回撤
- **修复**: 改为 `"000001.SH"`(上证指数)
### B9: report 命令只 sync 一只股票
- **症状**: generate_report 只调了 `dm.sync_daily("000001.SZ")`,其他 279 只不变
- **影响**: 日报的选股排名不会反映最新行情
- **状态**: 已加 sync 调用,范围仍为单只(全量 sync 走 `daily` 命令或 `warmup`
+59
View File
@@ -0,0 +1,59 @@
# CLAUDE-backtest.md — 回测引擎 + 参数优化
## VectorBTEngine (`finance/backtest/vectorbt/engine.py`)
只做多,10万/万三。
```python
from backtest.vectorbt.engine import VectorBTEngine
engine_bt = VectorBTEngine(initial_capital=100_000, commission=0.0003)
report = engine_bt.run(strategy, price_df, factor_df)
# → BacktestReport
report = engine_bt.run_cross_section(strategy, price_univ, factor_univ)
```
信号流:`1=buy, 0=sell, -1=hold``_signals_to_entries` → vbt.Portfolio.from_signalsdirection="longonly")。
## 策略 (`finance/backtest/strategies/`)
| 策略 | 参数 | 逻辑 |
|------|------|------|
| `SMACrossStrategy` | fast=5, slow=20 | 金叉买/死叉卖 |
| `RSIMeanRevertStrategy` | oversold=30, overbought=70 | 超卖买/超买卖 |
| `MomentumBreakoutStrategy` | lookback=20, exit=10 | 新高买/跌破卖 |
| `FactorCrossStrategy` | factor_column, buy/sell_threshold | 阈值交叉(通用) |
| `FactorRotationStrategy` | factor_name, top_n=5 | 排序选股 |
## 自定义策略
继承 `backtest/base.py:BaseStrategy`,实现 `generate_signals(factor_df) → pd.Series`
## 信号工具 (`backtest/signal.py`)
```python
factor_to_threshold_signal(series, buy, sell, direction)
cross_signal(fast, slow) # 金叉/死叉
factor_to_quantile_signal(...) # 分位数信号
```
## BacktestReport (`backtest/report.py`)
字段:total_return, cagr, max_drawdown, sharpe_ratio, calmar_ratio, annual_volatility, win_rate, profit_factor, total_trades, avg_hold_days, best/worst_trade_pct, equity_curve, drawdown_curve, monthly_returns, trades_df, stats_dict。`summary()` 一行摘要。
## OptunaEngine (`finance/optimizer/engine.py`)
```python
from optimizer.engine import OptunaEngine
from optimizer.space import rsi_revert_space
opt = OptunaEngine(bt_engine)
result = opt.optimize(StrategyClass, space, price_df, factor_df, metric="sharpe", n_trials=200)
# → OptimizationResult(best_params, best_value, best_report, trial_df, param_importance)
wf = opt.optimize_walk_forward(StrategyClass, space, price_df, factor_df,
train_window=756, test_window=252)
```
预置空间:`sma_cross_space`, `rsi_revert_space`, `momentum_breakout_space`, `factor_cross_space``optimizer/space.py`)。
目标指标:sharpe/cagr/calmar/total_return/return_over_dd。
+99
View File
@@ -0,0 +1,99 @@
# CLAUDE-data.md — 数据层 + 数据库
## DataManager (`finance/data/data_manager.py`)
双数据源:Tushare(优先)→ AkSharefallback)。DB 缓存优先。
```python
from data.data_manager import DataManager
dm = DataManager()
dm.init_db() # 首次建表(幂等)
```
### 关键方法
```python
stocks = dm.get_stock_list() # → 5524 只, DB 优先
daily = dm.get_daily("000001.SZ") # DB优先 → Tushare → AkShare
fina = dm.get_financial("000001.SZ") # 同花顺 + fallback
n = dm.sync_daily("000001.SZ") # 增量同步: latest>=today → 跳过
```
### 指数 vs 个股路由
`_try_fetch()` 自动检测 `is_index_code()`
- `000001.SH` / `399001.SZ``fetch_index_daily()`Tushare `index_daily` / AkShare `index_zh_a_hist`
- `000001.SZ` / `600519.SH``fetch_daily()`stock daily
### 数据源 (`finance/data/sources/`)
- `akshare_source.py``AkShareSource`(个股+指数+财务)
- `tushare_source.py``TushareSource`(需 `TUSHARE_TOKEN``available=False` 自动跳过)
## 数据库 (`finance/database/`)
### 连接
```python
from database.connection import get_engine, test_connection
# get_engine() 自动检测断连 → 运行 autossh.sh → 重建引擎
```
### 表结构(mac_ 前缀)
| 表 | 内容 | 主键 |
|----|------|------|
| `mac_stock_basic` | A股列表, 5524只 | ts_code |
| `mac_stock_daily` | 日线 OHLCV | (ts_code, trade_date) |
| `mac_stock_financial` | 财务指标 | (ts_code, end_date) |
| `mac_report` | 报告持久化 | id |
### DAO (`finance/database/dao.py`)
- `_DAILY_COLS` / `_FINA_COLS` — 入库前字段筛选
- `get_latest_trade_date(ts_code)` → 增量判断
- `save_report()` / `query_reports()` → 报告管理
### 配置
```bash
# SSH 隧道
bash shared/script/autossh.sh
# host: 127.0.0.1:13306 user: myquant database: myquant
```
## 缓存策略
- `sync_daily`: 已缓存且最新 → 0.04s 跳过
- 未缓存 → 提示 `agent_cli.py warmup`
- 每日增量只更新已缓存股票,未缓存统计跳过
## ⚠️ 已知 Bug 速查(避免重复踩坑)
### B1: `.env` 加载路径
- **症状**: Tushare available=False, QWEN_API_KEY 读不到
- **根因**: `load_dotenv()` 不传路径,从 CWD 找 .env,而非 `finance/`
- **修复**: `config/settings.py``Path(__file__).parent.parent / ".env"`
- **加固**: `tushare_source.py`, `qwen_client.py`, `news_source.py` 顶部 `import config.settings`
### B2: SSH 重连只生效一次
- **症状**: 第一次断连能恢复,第二次断连无法恢复
- **根因**: `_ssh_auto_attempted` 全局变量设 True 后永不重置
- **修复**: 移除该变量。`get_engine()` 每次断连都触发 `_reconnect_ssh()``dispose()` → 重建
### B3: 连接池半开连接
- **症状**: `_test_engine()` 返回 True 但实际查询报 `_read_bytes` 超时
- **根因**: 池中有活连接也有死连接,test 取到活的,查询取到死的
- **修复**: `pool_pre_ping=True` + `pool_recycle=600` + 断连时 `_engine.dispose()`
### B4: save_daily 主键冲突
- **症状**: `IntegrityError: Duplicate entry '000001.SZ-20260603'`
- **根因**: Tushare 返回的数据含已存在的日期,`append` 模式遇主键冲突
- **修复**: 写入前 `DELETE FROM mac_stock_daily WHERE ts_code IN (...) AND trade_date IN (...)`
### B5: 数据不更新排查清单
- 检查 SSH 隧道: `lsof -i :13306 | grep LISTEN`
- 检查 Tushare: `python -c "from data.sources.tushare_source import TushareSource; print(TushareSource().available)"`
- 检查最新数据: `from database.dao import get_latest_trade_date; print(get_latest_trade_date('000001.SZ'))`
- 手动触发 sync: `dm.sync_daily('000001.SZ')`
- 预期行为: T+1 数据产出,节假日无新数据属于正常
+55
View File
@@ -0,0 +1,55 @@
# CLAUDE-factors.md — 因子引擎 + 情绪因子
## FactorEngine (`finance/factors/engine.py`)
```python
from factors.registry import get_factor, list_factors
from factors.engine import FactorEngine
fe = FactorEngine(dm, sentiment_engine=sent)
factor_df = fe.compute("000001.SZ", [get_factor("momentum_20"), get_factor("rsi_14")])
# → DataFrame: index=trade_date, columns=[momentum_20, rsi_14]
cross = fe.compute_universe(factors, date="20250630", ts_codes=[...])
```
因子路由:`factor.category == "sentiment"` → SentimentEngine`isinstance(f, FUNDAMENTAL_FACTOR_TYPES)` → 注入财务数据。
## 因子注册表 (`finance/factors/registry.py`)
34 因子 / 12 分类:动量、RSI、MACD、量价、布林、ATR、均线、波动率、换手率、振幅、基本面、情绪。
```python
list_factors("动量") # → ['momentum_5','momentum_10','momentum_20','momentum_60']
get_factor("rsi_14") # → RSIFactor(period=14)
```
## 技术因子 (`finance/factors/technical/`)
10 类。每个继承 `BaseFactor`,实现 `calculate(df) → pd.Series`。停牌跳过,新股不足 N 日返回 NaN。
## 基本面因子 (`finance/factors/fundamental/`)
- `roe.py` — ROEFactor(同花顺 `stock_financial_abstract_ths`
- `pe_pb.py` — PEFactor/PBFactor/EPFactorclose + 财务 EPS/BVPS map to daily
## 情绪因子 (`finance/factors/sentiment/`)
### SentimentEngine (`sentiment_engine.py`)
```python
sent = SentimentEngine(dm, qwen_client=client, news_source=news_src)
sent_df = sent.compute("000001.SZ", max_news=30)
# → DataFrame: news_sent_5, news_conf_5, sent_delta_5
```
### 新闻源 (`news_source.py`)
三数据源聚合:AkShare `stock_news_em` + DB `xwlb_daily_ext` + MCP `trendradar-news`
xwlb 日期处理:DB `news_date +1day`(晚间播出→次日影响)→ `align_news_to_trading_days`
AkShare 限频 → 自动 fallback。LIMIT 按日期跨度动态计算。
### Qwen 客户端 (`qwen_client.py`)
双后端:DashScope API + 本地 Ollama。金融情绪专家 prompt,返回 `{sentiment_score, confidence, impact_duration, key_topics}`。配置:`.env` 中的 `QWEN_API_KEY``QWEN_LOCAL_BASE_URL`
### 情绪因子 (`sentiment_factor.py`)
- `NewsSentimentFactor(window, decay)` — 时间衰减加权情绪
- `SentimentConfidenceFactor(window)` — score × confidence 加权
- `SentimentMomentumFactor(period)` — 情绪变化方向
+45
View File
@@ -0,0 +1,45 @@
# CLAUDE-ml.md — ML 模型层
## FeatureEngine (`finance/models/features.py`)
因子 → 特征矩阵 + 标签。防前视偏差。
```python
from models.features import FeatureEngine
fe = FeatureEngine(lookahead=5, label_type="regression")
X, y = fe.build(factor_df, price_df, fit=True)
# fit=True: Winsorize(1%/99%) → ffill → median fill → RobustScaler.fit → 标签计算
# fit=False: 复用训练时的 scaler + 有效特征
```
## LightGBM (`finance/models/lightgbm/model.py`)
```python
from models.lightgbm.model import LightGBMModel
model = LightGBMModel(params={"n_estimators": 200, "learning_rate": 0.03}, eval_ratio=0.2)
model.fit(X_train, y_train) # val>=50 行才启用早停
pred = model.predict(X_test)
imp = model.get_feature_importance() # → DataFrame
cv = model.cv_evaluate(X, y, n_folds=5) # TimeSeriesSplit
```
## CatBoost (`finance/models/catboost/model.py`)
同接口。`get_feature_importance()` / `cv_evaluate()`
## ML 策略 (`finance/models/backtest_integration.py`)
```python
from models.backtest_integration import MLStrategy, MLBenchmark
strategy = MLStrategy(model, fe, buy_quantile=0.7, sell_quantile=0.3, rebalance_freq=5)
# 预测值分位 → 动态阈值 → 交易信号
benchmark = MLBenchmark([lgb, cb], fe, price_df, factor_df)
result = benchmark.run() # → DataFrame: model × (IC, return, sharpe, win_rate, trades)
```
## 重要约束
- lookahead 固定,不输入模型(防目标泄露)
- 单股票 IC≈0 是正常现象(噪声主导),多股票截面才是 ML 发挥价值的地方
- 特征工程严禁使用未来数据(RobustScaler fit 在训练集,transform 在测试集)
- 交叉验证用 TimeSeriesSplit(不 shuffle
+59
View File
@@ -0,0 +1,59 @@
# CLAUDE-reference.md — 因子 + 表结构速查
## 因子速查(34 个,12 分类)
| 注册名 | 类 | 参数 | 文件 |
|--------|-----|------|------|
| `momentum_5/10/20/60` | MomentumFactor | `period=N` | `factors/technical/momentum.py` |
| `rsi_7/14` | RSIFactor | `period=N` | `factors/technical/rsi.py` |
| `macd` | MACDFactor | `fast=12,slow=26,signal=9` | `factors/technical/macd.py` |
| `macd_5_35_5` | MACDFactor | `fast=5,slow=35,signal=5` | `factors/technical/macd.py` |
| `vol_ratio_5/20` | VolumeRatioFactor | `period=N` | `factors/technical/volume.py` |
| `vol_chg_5` | VolumeChangeFactor | `period=5` | `factors/technical/volume.py` |
| `boll` | BollingerPositionFactor | `period=20` | `factors/technical/bollinger.py` |
| `boll_width` | BollingerWidthFactor | `period=20` | `factors/technical/bollinger.py` |
| `atr_14` | ATRFactor | `period=14` | `factors/technical/atr.py` |
| `atr_ratio_14` | ATRRatioFactor | `period=14` | `factors/technical/atr.py` |
| `ma_cross_5_20` | MACrossFactor | `fast=5,slow=20` | `factors/technical/ma_cross.py` |
| `ma_cross_10_60` | MACrossFactor | `fast=10,slow=60` | `factors/technical/ma_cross.py` |
| `ma_dev_20/60` | MADevFactor | `period=N` | `factors/technical/ma_cross.py` |
| `volatility_20/60` | VolatilityFactor | `period=N` | `factors/technical/volatility.py` |
| `down_vol_20` | DownsideVolatilityFactor | `period=20` | `factors/technical/volatility.py` |
| `turnover_5` | TurnoverFactor | `period=5` | `factors/technical/turnover.py` |
| `turnover_chg_5` | TurnoverChangeFactor | `period=5` | `factors/technical/turnover.py` |
| `amplitude_5/20` | AmplitudeFactor | `period=N` | `factors/technical/amplitude.py` |
| `roe` | ROEFactor | — | `factors/fundamental/roe.py` |
| `pe` | PEFactor | — | `factors/fundamental/pe_pb.py` |
| `pb` | PBFactor | — | `factors/fundamental/pe_pb.py` |
| `ep` | EPFactor | — | `factors/fundamental/pe_pb.py` |
| `news_sent_5/20` | NewsSentimentFactor | `window=N,decay=0.3` | `factors/sentiment/sentiment_factor.py` |
| `news_conf_5` | SentimentConfidenceFactor | `window=5` | `factors/sentiment/sentiment_factor.py` |
| `sent_delta_5` | SentimentMomentumFactor | `period=5` | `factors/sentiment/sentiment_factor.py` |
## DB 表速查
| 表 (mac_) | 主键 | 关键列 | 用途 |
|-----------|------|--------|------|
| `stock_basic` | ts_code | ts_code, name, area, industry | 股票列表 (5524行) |
| `stock_daily` | (ts_code, trade_date) | open, high, low, close, vol, amount, turnover_rate | 日线行情 |
| `stock_financial` | (ts_code, end_date) | eps, bvps, roe, net_profit_margin, debt_to_assets | 财务指标 |
| `report` | id | report_date, title, subject_type, subject_code, content, is_active | 报告持久化 |
## 数据源接口速查
| 接口 | AkShareSource | TushareSource |
|------|--------------|---------------|
| 股票列表 | `ak.stock_info_a_code_name()` | `pro.stock_basic()` |
| 每日行情 | `ak.stock_zh_a_hist(symbol, period='daily')` | `pro.daily(ts_code,...)` |
| 指数行情 | `ak.index_zh_a_hist(symbol, period='daily')` | `pro.index_daily(ts_code,...)` |
| 财务指标 | `ak.stock_financial_abstract_ths(symbol)` | `pro.fina_indicator(ts_code,...)` |
| 复权因子 | — | `pro.adj_factor(ts_code,...)` |
## 常用快捷入口
```python
from factors.registry import get_factor, list_factors, list_categories
from database.dao import get_latest_trade_date, query_daily, save_daily
from database.connection import get_engine, test_connection
from reports.storage import save_report, query_reports
```
+120
View File
@@ -0,0 +1,120 @@
# CLAUDE.md
cc-cursor — Mac Mini 单机量化研究平台。全链路:Data → Factor → Backtest → Optimize → ML → Sentiment → Agent。
## 速查索引
| 模块 | 详情文件 | 核心入口 |
|------|---------|---------|
| 数据层 + 数据库 | `CLAUDE-data.md` | `from data.data_manager import DataManager` |
| 因子引擎 + 情绪 | `CLAUDE-factors.md` | `from factors.registry import get_factor` |
| 回测 + 优化 | `CLAUDE-backtest.md` | `from backtest.vectorbt.engine import VectorBTEngine` |
| ML 模型 | `CLAUDE-ml.md` | `from models.lightgbm.model import LightGBMModel` |
| Agent 系统 + CLI | `CLAUDE-agents.md` | `python finance/cli/agent_cli.py daily` |
## 工作区布局
| 目录 | 内容 |
|------|------|
| `finance/` | 核心量化引擎(**代码实际位置**)。代码内 import 用顶层名 `data.*`/`factors.*` 等 — 由 CLI 把 `finance/` 加入 sys.path;文件路径为 `finance/data/xxx.py` 等 |
| `djapi/` | Django API 子项目,有独立 `djapi/CLAUDE.md` |
| `shared/script/` | `autossh.sh` — MariaDB SSH 隧道 |
| `docs/` | `usage.md` / `usage.html` 使用指南、`news_report_api.md` 新闻接口文档;**新建 md 一律放这里** |
| `finance/strategy` `portfolio` `execution` `scheduler/` | 空壳占位(仅 `__init__.py`),逻辑未落地,别误以为有实现 |
| `finance/reports/` | 日报输出 `daily_YYYYMMDD.{md,html}` |
## 环境
```bash
conda activate quant # Python 3.11.13
bash shared/script/autossh.sh # DB SSH 隧道 (本地 13306 → 远程 3306)
# 环境变量在 finance/.env(示例见 finance/.env.example):QWEN / TUSHARE / DB / 情绪范围
```
## 任务→文档路由
| 任务类型 | 先读取 |
|---------|--------|
| 数据源/数据库/cache 相关 | `CLAUDE-data.md` |
| 因子/情绪/新闻相关 | `CLAUDE-factors.md` + `CLAUDE-reference.md` |
| 回测/优化/策略相关 | `CLAUDE-backtest.md` |
| ML 模型/特征工程相关 | `CLAUDE-ml.md` |
| Agent/CLI/报告相关 | `CLAUDE-agents.md` |
| 第三方库 API/参数 | `web_fetch` / `research` 查官方文档 |
| 因子名/类名/表结构速查 | `CLAUDE-reference.md` |
## 多步任务规则
复杂任务(涉及 3+ 文件或 2+ 模块)执行前:
1. 输出执行计划清单(步骤 + 每步验证方法)
2. 每步完成后验证通过才继续
3. 遇到失败先定位根因,不跳过
## 核心设计约束(必须遵守)
- 数据流:`Data → Factor → Model → Strategy → Backtest → Report`
- 策略层禁止直接访问 AkShare/Tushare → 全部通过 `DataManager`
- 模型层禁止直接访问数据库 → 全部通过 DataManager
- 指数代码规则:`.SH`=指数, `.SZ` 开头非 399=个股
- 数据源优先级:Tushare → AkShare (fallback)
- 当日数据未缓存 → 先 `dm.sync_daily`;增量同步只更新已缓存股票
- 修改多文件前先说明:文件清单、原因、影响;优先小范围修改
## CLI 常用命令
```bash
python finance/cli/agent_cli.py daily # 5步完整流程
python finance/cli/agent_cli.py picks 15 # 选股
python finance/cli/agent_cli.py risk # 风险评估
python finance/cli/agent_cli.py research # 因子研究
python finance/cli/agent_cli.py report 20260603 # 生成日报
python finance/cli/agent_cli.py warmup 50 # 首次预热缓存
python finance/cli/demo_data_manager.py --ts_code 600519.SH
python finance/cli/demo_factor_engine.py --ts_code 300750.SZ
python finance/cli/demo_backtest.py --ts_code 000001.SZ
python finance/cli/demo_optimizer.py --ts_code 000001.SZ --trials 100
python finance/cli/demo_ml.py --ts_code 000001.SZ --lookahead 5
python finance/cli/demo_sentiment.py --ts_code 600519.SH
python finance/cli/demo_sentiment_detail.py --ts_code 600519.SH --date 20260603
```
## 技术栈
| 组件 | 技术 | 环境 |
|------|------|------|
| 数据获取 | AkShare + Tushare (双源) | conda quant |
| 数据库 | MariaDB (SSH 隧道) | mac_ 前缀表 |
| 因子/特征 | pandas / numpy / sklearn | conda quant |
| 回测 | VectorBT 1.0 | conda quant |
| 优化 | Optuna 4.9 | conda quant |
| ML | LightGBM 4.6 + CatBoost 1.2 | conda quant |
| NLP | Qwen (DashScope / Ollama) | .env 配置 |
| Agent | 自研编排器 | finance/agents/ |
| API | Django 5.2 + uWSGI | djapi/ |
## 安全规范
### 禁止提交
- `.env`(含真实 key
- API Key`sk-*``TUSHARE_TOKEN` 等)
- Cookie / Session
- Token / 密钥
- 个人隐私数据(手机号、身份证、密码)
### 必须提供
- `.env.example` — 仅含占位符的示例配置,如:
```
TUSHARE_TOKEN=your_token_here
QWEN_API_KEY=sk-your-key-here
MAC_DB_PASSWORD=your_password_here
```
### 提交前检查
```bash
grep -r "sk-\|token\|_H(lU\|password" --include="*.py" --include="*.md" --include="*.yaml" | grep -v ".example\|your_token\|your_password"
```
-235
View File
@@ -1,235 +0,0 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users.
When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software.
A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public.
The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version.
An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license.
The precise terms and conditions for copying, distribution and modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based on the Program.
To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work.
A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source.
The Corresponding Source for a work in source code form is that same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures.
When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified it, and giving a relevant date.
b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices".
c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so.
A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways:
a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b.
d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d.
A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product.
"Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made.
If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM).
The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or authors of the material; or
e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors.
All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11).
However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice.
Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party.
If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it.
A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph.
Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the work with which it is combined will remain governed by version 3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of the GNU Affero General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU Affero General Public License, you may choose any version ever published by the Free Software Foundation.
If the Program specifies that a proxy can decide which future versions of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program.
Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found.
myquant
Copyright (C) 2026 simon
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer network, you should also make sure that it provides a way for users to get its source. For example, if your program is a web application, its interface could display a "Source" link that leads users to an archive of the code. There are many ways you could offer source, and different solutions will be better for different programs; see section 13 for the specific requirements.
You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU AGPL, see <http://www.gnu.org/licenses/>.
+227 -1
View File
@@ -1,2 +1,228 @@
# myquant # cc-cursor — Mac Mini 单机量化研究平台
从数据获取 → 因子计算 → 回测 → 参数优化 → ML 模型 → 情绪因子 → Agent 系统,全链路量化研究平台。
## 架构
```
cc-cursor/
├── finance/ # 核心量化引擎
│ ├── config/ # 全局配置(MariaDB / AkShare
│ ├── database/ # ORM 模型 + DAOmac_ 前缀表)
│ ├── data/ # DataManager 统一数据层
│ ├── factors/ # 因子引擎(34 因子 / 12 分类,含情绪因子)
│ ├── backtest/ # 回测引擎(VectorBT + 5 策略 + 截面回测)
│ ├── optimizer/ # Optuna 参数优化 + Walk-Forward
│ ├── models/ # LightGBM / CatBoost ML 模型
│ ├── agents/ # Agent 系统(4 Agent + 编排器 + CLI
│ ├── cli/ # 命令行 & 验证脚本
│ ├── reports/ # 自动日报输出目录
│ └── .env # 环境变量配置(API Key / 分析范围)
├── djapi/ # Django API 后端(已有)
├── mcp-servers/ # MCP ServerSerena
├── shared/ # 共享工具(SSH 隧道脚本)
├── docs/ # 文档 & 使用指南
└── .claude/ # Claude Code 配置
```
## 数据流
```
Agent 编排层
├── ResearchAgent ── 因子发现(IC/IC_IR 评估)
├── SelectionAgent ─ 多因子打分 + ML 预测
├── RiskAgent ────── 仓位控制 + 风险预警
└── ReportAgent ──── 自动日报生成
基础引擎层
DataManager ──→ FactorEngine ──→ BaseStrategy ──→ VectorBTEngine ──→ BacktestReport
│ │ │
│ FeatureEngine OptunaEngine
│ │ │
└──────→ LightGBM/CatBoost ←────────┘
情绪增强层
NewsSource(AkShare/DB/MCP) ──→ QwenClient ──→ SentimentFactor ──→ FactorEngine
```
全部通过 Service 层中转:策略不直连 AkShare,模型不直连数据库,Agent 不重建引擎。
---
## 开发进度
| Sprint | 模块 | 关键成果 | 状态 |
|--------|------|----------|------|
| Sprint 0 | 基础设施 | DataManager + MariaDB 3 表 | ✅ |
| Sprint 1 | 因子引擎 | 34 因子 / 12 分类 | ✅ |
| Sprint 2 | 回测引擎 | VectorBT + 5 策略 + 截面回测 | ✅ |
| Sprint 3 | 参数优化 | Optuna + Walk-Forward | ✅ |
| Sprint 4 | ML 模型 | LightGBM + CatBoost + 特征工程 | ✅ |
| Sprint 5 | 情绪因子 | Qwen + 三源新闻聚合 + 日期对齐 | ✅ |
| Sprint 6 | Agent 系统 | 4 Agent + 编排器 + CLI + 自动日报 | ✅ |
**全部 7 个 Sprint 已完成。**
---
## 功能模块
### 数据层 `finance/data/`
```python
from data.data_manager import DataManager
dm = DataManager(); dm.init_db()
stocks = dm.get_stock_list() # → 5,524 只
daily = dm.get_daily("000001.SZ") # → 日线
fina = dm.get_financial("000001.SZ") # → 财务数据
dm.sync_daily("000001.SZ") # → 增量同步
```
### 因子引擎 `finance/factors/`
```python
from factors.registry import get_factor, list_factors
from factors.engine import FactorEngine
engine = FactorEngine(dm)
factors = [get_factor("momentum_20"), get_factor("rsi_14")]
factor_df = engine.compute("000001.SZ", factors)
# → 34 个注册因子,12 个分类(动量/RSI/MACD/量价/布林/ATR/均线/波动率/换手率/振幅/基本面/情绪)
```
### 回测引擎 `finance/backtest/`
```python
from backtest.vectorbt.engine import VectorBTEngine
from backtest.strategies.rsi_mean_revert import RSIMeanRevertStrategy
engine_bt = VectorBTEngine(initial_capital=100_000, commission=0.0003)
report = engine_bt.run(RSIMeanRevertStrategy(oversold=30, overbought=70), price_df, factor_df)
# → 收益=29.4% 年化=4.3% 回撤=-19.1% 夏普=0.37 胜率=77.1%
```
5 个内置策略 + 自定义策略接口 + 截面回测 + BacktestReport 标准化报告。
### 参数优化 `finance/optimizer/`
```python
from optimizer.engine import OptunaEngine
from optimizer.space import rsi_revert_space
result = OptunaEngine(engine_bt).optimize(
RSIMeanRevertStrategy, rsi_revert_space, price_df, factor_df,
metric="sharpe", n_trials=200,
)
# → 最优参数: oversold=13, overbought=66
# → 夏普: 0.37→0.60 (+62%), 回撤: -19.1%→-1.8% (10倍改善)
```
6 种优化目标 + 4 个预置搜索空间 + Walk-Forward 滚动验证 + 快捷函数。
### ML 模型 `finance/models/`
```python
from models.features import FeatureEngine
from models.lightgbm.model import LightGBMModel
fe = FeatureEngine(lookahead=5)
X, y = fe.build(factor_df, price_df, fit=True)
model = LightGBMModel(params={"n_estimators": 200}).fit(X_train, y_train)
pred = model.predict(X_test) # → IC 评估 + 特征重要性 + 交叉验证 + ML 策略回测
```
Winsorize → 缺失填充 → RobustScaler → LightGBM/CatBoost 训练 → MLBenchmark 对比。
### 情绪因子 `finance/factors/sentiment/`
```python
from factors.sentiment.sentiment_engine import SentimentEngine
sent = SentimentEngine(dm)
sent_df = sent.compute("000001.SZ", max_news=20)
# → news_sent_5, news_conf_5, sent_delta_5
```
三数据源聚合(AkShare 个股新闻 + 新闻联播 DB + MCP trendradar-news)、日期对齐(非交易日→最近交易日)、xwlb 偏移(昨日新闻→今日使用)、DashScope + Ollama 双后端。
### Agent 系统 `finance/agents/`
```bash
python finance/cli/agent_cli.py daily # 完整每日流程
python finance/cli/agent_cli.py picks 15 # 选股 Top 15
python finance/cli/agent_cli.py risk # 风险评估
python finance/cli/agent_cli.py research # 因子研究
python finance/cli/agent_cli.py report # 生成日报
```
4 个 AgentResearch/Selection/Risk/Report+ 编排器 + 自动日报(reports/daily_YYYYMMDD.md)。
---
## 快速开始
```bash
# SSH 隧道
bash shared/script/autossh.sh
# Python 环境
conda activate quant # Python 3.11.13
# 每日 Agent 运行
python finance/cli/agent_cli.py daily
```
### 验证脚本
```bash
python finance/cli/demo_data_manager.py # Sprint 0 — DataManager
python finance/cli/demo_factor_engine.py # Sprint 1 — 因子引擎
python finance/cli/demo_backtest.py # Sprint 2 — 回测引擎
python finance/cli/demo_optimizer.py # Sprint 3 — 参数优化
python finance/cli/demo_ml.py # Sprint 4 — ML 模型
python finance/cli/demo_sentiment.py # Sprint 5 — 情绪因子
```
---
## 技术栈
| 组件 | 技术 | 版本 | 状态 |
|------|------|------|------|
| 数据获取 | AkShare | 1.18 | ✅ |
| 数据库 | MariaDB (SSH 隧道) | — | ✅ |
| 因子/特征 | pandas / numpy / sklearn | 3.0 / 2.4 / 1.9 | ✅ |
| 回测引擎 | VectorBT | 1.0 | ✅ |
| 参数优化 | Optuna | 4.9 | ✅ |
| ML 模型 | LightGBM / CatBoost | 4.6 / 1.2 | ✅ |
| NLP 情绪 | Qwen (DashScope / Ollama) | turbo / 2.5 | ✅ |
| Agent 框架 | 自研编排器 | — | ✅ |
| API 后端 | Django + uWSGI | 5.2 | 已有 |
| 代码分析 | Serena MCP | — | 已配置 |
---
## 设计原则
- **模块隔离**:各引擎通过统一接口交互,可替换实现(VectorBT → Backtrader
- **接口标准化**:因子 `calculate(df)→Series` / 策略 `generate_signals(df)→Series` / 模型 `fit/predict/save/load` / 优化 `optimize()→Result`
- **数据层统一**:策略/模型不直连数据源,全部通过 DataManager
- **Agent 不重建轮子**:Agent 通过依赖注入复用已有引擎,编排而非重建
- **防前视偏差**:时间序列交叉验证、expanding window 统计量
- **渐进演进**:全链路 7 个 Sprint 平滑推进,无推倒重写
## 文档
- [使用指南](./docs/usage.md) — 详细使用说明(12 章节,含代码示例)
- [使用指南 (HTML)](./docs/usage.html) — 网页版使用指南
## 子项目
- [djapi](./djapi/README.md) — Django API 后端,A 股数据 API + 新闻联播视频处理
## 数据库连接
```bash
bash shared/script/autossh.sh
# host: 127.0.0.1:13306 user: myquant database: myquant table_prefix: mac_
```
+83
View File
@@ -0,0 +1,83 @@
# continuation.md — cc-cursor 项目状态
生成时间:2026-06-07(全部 Sprint 完成 + 生产加固 + djapi 数据源归一化 + Git 初始化)
---
## Git 状态
- 仓库:https://github.com/Simon2046/myquant
- 分支:`main`
- commit`271a934` — Initial commit: cc-cursor 全链路量化研究平台
- 文件:293 个文件,59,598 行
- 已排除:`.env``mcp-servers/serena``__pycache__``.parquet``.db`
---
## 全部 Sprint 完成 ✅
| Sprint | 模块 | 状态 |
|--------|------|------|
| 0 | 基础设施(DataManager + MariaDB | ✅ |
| 1 | 因子引擎(34 因子 / 12 分类) | ✅ |
| 2 | VectorBT 回测(5 策略 + 截面 + BacktestReport | ✅ |
| 3 | Optuna 优化(+ Walk-Forward | ✅ |
| 4 | ML 模型(LightGBM + CatBoost + MLStrategy | ✅ |
| 5 | Qwen 情绪因子(三源新闻 + 日期对齐) | ✅ |
| 6 | Agent 系统(4 Agent + CLI + 日报 .md/.html | ✅ |
---
## 生产稳定性加固(14 项)
| # | 项 | 文件 |
|---|-----|------|
| 1 | Tushare 双数据源(优先) | `finance/data/data_manager.py` |
| 2 | 指数 vs 个股自动路由 | `finance/data/sources/akshare_source.py` |
| 3 | SSH 自动恢复(多次重连 + pool_pre_ping | `finance/database/connection.py` |
| 4 | save_daily 先删后插(防主键冲突) | `finance/database/dao.py` |
| 5 | load_dotenv 绝对路径 + 模块加固 | `finance/config/settings.py` + 3 文件 |
| 6 | 日报 5d/20d 修复(idx=-1→pos=len-1 | `finance/agents/report_agent.py` |
| 7 | RiskAgent 改用上证指数 | `finance/agents/risk_agent.py` |
| 8 | 日报增加"昨日对比" + 数据截止 | `finance/agents/report_agent.py` |
| 9 | mac_report 表 utf8mb4 + DATE + DATETIME | `finance/database/models.py` |
| 10 | 日报自动存入 DB + emoji 兼容 | `finance/reports/storage.py` + 8 CLI |
| 11 | CLAUDE-*.md 文档化 9 条已知 Bug | `CLAUDE-data.md` + `CLAUDE-agents.md` |
| 12 | demo 脚本全参数化 | `finance/cli/demo_*.py` |
| 13 | **djapi 数据源归一化(10→1 入口)** | `djapi/api/stock/data_source.py` |
| 14 | **indexDatas API 参数修正 + 容错** | `djapi/api/views.py` + `getIndexs.py` |
---
## djapi 数据源归一化
- 新增 `djapi/api/stock/data_source.py` — 统一入口
- `get_tushare_pro()` — 全局单例(线程安全)
- `get_daily()` — 双源 fallback (Tushare→AkShare)
- `get_mysql_db()` — MySQL 全局单例
- Token 兼容 `TUSHARE_TS_TOKEN` / `TUSHARE_TOKEN`
- 10 个模块迁移完成
- `getDivData_AK.py` 标记废弃
- `getIndexs.py` 修复:`index_dailybasic` 失败不阻塞,异常 raise 而非静默返回空
- `views.py` 修正:`indexDatas` 参数 `index_name``tscode`,描述从"股票代码"→"指数代码",新增 `_PARAM_INDEX_CODE`
- 已部署到 `api.doorcome.cn`
---
## CLI 命令
```bash
agent_cli.py daily / picks / risk / research / report / warmup
demo_*.py(全部支持 --ts_code --date 等参数)
```
## 文档
- `CLAUDE.md` — 入口 + 路由 + 多步任务规则
- `CLAUDE-data.md` — 数据层 + 5 条已知 Bug
- `CLAUDE-factors.md` — 因子引擎
- `CLAUDE-backtest.md` — 回测 + 优化
- `CLAUDE-ml.md` — ML 模型
- `CLAUDE-agents.md` — Agent + CLI + 4 条已知 Bug
- `CLAUDE-reference.md` — 因子/表结构速查
- `docs/usage.md` + `docs/usage.html` — 使用指南
+44
View File
@@ -0,0 +1,44 @@
# Django
DJANGO_SECRET_KEY=your-secret-key-here
DJANGO_DEBUG=True
# Tushare 股票数据
TUSHARE_TS_TOKEN=your-tushare-token
# MySQL
MYSQL_HOST=localhost
MYSQL_PORT=3306
MYSQL_USER=myquant
MYSQL_PASSWORD=your-mysql-password
MYSQL_DATABASE=myquant
# 日报结构化入库 (news_report / news_event) 只读查询
# 与 report_db_design.md §7 一致;密码必填,缺失时接口直接报错
NEWS_DB_HOST=127.0.0.1
NEWS_DB_PORT=3306
NEWS_DB_USER=myquant
NEWS_DB_PASSWORD=your-news-db-password
NEWS_DB_NAME=myquant
# DeepSeek AI
DEEPSEEK_API_KEY=your-deepseek-api-key
# 阿里 DashScope (ASR + Qwen)
DASHSCOPE_API_KEY=your-dashscope-api-key
# ========================
# 以下为 video 模块 AI 模型配置
# 修改后重启 uWSGI 生效
# ========================
# DeepSeek 模型名(视频分割 + 标题提取)
# 可用: deepseek-chat(推荐,支持 response_format, deepseek-reasoner
DEEPSEEK_MODEL=deepseek-chat
# DashScope ASR 语音识别模型
# 可用: paraformer-realtime-v2, paraformer-v2
DASHSCOPE_ASR_MODEL=paraformer-realtime-v2
# DashScope 大语言模型(文本纠错 + 文本分析)
# 可用: qwen-plus, qwen-max, qwen-turbo
DASHSCOPE_LLM_MODEL=qwen-plus
+33
View File
@@ -0,0 +1,33 @@
# Python
__pycache__/
*.py[cod]
*.pyo
*.egg-info/
dist/
build/
*.egg
# Django
db.sqlite3
*.log
*.log.old
# uWSGI
uwsgi.pid
*.sock
# Environment
.env
# IDE
.vscode/
.idea/
# OS
.DS_Store
Thumbs.db
# Video artifacts
api/video/xwlb_video/*.mp3
api/video/xwlb_video/*.mp4
api/video/audio_processing/*.wav
+16
View File
@@ -0,0 +1,16 @@
{
"mcpServers": {
"serena-djapi": {
"command": "uv",
"args": [
"run",
"--directory",
"/Users/summer/Downloads/cc-cursor/mcp-servers/serena",
"serena",
"start-mcp-server",
"--project",
"/Users/summer/Downloads/cc-cursor/djapi"
]
}
}
}
+2
View File
@@ -0,0 +1,2 @@
/cache
/project.local.yml
@@ -0,0 +1,23 @@
# Code Style & Conventions
## Python
- Django app: all business logic in `api/stock/`, not in views
- views.py is thin forwarding layer: extract params -> call function -> return Response
- Double import pattern for standalone scripts: try relative import first, fall back to absolute
- Use `viewFunc_tsCodeAndDate()` wrapper for ts_code + date_range endpoints
- Use `viewFunc_singleParam()` wrapper for single-param endpoints
- DRF `@api_view(['GET'])` + `@extend_schema` on all views
- DRF `Response` (not `JsonResponse`) — no `safe=False` parameter
- Configuration split: config.py (token) / strategy_config.py / scan_config.py
## Constraints
- `api/video/` is protected — do NOT modify unless user explicitly asks
- No python-dotenv dependency — use stdlib env loaders only
- Backward compatibility: keep re-exports when splitting modules
- Server `.env` file manages all secrets; uwsgi.ini only has DJANGO_SETTINGS_MODULE
## Secrets
- All API keys/tokens/passwords via os.getenv()
- Local: .env file (not committed)
- Server: /home/simon/myquant/djapi/.env
- Django loads via djapi/env_loader.py, video loads via api/video/env.py
@@ -0,0 +1,32 @@
# Project Overview
djapi is a Django 5.2 project providing financial data APIs for A-share stocks and CCTV news broadcast video processing.
## Tech Stack
- Python 3.10, Django 5.2, uWSGI, nginx
- Tushare (stock data), akshare (alternative stock data)
- DRF + drf-spectacular (API documentation)
- MySQL (business data), SQLite (Django admin only)
- yt-dlp + ffmpeg + pydub (video/audio processing)
- DashScope (ASR), DeepSeek API (AI text processing)
## Architecture
- Single Django app: `api`
- `api/stock/` — stock data module (Tushare/akshare -> pandas -> JsonResponse/DRF Response)
- `api/video/` — independent video processing pipeline (download -> audio -> ASR -> AI split -> MySQL)
- views.py is thin: extracts params, calls stock functions, returns Response
## Key Files
- `api/views.py` — all ~15 API views, using @api_view + @extend_schema
- `api/stock/stock_utils.py` — shared utilities: tscodeCheck, viewFunc_tsCodeAndDate, viewFunc_singleParam
- `api/stock/config.py` — Tushare token + re-exports from strategy_config, scan_config
- `api/serializers.py` — 13 DRF Serializer classes
- `djapi/env_loader.py` — .env file loader (stdlib, no python-dotenv)
- `api/video/env.py` — standalone .env loader for video module
- `api/utils/mysql_handler.py` — shared MySQLDB class
## Deployment
- Server: simon@doorcome.cn, path: /home/simon/myquant/djapi/
- Virtual env: /opt/miniconda/envs/django/
- uWSGI on port 5004, nginx reverse proxy
- Domains: api.doorcome.cn, echart.doorcome.cn
@@ -0,0 +1,44 @@
# Suggested Commands
## Development
```bash
python manage.py runserver 0.0.0.0:8000 # dev server
python manage.py check --deploy # check config
python manage.py test api # run tests
```
## uWSGI
```bash
uwsgi --ini uwsgi.ini # start
uwsgi --reload uwsgi.pid # hot reload
uwsgi --stop uwsgi.pid # stop
# On server:
/opt/miniconda/envs/django/bin/uwsgi --ini /home/simon/myquant/djapi/uwsgi.ini
kill $(lsof -ti:5004) # force stop
```
## Deploy
```bash
# Full sync (exclude production data)
rsync -avz --delete \
--exclude='.env' --exclude='db.sqlite3' \
--exclude='*.log' --exclude='uwsgi.pid' \
--exclude='__pycache__/' --exclude='*.pyc' \
--exclude='xwlb_video/' --exclude='audio_processing/' \
/Users/summer/Downloads/cc-cursor/djapi/ \
simon@doorcome.cn:/home/simon/myquant/djapi/
# Single file sync MUST use full target path
rsync -avz api/views.py simon@doorcome.cn:/home/simon/myquant/djapi/api/views.py
```
## API Docs
- /api/docs/ — Swagger UI
- /api/redoc/ — ReDoc
- /api/schema/ — OpenAPI JSON
## Video Processing
```bash
cd api/video
python main.py
```
+120
View File
@@ -0,0 +1,120 @@
# the name by which the project can be referenced within Serena
project_name: "djapi"
# list of languages for which language servers are started; choose from:
# al ansible bash clojure cpp
# cpp_ccls crystal csharp csharp_omnisharp dart
# elixir elm erlang fortran fsharp
# go groovy haskell haxe hlsl
# java json julia kotlin lean4
# lua luau markdown matlab msl
# nix ocaml pascal perl php
# php_phpactor powershell python python_jedi python_ty
# r rego ruby ruby_solargraph rust
# scala solidity swift systemverilog terraform
# toml typescript typescript_vts vue yaml
# zig
# (This list may be outdated. For the current list, see values of Language enum here:
# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py
# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
# Note:
# - For C, use cpp
# - For JavaScript, use typescript
# - For Free Pascal/Lazarus, use pascal
# Special requirements:
# Some languages require additional setup/installations.
# See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers
# When using multiple languages, the first language server that supports a given file will be used for that file.
# The first language is the default language and the respective language server will be used as a fallback.
# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
languages:
- typescript
- python
# the encoding used by text files in the project
# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
encoding: "utf-8"
# line ending convention to use when writing source files.
# Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default)
# This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings.
line_ending:
# The language backend to use for this project.
# If not set, the global setting from serena_config.yml is used.
# Valid values: LSP, JetBrains
# Note: the backend is fixed at startup. If a project with a different backend
# is activated post-init, an error will be returned.
language_backend:
# whether to use project's .gitignore files to ignore files
ignore_all_files_in_gitignore: true
# advanced configuration option allowing to configure language server-specific options.
# Maps the language key to the options.
# Have a look at the docstring of the constructors of the LS implementations within solidlsp (e.g., for C# or PHP) to see which options are available.
# No documentation on options means no options are available.
ls_specific_settings: {}
# list of additional paths to ignore in this project.
# Same syntax as gitignore, so you can use * and **.
# Note: global ignored_paths from serena_config.yml are also applied additively.
ignored_paths: []
# whether the project is in read-only mode
# If set to true, all editing tools will be disabled and attempts to use them will result in an error
# Added on 2025-04-18
read_only: false
# list of tool names to exclude.
# This extends the existing exclusions (e.g. from the global configuration)
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
excluded_tools: []
# list of tools to include that would otherwise be disabled (particularly optional tools that are disabled by default).
# This extends the existing inclusions (e.g. from the global configuration).
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
included_optional_tools: []
# fixed set of tools to use as the base tool set (if non-empty), replacing Serena's default set of tools.
# This cannot be combined with non-empty excluded_tools or included_optional_tools.
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
fixed_tools: []
# list of mode names that are to be activated by default, overriding the setting in the global configuration.
# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
# If the setting is undefined/empty, the default_modes from the global configuration (serena_config.yml) apply.
# Otherwise, this overrides the setting from the global configuration (serena_config.yml).
# Therefore, you can set this to [] if you do not want the default modes defined in the global config to apply
# for this project.
# This setting can, in turn, be overridden by CLI parameters (--mode).
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
default_modes:
# list of mode names to be activated additionally for this project, e.g. ["query-projects"]
# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
added_modes:
# initial prompt for the project. It will always be given to the LLM upon activating the project
# (contrary to the memories, which are loaded on demand).
initial_prompt: ""
# time budget (seconds) per tool call for the retrieval of additional symbol information
# such as docstrings or parameter information.
# This overrides the corresponding setting in the global configuration; see the documentation there.
# If null or missing, use the setting from the global configuration.
symbol_info_budget:
# list of regex patterns which, when matched, mark a memory entry as readonly.
# Extends the list from the global configuration, merging the two lists.
read_only_memory_patterns: []
# list of regex patterns for memories to completely ignore.
# Matching memories will not appear in list_memories or activate_project output
# and cannot be accessed via read_memory or write_memory.
# To access ignored memory files, use the read_file tool on the raw file path.
# Extends the list from the global configuration, merging the two lists.
# Example: ["_archive/.*", "_episodes/.*"]
ignored_memory_patterns: []
+105
View File
@@ -0,0 +1,105 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## 项目概述
djapi 是一个 Django 5.2 项目,提供金融数据 API 和新闻联播视频处理能力。部署在 Linux 服务器上,通过 uWSGI + nginx 对外服务。
## 常用命令
```bash
# 开发服务器
python manage.py runserver 0.0.0.0:8000
# uWSGI 管理
uwsgi --ini uwsgi.ini # 启动
uwsgi --reload uwsgi.pid # 热重载
uwsgi --stop uwsgi.pid # 停止
# 数据库操作(Django ORM 主要用于 admin,业务数据走 MySQL
python manage.py makemigrations
python manage.py migrate
python manage.py createsuperuser
# API 文档地址
# /api/docs/ - Swagger UI
# /api/redoc/ - ReDoc
# /api/schema/ - OpenAPI Schema
# 视频处理脚本(独立运行,非 Django 管理)
python api/video/main.py
```
## 架构
### Django 层(薄)
- 项目只有一个 app`api`
- views.py 仅做路由转发,每个 view 函数接收请求参数后直接调用 `api/stock/` 下的业务函数
- Django ORM 基本未使用(sqlite3 仅用于 admin),所有业务数据走 MySQL(通过 `mysqlHandle.py` 直连)
- DRF + drf_spectacular 已配置但 API 视图仍沿用原生 `JsonResponse`,未使用 DRF ViewSet/Serializer
### 子模块一:`api/stock/` — 股票数据
核心依赖 **Tushare** 获取 A 股数据,数据流是「Tushare API → pandas DataFrame → Django JsonResponse」。
- **`config.py`**:全局配置,包括 TS_TOKEN、默认日期范围、行业列表、扫描阈值等
- **`stock_utils.py`**:通用工具集,包含 tscodeCheck(股票代码格式校验/补全后缀)、dataCorrectNaN 填充)、dataMerge(按 ts_code+trade_date 合并 DataFrame)、viewFunc_tsCodeAndDate(通用 view 包装器:从 request 提取参数→调用 data_func→返回 JsonResponse
- **`stock_basic.py`**:日线行情(daily)、个股基本信息(stock_basic)、按行业查股票列表
- **`getStockParam.py`**:个股技术参数(市值等)
- **`getStockEp.py`**TTM EPS 和季度 EPS
- **`getIndexs.py`**:指数行情,支持按名称模糊查询指数代码
- **`stockMargin.py`**:融资融券数据
- **`getStockFina.py`**`FinanceData` 类,按财报日期获取资产负债表+利润表+现金流量表,计算运营/资产/负债/回报率等指标
- **`getStockDiv2.py`**`analyze_stock_dividend_and_price()` — 核心股息率计算。查分红记录→生成 TTM 分红序列→合并日线行情→计算 div_yield = cash_div_year/close,含毛刺平滑处理
- **`smoothBrush.py`**:滑动窗口中位数法检测并平滑毛刺数据
- **`divSearch.py`**:批量扫描全市场股息率,输出 CSV
- **`xwlbDaily.py`**:从 MySQL 查询新闻联播数据(`xwlb_daily``xwlb_daily_ext` 表)
- **`mysqlHandle.py`**`MySQLDB` 类封装 mysql-connector,提供 insert/query/update 方法
### 子模块二:`api/video/` — 新闻联播视频处理
离线批处理流水线:**抓取视频 → 下载 → 提取音频 → ASR 转文字 → AI 分割+取标题 → 入库**。
- **`getVideo5.py`**:主流程。生成 CCTV 节目页 URL → 解析完整版视频链接 → yt-dlp 下载视频 → ffmpeg 提取 MP3 → 调用音频识别 → 写入 MySQL
- **`audioRead.py`**:音频处理。MP3→WAV 转换、智能静音分割、DashScope Paraformer ASR 识别、Qwen 文本纠错
- **`deepseek.py`**`DeepSeekAPI` 类,带重试/降级机制的 DeepSeek API 封装
- **`ai.py`**:遗留的独立 AI 调用函数(deepseek_text, qwen_text),被 video 模块直接调用
- **`newsProcess.py`**post-processing —— 从 MySQL 取出当天原始识别文本,调用 DeepSeek 分割为独立新闻+生成标题,写入 `xwlb_daily_ext`
- **`main.py`**:定时任务入口,每天执行 `process_videos(today, today)`
- **`wasted/`**:废弃的旧版视频抓取脚本
- **`xwlb_video/`**:下载的视频和音频文件(服务器上)
- **`mysqlHandle.py`**video 子目录):与 stock 子目录功能相同的数据库连接类
### URL 路由
所有 API 端点挂载在 `/api/` 下,由 `api/urls.py` 定义,共约 20 个端点,按功能分为:
- 股票基础:`stockbasic/`, `stockinfo/`, `stockparam/`, `industrys/`
- 财务数据:`finance/`, `stockep/`, `quarterlyEps/`
- 行情+指数:`indexByName/`, `indexDatas/`
- 融资融券:`dailymargin/`, `stockmargin/`
- 分红:`getdiv/`
- 新闻联播:`xwlbNews/`, `xwlbFine/`
大多数 API 接受 ts_code、start_date、end_date 三个通用参数,经由 `viewFunc_tsCodeAndDate()` 统一处理。
### 部署
- 服务器用户 `simon`,项目路径 `/home/simon/myquant/djapi/`
- uWSGI 监听 127.0.0.1:5004,通过 socket 与 nginx 通信
- 虚拟环境:`/opt/miniconda/envs/django`
- 静态文件已 collect 到 `static/`,由 nginx 直接服务
- 生产域名:`api.doorcome.cn``echart.doorcome.cn`
- CORS 已配置,允许跨域 cookieSameSite=None
## 开发约束
- `api/video/` 是独立功能模块,默认不修改该目录下任何文件。仅当用户明确要求时才操作此目录。
## 注意事项
- `config.py` 中的 TS_TOKEN 和 `deepseek.py`/`ai.py`/`audioRead.py` 中的 API key、`mysqlHandle.py` 中的数据库密码均为硬编码 —— 生产环境应迁移到环境变量
- `api/stock/` 下的模块支持两种导入方式(相对导入和绝对导入),这是为了兼容「作为 Django app 被调用」和「直接命令行运行脚本」两种场景
- `api/video/` 模块设计为独立命令行运行,不依赖 Django 框架
- `db.sqlite3` 已提交到代码库,包含 Django admin 的用户数据
+123
View File
@@ -0,0 +1,123 @@
# djapi
Django 5.2 项目,提供 A 股金融数据 API 和新闻联播视频处理能力。
## 架构
```
djapi/
├── manage.py # Django 入口
├── djapi/ # 项目配置
│ ├── settings.py # Django 设置、CORS、DRF
│ ├── urls.py # 根路由 + OpenAPI schema
│ └── wsgi.py # WSGI 入口
├── api/ # 唯一 app
│ ├── views.py # 视图层(薄转发,调用 stock 模块)
│ ├── urls.py # /api/* 路由(~20 个端点)
│ ├── stock/ # 股票数据模块
│ │ ├── config.py # Tushare token、扫描参数
│ │ ├── stock_utils.py # 通用工具(tscode 校验、NaN 修正、view 包装器)
│ │ ├── stock_basic.py # 日线行情、基本信息、行业查询
│ │ ├── getStockParam.py # 个股参数
│ │ ├── getStockEp.py # TTM / 季度 EPS
│ │ ├── getIndexs.py # 指数行情
│ │ ├── stockMargin.py # 融资融券
│ │ ├── getStockFina.py # 财务报表分析
│ │ ├── getStockDiv2.py # 股息率计算(TTM 分红 + 毛刺平滑)
│ │ ├── smoothBrush.py # 数据平滑算法
│ │ ├── divSearch.py # 全市场股息率批量扫描
│ │ ├── xwlbDaily.py # 新闻联播数据查询
│ │ └── mysqlHandle.py # MySQL 连接封装
│ └── video/ # 新闻联播视频处理(独立模块)
│ ├── getVideo5.py # 主流程:抓取→下载→识别→入库
│ ├── audioRead.py # 音频转换、分割、ASR 识别
│ ├── deepseek.py # DeepSeek API 封装
│ ├── ai.py # AI 调用函数
│ ├── newsProcess.py # 新闻分割+标题提取
│ └── main.py # 定时任务入口
├── static/ # 静态文件(collect 后的 admin 资源)
├── uwsgi.ini # uWSGI 配置
└── requirements.txt # Python 依赖
```
### 数据流
**股票 API**`HTTP 请求``views.py` 提取参数 → `viewFunc_tsCodeAndDate()` 统一包装 → `stock/*.py` 调用 Tushare API → pandas DataFrame → JsonResponse
**视频处理**`main.py``getVideo5.py` 抓取 CCTV 视频页 → yt-dlp 下载 → ffmpeg 提取 MP3 → pydub 静音分割 → DashScope Paraformer ASR → MySQL → `newsProcess.py` 调用 DeepSeek 分割新闻+生成标题 → MySQL
### 外部依赖
| 服务 | 用途 | 相关文件 |
|------|------|----------|
| Tushare | A 股行情、财报、分红、指数 | `api/stock/*.py` |
| DeepSeek API | 新闻文本分割、摘要 | `api/video/deepseek.py`, `ai.py` |
| DashScope (Qwen) | ASR 语音识别、文本纠错 | `api/video/audioRead.py`, `ai.py` |
| MySQL | 股票数据、新闻联播数据 | `mysqlHandle.py`stock 和 video 各一份) |
| yt-dlp + ffmpeg | 视频下载、音频提取 | `api/video/getVideo5.py` |
## 快速开始
```bash
# 安装依赖
pip install -r requirements.txt
# 开发服务器
python manage.py runserver 0.0.0.0:8000
# 数据库初始化
python manage.py migrate
python manage.py createsuperuser
# uWSGI 部署
uwsgi --ini uwsgi.ini
uwsgi --reload uwsgi.pid
uwsgi --stop uwsgi.pid
```
## API 概览
基础 URL`/api/`
| 端点 | 参数 | 说明 |
|------|------|------|
| `stockbasic/` | tscode, start_date, end_date | 日线行情 |
| `stockinfo/` | tscode | 个股基本信息 |
| `stockparam/` | tscode, start_date, end_date | 个股参数(市值等) |
| `industrys/` | industry | 按行业查股票列表 |
| `indexByName/` | index_name | 按名称查指数 |
| `indexDatas/` | index_name, start_date, end_date | 指数日行情 |
| `stockep/` | tscode, start_date, end_date | TTM EPS |
| `quarterlyEps/` | tscode, start_date, end_date | 季度 EPS |
| `finance/` | tscode, start_date, end_date | 财务报表分析 |
| `getdiv/` | tscode, start_date, end_date | 股息率(含 TTM |
| `dailymargin/` | trade_date, exchange_id | 每日融资融券汇总 |
| `stockmargin/` | tscode, start_date, end_date | 个股融资融券 |
| `xwlbNews/` | start_date, end_date | 新闻联播(原始识别文本) |
| `xwlbFine/` | start_date, end_date | 新闻联播(AI 分割后) |
| `news/reports/` | report_type, start_date, end_date, id | 日报查询(默认最近 24h;传 id 返回单份详情含事件) |
| `news/events/` | days, importance, report_type, section, limit | 重要事件聚合(跨日报,最近 N 天 importance≥阈值) |
日报查询接口详细说明见 [`docs/news_report_api.md`](../docs/news_report_api.md)(表结构见 `djapi/docs/db_schema.md`)。
API 文档(Swagger):`/api/docs/`
OpenAPI Schema`/api/schema/`
## 部署
- 服务器路径:`/home/simon/myquant/djapi/`
- uWSGI 监听 `127.0.0.1:5004`nginx 反向代理
- 域名:`api.doorcome.cn``echart.doorcome.cn`
- 虚拟环境:`/opt/miniconda/envs/django`
- Python 版本:3.10
## TODO
- [ ] 将硬编码的 API Key / Token / 数据库密码迁移到环境变量
- [ ] 视图片段从 `JsonResponse` 迁移到 DRF ViewSet + Serializer,完善 Swagger 文档
- [ ] 添加接口限流和认证机制
- [ ] 补充单元测试(当前 tests.py 为空)
- [ ] `config.py` 中的交易所、行业列表等改为可动态配置
- [ ] video 模块的 `db.sqlite3` 不应随代码提交,添加到 `.gitignore`
- [ ] 数据库密码、API Key 出现在多个 `__pycache__/*.pyc` 中,需要清理历史
- [ ] video 模块中废弃脚本(`wasted/`)确认后清理
+12
View File
@@ -0,0 +1,12 @@
#api/__init__.py
from . import stock
__version__ = "0.1.0"
__all__ = ["stock"]
# 初始化代码
def init():
pass
if __name__ == "__main__":
init()
+3
View File
@@ -0,0 +1,3 @@
from django.contrib import admin
# Register your models here.
+6
View File
@@ -0,0 +1,6 @@
from django.apps import AppConfig
class ApiConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'api'
View File
+22
View File
@@ -0,0 +1,22 @@
from django.db import models
# Create your models here.
class Book(models.Model):
title = models.CharField(max_length=100)
author = models.CharField(max_length=100)
published_date = models.DateField()
isbn_number = models.CharField(max_length=13)
def __str__(self):
return self.title
# admin.py
from django.contrib import admin
from .models import Book
class BookAdmin(admin.ModelAdmin):
list_display = ('title', 'author', 'published_date')
search_fields = ('title', 'author')
list_filter = ('published_date',)
admin.site.register(Book, BookAdmin)
View File
+168
View File
@@ -0,0 +1,168 @@
"""
news_report / news_event 只读查询层(日报结构化入库,见 docs/db_schema.md)。
连接配置来自环境变量(与 docs/report_db_design.md §7 保持一致):
NEWS_DB_HOST / NEWS_DB_PORT / NEWS_DB_USER / NEWS_DB_PASSWORD / NEWS_DB_NAME
NEWS_DB_PASSWORD 缺失时直接报错,禁止默认密码。
所有 SQL 均为 MariaDB 方言、参数化查询(防 SQL 注入),不依赖 ORM。
"""
import json
import os
from datetime import date, datetime, timedelta
import mysql.connector
_REPORT_FIELDS = (
"id, report_date, report_type, file_name, generated_at, "
"ai_summary, stats, created_at"
)
_EVENT_FIELDS = (
"id, report_id, section, rank, importance, event_type, title, "
"summary, sentiment, source, url"
)
def load_db_config() -> dict:
"""从环境变量读取 NEWS_DB_* 连接配置,密码缺失时抛错。"""
password = os.getenv("NEWS_DB_PASSWORD")
if not password:
raise RuntimeError(
"NEWS_DB_PASSWORD 未设置,禁止使用默认密码连接 news 库"
)
return {
"host": os.getenv("NEWS_DB_HOST", "127.0.0.1"),
"port": int(os.getenv("NEWS_DB_PORT", "3306")),
"user": os.getenv("NEWS_DB_USER", "myquant"),
"password": password,
"database": os.getenv("NEWS_DB_NAME", "myquant"),
"charset": "utf8mb4",
}
def _connect():
return mysql.connector.connect(**load_db_config())
def _row_to_dict(row: dict) -> dict:
"""序列化行:stats JSON 解析、日期/时间转 ISO 字符串。"""
d = dict(row)
if d.get("stats") is not None:
try:
d["stats"] = json.loads(d["stats"])
except (TypeError, ValueError):
d["stats"] = None
for k, v in d.items():
if isinstance(v, (date, datetime)):
d[k] = v.isoformat()
return d
def fetch_reports(
report_type: str | None = None,
start_date: date | None = None,
end_date: date | None = None,
report_id: int | None = None,
):
"""
日报查询。
report_id 给定 → 返回单份详情 dict(含 events,按 section, rank 排序);
不存在返回 None。
否则 → 返回列表:每天每类型取最新一份(MAX(generated_at) 子查询),
仅主表字段(轻量,不带 events)。
"""
conn = _connect()
try:
cur = conn.cursor(dictionary=True)
if report_id is not None:
cur.execute(
f"SELECT {_REPORT_FIELDS} FROM news_report WHERE id = %s",
(report_id,),
)
row = cur.fetchone()
if row is None:
return None
report = _row_to_dict(row)
cur.execute(
"SELECT id, section, rank, importance, event_type, title, "
"summary, sentiment, source, url "
"FROM news_event WHERE report_id = %s ORDER BY section, rank",
(report_id,),
)
report["events"] = [dict(r) for r in cur.fetchall()]
return report
where, params = [], []
if report_type:
where.append("r.report_type = %s")
params.append(report_type)
if start_date:
where.append("r.report_date >= %s")
params.append(start_date.isoformat())
if end_date:
where.append("r.report_date <= %s")
params.append(end_date.isoformat())
cond = (" WHERE " + " AND ".join(where)) if where else ""
sql = (
"SELECT r.id, r.report_date, r.report_type, r.file_name, "
"r.generated_at, r.ai_summary, r.stats, r.created_at "
"FROM news_report r "
"JOIN ("
" SELECT report_date, report_type, MAX(generated_at) AS g "
" FROM news_report GROUP BY report_date, report_type"
") t ON r.report_date = t.report_date "
" AND r.report_type = t.report_type "
" AND r.generated_at = t.g"
+ cond
+ " ORDER BY r.report_date DESC, r.report_type"
)
cur.execute(sql, tuple(params))
return [_row_to_dict(r) for r in cur.fetchall()]
finally:
conn.close()
def fetch_important_events(
days: int = 7,
importance: int = 4,
report_type: str | None = None,
section: str | None = None,
limit: int = 100,
) -> list:
"""
跨日报重要事件聚合检索(最近 N 天,importance >= 阈值)。
按 importance DESC, report_date DESC 排序。
"""
conn = _connect()
try:
cur = conn.cursor(dictionary=True)
since = (date.today() - timedelta(days=days)).isoformat()
where = ["r.report_date >= %s", "e.importance >= %s"]
params = [since, int(importance)]
if report_type:
where.append("r.report_type = %s")
params.append(report_type)
if section:
where.append("e.section = %s")
params.append(section)
sql = (
"SELECT r.report_date, r.report_type, e.id, e.section, e.rank, "
"e.importance, e.event_type, e.title, e.summary, e.sentiment, "
"e.source, e.url "
"FROM news_event e "
"JOIN news_report r ON r.id = e.report_id "
"WHERE " + " AND ".join(where)
+ " ORDER BY e.importance DESC, r.report_date DESC, "
"e.section, e.rank "
+ "LIMIT %s"
)
params.append(int(limit))
cur.execute(sql, tuple(params))
return [dict(r) for r in cur.fetchall()]
finally:
conn.close()
+50
View File
@@ -0,0 +1,50 @@
"""日报查询 API 的 OpenAPI 文档 serializer(只读,不用于反序列化)。"""
from rest_framework import serializers
class EventSerializer(serializers.Serializer):
"""日报事件明细(news_event 一行)"""
id = serializers.IntegerField()
section = serializers.CharField()
rank = serializers.IntegerField()
importance = serializers.IntegerField(allow_null=True)
event_type = serializers.CharField(allow_null=True)
title = serializers.CharField()
summary = serializers.CharField(allow_null=True)
sentiment = serializers.CharField(allow_null=True)
source = serializers.CharField(allow_null=True)
url = serializers.CharField(allow_null=True)
class ReportListSerializer(serializers.Serializer):
"""日报列表项(news_report 主表字段)"""
id = serializers.IntegerField()
report_date = serializers.CharField()
report_type = serializers.CharField()
file_name = serializers.CharField()
generated_at = serializers.CharField()
ai_summary = serializers.CharField(allow_null=True)
stats = serializers.JSONField(allow_null=True)
created_at = serializers.CharField()
class ReportDetailSerializer(ReportListSerializer):
"""日报详情(主表字段 + 事件列表)"""
events = EventSerializer(many=True)
class ImportantEventSerializer(serializers.Serializer):
"""跨日报重要事件聚合(news_event JOIN news_report"""
report_date = serializers.CharField()
report_type = serializers.CharField()
id = serializers.IntegerField()
section = serializers.CharField()
rank = serializers.IntegerField()
importance = serializers.IntegerField(allow_null=True)
event_type = serializers.CharField(allow_null=True)
title = serializers.CharField()
summary = serializers.CharField(allow_null=True)
sentiment = serializers.CharField(allow_null=True)
source = serializers.CharField(allow_null=True)
url = serializers.CharField(allow_null=True)
+149
View File
@@ -0,0 +1,149 @@
"""
日报查询 API 测试(GET /api/news/reports/ 与 /api/news/events/)。
通过 mock 数据库查询层(api.report.query.*),只验证视图层
参数解析 / 默认窗口 / 响应结构 / 错误处理,不依赖真实 MySQL。
"""
from datetime import timedelta
from unittest.mock import patch
from django.test import TestCase
from django.utils import timezone
from rest_framework.test import APIClient
class NewsReportsAPITest(TestCase):
"""GET /api/news/reports/ 日报查询"""
def setUp(self):
self.client = APIClient()
self.url = '/api/news/reports/'
@patch('api.report.query.fetch_reports', return_value=[])
def test_default_window_last_24h(self, mock_fetch):
resp = self.client.get(self.url)
self.assertEqual(resp.status_code, 200)
self.assertEqual(resp.json(), [])
kwargs = mock_fetch.call_args.kwargs
now = timezone.now()
self.assertEqual(kwargs['start_date'], (now - timedelta(hours=24)).date())
self.assertEqual(kwargs['end_date'], now.date())
self.assertIsNone(kwargs['report_type'])
self.assertIsNone(kwargs['report_id'])
@patch('api.report.query.fetch_reports',
return_value=[{'id': 1, 'report_date': '2026-08-01', 'report_type': 'finance'}])
def test_report_type_and_date_range(self, mock_fetch):
resp = self.client.get(self.url, {
'report_type': 'finance',
'start_date': '2026-07-01',
'end_date': '2026-08-03',
})
self.assertEqual(resp.status_code, 200)
self.assertEqual(len(resp.json()), 1)
kwargs = mock_fetch.call_args.kwargs
self.assertEqual(kwargs['report_type'], 'finance')
self.assertEqual(str(kwargs['start_date']), '2026-07-01')
self.assertEqual(str(kwargs['end_date']), '2026-08-03')
@patch('api.report.query.fetch_reports')
def test_detail_by_id(self, mock_fetch):
mock_fetch.return_value = {
'id': 10, 'report_date': '2026-08-03', 'report_type': 'finance',
'events': [{'id': 1, 'title': '事件一'}],
}
resp = self.client.get(self.url, {'id': '10'})
self.assertEqual(resp.status_code, 200)
self.assertEqual(resp.json()['id'], 10)
self.assertEqual(mock_fetch.call_args.kwargs['report_id'], 10)
@patch('api.report.query.fetch_reports', return_value=None)
def test_detail_not_found(self, mock_fetch):
resp = self.client.get(self.url, {'id': '99999'})
self.assertEqual(resp.status_code, 404)
self.assertIn('error', resp.json())
def test_invalid_report_type(self):
resp = self.client.get(self.url, {'report_type': 'xxx'})
self.assertEqual(resp.status_code, 400)
def test_invalid_date_format(self):
resp = self.client.get(self.url, {'start_date': '2026/07/01'})
self.assertEqual(resp.status_code, 400)
def test_invalid_id(self):
resp = self.client.get(self.url, {'id': 'abc'})
self.assertEqual(resp.status_code, 400)
def test_non_positive_id(self):
resp = self.client.get(self.url, {'id': '0'})
self.assertEqual(resp.status_code, 400)
@patch('api.report.query.fetch_reports', side_effect=RuntimeError('db down'))
def test_query_error_500(self, mock_fetch):
resp = self.client.get(self.url)
self.assertEqual(resp.status_code, 500)
self.assertIn('error', resp.json())
class NewsEventsAPITest(TestCase):
"""GET /api/news/events/ 重要事件聚合"""
def setUp(self):
self.client = APIClient()
self.url = '/api/news/events/'
@patch('api.report.query.fetch_important_events', return_value=[])
def test_defaults(self, mock_fetch):
resp = self.client.get(self.url)
self.assertEqual(resp.status_code, 200)
self.assertEqual(resp.json(), [])
kwargs = mock_fetch.call_args.kwargs
self.assertEqual(kwargs['days'], 7)
self.assertEqual(kwargs['importance'], 4)
self.assertEqual(kwargs['limit'], 100)
self.assertIsNone(kwargs['report_type'])
self.assertIsNone(kwargs['section'])
@patch('api.report.query.fetch_important_events',
return_value=[{'id': 1, 'title': '重要事件', 'importance': 5}])
def test_filters(self, mock_fetch):
resp = self.client.get(self.url, {
'days': '3', 'importance': '5',
'report_type': 'intl', 'section': 'intl', 'limit': '10',
})
self.assertEqual(resp.status_code, 200)
self.assertEqual(len(resp.json()), 1)
kwargs = mock_fetch.call_args.kwargs
self.assertEqual(kwargs['days'], 3)
self.assertEqual(kwargs['importance'], 5)
self.assertEqual(kwargs['limit'], 10)
self.assertEqual(kwargs['report_type'], 'intl')
self.assertEqual(kwargs['section'], 'intl')
def test_invalid_days(self):
resp = self.client.get(self.url, {'days': 'abc'})
self.assertEqual(resp.status_code, 400)
def test_days_out_of_range(self):
resp = self.client.get(self.url, {'days': '0'})
self.assertEqual(resp.status_code, 400)
def test_invalid_importance(self):
resp = self.client.get(self.url, {'importance': '9'})
self.assertEqual(resp.status_code, 400)
def test_invalid_section(self):
resp = self.client.get(self.url, {'section': 'foo'})
self.assertEqual(resp.status_code, 400)
def test_invalid_report_type(self):
resp = self.client.get(self.url, {'report_type': 'xxx'})
self.assertEqual(resp.status_code, 400)
@patch('api.report.query.fetch_important_events', side_effect=RuntimeError('db down'))
def test_query_error_500(self, mock_fetch):
resp = self.client.get(self.url)
self.assertEqual(resp.status_code, 500)
self.assertIn('error', resp.json())
+147
View File
@@ -0,0 +1,147 @@
"""
日报查询 API 视图。
- GET /api/news/reports/ 日报查询(默认最近 24 小时;传 id 返回单份详情含事件)
- GET /api/news/events/ 跨日报重要事件聚合(最近 N 天 importance >= 阈值)
"""
from datetime import datetime, timedelta
from django.utils import timezone
from drf_spectacular.utils import OpenApiParameter, extend_schema
from rest_framework.decorators import api_view
from rest_framework.response import Response
from . import query as report_query
from .serializers import (
ImportantEventSerializer,
ReportDetailSerializer,
ReportListSerializer,
)
_REPORT_TYPES = ("finance", "intl")
_SECTIONS = ("xwlb", "news", "cninfo", "intl")
def _parse_date_param(request, name):
"""解析 YYYY-MM-DD 参数,非法格式抛 ValueError。"""
raw = request.GET.get(name)
if not raw:
return None
try:
return datetime.strptime(raw, "%Y-%m-%d").date()
except ValueError:
raise ValueError(f"{name} 格式错误,应为 YYYY-MM-DD")
@extend_schema(
parameters=[
OpenApiParameter(name='report_type', type=str, required=False,
description='日报类型:finance | intl(默认两者)'),
OpenApiParameter(name='start_date', type=str, required=False,
description='起始日期 YYYY-MM-DD(默认:当前时间往前 24 小时)'),
OpenApiParameter(name='end_date', type=str, required=False,
description='结束日期 YYYY-MM-DD(默认今天)'),
OpenApiParameter(name='id', type=int, required=False,
description='日报 id,指定时返回单份详情(含事件,按板块/序号排序)'),
],
responses={200: ReportDetailSerializer},
description='AI 财经日报查询:默认返回最近 24 小时的日报列表(每天每类型取最新一份);'
'传 id 返回单份详情含事件',
tags=['日报'],
)
@api_view(['GET'])
def news_reports(request):
try:
report_id = request.GET.get('id')
if report_id is not None:
report_id = int(report_id)
if report_id <= 0:
raise ValueError("id 必须为正整数")
report_type = request.GET.get('report_type')
if report_type and report_type not in _REPORT_TYPES:
raise ValueError("report_type 仅支持 finance / intl")
start_date = _parse_date_param(request, 'start_date')
end_date = _parse_date_param(request, 'end_date')
except ValueError as e:
return Response({'error': str(e)}, status=400)
now = timezone.now()
start_date = start_date or (now - timedelta(hours=24)).date()
end_date = end_date or now.date()
try:
data = report_query.fetch_reports(
report_type=report_type,
start_date=start_date,
end_date=end_date,
report_id=report_id,
)
except Exception as e:
return Response({'error': f'查询失败: {e}'}, status=500)
if report_id is not None:
if data is None:
return Response({'error': f'日报 id={report_id} 不存在'}, status=404)
return Response(data)
return Response(data)
def _get_int_param(request, name, default, lo, hi):
"""解析整数参数并校验范围,非法抛 ValueError。"""
raw = request.GET.get(name)
if raw is None:
return default
try:
value = int(raw)
except ValueError:
raise ValueError(f"{name} 必须为整数")
if not (lo <= value <= hi):
raise ValueError(f"{name} 需在 {lo}~{hi} 之间")
return value
@extend_schema(
parameters=[
OpenApiParameter(name='days', type=int, required=False, default=7,
description='最近 N 天(1~365'),
OpenApiParameter(name='importance', type=int, required=False, default=4,
description='最低重要度(1~5'),
OpenApiParameter(name='report_type', type=str, required=False,
description='日报类型:finance | intl(默认两者)'),
OpenApiParameter(name='section', type=str, required=False,
description='板块:xwlb | news | cninfo | intl(默认全部)'),
OpenApiParameter(name='limit', type=int, required=False, default=100,
description='返回条数上限(1~500'),
],
responses={200: ImportantEventSerializer(many=True)},
description='跨日报重要事件聚合:最近 N 天 importance >= 阈值的事件,'
'按重要度、日期降序',
tags=['日报'],
)
@api_view(['GET'])
def news_events(request):
try:
days = _get_int_param(request, 'days', 7, 1, 365)
importance = _get_int_param(request, 'importance', 4, 1, 5)
limit = _get_int_param(request, 'limit', 100, 1, 500)
report_type = request.GET.get('report_type')
if report_type and report_type not in _REPORT_TYPES:
raise ValueError("report_type 仅支持 finance / intl")
section = request.GET.get('section')
if section and section not in _SECTIONS:
raise ValueError("section 仅支持 xwlb / news / cninfo / intl")
except ValueError as e:
return Response({'error': str(e)}, status=400)
try:
data = report_query.fetch_important_events(
days=days,
importance=importance,
report_type=report_type,
section=section,
limit=limit,
)
except Exception as e:
return Response({'error': f'查询失败: {e}'}, status=500)
return Response(data)
+145
View File
@@ -0,0 +1,145 @@
from rest_framework import serializers
class StockDailySerializer(serializers.Serializer):
"""日线行情(stockbasic"""
ts_code = serializers.CharField()
trade_date = serializers.CharField()
open = serializers.FloatField()
high = serializers.FloatField()
low = serializers.FloatField()
close = serializers.FloatField()
pre_close = serializers.FloatField()
change = serializers.FloatField()
pct_chg = serializers.FloatField()
vol = serializers.FloatField()
amount = serializers.FloatField()
class StockInfoSerializer(serializers.Serializer):
"""个股基本信息(stockinfo"""
ts_code = serializers.CharField()
symbol = serializers.CharField()
name = serializers.CharField()
area = serializers.CharField()
industry = serializers.CharField()
market = serializers.CharField()
list_date = serializers.CharField()
fullname = serializers.CharField()
enname = serializers.CharField()
exchange = serializers.CharField()
curr_type = serializers.CharField()
list_status = serializers.CharField()
is_hs = serializers.CharField()
class IndustryStockSerializer(serializers.Serializer):
"""行业股票列表(industrys"""
ts_code = serializers.CharField()
name = serializers.CharField()
class StockParamSerializer(serializers.Serializer):
"""个股参数(stockparam"""
ts_code = serializers.CharField()
trade_date = serializers.CharField()
close = serializers.FloatField()
turnover_rate = serializers.FloatField()
turnover_rate_f = serializers.FloatField()
volume_ratio = serializers.FloatField()
pe = serializers.FloatField()
pe_ttm = serializers.FloatField()
pb = serializers.FloatField()
ps = serializers.FloatField()
ps_ttm = serializers.FloatField()
dv_ratio = serializers.FloatField()
dv_ttm = serializers.FloatField()
total_share = serializers.FloatField()
float_share = serializers.FloatField()
free_share = serializers.FloatField()
total_mv = serializers.FloatField()
circ_mv = serializers.FloatField()
class StockEpSerializer(serializers.Serializer):
"""TTM EPSstockep"""
ts_code = serializers.CharField()
trade_date = serializers.CharField()
eps_ttm = serializers.FloatField()
class QuarterlyEpsSerializer(serializers.Serializer):
"""季度 EPSquarterlyEps"""
ts_code = serializers.CharField()
trade_date = serializers.CharField()
eps = serializers.FloatField()
report_date = serializers.CharField()
class IndexInfoSerializer(serializers.Serializer):
"""指数信息(indexByName"""
index_code = serializers.CharField()
name = serializers.CharField()
fullname = serializers.CharField()
market = serializers.CharField()
publisher = serializers.CharField()
index_type = serializers.CharField()
category = serializers.CharField()
list_date = serializers.CharField()
class IndexDailySerializer(serializers.Serializer):
"""指数日行情(indexDatas"""
ts_code = serializers.CharField()
trade_date = serializers.CharField()
close = serializers.FloatField()
open = serializers.FloatField()
high = serializers.FloatField()
low = serializers.FloatField()
pre_close = serializers.FloatField()
change = serializers.FloatField()
pct_chg = serializers.FloatField()
vol = serializers.FloatField()
amount = serializers.FloatField()
class MarginDailySerializer(serializers.Serializer):
"""每日融资融券汇总(dailyMargin"""
trade_date = serializers.CharField()
exchange_id = serializers.CharField()
rzye = serializers.FloatField()
rqye = serializers.FloatField()
rzrqye = serializers.FloatField()
class StockMarginSerializer(serializers.Serializer):
"""个股融资融券(stockMargin"""
ts_code = serializers.CharField()
trade_date = serializers.CharField()
rzye = serializers.FloatField()
rqye = serializers.FloatField()
rzrqye = serializers.FloatField()
class FinanceDataSerializer(serializers.Serializer):
"""财务报表分析(finance"""
ts_code = serializers.CharField()
period = serializers.CharField()
class DividendSerializer(serializers.Serializer):
"""股息率(getdiv"""
ts_code = serializers.CharField()
trade_date = serializers.CharField()
close = serializers.FloatField()
cash_div_tax = serializers.FloatField()
cash_div_year = serializers.FloatField()
div_yield = serializers.FloatField()
class XwlbNewsSerializer(serializers.Serializer):
"""新闻联播(xwlbNews / xwlbFine"""
news_days = serializers.CharField()
daily_sub_id = serializers.IntegerField()
news_improve = serializers.CharField()
news_title = serializers.CharField()
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+9
View File
@@ -0,0 +1,9 @@
# api/stock/__init__.py
from . import config
from . import getIndexs
from . import getStockEp
from . import getStockParam
from . import stock_basic
from . import stock_utils
__all__ = ["config","getIndexs","getStockEp","getStockParam","stock_basic","stock_utils"]
+21
View File
@@ -0,0 +1,21 @@
import os
# Tushare API Token(来自环境变量)
TS_TOKEN = os.getenv('TUSHARE_TS_TOKEN', '')
# 股票代码
TS_CODE = '002273.SZ'
# 数据日期范围
START_DATE = '20200101'
END_DATE = '20251231'
# 常规配置
PRECISION_CONFIG = 4 #小数点后精度默认配置
# 策略参数 — 拆分自 strategy_config
from .strategy_config import * # noqa: F401, F403
# 扫描配置 — 拆分自 scan_config
from .scan_config import * # noqa: F401, F403
+183
View File
@@ -0,0 +1,183 @@
"""
统一数据源模块 — djapi/api/stock/ 的单一数据入口。
归一化 Tushare / AkShare / MySQL 三种数据源。
所有模块通过此入口获取数据连接,不再各自创建 pro 实例。
用法:
from .data_source import get_tushare_pro, get_mysql_db
pro = get_tushare_pro()
df = pro.daily(ts_code='000001.SZ', ...)
db = get_mysql_db()
rows = db.query("SELECT * FROM xwlb_daily WHERE ...")
扩展新数据源:
class NewSource: ...
_sources['new'] = NewSource()
def get_new_source(): return _sources['new']
"""
import os
import threading
import tushare as ts
# 延迟加载 AkShare(避免不必要的导入开销)
_akshare = None
def _get_akshare():
global _akshare
if _akshare is None:
import akshare as ak
_akshare = ak
return _akshare
# ═══════════════════════════════════════════════════════════
# Token 加载
# ═══════════════════════════════════════════════════════════
def get_ts_token() -> str:
"""获取 Tushare Token,优先级:环境变量 TUSHARE_TS_TOKEN > TUSHARE_TOKEN > config.py"""
token = os.getenv("TUSHARE_TS_TOKEN", "") or os.getenv("TUSHARE_TOKEN", "")
if not token:
try:
from .config import TS_TOKEN
token = TS_TOKEN
except ImportError:
try:
from config import TS_TOKEN
token = TS_TOKEN
except ImportError:
pass
return token
# ═══════════════════════════════════════════════════════════
# Tushare Pro 连接池(线程安全单例)
# ═══════════════════════════════════════════════════════════
_pro_lock = threading.Lock()
_pro = None
def get_tushare_pro():
"""获取 Tushare pro_api 实例(全局单例,线程安全)。"""
global _pro
if _pro is not None:
return _pro
with _pro_lock:
if _pro is not None:
return _pro
token = get_ts_token()
ts.set_token(token)
_pro = ts.pro_api()
return _pro
def reset_tushare_pro():
"""重置 Tushare 连接(token 变更时调用)。"""
global _pro
with _pro_lock:
_pro = None
# ═══════════════════════════════════════════════════════════
# MySQL 连接
# ═══════════════════════════════════════════════════════════
_mysql_db = None
def get_mysql_db():
"""获取 MySQLDB 实例(全局单例)。"""
global _mysql_db
if _mysql_db is not None:
return _mysql_db
try:
from ..utils.mysql_handler import MySQLDB
except (ImportError, ValueError):
try:
from utils.mysql_handler import MySQLDB
except ImportError:
return None
_mysql_db = MySQLDB()
return _mysql_db
# ═══════════════════════════════════════════════════════════
# 日线行情 — 双源 fallbackTushare → AkShare
# ═══════════════════════════════════════════════════════════
def get_daily(ts_code: str, start_date: str, end_date: str, source: str = "tushare"):
"""
获取个股日线行情。
参数:
ts_code: 如 '000001.SZ'
start_date: YYYYMMDD
end_date: YYYYMMDD
source: 'tushare' | 'akshare' | 'auto' (tushare优先)
返回:
pd.DataFrame (trade_date, open, high, low, close, vol, amount, ...)
"""
import pandas as pd
if source == "auto":
# Tushare 优先
try:
df = get_daily(ts_code, start_date, end_date, source="tushare")
if df is not None and not df.empty:
return df
except Exception:
pass
return get_daily(ts_code, start_date, end_date, source="akshare")
if source == "tushare":
pro = get_tushare_pro()
df = pro.daily(
ts_code=ts_code, start_date=start_date, end_date=end_date,
fields="ts_code,trade_date,open,high,low,close,pre_close,change,pct_chg,vol,amount"
)
if df is not None and not df.empty:
df["trade_date"] = df["trade_date"].astype(str)
return df
if source == "akshare":
symbol = ts_code.replace(".SZ", "").replace(".SH", "").replace(".BJ", "")
ak = _get_akshare()
df = ak.stock_zh_a_hist(
symbol=symbol, period="daily",
start_date=start_date, end_date=end_date, adjust="qfq"
)
if df is not None and not df.empty:
df = df.rename(columns={
"日期": "trade_date", "开盘": "open", "收盘": "close",
"最高": "high", "最低": "low", "成交量": "vol", "成交额": "amount",
"涨跌幅": "pct_chg", "涨跌额": "change",
})
df["ts_code"] = ts_code
df["trade_date"] = df["trade_date"].astype(str)
return df if df is not None else pd.DataFrame()
raise ValueError("Unknown source: {}".format(source))
# ═══════════════════════════════════════════════════════════
# 扩展点:未来新增数据源
# ═══════════════════════════════════════════════════════════
#
# 1. 在 _sources dict 中注册新源
# 2. 实现与 get_daily() 相同签名的函数
# 3. 在 get_daily(source=...) 中添加路由
#
# _sources = {
# "tushare": TushareDailySource(),
# "akshare": AkShareDailySource(),
# "wind": WindDailySource(), # 未来
# "joinquant": JoinQuantSource(), # 未来
# }
+219
View File
@@ -0,0 +1,219 @@
import pandas as pd
from config import START_DATE, END_DATE, PRECISION_CONFIG
from stock_utils import *
from getStockDiv2 import analyze_stock_dividend_and_price
from getStockParam import getStockParam
from .data_source import get_tushare_pro
import datetime
pro = get_tushare_pro()
# --- 数据输出配置 ---
OUTPUT_TO_CSV = True # 开关:是否输出到CSV
OUTPUT_TO_MYSQL = False # 开关:是否输出到MySQL (需要额外配置数据库连接)
# --- 数据库输出函数 (伪代码) ---
def save_to_mysql(dataframe, table_name):
"""
将DataFrame保存到MySQL数据库的伪代码函数。
实际使用时需要配置数据库连接。
"""
if not OUTPUT_TO_MYSQL:
print("MySQL输出已禁用。")
return
print(f"正在将数据保存到MySQL表 {table_name}...")
# --- 伪代码开始 ---
# import pymysql # 或其他数据库连接库
#
# connection = pymysql.connect(host='your_host',
# user='your_user',
# password='your_password',
# database='your_database',
# charset='utf8mb4')
# try:
# with connection.cursor() as cursor:
# # 创建表 (如果不存在)
# # create_table_sql = "..."
# # cursor.execute(create_table_sql)
#
# # 遍历DataFrame并插入数据
# for index, row in dataframe.iterrows():
# # 注意:需要处理SQL注入风险,最好使用参数化查询
# insert_sql = f"INSERT INTO {table_name} (...) VALUES (...)"
# cursor.execute(insert_sql, tuple(row))
# connection.commit()
# print(f"成功保存 {len(dataframe)} 条记录到MySQL。")
# except Exception as e:
# print(f"保存到MySQL时出错: {e}")
# connection.rollback()
# finally:
# connection.close()
# --- 伪代码结束 ---
print("MySQL保存操作完成 (伪代码)。")
# --- CSV输出函数 ---
def save_to_csv(dataframe, filename, mode='w', header=True):
"""将DataFrame保存到CSV文件"""
if not OUTPUT_TO_CSV:
print("CSV输出已禁用。")
return
try:
dataframe.to_csv(filename, index=False, encoding='utf-8', mode=mode, header=header)
print(f"成功保存 {len(dataframe)} 条记录到 {filename}")
except Exception as e:
print(f"保存CSV文件 {filename} 时出错: {e}")
'''
写一个函数,调用上面的函数,传入股票代码和起止日期,计算div_yield的最大值,最小值,均值,最新值,以及标准差,返回pandas的Series
'''
def calculate_dividend_yield_stats(ts_code, start_date=START_DATE, end_date=END_DATE):
"""
计算股息率的统计指标
参数:
ts_code (str): 股票代码
start_date (str): 起始日期
end_date (str): 结束日期
返回:
pd.Series: 包含股息率统计指标的Series
"""
df = analyze_stock_dividend_and_price(ts_code, start_date, end_date)
if df.empty or 'div_yield' not in df.columns:
return pd.Series(dtype=float)
div_yield_data = df['div_yield']
stats = pd.Series({
'ts_code': ts_code,
'max': div_yield_data.max(),
'min': div_yield_data.min(),
'mean': div_yield_data.mean(),
'latest': div_yield_data.iloc[-1] if len(div_yield_data) > 0 else 0,
'std': div_yield_data.std(),
'zero_count': (div_yield_data == 0).sum()
})
return stats
# --- 主处理函数 ---
def process_stock_dividend_batch(start_dt='20200101', end_dt='20251231', dataInterval=10, exchange='SSE'):
"""
批量处理股票股息率数据,并分批输出。
"""
print(f"开始处理 {exchange} 交易所的股票数据,时间范围: {start_dt} - {end_dt}")
try:
df_stocks = get_stock_basic(exchange=exchange)
if df_stocks.empty:
print("未获取到股票基础数据")
return
except Exception as e:
print(f"获取股票基础数据时出错: {e}")
return
results_batch = []
all_results = []
processed_count = 0
total_stocks = len(df_stocks)
csv_filename = f"{exchange}_dividend_yield_stats_batch.csv"
csv_first_write = True # 用于控制CSV文件头只写入一次
for idx, row in df_stocks.iterrows():
stock_code = row['ts_code']
print(f"正在处理 {stock_code} ({idx+1}/{total_stocks})")
try:
raw_stats = calculate_dividend_yield_stats(stock_code, start_date=start_dt, end_date=end_dt)
# 确保我们最终得到一个 Series (统计数据)
stats_series = None
if isinstance(raw_stats, pd.Series):
# 如果直接返回了 Series
stats_series = raw_stats
elif isinstance(raw_stats, pd.DataFrame) and not raw_stats.empty:
# 如果返回了 DataFrame,则取第一行
stats_series = raw_stats.iloc[0]
else:
# 如果返回了空DataFrame, None, 或其他类型
print(f"股票 {stock_code} 的 calculate_dividend_yield_stats 返回了空数据或无效类型 ({type(raw_stats)})。跳过...")
continue
yesterday = (datetime.datetime.now() - datetime.timedelta(days=1)).strftime("%Y%m%d")
stockParam = getStockParam(stock_code, START_DATE=yesterday, END_DATE=yesterday)
# 选择需要的字段与下方合并
if not stockParam.empty:
selected_params_dict = {
'ts_code': stockParam['ts_code'].iloc[0],
'trade_date': stockParam['trade_date'].iloc[0],
'total_mv': stockParam['total_mv'].iloc[0],
'circ_mv': stockParam['circ_mv'].iloc[0]
}
else:
selected_params_dict = { 'ts_code': stock_code, 'trade_date': None, 'total_mv': None,'circ_mv': None}
# --- 修改点2:简化合并逻辑 ---
# 现在 stats_series 已确认是 Series,可以直接合并
# 使用字典合并确保结果清晰且 stats 覆盖同名项
combined_dict = {**row.to_dict(), **stats_series.to_dict(), **selected_params_dict}
combined_series = pd.Series(combined_dict)
print(f"合并结果: {combined_series}")
print(f"{stock_code}-{start_dt} -- {end_dt} 处理完成。")
results_batch.append(combined_series)
all_results.append(combined_series)
except Exception as e:
print(f"处理股票 {stock_code} 时出错: {e}")
# 可以选择在这里添加错误记录到 results_batch 或 all_results
continue
processed_count += 1
# 检查是否达到批次大小
if processed_count % dataInterval == 0 and results_batch:
print(f"已处理 {processed_count} 支股票,达到批次大小 {dataInterval},开始输出...")
df_batch = pd.DataFrame(results_batch)
# 输出到CSV (追加模式)
save_to_csv(df_batch, csv_filename, mode='a', header=csv_first_write)
if csv_first_write: csv_first_write = False # 之后的批次不再写入header
# 输出到MySQL (伪代码)
save_to_mysql(df_batch, 'stock_dividend_stats')
# 清空批次缓存
results_batch = []
print("--- 批次处理完成 ---")
# 处理最后一批不足dataInterval的数据
if results_batch:
print(f"处理剩余 {len(results_batch)} 支股票...")
df_final_batch = pd.DataFrame(results_batch)
save_to_csv(df_final_batch, csv_filename, mode='a', header=csv_first_write)
save_to_mysql(df_final_batch, 'stock_dividend_stats')
print("--- 最终批次处理完成 ---")
# (可选) 将所有结果一次性保存到一个完整的CSV文件
if all_results:
final_csv_filename = f"{exchange}_dividend_yield_stats_final.csv"
print(f"正在保存所有 {len(all_results)} 条结果到 {final_csv_filename}...")
df_final = pd.DataFrame(all_results)
save_to_csv(df_final, final_csv_filename, mode='w', header=True)
save_to_mysql(df_final, 'stock_dividend_stats_final')
print("所有数据处理并保存完成。")
else:
print("无有效结果数据可保存")
# --- 程序入口 ---
if __name__ == "__main__":
# 可以通过修改这里的参数来调用函数
process_stock_dividend_batch(start_dt='20150101', end_dt='20251231', dataInterval=10, exchange='SSE')
process_stock_dividend_batch(start_dt='20150101', end_dt='20251231', dataInterval=10, exchange='SZSE')
+145
View File
@@ -0,0 +1,145 @@
import pandas as pd
# 导入当站目录的config文件
try:
# 尝试相对导入(作为包的一部分)
from .config import TS_TOKEN, START_DATE, END_DATE, PRECISION_CONFIG
from .stock_utils import *
except (ImportError, SystemError):
# 失败则使用绝对导入(直接运行脚本)
from config import TS_TOKEN, START_DATE, END_DATE, PRECISION_CONFIG
from stock_utils import *
from .data_source import get_tushare_pro
pro = get_tushare_pro()
def get_index_by_name(index_name: str) -> pd.DataFrame:
"""
通过指数名称查询指数基本信息(使用tushare的index_basic接口)
Args:
index_name (str): 要查询的指数名称(支持模糊匹配,如"上证"
Returns:
pd.DataFrame: 包含查询结果的DataFrame,列包括:
- ts_code: 指数代码
- name: 指数名称
- fullname: 指数全称
- market: 市场
- publisher: 发布方
- index_type: 指数类型
- etc.
Raises:
Exception: 当tushare接口调用失败时抛出异常
Example:
>>> df = get_index_by_name("上证50")
>>> print(df[['ts_code', 'name']])
"""
try:
# 调用tushare接口查询指数信息
df = pro.index_basic(name=index_name)
# 检查返回结果是否为空
if df.empty:
print(f"未找到名称包含'{index_name}'的指数")
return pd.DataFrame() # 返回空DataFrame保持类型一致
return df
except Exception as e:
print(f"查询指数信息失败: {str(e)}")
return pd.DataFrame()
def get_index_daily_data(ts_code: str, start_date: str = START_DATE, end_date: str = END_DATE) -> pd.DataFrame:
"""
通过指数代码查询日线行情数据(合并基本行情和扩展行情)
Args:
ts_code (str): 指数代码(如"000001.SH"
start_date (str): 开始日期(格式"YYYYMMDD",默认使用config中的START_DATE
end_date (str): 结束日期(格式"YYYYMMDD",默认使用config中的END_DATE
Returns:
pd.DataFrame: 合并后的日线行情数据,包含以下列(示例):
- ts_code: 指数代码
- trade_date: 交易日期
- close: 收盘点位
- open: 开盘点位
- high: 最高点位
- low: 最低点位
- pe: 市盈率
- pb: 市净率
- etc.
Raises:
Exception: 当tushare接口调用失败或数据合并失败时抛出异常
Example:
>>> df = get_index_daily_data("000001.SH")
>>> print(df[['trade_date', 'close', 'pe']].head())
"""
try:
start_date = date_format_correction(start_date)
end_date = date_format_correction(end_date)
# 1. 查询日线基本行情(核心数据,必须成功)
daily_df = pro.index_daily(
ts_code=ts_code,
start_date=start_date,
end_date=end_date,
fields="ts_code,trade_date,open,high,low,close,pre_close,change,pct_chg,vol,amount"
)
if daily_df is None or daily_df.empty:
print("未找到指数{}{}{}期间的行情数据".format(ts_code, start_date, end_date))
return pd.DataFrame()
# 2. 尝试查询扩展行情(PE/PB 等,需要高权限,失败不阻塞)
try:
daily_basic_df = pro.index_dailybasic(
ts_code=ts_code,
start_date=start_date,
end_date=end_date,
fields="ts_code,trade_date,total_mv,float_mv,pe,pe_ttm,pb,turnover_rate,turnover_rate_f"
)
if daily_basic_df is not None and not daily_basic_df.empty:
daily_df = pd.merge(
daily_df, daily_basic_df,
on=['ts_code', 'trade_date'],
how='left'
)
except Exception as e:
print("index_dailybasic 不可用(权限不足或接口变更): {}".format(e))
return daily_df
except Exception as e:
print("查询指数日线行情失败: {}".format(e))
raise # 抛出而非静默返回空,让 view 层返回错误信息
if __name__ == "__main__":
# 测试指数查询
print("测试指数查询:")
index_df = get_index_by_name("中证500")
if not index_df.empty:
print(index_df[['ts_code', 'name']].head())
else:
print("未查询到指数信息")
'''# 测试指数行情获取
print("\n测试指数日线行情:")
test_code = "000016.SH" # 上证50指数代码
daily_data = get_index_daily_data(test_code, start_date="20230101", end_date="20231231")
if not daily_data.empty:
print(f"获取到{daily_data.shape[0]}条数据")
# 显示关键列的前5行
print(daily_data[['trade_date', 'close', 'pe', 'pb']].head())
# 数据完整性检查
print("\n数据完整性检查:")
print(daily_data[['close', 'pe']].describe())
else:
print(f"未获取到指数{test_code}的行情数据")'''
+135
View File
@@ -0,0 +1,135 @@
import pandas as pd
import numpy as np
from datetime import datetime
try:
# 尝试相对导入(作为包的一部分)
from .config import TS_TOKEN, START_DATE, END_DATE, PRECISION_CONFIG
from .stock_utils import *
except (ImportError, SystemError):
# 失败则使用绝对导入(直接运行脚本)
from config import TS_TOKEN, START_DATE, END_DATE, PRECISION_CONFIG
from stock_utils import *
'''
给定个股代码和起止日期:
1. 调用tushare接口查询个股分红数据,接口文档:https://tushare.pro/document/2?doc_id=103 查询日期范围内的所有分红记录
2. 返回调整后的分红记录,包含以下字段:
- ts_code: 股票代码
- end_date: 分红年度
- ann_date: 公告日期
- ex_date: 除权除息日
- cash_div_tax: 每股现金分红(含税)
- div_proc: 实施进度,仅筛选 div_proc='实施'的记录
3. 调用tushare 的trade_cal接口,查询起止日期内的所有交易日
4. 建一个新的df,在起止日期内填充,规则:
- 下列运算过程中将起始时间往前推 gap_days,赋值360天。运算结束后,截取起止时间内的数据
- trade_date: 以交易日历为准,填充所有交易日。
- 填入当ex_date=trade_date的日期,填入:ex_date,cash_div_tax,其余日期ex_date留空,cash_div_tax 置 0
- cash_div_year:逐行计算填入,以trade_date往前计算过去gap_days天内的cash_div_tax之和
5. 调用tushare的个股日线行情接口:https://tushare.pro/document/2?doc_id=27,查询起止日期内的个股日线行情数据,包含以下字段:
- ts_code: 股票代码
- trade_date: 交易日期
- close: 收盘价
6. 将分红数据和日线行情数据按交易日期合并,得到最终结果,包含以下字段:
- ts_code: 股票代码
- trade_date: 交易日期
- close: 收盘价
- cach_div_tax: 每股现金分红(含税)
- cach_div_year: 每股现金TTM年度分红(含税)
- div_yield: 股息率,计算公式为 (cach_div_year / close) * 100,保留PRECISION_CONFIG位小数,如果cash_div_year为0则div_yield也为0
7. 返回最终结果的DataFrame
'''
# 初始化tushare
from .data_source import get_tushare_pro
pro = get_tushare_pro()
def get_dividend_yield(ts_code, start_date, end_date):
# 检查日期范围是否超过当前日期,若超过则调整为当前日期
today = datetime.now().strftime("%Y%m%d")
if end_date > today:
end_date = today
if start_date > today:
start_date = today
gap_days = 360 # 定义TTM计算窗口期为360天
# 1. 获取分红数据:查询指定股票的分红信息
df_div = pro.dividend(ts_code=ts_code, fields='ts_code,end_date,ann_date,ex_date,cash_div_tax,div_proc')
# 筛选实施状态的分红记录
df_div = df_div[df_div['div_proc'] == '实施']
print(f"分红原始数据:\n {df_div.to_string()} ")
# 2. 获取交易日历:查询指定日期范围内的开盘日
df_cal = pro.trade_cal(exchange='', start_date=start_date, end_date=end_date, is_open='1')
trade_dates = df_cal['cal_date'].tolist()
# 3. 创建基础DataFrame:将计算窗口期向前扩展gap_days天
extended_start = pd.to_datetime(start_date) - pd.Timedelta(days=gap_days)
extended_start_str = extended_start.strftime('%Y%m%d')
# 获取扩展后的交易日历
df_cal_ext = pro.trade_cal(exchange='', start_date=extended_start_str, end_date=end_date, is_open='1')
df_base = pd.DataFrame({'trade_date': df_cal_ext['cal_date']})
# 4. 合并分红数据:将分红信息按除权除息日合并到基础交易日历
df_base = df_base.merge(df_div[['ex_date', 'cash_div_tax']],
left_on='trade_date', right_on='ex_date', how='left')
# 填充空值:无分红日期现金分红设为0
df_base['cash_div_tax'] = df_base['cash_div_tax'].fillna(0)
# 5. 计算TTM年度分红:滚动计算过去gap_days天的现金分红总和
# 修正滚动窗口计算:使用固定窗口大小,确保在窗口期内正确累加
df_base['cash_div_year'] = df_base['cash_div_tax'].rolling(window=gap_days, min_periods=0).sum()
print(f"分红填充数据S1\n {df_base.to_string()} ")
# 6. 截取指定日期范围:保留原始查询日期范围内的数据
df_base = df_base[df_base['trade_date'] >= start_date]
print(f"分红填充数据S2\n {df_base.to_string()} ")
# 7. 获取日线行情数据:查询指定股票的日线收盘价
df_daily = pro.daily(ts_code=ts_code, start_date=start_date, end_date=end_date,
fields='ts_code,trade_date,close')
# 8. 合并数据:将行情数据与分红数据按交易日合并
result = df_base.merge(df_daily, on='trade_date', how='left')
# 填充股票代码:确保所有行都有股票代码
result['ts_code'] = result['ts_code'].fillna(ts_code)
# 9. 计算股息率:当TTM分红>0时计算(分红/收盘价)*100,否则为0
result['div_yield'] = np.where(
result['cash_div_year'] > 0,
(result['cash_div_year'] / result['close']) * 100,
0
)
# 精度处理:保留配置指定的小数位数
result['div_yield'] = result['div_yield'].round(PRECISION_CONFIG)
# 10. 整理列顺序:选择并排列最终输出的列
result = result[['ts_code', 'trade_date', 'close', 'cash_div_tax', 'cash_div_year', 'div_yield']]
return result
if __name__ == "__main__":
# 测试代码
test_code = "600900.SH"
test_start = "20230101"
test_end = "20251231"
try:
result = get_dividend_yield(test_code, test_start, test_end)
print(f"股票 {test_code} 的分红股息率数据:")
print(result.head(100))
print(f"\n数据形状: {result.shape}")
print(f"\n数据列名: {result.columns.tolist()}")
# 检查是否有分红数据
if not result.empty:
print(f"\n股息率统计:")
print(result['div_yield'].describe())
else:
print("未找到分红数据")
except Exception as e:
print(f"测试过程中出现错误: {e}")
+179
View File
@@ -0,0 +1,179 @@
import pandas as pd
import numpy as np
from datetime import datetime
try:
from .config import TS_TOKEN, START_DATE, END_DATE, PRECISION_CONFIG
from .stock_utils import *
from .smoothBrush import smooth_dataframe_brush
except (ImportError, SystemError):
from config import TS_TOKEN, START_DATE, END_DATE, PRECISION_CONFIG
from stock_utils import *
from smoothBrush import smooth_dataframe_brush
from .data_source import get_tushare_pro
pro = get_tushare_pro()
'''
给定个股代码和起止日期:
1. 调用tushare接口查询个股分红数据,接口文档:https://tushare.pro/document/2?doc_id=103 查询日期范围内的所有分红记录
2. 返回调整后的分红记录,包含以下字段:
- ts_code: 股票代码
- end_date: 分红年度
- ann_date: 公告日期
- ex_date: 除权除息日
- cash_div_tax: 每股现金分红(含税)
- div_proc: 实施进度,仅筛选 div_proc='实施'的记录
3. 调用tushare 的trade_cal接口,查询起止日期内的所有交易日
4. 建一个新的df,在起止日期内填充,规则:
- 下列运算过程中将起始时间往前推 gap_days,赋值360天。运算结束后,截取起止时间内的数据
- trade_date: 以交易日历为准,填充所有交易日。
- 填入当ex_date=trade_date的日期,填入:ex_date,cash_div_tax,其余日期ex_date留空,cash_div_tax 置 0
- cash_div_year:逐行计算填入,以trade_date往前计算过去gap_days天内的cash_div_tax之和
5. 调用tushare的个股日线行情接口:https://tushare.pro/document/2?doc_id=27,查询起止日期内的个股日线行情数据,包含以下字段:
- ts_code: 股票代码
- trade_date: 交易日期
- close: 收盘价
6. 将分红数据和日线行情数据按交易日期合并,得到最终结果,包含以下字段:
- ts_code: 股票代码
- trade_date: 交易日期
- close: 收盘价
- cach_div_tax: 每股现金分红(含税)
- cach_div_year: 每股现金TTM年度分红(含税)
- div_yield: 股息率,计算公式为 (cach_div_year / close) * 100,保留PRECISION_CONFIG位小数,如果cash_div_year为0则div_yield也为0
7. 返回最终结果的DataFrame
'''
def analyze_stock_dividend_and_price(ts_code, start_date=START_DATE, end_date=END_DATE):
# 检查日期范围是否超过当前日期,若超过则调整为当前日期
start_date=date_format_correction(start_date)
end_date=date_format_correction(end_date)
today = datetime.now().strftime("%Y%m%d")
if end_date > today: end_date = today
if start_date > today: start_date = today
GAP_DAYS = 360 # 定义TTM计算窗口期为360天
"""
分析个股分红与行情数据。
参数:
ts_code (str): 股票代码,例如 '000001.SZ'
start_date (str): 起始日期,格式 'YYYYMMDD'
end_date (str): 结束日期,格式 'YYYYMMDD'
返回:
pd.DataFrame: 包含合并后数据的DataFrame。
"""
# 1. 调用tushare接口查询个股分红数据
try:
df_div_raw = pro.dividend(ts_code=ts_code)
# 2. 筛选并调整分红记录
# 筛选实施进度为'实施'的记录,并在指定日期范围内
df_div_filtered = df_div_raw[
(df_div_raw['div_proc'] == '实施')
].copy()
df_div_adjusted = df_div_filtered[[
'ts_code', 'end_date', 'ann_date', 'ex_date', 'cash_div_tax'
]].reset_index(drop=True)
except Exception as e:
print(f"获取或处理分红数据时出错: {e}")
return pd.DataFrame() # 返回空DataFrame
# 3. 调用tushare 的trade_cal接口,查询交易日
try:
# 运算时起始时间往前推 GAP_DAYS
calc_start_date = pd.to_datetime(start_date) - pd.Timedelta(days=GAP_DAYS)
calc_start_date_str = calc_start_date.strftime('%Y%m%d')
df_cal = pro.trade_cal(exchange='', start_date=calc_start_date_str, end_date=end_date)
# 筛选交易日
trade_dates_all = df_cal[df_cal['is_open'] == 1]['cal_date'].sort_values().tolist()
except Exception as e:
print(f"获取交易日历时出错: {e}")
return pd.DataFrame()
# 4. 建立新df并填充
df_div_processed = pd.DataFrame({'trade_date': trade_dates_all})
df_div_processed['trade_date'] = pd.to_datetime(df_div_processed['trade_date'], format='%Y%m%d')
# 将原始分红数据的ex_date也转为datetime以便合并
df_div_adjusted['ex_date'] = pd.to_datetime(df_div_adjusted['ex_date'], format='%Y%m%d')
# 合并分红数据到交易日历
df_merged_temp = df_div_processed.merge(df_div_adjusted[['ex_date', 'cash_div_tax']],
left_on='trade_date', right_on='ex_date', how='left')
df_merged_temp.drop('ex_date', axis=1, inplace=True)
# 填充空值
df_merged_temp['cash_div_tax'] = df_merged_temp['cash_div_tax'].fillna(0.0)
# 计算 cash_div_year (TTM) — 向量化 rolling 窗口
df_merged_temp = df_merged_temp.sort_values('trade_date').reset_index(drop=True)
df_temp = df_merged_temp.set_index('trade_date')
df_temp['cash_div_year'] = df_temp['cash_div_tax'].rolling(f'{GAP_DAYS}D', min_periods=1).sum()
df_merged_temp['cash_div_year'] = df_temp['cash_div_year'].values
# 截取原始请求的起止时间内的数据
start_date_dt = pd.to_datetime(start_date, format='%Y%m%d')
end_date_dt = pd.to_datetime(end_date, format='%Y%m%d')
df_div_final = df_merged_temp[
(df_merged_temp['trade_date'] >= start_date_dt) &
(df_merged_temp['trade_date'] <= end_date_dt)
].copy()
df_div_final['trade_date'] = df_div_final['trade_date'].dt.strftime('%Y%m%d')
# 5. 调用tushare的个股日线行情接口
try:
df_daily = pro.daily(ts_code=ts_code, start_date=start_date, end_date=end_date)
df_daily = df_daily[['ts_code', 'trade_date', 'close']].sort_values('trade_date').reset_index(drop=True)
except Exception as e:
print(f"获取日线行情数据时出错: {e}")
return pd.DataFrame()
# 6. 合并分红数据和日线行情数据
df_result = df_daily.merge(df_div_final[['trade_date', 'cash_div_tax', 'cash_div_year']],
on='trade_date', how='left')
# 填充因合并可能产生的NaN(例如,某日有行情但无分红记录)
df_result['cash_div_tax'] = df_result['cash_div_tax'].fillna(0.0)
df_result['cash_div_year'] = df_result['cash_div_year'].fillna(0.0)
# 毛刺平滑处理 cash_div_year 列
df_result=smooth_dataframe_brush(df_result, target_columns=['cash_div_year'], window_size=31, threshold_factor=0.5, max_brush_length=15 )
# 恢复原始日期顺序
df_result = df_result.sort_values('trade_date').reset_index(drop=True)
# 计算股息率
df_result['div_yield'] = 0.0
mask_non_zero_price = df_result['close'] > 0
mask_non_zero_div_year = df_result['cash_div_year'] > 0
# 只对收盘价大于0且TTM分红大于0的记录计算股息率
valid_mask = mask_non_zero_price & mask_non_zero_div_year
df_result.loc[valid_mask, 'div_yield'] = (
(df_result.loc[valid_mask, 'cash_div_year'] / df_result.loc[valid_mask, 'close']) * 100
).round(PRECISION_CONFIG)
# 7. 返回最终结果
# 重命名字段以匹配要求 (注意: 题目中'cach_div_tax'应为'cash_div_tax')
#df_result.rename(columns={'cash_div_tax': 'cach_div_tax', 'cash_div_year': 'cach_div_year'}, inplace=True)
final_columns = ['ts_code', 'trade_date', 'close', 'cash_div_tax', 'cash_div_year', 'div_yield']
df_final = df_result[final_columns]
return df_final
# --- 示例用法 ---
if __name__ == "__main__":
start_dt = '2020-01-01'
end_dt = '2025-12-31'
ts_code = '000001.SZ'
result = analyze_stock_dividend_and_price(ts_code, start_dt, end_dt)
print(f"股票 {ts_code} 的分红与行情数据分析结果:")
print(result.head(10))
print(f"\n数据总行数: {len(result)}")
+393
View File
@@ -0,0 +1,393 @@
import pandas as pd
# 将项目根目录添加到 sys.path
#project_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
#sys.path.append(project_root)
# 导入当站目录的config文件
try:
# 尝试相对导入(作为包的一部分)
from .config import TS_TOKEN, START_DATE, END_DATE, PRECISION_CONFIG
from .stock_utils import *
except (ImportError, SystemError):
# 失败则使用绝对导入(直接运行脚本)
from config import TS_TOKEN, START_DATE, END_DATE, PRECISION_CONFIG
from stock_utils import *
from .data_source import get_tushare_pro
pro = get_tushare_pro()
def getStockEp(TS_CODE,start_date=START_DATE,end_date=END_DATE):
"""
从tushare接口获取单只股票的财务数据,计算并填充每日的每股收益(EP)指标。
Parameters:
TS_CODE (str): 股票代码,格式为 '股票代码.SZ''股票代码.SH',例如 '000001.SZ'
START_DATE (str): 开始日期,格式为 'YYYYMMDD'
END_DATE (str): 结束日期,格式为 'YYYYMMDD'
Returns:
pd.DataFrame: 包含以下字段的DataFrame:
ts_code: 股票代码
ann_date: 财报公告日期
end_date: 财报结束日期/填充日期
basic_eps: 基本每股收益
diluted_eps: 稀释每股收益
trade_date: 实际交易日(来自日线数据)
close: 收盘价
basic_ep: 基本EP值(basic_eps/close)
diluted_ep: 稀释EP值(diluted_eps/close)
Raises:
Exception: 如果从Tushare接口获取数据时发生错误。
"""
'''
1. 调用get_trading_dates函数,填充完整df_income 变量内部end_date,规则为:以end_date为限,往前天从到上一个财报日期为止(不包含上一个财报日期),期间填充basic_eps,diluted_eps值为报告日的相同值
2. 新增一个basic_ep列,数据为:basic_eps/当天交易日期的股价,当天交易日期的股价来自df_daily的close列
3. 新增一个diluted_ep列,数据为:diluted_eps/当天交易日期的股价,当天交易日期的股价来自df_daily的close列
'''
try:
# 获取股票财务数据
df_income = pro.income(ts_code=TS_CODE, start_date=start_date, end_date=end_date,
fields='ts_code,ann_date,end_date,basic_eps,diluted_eps')
# 填充df_income的end_date区间数据
if not df_income.empty:
# 按end_date分组并排序财报数据
df_income = df_income.sort_values('end_date')
grouped = df_income.groupby('end_date')
# 存储填充后的数据
filled_data = []
# 遍历每个财报期间
for end_date, group in grouped:
# 获取该日期到上一个财报日的所有交易日期
trading_dates = get_trading_dates(end_date)
# 填充数据
for date in trading_dates:
# 将日期对象转换为字符串格式(YYYYMMDD)
# 如果date是日期对象(有strftime方法),则调用strftime格式化
# 否则直接使用原值(假设已经是字符串格式)
date_str = date.strftime('%Y%m%d') if hasattr(date, 'strftime') else date
filled_row = {
'ts_code': TS_CODE,
'ann_date': group['ann_date'].iloc[0],
'end_date': date_str,
'basic_eps': group['basic_eps'].iloc[0],
'diluted_eps': group['diluted_eps'].iloc[0]
}
filled_data.append(filled_row)
# 合并填充后的数据
df_income = pd.DataFrame(filled_data)
# 获取股票日线数据以获取股价
min_end_date = df_income['end_date'].min()
df_daily = pro.daily(ts_code=TS_CODE, start_date=min_end_date, end_date=END_DATE)
# 2. 计算basic_ep和diluted_ep
if not df_income.empty and not df_daily.empty:
# 修改原因:原代码使用left_on='end_date'和right_on='trade_date'导致匹配失败
# 解决方案:将df_daily的trade_date转换为字符串格式再合并
df_daily['trade_date_str'] = df_daily['trade_date'].astype(str)
df_income = pd.merge(
df_income,
df_daily[['trade_date_str', 'close']],
left_on='end_date',
right_on='trade_date_str',
how='left'
)
# 将合并后的trade_date_str重命名为trade_date
df_income.rename(columns={'trade_date_str': 'trade_date'}, inplace=True)
# 计算basic_ep和diluted_ep
df_income['basic_ep'] = (df_income['basic_eps'] / df_income['close']).round(PRECISION_CONFIG)
df_income['diluted_ep'] = (df_income['diluted_eps'] / df_income['close']).round(PRECISION_CONFIG)
#print("df_income:",df_income)
# 按日期截断
#df_income = df_income[(df_income['trade_date'] >= START_DATE) & (df_income['trade_date'] <= END_DATE)]
# 数据修正
cols = ['basic_eps', 'diluted_eps', 'close', 'basic_ep', 'diluted_ep']
df_income = dataCorrect(df_income, cols).sort_values('trade_date')
return df_income
except Exception as e:
# 捕获并处理异常
print(f"错误: 获取股票 {TS_CODE} 的EP数据时发生错误: {e}")
return pd.DataFrame()
def getStockEp_ttm(TS_CODE,start_date=START_DATE,end_date=END_DATE):
"""
获取股票的TTM(最近12个月)每股收益与股价比率(EP)数据
Parameters:
TS_CODE (str): 股票代码,格式为 '股票代码.SZ''股票代码.SH',例如 '000001.SZ'
START_DATE (str): 开始日期,格式为 'YYYYMMDD'
END_DATE (str): 结束日期,格式为 'YYYYMMDD'
Returns:
pd.DataFrame: 包含以下字段的DataFrame:
ts_code: 股票代码
trade_date: 交易日
close: 收盘价
basic_ep_ttm: 基本EP_TTM值(basic_eps_q_ttm/close)
diluted_ep_ttm: 稀释EP_TTM值(diluted_eps_q_ttm/close)
basic_eps_q_ttm: 基本每股收益TTM值
diluted_eps_q_ttm: 稀释每股收益TTM值
Raises:
Exception: 如果从Tushare接口获取数据时发生错误
"""
try:
start_date=date_format_correction(start_date) #confirm date as yyyymmdd
end_date=date_format_correction(end_date) #confirm date as yyyymmdd
df_eps = get_quarterly_eps(tscodeCheck(TS_CODE), start_date, end_date)
df_eps_ttm = calculate_ttm_eps(df_eps)
df_eps_ttm=fill_trading_dates_with_eps(df_eps_ttm,start_date=start_date,end_date=end_date)
df_daily = pro.daily(ts_code=TS_CODE, start_date=start_date, end_date=end_date)
df_eps_ttm = pd.merge(
df_eps_ttm,
df_daily[['trade_date', 'close']],
on='trade_date',
how='left'
)
df_eps_ttm['basic_ep_ttm'] = (100*df_eps_ttm['basic_eps_q_ttm'] / df_eps_ttm['close']).round(PRECISION_CONFIG)
df_eps_ttm['diluted_ep_ttm'] = (100*df_eps_ttm['diluted_eps_q_ttm'] / df_eps_ttm['close']).round(PRECISION_CONFIG)
# 比较最大日期并补充数据, 解决当ep不存在,close值也不会显示的问题
if not df_eps_ttm.empty and not df_daily.empty:
max_eps_date = df_eps_ttm['trade_date'].max()
max_daily_date = df_daily['trade_date'].max()
print(f"max_eps_date:{max_eps_date}")
print(f"max_daily_date:{max_daily_date}")
if max_eps_date != max_daily_date:
# 获取需要补充的日期范围
mask = (df_daily['trade_date'] > max_eps_date) & (df_daily['trade_date'] <= max_daily_date)
additional_data = df_daily.loc[mask, ['ts_code', 'trade_date', 'close']].copy()
# 补充空列
for col in df_eps_ttm.columns:
if col not in ['ts_code', 'trade_date', 'close']:
additional_data[col] = ''
print(additional_data)
# 合并数据
df_eps_ttm = pd.concat([df_eps_ttm, additional_data], ignore_index=True)
cols=["close","basic_ep_ttm","diluted_ep_ttm","basic_eps_q_ttm","diluted_eps_q_ttm"]
df_eps_ttm = dataCorrect(df_eps_ttm,cols)
return df_eps_ttm[['ts_code', 'trade_date', 'close', 'basic_ep_ttm', 'diluted_ep_ttm', 'basic_eps_q_ttm', 'diluted_eps_q_ttm']]
except Exception as e:
# 捕获并处理异常
print(f"错误: 获取股票 {TS_CODE} 的EP_TTM数据时发生错误: {e}")
return pd.DataFrame()
def get_quarterly_eps(TS_CODE, start_date=START_DATE, end_date=END_DATE):
"""
获取单季度每股收益数据
Parameters:
TS_CODE (str): 股票代码
START_DATE (str): 开始日期(YYYYMMDD)
END_DATE (str): 结束日期(YYYYMMDD)
Returns:
pd.DataFrame: 包含以下字段的DataFrame:
ts_code: 股票代码
ann_date: 财报公告日期
end_date: 财报结束日期(YYYYMMDD格式)
basic_eps: 累计基本每股收益
diluted_eps: 累计稀释每股收益
basic_eps_q: 单季度基本每股收益
diluted_eps_q: 单季度稀释每股收益
"""
try:
# 扩展日期范围往前三个季度
extended_start = (pd.to_datetime(start_date) - pd.DateOffset(months=12)).strftime('%Y%m%d')
# 获取原始财务数据
df = pro.income(ts_code=TS_CODE, start_date=extended_start, end_date=end_date,
fields='ts_code,ann_date,end_date,basic_eps,diluted_eps')
if df.empty:
return pd.DataFrame()
# 按财报日期排序并去重(解决重复数据问题)
df = df.sort_values('end_date').drop_duplicates(subset=['end_date'], keep='last') # 修改原因:确保每个end_date只保留最新数据
# 计算单季度数据
df['basic_eps_q'] = df['basic_eps']
df['diluted_eps_q'] = df['diluted_eps']
# 非第一季度数据需要减去上季度数据
mask = ~df['end_date'].str.endswith('0331')
df.loc[mask, 'basic_eps_q'] = df['basic_eps'].diff()
df.loc[mask, 'diluted_eps_q'] = df['diluted_eps'].diff()
# 过滤掉非季报数据(保留3/6/9/12月数据)
df = df[df['end_date'].str.endswith(('0331', '0630', '0930', '1231'))]
# 删除最小日期的数据行
if not df.empty:
df = df[df['end_date'] != df['end_date'].min()]
df['basic_eps_q'] = df['basic_eps_q'].round(PRECISION_CONFIG)
df['diluted_eps_q'] = df['diluted_eps_q'].round(PRECISION_CONFIG)
# 重置行号并保留原始EPS值
#return df[['ts_code', 'ann_date', 'end_date', 'basic_eps', 'diluted_eps', 'basic_eps_q', 'diluted_eps_q']].reset_index(drop=True)
#重置行号,去掉原始EPS值
return df[['ts_code', 'ann_date', 'end_date', 'basic_eps_q', 'diluted_eps_q']].reset_index(drop=True)
except Exception as e:
print(f"获取季度EPS数据出错: {e}")
return pd.DataFrame()
def calculate_ttm_eps(df):
"""
计算EPS指标的TTM(最近12个月)值
Parameters:
df (pd.DataFrame): get_quarterly_eps函数返回的DataFrame,包含以下列:
- ts_code: 股票代码
- ann_date: 公告日期
- end_date: 财报结束日期
- basic_eps: 基本每股收益(累计)
- diluted_eps: 稀释每股收益(累计)
- basic_eps_q: 单季度基本每股收益
- diluted_eps_q: 单季度稀释每股收益
Returns:
pd.DataFrame: 包含原始数据和TTM计算结果的DataFrame,新增以下列:
- basic_eps_ttm: 基本每股收益TTM值
- diluted_eps_ttm: 稀释每股收益TTM值
- basic_eps_q_ttm: 单季度基本每股收益TTM值
- diluted_eps_q_ttm: 单季度稀释每股收益TTM值
"""
if df.empty:
return df
try:
# 确保数据按end_date降序排列
df = df.sort_values('end_date', ascending=False).reset_index(drop=True)
# 初始化TTM结果列
#df['basic_eps_ttm'] = None
#df['diluted_eps_ttm'] = None
df['basic_eps_q_ttm'] = None
df['diluted_eps_q_ttm'] = None
# 遍历每一行数据计算TTM
for i in range(len(df)):
# 检查是否有足够的后续数据(至少3个季度)
if i + 3 >= len(df):
continue # 数据不足,跳过计算
# 计算TTM值(当前季度+后续3个季度)
#df.at[i, 'basic_eps_ttm'] = df.loc[i:i+3, 'basic_eps'].sum()
# df.at[i, 'diluted_eps_ttm'] = df.loc[i:i+3, 'diluted_eps'].sum()
df.at[i, 'basic_eps_q_ttm'] = df.loc[i:i+3, 'basic_eps_q'].sum()
df.at[i, 'diluted_eps_q_ttm'] = df.loc[i:i+3, 'diluted_eps_q'].sum()
# 四舍五入保留指定小数位数
ttm_cols = [ 'basic_eps_q_ttm', 'diluted_eps_q_ttm']
df[ttm_cols] = df[ttm_cols].round(PRECISION_CONFIG)
# 删除最后三行数据,因为最后三行ttm数据为None
df = df.iloc[:-3]
return df
except Exception as e:
print(f"计算TTM值时出错: {e}")
return pd.DataFrame()
def fill_trading_dates_with_eps(df_ttm,start_date=START_DATE,end_date=END_DATE):
"""
填充交易日期并保留EPS值
参数:
df_ttm (pd.DataFrame): calculate_ttm_eps函数返回的DataFrame,包含以下列:
- end_date: 财报结束日期(YYYYMMDD格式)
- basic_eps_ttm: 基本每股收益TTM值
- diluted_eps_ttm: 稀释每股收益TTM值
- basic_eps_q_ttm: 单季度基本每股收益TTM值
- diluted_eps_q_ttm: 单季度稀释每股收益TTM值
返回:
pd.DataFrame: 包含填充后的交易日期和对应EPS值的DataFrame
异常处理:
- 输入为空DataFrame时直接返回
- Tushare接口调用失败时返回原始数据
"""
if df_ttm.empty:
return df_ttm
try:
# 1. 准备数据: 按end_date排序并转换为datetime格式
df_ttm = df_ttm.sort_values('end_date')
df_ttm['end_date_dt'] = pd.to_datetime(df_ttm['end_date'])
# 2. 获取所有需要填充的日期区间
date_ranges = []
for i in range(len(df_ttm)-1):
s_date = df_ttm['end_date_dt'].iloc[i]
e_date = df_ttm['end_date_dt'].iloc[i+1]
date_ranges.append((s_date, e_date))
# 检查是否需要添加最后一个区间
if df_ttm['end_date_dt'].max() < pd.to_datetime(end_date):
next_report_date = pd.to_datetime(get_next_report_date(df_ttm['end_date_dt'].max()))
next_report_date = min(next_report_date, pd.to_datetime(end_date))
date_ranges.append((df_ttm['end_date_dt'].max(), next_report_date))
# 3. 获取交易所交易日历
exchange = 'SZSE' if df_ttm['ts_code'].iloc[0].endswith('SZ') else 'SSE' # 可以不用考虑SH,SZ,BJ, 理论上日历应该是一样的
trade_cal = pro.trade_cal(exchange=exchange,
start_date=start_date,
end_date=end_date)
# 过滤出交易日
trade_cal = trade_cal[trade_cal['is_open'] == 1]
trade_cal['cal_date_dt'] = pd.to_datetime(trade_cal['cal_date'])
# 4. 填充每个区间内的交易日
filled_data = []
#eps_cols = ['basic_eps_ttm', 'diluted_eps_ttm', 'basic_eps_q_ttm', 'diluted_eps_q_ttm']
# 首先添加原始数据
for _, row in df_ttm.iterrows():
filled_data.append(row.to_dict())
# 然后填充区间数据
for start_date, end_date in date_ranges:
# 获取该区间内的所有交易日
mask = (trade_cal['cal_date_dt'] > start_date) & (trade_cal['cal_date_dt'] < end_date)
dates_in_range = trade_cal[mask]['cal_date_dt']
# 使用较小的日期(即start_date)的EPS值填充
ref_row = df_ttm[df_ttm['end_date_dt'] == start_date].iloc[0]
for date in dates_in_range:
new_row = ref_row.copy()
new_row['end_date'] = date.strftime('%Y%m%d')
new_row['end_date_dt'] = date
filled_data.append(new_row)
# 5. 转换为DataFrame并整理
result = pd.DataFrame(filled_data)
result = result.sort_values('end_date_dt')
# 删除临时列并重置索引
result = result.drop(columns=['end_date_dt']).reset_index(drop=True)
result = result.drop(columns=['ann_date']) #去掉ann_date列
result = result.rename(columns={'end_date': 'trade_date'}) #重命名 end_date 为trade_date
# 过滤掉非交易日
result = result[result['trade_date'].isin(trade_cal['cal_date'])]
return result
except Exception as e:
print(f"填充交易日期时出错: {e}, 返回原始数据")
return df_ttm
if __name__ == '__main__':
#df = getStockEp(tscodeCheck('688469'))
'''df = get_quarterly_eps(tscodeCheck('688556'), '20230101', END_DATE)
print(df)
df2 = calculate_ttm_eps(df)
print(df2)
df3=fill_trading_dates_with_eps(df2)
print(df3)'''
df4= getStockEp_ttm(tscodeCheck('300316'), '20230101', END_DATE)
print(df4)
+288
View File
@@ -0,0 +1,288 @@
import pandas as pd
import numpy as np
try:
from .config import TS_TOKEN, START_DATE, END_DATE, PRECISION_CONFIG
from .stock_utils import *
except (ImportError, SystemError):
from config import TS_TOKEN, START_DATE, END_DATE, PRECISION_CONFIG
from stock_utils import *
from .data_source import get_tushare_pro
class FinanceData:
def __init__(self, token=None):
"""
初始化 Tushare 接口
:param token: 可选,不再使用(保留兼容)
"""
self.pro = get_tushare_pro()
self.ts_code = None
self.fin_date = None
self.unit_factor = 100000000 # 单位换算因子 (元 -> 亿)
def get_finance_data(self):
"""
获取完整的财务数据并计算指标
:param ts_code: 股票代码 (如 '002273.SZ')
:param fin_date: 财报日期 (如 '20220331')
:return: 包含财务指标的字典
"""
# 获取各类财务数据 (返回 DataFrame)
balance = self.get_balance()
pre_balance = self.get_pre_balance()
income = self.get_income()
cash = self.get_cashflow()
balance = balance.where(balance.notna(), 0) # NaN替换为0
pre_balance = pre_balance.where(pre_balance.notna(), 0) # NaN替换为0
income = income.where(income.notna(), 0) # NaN替换为0
cash = cash.where(cash.notna(), 0) # NaN替换为0
# 初始化结果字典
data = {
'ts_code': self.ts_code,
'period': self.fin_date,
'--运营数据--': ''
}
# 运营数据计算
cash_equ = cash['c_cash_equ_end_period'].iloc[0]
inventories = balance['inventories'].iloc[0]
total_assets = balance['total_assets'].iloc[0]
# 应收账票 = 应收账款 + 应收票据
accounts_receiv = float(balance['accounts_receiv'].iloc[0]) + float(balance['notes_receiv'].iloc[0])
prepayment = float(balance['prepayment'].iloc[0])
data.update({
'现金额-亿': cash_equ / self.unit_factor,
'现金占比率': cash_equ / total_assets,
'存货-亿': inventories / self.unit_factor,
'存货占比率': inventories / total_assets,
'应收账票-亿': accounts_receiv / self.unit_factor,
'应收账票占比率': accounts_receiv / total_assets,
'预付款-亿': prepayment / self.unit_factor,
'预付款占比率': prepayment / total_assets,
'运营占比率': (cash_equ + inventories + accounts_receiv + prepayment) / total_assets
})
# 资产分布计算
fix_assets = float(balance['fix_assets'].iloc[0]) if 'fix_assets' in balance else 0
intan_assets = float(balance['intan_assets'].iloc[0]) if 'intan_assets' in balance else 0
lt_eqt_invest = float(balance['lt_eqt_invest'].iloc[0]) if 'lt_eqt_invest' in balance else 0
data.update({
'--资产分布--': '',
'固定资产-亿': fix_assets / self.unit_factor,
'固定资产占比率': fix_assets / total_assets,
'无形资产-亿': intan_assets / self.unit_factor,
'无形资产占率': intan_assets / total_assets,
'股权投资-亿': lt_eqt_invest / self.unit_factor,
'股权投资占比率': lt_eqt_invest / total_assets,
'投资占比率': (fix_assets + intan_assets + lt_eqt_invest) / total_assets
})
# 负债分布计算
acct_payable = float(balance['acct_payable'].iloc[0]) if 'acct_payable' in balance else 0
notes_payable = float(balance['notes_payable'].iloc[0]) if 'notes_payable' in balance else 0
adv_receipts = float(balance['adv_receipts'].iloc[0]) if 'adv_receipts' in balance else 0
st_borr = float(balance['st_borr'].iloc[0]) if 'st_borr' in balance else 0
lt_borr = float(balance['lt_borr'].iloc[0]) if 'lt_borr' in balance else 0
bond_payable = float(balance['bond_payable'].iloc[0]) if 'bond_payable' in balance else 0
biz_liab = acct_payable + notes_payable + adv_receipts
fin_liab = st_borr + lt_borr + bond_payable
zcfz = (float(balance['total_cur_liab'].iloc[0]) + float(balance['total_ncl'].iloc[0])) / total_assets
data.update({
'--负债分布--': '',
'经营负债-亿': biz_liab / self.unit_factor,
'经营负债占比率': biz_liab / total_assets,
'金融负债-亿': fin_liab / self.unit_factor,
'金融负债占比率': fin_liab / total_assets,
'资产负债率': zcfz
})
# 运营能力计算
total_days = self.get_total_days()
oper_cost = income['oper_cost'].iloc[0]
revenue = income['revenue'].iloc[0]
# 存货周转天数 (防除零)
#avg_inventories = (float(pre_balance.get('inventories', 0)) + inventories) / 2
avg_inventories = (float(pre_balance['inventories'].iloc[0]) + inventories) / 2
days_1 = total_days / (oper_cost / avg_inventories) if avg_inventories > 0 else 0
# 应收周转天数 (防除零)
avg_receiv = (float(pre_balance['accounts_receiv'].iloc[0]) + accounts_receiv) / 2
days_2 = total_days / (revenue / avg_receiv) if avg_receiv > 0 else 0
data.update({
'--运营能力--': '',
'存货周转天数': days_1,
'应收周转天数': days_2,
'营业周期': days_1 + days_2
})
# 管理费分布计算
gross_profit = revenue - oper_cost
gross_margin = gross_profit / revenue if revenue > 0 else 0
data.update({
'--管理费分布--': '',
'毛利额': gross_profit / self.unit_factor,
'毛利率': gross_margin,
'营业税金率': float(income['biz_tax_surchg'].iloc[0]) / float(revenue) if revenue > 0 else 0,
'销售费用率': float(income['sell_exp'].iloc[0]) / float(revenue) if revenue > 0 else 0,
'研发费用率': float(income['rd_exp'].iloc[0]) / float(revenue) if revenue > 0 else 0,
'管理费用率': float(income['admin_exp'].iloc[0]) / float(revenue) if revenue > 0 else 0,
'净利润': float(income['n_income'].iloc[0]) / self.unit_factor,
'净利润率': float(income['n_income'].iloc[0]) / float(revenue) if revenue > 0 else 0
})
# 权益及回报率计算
data.update({
'--权益及回报率--': '',
'总资产-亿': total_assets / self.unit_factor,
'销售收入-亿': revenue / self.unit_factor,
'总资产周转率': revenue / total_assets if total_assets > 0 else 0,
'总资产回报率': float(income['n_income'].iloc[0]) / total_assets if total_assets > 0 else 0,
'权益乘数': 1 / (1 - zcfz) if zcfz < 1 else 0,
})
# 计算ROE (净资产回报率)
roa = data['总资产回报率']
data['净资产回报率'] = roa * data['权益乘数']
# 格式化比率数据
for key in list(data.keys()):
if isinstance(data[key], float):
if key.endswith(''):
data[key] = f"{data[key] * 100:.2f}%"
else:
data[key] = round(data[key], 2)
return pd.Series(data)
def get_balance(self):
"""获取资产负债表数据"""
fields = [
'ts_code', 'end_date', 'total_assets', 'fix_assets', 'intan_assets',
'lt_eqt_invest', 'inventories', 'accounts_receiv', 'notes_receiv',
'prepayment', 'acct_payable', 'notes_payable', 'adv_receipts',
'st_borr', 'lt_borr', 'bond_payable', 'total_cur_liab', 'total_ncl'
]
return self.pro.balancesheet(
ts_code=self.ts_code,
period=self.fin_date,
fields=fields
)
def get_pre_balance(self):
"""获取上年度资产负债表数据"""
pre_date = f"{int(self.fin_date[:4]) - 1}1231" # 上年末日期
fields = ['inventories', 'accounts_receiv', 'notes_receiv']
return self.pro.balancesheet(
ts_code=self.ts_code,
period=pre_date,
fields=fields
) # 直接返回Series
def get_income(self):
"""获取利润表数据"""
fields = [
'revenue', 'oper_cost', 'biz_tax_surchg', 'sell_exp', 'fin_exp',
'admin_exp', 'n_income', 'rd_exp'
]
return self.pro.income(
ts_code=self.ts_code,
period=self.fin_date,
fields=fields
)
def get_cashflow(self):
"""获取现金流量表数据"""
#c_cash_equ_end_period 期末现金及现金等价物余额
#end_bal_cash 现金的期末余额
fields = ['c_cash_equ_end_period', 'end_bal_cash']
return self.pro.cashflow(
ts_code=self.ts_code,
period=self.fin_date,
fields=fields
)
def get_total_days(self):
"""根据财报类型返回计算周转率的天数"""
quarter = self.fin_date[4:6]
return {
'03': 90, # Q1
'06': 180, # H1
'09': 270, # Q1-Q3
'12': 360 # 全年
}.get(quarter, 360)
#循环调用类
def get_finance_data_range( ts_code, start_date=START_DATE, end_date=END_DATE,TS_TOKEN=TS_TOKEN):
"""
获取指定日期范围内的所有财报数据
:param ts_code: 股票代码
:param start_date: 开始日期 (yyyyMMdd)
:param end_date: 结束日期 (yyyyMMdd)
:return: 合并后的财报数据列表
"""
results = pd.DataFrame()
# 生成所有可能的财报日期 (季度末)
years = range(int(start_date[:4]), int(end_date[:4]) + 1)
report_dates = []
for year in years:
report_dates.extend([
f"{year}0331", # Q1
f"{year}0630", # Q2
f"{year}0930", # Q3
f"{year}1231" # Q4
])
# 筛选在指定日期范围内的财报日期
report_dates = [
date for date in report_dates
if start_date <= date <= end_date
]
report_dates=get_released_report_dates(start_date,end_date)
# 按日期顺序获取财报数据
analyzer = FinanceData(TS_TOKEN)
analyzer.ts_code=tscodeCheck(ts_code)
for fin_date in sorted(report_dates):
analyzer.fin_date = fin_date
try:
data = analyzer.get_finance_data()
results = pd.concat([pd.DataFrame(results), data.to_frame().T], ignore_index=True) if len(results) > 0 else data.to_frame().T
except Exception as e:
print(f"获取 {ts_code} {fin_date} 财报数据失败: {str(e)}")
continue
return results
# 使用示例
if __name__ == "__main__":
# 导入当站目录的config文件
try:
# 尝试相对导入(作为包的一部分)
from .config import TS_TOKEN, START_DATE, END_DATE, PRECISION_CONFIG
except (ImportError, SystemError):
# 失败则使用绝对导入(直接运行脚本)
from config import TS_TOKEN, START_DATE, END_DATE, PRECISION_CONFIG
#token = "your_tushare_token" # 替换为实际token
#analyzer = FinanceData(TS_TOKEN)
# 获取002273.SZ在2022Q1的财务数据
#analyzer.ts_code = '300316.SZ'
#analyzer.fin_date = '20240630'
#result = analyzer.get_finance_data()
result=get_finance_data_range("300316",start_date='20200101',end_date='2025060')
print(result)
#print(pd.Series(result))
+70
View File
@@ -0,0 +1,70 @@
import pandas as pd
from .data_source import get_tushare_pro
# 导入当站目录的config文件
try:
# 尝试相对导入(作为包的一部分)
from .config import TS_TOKEN, START_DATE, END_DATE
from .stock_utils import dataCorrect
except (ImportError, SystemError):
# 失败则使用绝对导入(直接运行脚本)
from config import TS_TOKEN, START_DATE, END_DATE
from stock_utils import dataCorrect
def getStockParam(TS_CODE,START_DATE=START_DATE,END_DATE=END_DATE):
pro = get_tushare_pro()
"""
从tushare daily_basic接口获取单只股票的所有基础数据,并进行数据修正。
Parameters:
TS_CODE (str): 股票代码,格式为 '股票代码.SZ''股票代码.SH',例如 '000001.SZ'
Returns:
pd.DataFrame: 包含股票基础数据的DataFrame,字段说明如下:
ts_code: 股票代码
trade_date: 交易日期
close: 收盘价
turnover_rate: 换手率(%
turnover_rate_f: 换手率(自由流通股)
volume_ratio: 量比
pe: 市盈率(总市值/净利润,亏损的PE为空)
pe_ttm: 市盈率(TTM,亏损的PE为空)
pb: 市净率(总市值/净资产)
ps: 市销率(总市值/营业收入)
ps_ttm: 市销率(TTM
dv_ratio: 股息率(%
dv_ttm: 股息率(TTM
total_share: 总股本(万股)
float_share: 流通股本(万股)
free_share: 自由流通股本(万股)
total_mv: 总市值(万元)
circ_mv: 流通市值(万元)
Raises:
Exception: 如果从Tushare接口获取数据时发生错误。
"""
try:
# 获取股票基础数据
df = pro.daily_basic(ts_code=TS_CODE, start_date=START_DATE, end_date=END_DATE)
# 需要修正的列
cols = ['close', 'turnover_rate', 'turnover_rate_f', 'volume_ratio', 'pe', 'pe_ttm',
'pb', 'ps', 'ps_ttm', 'dv_ratio', 'dv_ttm', 'total_share', 'float_share',
'free_share', 'total_mv', 'circ_mv']
# 调用dataCorrect函数进行数据修正
df = dataCorrect(df, cols)
return df
except Exception as e:
# 捕获并处理异常
print(f"错误: 获取股票 {TS_CODE} 的基础数据时发生错误: {e}")
return pd.DataFrame()
if __name__ == "__main__":
# 简单测试
import datetime
yesterday = (datetime.datetime.now() - datetime.timedelta(days=1)).strftime("%Y%m%d")
test_df = getStockParam("601398.SH", START_DATE=yesterday, END_DATE=yesterday)
print(test_df)
+1
View File
@@ -0,0 +1 @@
from ..utils.mysql_handler import MySQLDB # noqa: F401 — 向后兼容重新导出
+15
View File
@@ -0,0 +1,15 @@
# 扫描配置
# 指定行业板块
INDUSTRIES = ["软件服务", "运输设备", "电气设备", "元器件", "火力发电",
"医药商业", "汽车配件", "新型电力", "铅锌", "通信设备", "IT设备",
"工程机械", "证券", "生物制药", "百货", "食品", "机械基件",
"汽车整车", "煤炭开采", "白酒", "", "", "小金属",
"互联网", "航空", "超市连锁", "轻工机械", "电器仪表", "半导体",
"公共交通", "电信运营"]
# 判断阈值
PRICE_VOLATILITY_THRESHOLD = 5 # 价格波动幅度均值上限(百分比)
MA20_STD_THRESHOLD = 0.05 # 移动平均线标准差上限(相对于均值的百分比)
ATR_THRESHOLD = 0.02 # ATR均值上限(相对于收盘价均值的百分比)
BOLLINGER_BAND_WIDTH_THRESHOLD = 5 # 布林带宽度均值上限(百分比)
+190
View File
@@ -0,0 +1,190 @@
import pandas as pd
import numpy as np
def smooth_series_brush(series: pd.Series, window_size: int = 7, threshold_factor: float = 0.5, max_brush_length: int = 5) -> pd.Series:
"""
平滑处理pandas序列中的连续毛刺数据,使用前值或后值填充。
参数:
series (pd.Series): 输入的pandas序列。
window_size (int): 用于检测毛刺的滑动窗口大小。必须为奇数。默认为7。
threshold_factor (float): 判断毛刺的阈值因子。如果 abs(value - median) / median > threshold_factor,则认为是毛刺。默认为0.5 (50%)。
max_brush_length (int): 允许的最大连续毛刺长度。超过此长度的连续点将不被处理。默认为5。
返回:
pd.Series: 处理后的平滑序列。
"""
if not isinstance(series, pd.Series):
raise TypeError("输入必须是 pandas Series 对象。")
if window_size % 2 == 0:
raise ValueError("window_size 必须是奇数。")
# 创建副本以避免修改原始数据
smoothed_series = series.copy()
# 用于标记是否为毛刺的布尔序列
is_brush = pd.Series([False] * len(series), index=series.index)
half_window = window_size // 2
# --- 第一步:检测毛刺 ---
for i in range(len(series)):
start_idx = max(0, i - half_window)
end_idx = min(len(series), i + half_window + 1)
# 获取当前窗口数据
window_data = series.iloc[start_idx:end_idx]
if len(window_data) < 2:
continue
# 计算窗口中位数
window_median = window_data.median()
# 避免除以零
if window_median == 0:
continue
current_value = series.iloc[i]
# 计算偏差比例
deviation_ratio = abs(current_value - window_median) / abs(window_median)
# 如果偏差超过阈值,则标记为毛刺
if deviation_ratio > threshold_factor:
is_brush.iloc[i] = True
# --- 第二步:处理连续的毛刺段 ---
# 使用 cumsum 技巧识别连续毛刺段
brush_groups = (is_brush != is_brush.shift()).cumsum() * is_brush
# 遍历每个被标记为毛刺的组
for group_id in brush_groups[brush_groups != 0].unique():
if pd.isna(group_id):
continue
brush_indices = brush_groups[brush_groups == group_id].index
# 检查连续毛刺长度
if len(brush_indices) > max_brush_length:
print(f"警告: 发现长度为 {len(brush_indices)} 的连续毛刺段 (超过 max_brush_length={max_brush_length}),将不进行平滑处理。")
continue
# --- 平滑处理:使用前值或后值填充 ---
# 查找前一个非毛刺点
prev_valid_val = None
start_loc = series.index.get_loc(brush_indices[0])
for j in range(start_loc - 1, -1, -1):
if not is_brush.iloc[j]:
prev_valid_val = series.iloc[j]
break
# 查找后一个非毛刺点
next_valid_val = None
end_loc = series.index.get_loc(brush_indices[-1])
for j in range(end_loc + 1, len(series)):
if not is_brush.iloc[j]:
next_valid_val = series.iloc[j]
break
# 决定使用哪个值填充
if prev_valid_val is not None:
fill_value = prev_valid_val
elif next_valid_val is not None:
fill_value = next_valid_val
else:
print(f"警告: 毛刺段 {brush_indices} 无有效邻居,使用全局中位数填充。")
fill_value = series.median()
# 用 fill_value 填充整个毛刺段
for idx in brush_indices:
smoothed_series.loc[idx] = fill_value
return smoothed_series
def smooth_dataframe_brush(df: pd.DataFrame, target_columns: list, **kwargs) -> pd.DataFrame:
"""
对DataFrame中的指定列进行毛刺平滑处理。
参数:
df (pd.DataFrame): 输入的pandas DataFrame。
target_columns (list): 需要去毛刺处理的列名列表。
**kwargs: 传递给 smooth_series_brush 函数的参数 (如 window_size, threshold_factor, max_brush_length)。
返回:
pd.DataFrame: 处理后的DataFrame,指定列已平滑,其余列不变。
"""
if not isinstance(df, pd.DataFrame):
raise TypeError("输入必须是 pandas DataFrame 对象。")
# 创建副本以避免修改原始数据
result_df = df.copy()
# 检查目标列是否都存在于DataFrame中
missing_cols = [col for col in target_columns if col not in df.columns]
if missing_cols:
raise ValueError(f"以下列不在DataFrame中: {missing_cols}")
# 对每个目标列应用平滑函数
for col in target_columns:
print(f"正在处理列: {col}")
try:
# 应用去毛刺函数
result_df[col] = smooth_series_brush(df[col], **kwargs)
except Exception as e:
print(f"处理列 {col} 时出错: {e}")
# 可以选择保留原始数据或抛出异常
# 这里选择保留原始数据
continue
return result_df
# --- 示例 ---
if __name__ == "__main__":
# 1. 创建示例 DataFrame
dates = pd.date_range('2023-01-01', periods=20, freq='D')
# 需要处理的列
values_to_smooth = [10, 11, 10.5, 12, 11.8, 50, 12.1, 11.9, 10, 10.2,
9.8, 100, 105, 99, 10.1, 9.9, 10.3, 5, 10.2, 10.1]
# 不需要处理的列 (例如,另一个传感器数据)
other_data = np.random.randn(20).cumsum() + 100 # 累积和,模拟趋势
# 构建 DataFrame
df_original = pd.DataFrame({
'Date': dates,
'Sensor_A': values_to_smooth, # 需要去毛刺
'Sensor_B': other_data, # 不需要处理
'Other_Info': range(20) # 其他信息,不需要处理
})
# 设置日期为索引 (常见做法)
df_original.set_index('Date', inplace=True)
print("原始 DataFrame:")
print(df_original.head(10))
print("\n" + "="*50 + "\n")
# 2. 应用平滑函数
# 指定需要处理的列
columns_to_smooth = ['Sensor_A']
# 调用新函数
df_smoothed = smooth_dataframe_brush(
df_original,
target_columns=columns_to_smooth,
window_size=5,
threshold_factor=0.3,
max_brush_length=5
)
print("平滑后的 DataFrame:")
print(df_smoothed.head(10))
print("\n" + "="*50 + "\n")
# 3. 比较
comparison_df = df_original.copy()
comparison_df['Sensor_A_Smoothed'] = df_smoothed['Sensor_A']
comparison_df['Difference'] = comparison_df['Sensor_A'] - comparison_df['Sensor_A_Smoothed']
print("对比 (原始 Sensor_A vs 平滑后 vs 差异):")
print(comparison_df[['Sensor_A', 'Sensor_A_Smoothed', 'Difference']].head(10))
+125
View File
@@ -0,0 +1,125 @@
import pandas as pd
# 将项目根目录添加到 sys.path
#project_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
#sys.path.append(project_root)
# 导入当站目录的config文件
try:
# 尝试相对导入(作为包的一部分)
from .config import TS_TOKEN, START_DATE, END_DATE
from .stock_utils import dataCorrect,get_trading_dates,tscodeCheck
except (ImportError, SystemError):
# 失败则使用绝对导入(直接运行脚本)
from config import TS_TOKEN, START_DATE, END_DATE
from stock_utils import dataCorrect,get_trading_dates,tscodeCheck
from .data_source import get_tushare_pro
pro = get_tushare_pro()
def getStockMargin(tscode: str, start_date: str=START_DATE, end_date: str=END_DATE) -> pd.DataFrame:
"""
获取指定股票代码在时间区间内的每日融资明细数据
参数:
tscode (str): 股票代码,格式如 '600000.SH'
start_date (str): 开始日期,格式 'YYYY-MM-DD'
end_date (str): 结束日期,格式 'YYYY-MM-DD'
返回:
pd.DataFrame: 包含融资明细数据的DataFrame,列包括:
- trade_date: 交易日期
- tscode: 股票代码
- rzye: 融资余额(元)
- rqye float 融券余额(元)
- rzmre float 融资买入额(元)
- rqyl float 融券余量(股)
- rzche float 融资偿还额(元)
- rqchl float 融券偿还量(股)
- rqmcl float 融券卖出量(股,份,手)
- rzrqye float 融资融券余额(元)
异常处理:
- 若tushare接口调用失败,打印错误信息并返回空DataFrame
"""
try:
tscode=tscodeCheck(tscode)
# 调用tushare接口
df = pro.margin_detail(ts_code=tscode, start_date=start_date, end_date=end_date)
cols=["rzye","rqye","rzmre","rqyl","rzche","rqchl","rqmcl","rzrqye"]
df = dataCorrect(df,cols)
if df.empty:
print(f"未找到{tscode}{start_date}{end_date}期间的融资数据")
return df
except Exception as e:
print(f"获取融资数据失败: {e}")
return pd.DataFrame()
def getDailyMargin(trade_date: str = None, start_date: str = None, end_date: str = None, exchange_id: str = None) -> pd.DataFrame:
"""
获取指定日期或时间区间内的每日融资明细数据
参数:
trade_date (str, optional): 指定单个交易日,格式 'YYYY-MM-DD'。与start_date/end_date互斥
start_date (str, optional): 开始日期,格式 'YYYY-MM-DD'。需与end_date同时使用
end_date (str, optional): 结束日期,格式 'YYYY-MM-DD'。需与start_date同时使用
exchange_id (str, optional): 交易所代码,如 'SSE'(上交所)、'SZSE'(深交所)
返回:
pd.DataFrame: 包含每日融资明细数据的DataFrame,列包括:
- exchange_id: 交易所代码
- trade_date: 交易日期
- rzye: 融资余额(元)
- rqye: 融券余额(元)
- rzmre: 融资买入额(元)
- rqyl: 融券余量(股)
- rzche: 融资偿还额(元)
- rqchl: 融券偿还量(股)
- rqmcl: 融券卖出量(股,份,手)
- rzrqye: 融资融券余额(元)
异常处理:
- 若参数组合无效,打印错误信息并返回空DataFrame
- 若tushare接口调用失败,打印错误信息并返回空DataFrame
"""
try:
# 日期格式转换
if trade_date:
trade_date = trade_date.replace("-", "")
if start_date:
start_date = start_date.replace("-", "")
if end_date:
end_date = end_date.replace("-", "")
# 参数校验
if trade_date and (start_date or end_date):
print("错误:trade_date不能与start_date/end_date同时使用")
return pd.DataFrame()
if (start_date and not end_date) or (end_date and not start_date):
print("错误:start_date和end_date必须同时使用")
return pd.DataFrame()
# 调用tushare接口
df = pro.margin(exchange_id=exchange_id,
trade_date=trade_date,
start_date=start_date,
end_date=end_date)
if df.empty:
print("未找到符合条件的融资数据")
return df
except Exception as e:
print(f"获取每日融资数据失败: {e}")
return pd.DataFrame()
if __name__ == "__main__":
# 测试getStockMargin函数
test_code = "600000.SH"
test_start = "20150101"
test_end = "20250731"
result = getStockMargin(test_code, test_start, test_end)
print("返回结果示例:")
print(result.head())
if not result.empty:
print(f"\n返回数据行数: {len(result)}")
+163
View File
@@ -0,0 +1,163 @@
import pandas as pd
# 将项目根目录添加到 sys.path
#project_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
#sys.path.append(project_root)
# 导入当站目录的config文件
try:
# 尝试相对导入(作为包的一部分)
from .config import TS_TOKEN, START_DATE, END_DATE
from .stock_utils import dataCorrect,get_trading_dates,tscodeCheck
except (ImportError, SystemError):
# 失败则使用绝对导入(直接运行脚本)
from config import TS_TOKEN, START_DATE, END_DATE
from stock_utils import dataCorrect,get_trading_dates,tscodeCheck
from .data_source import get_tushare_pro
pro = get_tushare_pro()
def getStockBasic(TS_CODE,START_DATE=START_DATE,END_DATE=END_DATE):
"""
从tushare daily接口获取单只股票的所有返回数据,并进行数据修正。
Parameters:
TS_CODE (str): 股票代码,格式为 '股票代码.SZ''股票代码.SH',例如 '000001.SZ'
Returns:
pd.DataFrame: 包含股票日线数据的DataFrame,字段说明如下:
ts_code: 股票代码
trade_date: 交易日期
open: 开盘价
high: 最高价
low: 最低价
close: 收盘价
pre_close: 前日收盘价
change: 涨跌额
pct_chg: 涨跌幅(百分比)
vol: 成交量(手)
amount: 成交额(千元)
Raises:
Exception: 如果从Tushare接口获取数据时发生错误。
"""
try:
# 获取股票日线数据
df = pro.daily(ts_code=TS_CODE, start_date=START_DATE, end_date=END_DATE)
# 需要修正的列
cols = ['open', 'high', 'low', 'close', 'pre_close', 'change', 'pct_chg', 'vol', 'amount']
# 调用dataCorrect函数进行数据修正
df = dataCorrect(df, cols)
return df
except Exception as e:
# 捕获并处理异常
print(f"错误: 获取股票 {TS_CODE} 的日线数据时发生错误: {e}")
return pd.DataFrame()
def getStockInfo(TS_CODE):
"""
从tushare的stock_basic接口获取单只股票的基本信息。
Parameters:
TS_CODE (str): 股票代码,格式为 '股票代码.SZ''股票代码.SH',例如 '000001.SZ'
Returns:
pd.DataFrame: 包含股票基本信息的DataFrame,字段说明如下:
ts_code: 股票代码
symbol: 股票代码(不带后缀)
name: 股票名称
area: 所在地域
industry: 所属行业
market: 市场类型(主板/创业板/科创板等)
list_date: 上市日期
fullname: 股票全称
enname: 英文全称
exchange: 交易所代码
curr_type: 交易货币
list_status: 上市状态
is_hs: 是否沪深港通标的
Raises:
Exception: 如果从Tushare接口获取数据时发生错误。
"""
try:
TS_CODE=tscodeCheck(TS_CODE)
# 调用stock_basic接口获取股票基本信息
df = pro.stock_basic(ts_code=TS_CODE)
# 如果没有获取到数据,返回空的DataFrame
if df.empty:
print(f"警告: 未找到股票 {TS_CODE} 的基本信息")
return pd.DataFrame()
return df
except Exception as e:
# 捕获并处理异常
print(f"错误: 获取股票 {TS_CODE} 的基本信息时发生错误: {e}")
return pd.DataFrame()
def getStockListByIndustry(industry):
"""
根据行业名称获取股票列表。
该函数首先通过 `index_classify` 接口获取行业分类的级别和行业代码,
然后使用 `index_member_all` 接口获取该行业下的所有股票列表。
Parameters:
industry (str): 行业名称,例如 "银行""医药" 等。
Returns:
pd.DataFrame: 包含股票代码 (ts_code) 和股票名称 (name) 的 DataFrame。
如果未找到匹配的行业或股票,返回空的 DataFrame。
Raises:
Exception: 如果从 Tushare 接口获取数据时发生错误。
"""
try:
# 1. 获取行业分类信息
# 使用 index_classify 接口获取所有行业分类信息
industry_df = pro.index_classify(level='', src='SW2021')
# 过滤出与给定行业名称匹配的行业
industry_info = industry_df[industry_df['industry_name'] == industry]
# 如果没有找到匹配的行业,返回空的 DataFrame
if industry_info.empty:
print(f"警告: 未找到行业 '{industry}' 的分类信息")
return pd.DataFrame(columns=['ts_code', 'name'])
# 获取行业代码和级别
industry_code = industry_info.iloc[0]['index_code']
industry_level = industry_info.iloc[0]['level']
print(industry_info)
# 2. 根据行业级别调用 index_member_all 接口
if industry_level == 'L1':
stock_list_df = pro.index_member_all(l1_code=industry_code)
elif industry_level == 'L2':
stock_list_df = pro.index_member_all(l2_code=industry_code)
elif industry_level == 'L3':
stock_list_df = pro.index_member_all(l3_code=industry_code)
else:
print(f"警告: 未知的行业级别 '{industry_level}'")
return pd.DataFrame(columns=['ts_code', 'name'])
# 如果没有找到股票,返回空的 DataFrame
if stock_list_df.empty:
print(f"警告: 行业 '{industry}' 下没有找到股票")
return pd.DataFrame(columns=['ts_code', 'name'])
# 3. 返回股票代码和名称
return stock_list_df[['ts_code', 'name']]
except Exception as e:
# 捕获并处理异常
print(f"错误: 获取行业 '{industry}' 的股票列表时发生错误: {e}")
return pd.DataFrame(columns=['ts_code', 'name'])
if __name__ == '__main__':
df = getStockListByIndustry('果蔬加工')
print(df)
+363
View File
@@ -0,0 +1,363 @@
import pandas as pd
from django.http import HttpResponse
from rest_framework.response import Response
try:
from .config import TS_TOKEN, START_DATE, END_DATE
except (ImportError, SystemError):
from config import TS_TOKEN, START_DATE, END_DATE
from .data_source import get_tushare_pro
# 统一入口(全局单例,向后兼容旧代码直接访问 pro)
pro = get_tushare_pro()
def dataCorrect(df, columns=None):
"""
检查并修正DataFrame中的NaN或None值。
1. 如果前后有数字,该字段取前后值的均值填入
2. 如果仅前或后有数字,该字段取copy前或后数字
3. 若前后都为NaN,则设为0
4. 若出发点修改,打印修改情况到屏幕
Parameters:
df (pd.DataFrame): 要处理的数据框
columns (list): 要检查的列名列表,如果为None则检查所有列
"""
# 如果没有指定列,则检查所有列
if columns is None:
columns = df.columns
# 遍历指定列
for col in columns:
# 遍历每一行
for i in range(len(df)):
# 检查当前值是否为NaN或None
if pd.isna(df.at[i, col]):
# 获取前后值
prev_val = df.at[i-1, col] if i > 0 else None
next_val = df.at[i+1, col] if i < len(df) - 1 else None
# 检查前后值是否为有效数字
prev_valid = prev_val is not None and not pd.isna(prev_val)
next_valid = next_val is not None and not pd.isna(next_val)
# 如果前后都有值,取均值
if prev_valid and next_valid:
new_val = (prev_val + next_val) / 2
df.at[i, col] = new_val
#print(f"修正 {col} 列第 {i} 行: 前后值均值填充为 {new_val}")
# 如果只有前值,取前值
elif prev_valid:
df.at[i, col] = prev_val
#print(f"修正 {col} 列第 {i} 行: 前值填充为 {prev_val}")
# 如果只有后值,取后值
elif next_valid:
df.at[i, col] = next_val
#print(f"修正 {col} 列第 {i} 行: 后值填充为 {next_val}")
# 如果前后都没有有效值,设为0
else:
df.at[i, col] = 0
#print(f"修正 {col} 列第 {i} 行: 前后均无有效值,设为0")
return df
def dataMerge(*dfs):
"""
合并多个股票数据DataFrame,确保合并后的数据ts_code和trade_date匹配。
Parameters:
*dfs: 可变数量的DataFrame参数,每个DataFrame应包含ts_code和trade_date列
Returns:
pd.DataFrame: 合并后的DataFrame,包含所有输入DataFrame的列。如果某行的ts_code或trade_date不匹配,则用NaN填充。
"""
if len(dfs) < 2:
raise ValueError("至少需要提供2个DataFrame进行合并")
# 检查所有DataFrame是否都有ts_code和trade_date列
for df in dfs:
if 'ts_code' not in df.columns or 'trade_date' not in df.columns:
raise ValueError("所有DataFrame必须包含ts_code和trade_date列")
# 初始化合并结果为第一个DataFrame
merged_df = dfs[0].copy()
# 逐个合并剩余的DataFrame
for df in dfs[1:]:
merged_df = pd.merge(merged_df, df, on=['ts_code', 'trade_date'], how='outer')
return merged_df
def tscodeCheck(tscode):
"""
检查并修正股票代码格式。
Parameters:
tscode (str): 股票代码
Returns:
str: 格式正确的股票代码 (如 '000001.SZ')
Raises:
ValueError: 如果输入不符合要求
"""
if not isinstance(tscode, str):
raise ValueError("输入必须是字符串")
tscode = tscode.upper() # 转为大写
# 检查长度
if len(tscode) < 6:
raise ValueError("股票代码长度不能小于6位")
elif len(tscode) == 6:
if not tscode.isdigit():
raise ValueError("6位股票代码必须全为数字")
# 根据开头添加后缀
if tscode.startswith('00') or tscode.startswith('3'):
return f"{tscode}.SZ"
elif tscode.startswith(('60','68')): # 68为科创板
return f"{tscode}.SH"
elif tscode.startswith(('8', '9')):
return f"{tscode}.BJ"
else:
raise ValueError("未知的6位股票代码开头")
elif len(tscode) == 9:
prefix = tscode[:6]
suffix = tscode[-3:]
if not prefix.isdigit():
raise ValueError("9位股票代码前6位必须为数字")
if suffix not in ('.SZ', '.SH', '.BJ'):
raise ValueError("9位股票代码后缀必须是.SZ/.SH/.BJ")
return tscode
else:
raise ValueError("股票代码长度必须为6位或9位")
def get_trading_dates(trade_date):
"""
获取上一个季度财报日期到给定日期的所有交易日期
Parameters:
trade_date (str): 给定日期 (格式yyyymmdd)
Returns:
list: 交易日期列表 (格式yyyymmdd)
"""
# 将输入日期转为datetime
try:
current_date = pd.to_datetime(trade_date, format='%Y%m%d')
except:
raise ValueError("trade_date格式应为yyyymmdd")
# 计算上一个季度财报日期 (3/31, 6/30, 9/30, 12/31)
year = current_date.year
month = current_date.month
if month < 4:
prev_report_date = pd.Timestamp(year-1, 12, 31)
elif month < 7:
prev_report_date = pd.Timestamp(year, 3, 31)
elif month < 10:
prev_report_date = pd.Timestamp(year, 6, 30)
else:
prev_report_date = pd.Timestamp(year, 9, 30)
# 获取交易日历
calendar_df = pro.trade_cal(exchange='', start_date=prev_report_date.strftime('%Y%m%d'), end_date=trade_date)
# 过滤交易日历
calendar_df['cal_date_dt'] = pd.to_datetime(calendar_df['cal_date'], format='%Y%m%d')
filtered_dates = calendar_df[
(calendar_df['cal_date_dt'] > prev_report_date) &
(calendar_df['cal_date_dt'] <= current_date) &
(calendar_df['is_open'] == 1)
]
# 返回日期列表 (格式yyyymmdd)
return filtered_dates['cal_date'].tolist()
def get_next_report_date(trade_date):
"""
获取给定日期的下一个财报日期
Parameters:
trade_date (str): 给定日期 (格式yyyymmdd)
Returns:
str: 下一个财报日期 (格式yyyymmdd)
"""
try:
current_date = pd.to_datetime(trade_date, format='%Y%m%d')
except:
raise ValueError("trade_date格式应为yyyymmdd")
year = current_date.year
month = current_date.month
day = current_date.day
if month < 3 or (month == 3 and day < 31):
return f"{year}0331"
elif month < 6 or (month == 6 and day < 30):
return f"{year}0630"
elif month < 9 or (month == 9 and day < 30):
return f"{year}0930"
elif month < 12 or (month == 12 and day < 31):
return f"{year}1231"
else:
return f"{year+1}0331"
def viewFunc_singleParam(request, data_func, param_name='tscode', default_value=None):
"""
通用单参数视图包装器。
:param request: Django request 对象
:param data_func: 接收单个参数的数据处理函数
:param param_name: URL 查询参数名
:param default_value: 参数默认值
:return: Response
"""
param = request.GET.get(param_name, default_value)
if not param:
return Response({'error': f'缺少 {param_name} 参数'}, status=400)
try:
data = data_func(param)
dict_data = data.to_dict(orient='records') if isinstance(data, pd.DataFrame) else data
return Response(dict_data)
except ImportError:
return Response({'error': f'模块不存在'}, status=500)
except Exception as e:
return Response({'error': str(e)}, status=500)
def viewFunc_tsCodeAndDate(request,data_func):
"""
通用数据处理函数
:param request: Django request对象
:param data_func: 数据处理函数(需返回Response)
:return: Response
"""
#通过url 传入 tscode 变量
tscode = request.GET.get('tscode','000001.SZ') # 从URL获取industry参数
tscode = tscodeCheck(tscode)
if not tscode:
return Response({'error': '缺少 tscode 参数'}, status=400)
start_date = request.GET.get('start_date', START_DATE)
end_date = request.GET.get('end_date', END_DATE)
try:
data = data_func(tscode, start_date, end_date) # 调用函数
dict_data = data.to_dict(orient='records') if isinstance(data, pd.DataFrame) else data
return Response(dict_data)
except ImportError:
return Response({'error': '模块 stock_basic 不存在'}, status=500)
except Exception as e:
return Response({'error': str(e)}, status=500)
def date_format_correction(date_str):
"""
日期格式矫正函数:
1. 如果输入格式为yyyy-mm-dd则返回yyyymmdd格式
2. 如果输入格式已经是yyyymmdd则直接返回
3. 其他情况返回None
Parameters:
date_str (str): 日期字符串
Returns:
str: yyyymmdd格式的日期字符串,如果输入格式不正确则返回None
"""
try:
# 先尝试解析yyyymmdd格式
pd.to_datetime(date_str, format='%Y%m%d')
return date_str
except ValueError:
try:
# 再尝试解析yyyy-mm-dd格式
date_obj = pd.to_datetime(date_str, format='%Y-%m-%d')
return date_obj.strftime('%Y%m%d')
except ValueError:
return None
from datetime import datetime
def get_released_report_dates(start_date: str, end_date: str) -> list:
"""
根据起止日期生成所有财报季末日,并去除尚未可能披露的日期。
:param start_date: 开始日期,格式为 "YYYYMMDD"
:param end_date: 结束日期,格式为 "YYYYMMDD"
:param today: 当前日期,格式为 "YYYYMMDD",默认为系统当前日期
:return: 已披露的财报季末日期列表,格式为 ["20240331", "20240630", ...]
"""
today = datetime.today()
# 财报发布日期最晚规则
report_deadlines = {
"0331": "0430",
"0630": "0831",
"0930": "1031",
"1231": "0430" # 次年4月30日
}
# 生成所有财报季度末日期
years = range(int(start_date[:4]), int(end_date[:4]) + 1)
report_dates = [
f"{year}{quarter}"
for year in years
for quarter in ["0331", "0630", "0930", "1231"]
]
# 筛选日期范围内的
report_dates = [d for d in report_dates if start_date <= d <= end_date]
def is_report_released(report_date_str):
report_dt = datetime.strptime(report_date_str, "%Y%m%d")
year = report_dt.year
q_end = report_date_str[4:]
if q_end == "1231":
deadline = datetime.strptime(f"{year+1}0430", "%Y%m%d")
else:
deadline = datetime.strptime(f"{year}{report_deadlines[q_end]}", "%Y%m%d")
return today >= deadline
# 返回已发布的日期
return [d for d in report_dates if is_report_released(d)]
'''
写一个函数,调用stock_basic API(说明文档:https://tushare.pro/document/2?doc_id=25) 获取个股清单
给定交易所代码,仅查询当前仍然上市的股票
返回:
- ts_code: TS代码
- symbol: 股票代码
- name: 股票名称
- fullname: 股票全称
- exchange: 交易所代码 SSE上交所 SZSE深交所 BSE北交所
- list_status: 上市状态 L上市 D退市 P暂停上市
'''
def get_stock_basic(exchange='SSE'):
"""
获取指定交易所的上市股票基本信息
Parameters:
exchange (str): 交易所代码 SSE上交所 SZSE深交所 BSE北交所
Returns:
pd.DataFrame: 包含股票基本信息的DataFrame
"""
# 调用stock_basic接口
df = pro.stock_basic(exchange=exchange, list_status='L',
fields='ts_code,symbol,name,fullname,exchange,list_status')
return df
if __name__ == "__main__":
# 测试当前日期之前的财报
print("\n测试2: 当前日期之前的财报")
result = get_released_report_dates("20230101", "20251230")
print(f"结果: {result}")
+13
View File
@@ -0,0 +1,13 @@
# 策略参数
PFAST = 10 # 快速移动平均线周期
PSLOW = 30 # 慢速移动平均线周期
STOP_LOSS = 0.05 # 止损比例
TAKE_PROFIT = 0.10 # 止盈比例
# 初始资金
INITIAL_CASH = 100000.0
# 佣金费率
COMMISSION_BUY = 0.001 # 买入佣金费率
COMMISSION_SELL = 0.002 # 卖出佣金费率
+80
View File
@@ -0,0 +1,80 @@
'''
写一个方法:
1. 接收日期范围
2. 根据日期范围从表:xwlb_daily 查询数据:news_days, daily_sub_id, news_improve, news_title
按news_days desc, daily_sub_id asc 排序
3. 调用mysqlHandle.py 里的查询方法查询数据(仔细阅读video/mysqlHandle)
'''
# 调用mysqlHandle中的查询方法
try:
from .mysqlHandle import MySQLDB
except (ImportError, SystemError):
from mysqlHandle import MySQLDB
import pandas as pd
def get_xwlb(start_date, end_date):
"""
根据日期范围查询新闻联播数据
Args:
start_date: 开始日期
end_date: 结束日期
Returns:
查询结果列表
"""
# SQL注入警告:使用参数化查询防止SQL注入
sql = "xwlb_daily"
columns = "news_days, daily_sub_id, news_improve, news_title"
where = "news_days >= %s AND news_days <=%s order by news_days desc, daily_sub_id asc"
params = (start_date, end_date)
try:
db = MySQLDB()
result = db.query_data(sql, columns, where, params)
print(f"查询到 {len(result)} 条记录")
finally:
# 关闭连接
db.close()
# 转换为pandas DataFrame
df = pd.DataFrame(result)
return df
def get_xwlb_fine(start_date, end_date):
"""
根据日期范围查询新闻联播数据
Args:
start_date: 开始日期
end_date: 结束日期
Returns:
查询结果列表
"""
# SQL注入警告:使用参数化查询防止SQL注入
sql = "xwlb_daily_ext"
columns = "news_date as news_days, sub_id as daily_sub_id, news_content as news_improve, news_title"
where = "news_date >= %s AND news_date <=%s order by news_date desc, sub_id asc"
params = (start_date, end_date)
try:
db = MySQLDB()
result = db.query_data(sql, columns, where, params)
print(f"查询到 {len(result)} 条记录")
finally:
# 关闭连接
db.close()
# 转换为pandas DataFrame
df = pd.DataFrame(result)
return df
if __name__ == "__main__":
# 测试代码
start_date = "2025-01-01"
end_date = "2025-01-31"
result = get_xwlb(start_date, end_date)
print("查询结果:")
print(result.head())
print(f"总记录数:{len(result)}")
+10
View File
@@ -0,0 +1,10 @@
<!DOCTYPE html>
<html>
<head>
<title>Django Home Page</title>
</head>
<body>
<h1>Welcome to Django Home Page!</h1>
<p>This is the main page of our Django project.</p>
</body>
</html>
+87
View File
@@ -0,0 +1,87 @@
from django.test import TestCase
from .stock.stock_utils import tscodeCheck, date_format_correction, get_released_report_dates
class TscodeCheckTest(TestCase):
"""股票代码格式校验测试"""
def test_sz_6digit(self):
self.assertEqual(tscodeCheck('000001'), '000001.SZ')
def test_sz_6digit_300(self):
self.assertEqual(tscodeCheck('300750'), '300750.SZ')
def test_sh_6digit_60(self):
self.assertEqual(tscodeCheck('600000'), '600000.SH')
def test_sh_6digit_68(self):
self.assertEqual(tscodeCheck('688001'), '688001.SH')
def test_bj_6digit(self):
self.assertEqual(tscodeCheck('830799'), '830799.BJ')
def test_9digit_pass_through(self):
self.assertEqual(tscodeCheck('000001.SZ'), '000001.SZ')
def test_lowercase_to_uppercase(self):
self.assertEqual(tscodeCheck('000001.sz'), '000001.SZ')
def test_invalid_length_short(self):
with self.assertRaises(ValueError):
tscodeCheck('12345')
def test_invalid_length_long(self):
with self.assertRaises(ValueError):
tscodeCheck('1234567890')
def test_invalid_suffix(self):
with self.assertRaises(ValueError):
tscodeCheck('000001.XX')
def test_not_string(self):
with self.assertRaises(ValueError):
tscodeCheck(123456)
def test_unknown_prefix(self):
with self.assertRaises(ValueError):
tscodeCheck('500001')
class DateFormatCorrectionTest(TestCase):
"""日期格式校正测试"""
def test_yyyymmdd_passthrough(self):
self.assertEqual(date_format_correction('20240115'), '20240115')
def test_yyyy_mm_dd_conversion(self):
self.assertEqual(date_format_correction('2024-01-15'), '20240115')
def test_invalid_format(self):
self.assertIsNone(date_format_correction('15/01/2024'))
def test_empty_string(self):
self.assertIsNone(date_format_correction(''))
class GetReleasedReportDatesTest(TestCase):
"""财报发布日期测试"""
def test_returns_list(self):
result = get_released_report_dates('20230101', '20231231')
self.assertIsInstance(result, list)
def test_all_dates_yyyymmdd_format(self):
result = get_released_report_dates('20230101', '20231231')
for d in result:
self.assertEqual(len(d), 8)
self.assertTrue(d.isdigit())
def test_start_after_end_returns_empty(self):
result = get_released_report_dates('20251231', '20230101')
self.assertEqual(result, [])
def test_single_year_quarters(self):
"""单年内应该返回最多4个季末日期"""
result = get_released_report_dates('20200101', '20201231')
for d in result:
self.assertTrue(d.endswith(('0331', '0630', '0930', '1231')))
+29
View File
@@ -0,0 +1,29 @@
from django.urls import path
from . import views
from .report import views as report_views
urlpatterns = [
# 其他 URL 路由
path('python-version/', views.python_version, name='python_version'),
path('', views.home, name='home'), # 根 URL 映射到主页视图
path('stockbasic/', views.stockbasic, name='stockbasic'),
path('stockparam/', views.stockparam, name='stockparam'),
path('industrys/', views.industrys, name='industrys'),
path('stockinfo/', views.stockInfo, name='stockInfo'),
path('stockep/', views.stockep, name='stockep'),
path('quarterlyEps/', views.quarterlyEps, name='quarterlyEps'),
path('indexByName/', views.indexByName, name='indexByName'),
path('indexDatas/', views.indexDatas, name='indexDatas'),
path('dailymargin/', views.dailyMargin, name='dailyMargin'),
path('stockmargin/', views.stockMargin, name='stockMargin'),
path('stockep/', views.stockep, name='stockep'),
path('finance/', views.getFinaData, name='getFinaData'),
path('getdiv/', views.getDivData, name='getDivData'),
path('xwlbNews/', views.xwlbNews, name='xwlbNews'),
path('xwlbFine/', views.xwlbFine, name='xwlbFine'),
# 日报查询(news_report / news_event
path('news/reports/', report_views.news_reports, name='news_reports'),
path('news/events/', report_views.news_events, name='news_events'),
]
View File
+94
View File
@@ -0,0 +1,94 @@
import os
import mysql.connector
from mysql.connector import Error
class MySQLDB:
def __init__(self, host=None, port=None, username=None, password=None, database=None):
self.host = host or os.getenv('MYSQL_HOST', 'localhost')
self.port = port or int(os.getenv('MYSQL_PORT', '3306'))
self.username = username or os.getenv('MYSQL_USER', 'myquant')
self.password = password or os.getenv('MYSQL_PASSWORD', '')
self.database = database or os.getenv('MYSQL_DATABASE', 'myquant')
self.connection = None
self.connect()
def connect(self):
"""连接数据库"""
try:
self.connection = mysql.connector.connect(
host=self.host,
port=self.port,
user=self.username,
password=self.password,
database=self.database
)
if self.connection.is_connected():
print("成功连接到MySQL数据库")
except Error as e:
print(f"连接错误: {e}")
def insert_data(self, table, data):
"""插入数据"""
try:
cursor = self.connection.cursor()
columns = ', '.join(data.keys())
placeholders = ', '.join(['%s'] * len(data))
query = f"INSERT INTO {table} ({columns}) VALUES ({placeholders})"
cursor.execute(query, tuple(data.values()))
self.connection.commit()
print(f"成功插入数据,影响行数: {cursor.rowcount}")
return cursor.lastrowid
except Error as e:
print(f"插入错误: {e}")
return None
finally:
if cursor:
cursor.close()
def query_data(self, table, columns="*", where=None, params=None):
"""查询数据"""
try:
cursor = self.connection.cursor(dictionary=True)
query = f"SELECT {columns} FROM {table}"
if where:
query += f" WHERE {where}"
print(query)
cursor.execute(query, params or ())
result = cursor.fetchall()
return result
except Error as e:
print(f"查询错误: {e}")
return []
finally:
if cursor:
cursor.close()
def update_data(self, table, data, where, params=None):
"""更新数据"""
try:
cursor = self.connection.cursor()
set_clause = ', '.join([f"{key} = %s" for key in data.keys()])
query = f"UPDATE {table} SET {set_clause} WHERE {where}"
all_params = tuple(data.values()) + (params if params else ())
cursor.execute(query, all_params)
self.connection.commit()
print(f"成功更新数据,影响行数: {cursor.rowcount}")
return cursor.rowcount
except Error as e:
print(f"更新错误: {e}")
return 0
finally:
if cursor:
cursor.close()
def close(self):
"""关闭数据库连接"""
if self.connection and self.connection.is_connected():
self.connection.close()
print("数据库连接已关闭")
+439
View File
@@ -0,0 +1,439 @@
import env # 加载 .env 到环境变量
import os
import dashscope
import pydub
from pydub import AudioSegment
from pydub.silence import split_on_silence
from dashscope.audio.asr import Recognition
from dashscope import Generation
from http import HTTPStatus
from mysqlHandle import MySQLDB
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
# 设置环境变量
# os.environ["DASHSCOPE_API_KEY"] = "sk-your-dashscope-key"
def convert_mp3_to_wav(mp3_path, output_wav_path):
"""
将MP3文件转换为16kHz单声道WAV格式,这是Qwen3-ASR-Flash模型的推荐格式
参数:
mp3_path (str): MP3文件路径
output_wav_path (str): 输出WAV文件路径
返回值:
str: 转换后的WAV文件路径
"""
logger.info(f"开始转换MP3到WAV: {mp3_path}")
# 加载MP3文件
audio = AudioSegment.from_file(mp3_path, format="mp3")
# 转换为16kHz采样率、单声道、16位深度
audio = audio.set_frame_rate(16000).set_channels(1)
# 导出为WAV格式
audio.export(output_wav_path, format="wav")
logger.info(f"✓ MP3转换完成: {output_wav_path}")
#print(f"✓ MP3转换完成: {output_wav_path}")
return output_wav_path
def split_audio_by_fixed_duration(audio_path, chunk_duration, output_folder):
"""
将音频文件按固定时长分割成多个片段
参数:
audio_path (str): 音频文件路径
chunk_duration (int): 分片时长(毫秒)
output_folder (str): 输出文件夹路径
返回值:
list: 分片文件路径列表
"""
# 加载音频文件
audio = AudioSegment.from_file(audio_path)
# 计算总时长(毫秒)
total_duration = len(audio)
# 分片数
num_chunks = total_duration // chunk_duration + 1
# 存储分片文件路径
chunks = []
# 创建输出文件夹
os.makedirs(output_folder, exist_ok=True)
logger.info(f"开始音频分割,总时长: {total_duration/1000:.1f}秒,将分割为{num_chunks}个片段")
for i in range(num_chunks):
# 计算当前分片的起始和结束时间
start_time = i * chunk_duration
end_time = (i + 1) * chunk_duration
# 提取分片音频
chunk = audio[start_time:end_time]
# 生成文件名
chunk_name = f"chunk_{i}.wav"
chunk_path = os.path.join(output_folder, chunk_name)
# 导出分片音频
chunk.export(chunk_path, format="wav")
chunks.append(chunk_path)
# 打印处理进度
progress = (i + 1) / num_chunks * 100
logger.info(f"✓ 已完成分片 {i+1}/{num_chunks} ({progress:.1f}%)")
logger.info(f"✓ 音频分割完成,共生成{len(chunks)}个分片文件")
return chunks
def split_audio_by_smart_silence(audio_path, min_silence_len, silence_thresh, output_folder):
"""
将音频文件按智能静音检测方式分割成多个片段,每段不超过3分钟
参数:
audio_path (str): 音频文件路径
min_silence_len (int): 最小静音长度(毫秒)
silence_thresh (int): 静音阈值(dBFS)
output_folder (str): 输出文件夹路径
返回值:
list: 分片文件路径列表
"""
# 加载音频文件
audio = AudioSegment.from_file(audio_path, format="wav")
# 按静音分割
segments = split_on_silence(
audio,
# 静音超过700毫秒则分割
min_silence_len=min_silence_len,
# 静音阈值为-40dBFS
silence_thresh=silence_thresh,
# 保留静音部分
keep_silence=400
)
logger.info(f"✓ 静音分割完成,共{len(segments)}个初始片段")
# 合并过短的片段
merged_segments = []
current_segment = None
for segment in segments:
if current_segment is None:
current_segment = segment
else:
# 合并当前片段和新片段
temp_segment = current_segment + segment
# 如果合并后的片段超过3分钟,则单独保存当前片段
if len(temp_segment) > 180000: # 3分钟=180,000毫秒
merged_segments.append(current_segment)
current_segment = segment
else:
current_segment = temp_segment
# 添加最后一个片段
if current_segment is not None:
merged_segments.append(current_segment)
logger.info(f"✓ 片段合并完成,共{len(merged_segments)}个最终片段")
# 存储分片文件路径
chunks = []
# 创建输出文件夹
os.makedirs(output_folder, exist_ok=True)
logger.info(f"开始导出音频片段到: {output_folder}")
for i, segment in enumerate(merged_segments):
# 生成文件名
chunk_name = f"chunk_{i}.wav"
chunk_path = os.path.join(output_folder, chunk_name)
# 导出分片音频
segment.export(chunk_path, format="wav")
chunks.append(chunk_path)
# 打印处理进度
progress = (i + 1) / len(merged_segments) * 100
logger.info(f"✓ 已完成分片 {i+1}/{len(merged_segments)} ({progress:.1f}%)")
logger.info(f"✓ 智能静音分割完成,共生成{len(chunks)}个分片文件")
return chunks
def transcribe_audio(audio_path):
"""
使用Paraformer实时语音识别模型(通过本地文件)转录音频文件
参数:
audio_path (str): 音频文件路径(必须是16kHz单声道WAV)
返回值:
str: 识别文本,如果失败返回空字符串
"""
try:
# 确保音频文件存在
if not os.path.exists(audio_path):
logger.error(f"音频文件不存在: {audio_path}")
return ['', '']
dashscope.api_key = os.getenv('DASHSCOPE_API_KEY', '')
# 创建识别对象
recognition = Recognition(
model=os.getenv('DASHSCOPE_ASR_MODEL', 'paraformer-realtime-v2'),
format='wav',
sample_rate=16000,
language_hints=['zh','en'], # 中文和英文
callback=None
)
# 调用识别
logger.info(f"开始识别音频: {audio_path}")
result = recognition.call(audio_path)
text=[]
if result.status_code == HTTPStatus.OK:
# 提取识别结果
logger.info(f"{audio_path} 识别成功")
sentence = result.get_sentence()
text.append(merge_transcripts(sentence))
logger.info(f"识别文本长度: {len(text[0])}")
text.append(analyze_and_correct_text(text[0]))
return text
else:
logger.error(f"❌ 任务失败: {result.message}")
return ['', '']
except Exception as e:
logger.error(f"识别过程中发生异常: {e}")
return ['', '']
def merge_transcripts(transcripts):
"""
将多段识别文本合并成完整句子(保留原始段落逻辑,用空格连接)
参数:
transcripts (list): 识别结果列表,每个元素为字典{'text': '识别文本'}
返回:
str: 合并后的完整文本
"""
# 输入参数检查
if not transcripts:
return ""
# 确保transcripts是可迭代对象
if not hasattr(transcripts, '__iter__'):
return ""
try:
# 提取所有有效的text字段
texts = []
for t in transcripts:
try:
# 检查是否为字典类型且包含text字段
if isinstance(t, dict) and 'text' in t and t['text']:
text = t['text']
# 确保text是字符串类型
if isinstance(text, str) and text.strip():
texts.append(text.strip())
except (KeyError, TypeError, AttributeError):
# 忽略单个元素的处理错误,继续处理其他元素
continue
# 用空格连接所有段落(根据实际需求可调整连接符)
return " ".join(texts) if texts else ""
except Exception as e:
logger.error(f"合并转录文本时发生错误: {e}")
return ""
def text_correction(text):
"""
使用通义千问模型修正文本中的错误和标点符号
参数:
text (str): 需要修正的文本
返回值:
str: 修正后的文本
"""
logger.info("开始文本修正...")
# 构建修正提示词
correction_prompt = """请仔细检查以下文本,修正其中的错误:
1. 错别字和语法错误
2. 标点符号使用错误
3. 语句不通顺的地方
4. 逻辑不清晰的部分
请直接返回修正后的完整文本,不要添加任何解释说明。"""
# 构建消息列表
messages = [
{"role": "system", "content": "你是一个专业的文本校对助手,擅长修正文本中的各种错误。"},
{"role": "user", "content": correction_prompt},
{"role": "user", "content": text}
]
logger.info("调用通义千问模型进行文本修正...")
# 调用DashScope文本生成接口
response = Generation.call(
model=os.getenv('DASHSCOPE_LLM_MODEL', 'qwen-plus'),
messages=messages,
max_tokens=30000,
temperature=0.1, # 使用较低的温度以提高确定性
top_p=0.5
)
# 检查API调用是否成功
if response.status_code != 200:
logger.warning(f"❌ 文本修正API调用失败: {response.message}")
raise Exception(f"文本修正API调用失败: {response.message}")
logger.info("✓ 文本修正完成")
# 返回修正后的文本
return response.output.text
def analyze_and_correct_text(text):
"""
分析文本并自动修正错误
参数:
text (str): 待分析和修正的文本
prompt (str): 分析提示词
返回值:
tuple: (修正后的文本, 分析结果)
"""
logger.info("开始文本分析和修正流程...")
# 首先修正文本错误
corrected_text = text_correction(text)
if corrected_text is None:
logger.warning("文本修正返回 None,使用原始文本")
corrected_text = text
logger.info(f"原始文本长度: {len(text)}")
logger.info(f"修正后文本长度: {len(corrected_text)}")
# 使用修正后的文本进行分析
# analysis_result = analyze_text(corrected_text, prompt)
return corrected_text
def analyze_text(text, prompt):
"""
使用通义千问模型分析文本
参数:
text (str): 待分析文本
prompt (str): 分析提示词
返回值:
str: 分析结果
"""
logger.info("开始文本分析...")
# 设置系统提示
system_prompt = "你是一个专业的文本分析助手,擅长根据提示词对长文本进行深入分析。"
# 构建消息列表
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": prompt},
{"role": "user", "content": text}
]
logger.info("调用通义千问模型进行文本分析...")
# 调用DashScope文本生成接口
response = Generation.call(
model=os.getenv('DASHSCOPE_LLM_MODEL', 'qwen-plus'),
messages=messages,
max_tokens=8190, # 控制生成文本的最大长度
temperature=0.3, # 控制生成文本的确定性
top_p=0.7 # 控制生成文本的多样性
)
# 检查API调用是否成功
if response.status_code != 200:
logger.error(f"❌ API调用失败: {response.message}")
raise Exception(f"API调用失败: {response.message}")
logger.info("✓ 文本分析完成")
# 返回分析结果
return response.output.text
def process_long_audio(mp3_path, output_folder, date_str):
"""
处理长音频文件,分割、识别并分析
参数:
mp3_path (str): MP3文件路径
prompt (str): 分析提示词
output_folder (str): 输出文件夹路径
返回值:
str: 分析结果
"""
logger.info("开始处理长音频...")
# 转换MP3为WAV格式
logger.info("步骤1/4: 转换MP3为WAV格式")
wav_path = convert_mp3_to_wav(
mp3_path, os.path.join(output_folder, "input.wav")
)
# 分割音频
# 可以选择固定分片或智能静音分割
# chunks = split_audio_by_fixed_duration(wav_path, 180000, output_folder)
logger.info("步骤2/4: 智能静音分割音频")
chunks = split_audio_by_smart_silence(
wav_path, 700, -40, output_folder
)
# 存储所有识别文本
transcribed_text = ""
# 识别每个分片
logger.info(f"步骤3/4: 开始识别音频分片,共{len(chunks)}个分片")
for i, chunk_path in enumerate(chunks):
try:
logger.info(f"识别进度: {i+1}/{len(chunks)} ({((i+1)/len(chunks)*100):.1f}%)")
# 调用音频识别API
text = transcribe_audio(chunk_path)
"""
if not text[1].startswith('今天的新闻联播节目播送完毕'):
prompt='请分析所给文本的新闻内容,返回一个简短标题'
text.append(analyze_text(text[1],prompt))
else:
text.append('')
"""
# 新闻标题留空
text.append('')
# 添加到总文本
#transcribed_text += text + "\n"
# 删除临时文件
os.remove(chunk_path)
# 初始化数据库连接
db = MySQLDB() # 使用默认参数连接数据库
try:
# 插入数据示例
user_data = {
"news_days": date_str,
"daily_sub_id": i,
"news_raw": text[0],
"news_improve": text[1],
"news_title": text[2]
}
user_id = db.insert_data("xwlb_daily", user_data)
finally:
# 关闭连接
db.close()
except Exception as e:
logger.error(f"识别失败: {chunk_path}, 错误: {e}")
# 可以在这里添加重试逻辑
# 分析识别文本
"""
print("步骤4/4: 分析识别文本")
print(f"识别文本长度: {len(transcribed_text)}")
print(f"识别文本内容: {transcribed_text}")
if len(transcribed_text) < 1:
print("识别文本为空,跳过分析处理")
return "识别文本为空,无法进行分析"
analysis_result = analyze_text(transcribed_text, prompt)
"""
logger.info("✓ 长音频处理完成")
# 返回分析结果
return ""
# 使用示例
if __name__ == "__main__":
# MP3文件路径
mp3_path = "20251002.mp3"
# 分析提示词
prompt = "请总结这段由中国中央电视台新闻联播音频转为文字的文本,理解其主要内容并提取其中的关键信息。"
# 输出文件夹
output_folder = "audio_processing"
# 处理长音频
try:
result = process_long_audio(
mp3_path, prompt, output_folder
)
# 打印分析结果
print("分析结果:\n")
print(result)
except Exception as e:
print(f"处理失败: {e}")
+271
View File
@@ -0,0 +1,271 @@
import env # 加载 .env 到环境变量
import requests
import json
import time
import logging
from typing import Optional, Dict, Any
import os
# 配置日志
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
class DeepSeekAPI:
def __init__(self, api_key: Optional[str] = None):
"""
初始化DeepSeek API客户端
Args:
api_key: DeepSeek API密钥,如果为None则从环境变量获取
"""
self.api_key = api_key or os.getenv('DEEPSEEK_API_KEY')
if not self.api_key:
logger.warning("API密钥未提供且环境变量DEEPSEEK_API_KEY未设置")
self.api_url = "https://api.deepseek.com/v1/chat/completions"
self.max_retries = 3
self.retry_delay = 2 # 秒
# 默认系统提示词
self.default_system_prompt = """你是一个专业的AI助手,能够准确理解用户需求并提供高质量的回答。
请根据用户的输入进行适当的处理和分析,保持回答的专业性和准确性。注意:所处理文字来自中央电视台新闻联播节目转文字,请在内容审查时重点考虑。"""
def _handle_api_error(self, response: requests.Response) -> str:
"""
处理API错误响应
Args:
response: API响应对象
Returns:
错误描述信息
"""
error_msg = f"API请求失败: {response.status_code} {response.reason}"
try:
error_data = response.json()
if 'error' in error_data:
error_msg += f" - {error_data['error'].get('message', '未知错误')}"
logger.error(f"API错误详情: {error_data}")
except json.JSONDecodeError:
error_msg += f" - 响应内容: {response.text[:200]}"
return error_msg
def _make_api_request(self, payload: Dict[str, Any]) -> Dict[str, Any]:
"""
发送API请求并处理响应
Args:
payload: 请求数据
Returns:
API响应数据
Raises:
Exception: 当所有重试都失败时抛出异常
"""
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {self.api_key}"
}
last_exception = None
for attempt in range(self.max_retries):
try:
logger.info(f"发送API请求 (尝试 {attempt + 1}/{self.max_retries})")
response = requests.post(
self.api_url,
headers=headers,
json=payload,
timeout=60 # 60秒超时
)
if response.status_code == 200:
return response.json()
elif response.status_code == 400:
# 400错误通常是请求格式问题,不需要重试
error_msg = self._handle_api_error(response)
raise Exception(f"请求参数错误: {error_msg}")
elif response.status_code == 401:
# 401未授权错误,不需要重试
raise Exception("API密钥无效或未授权,请检查您的API密钥")
elif response.status_code == 429:
# 速率限制,需要重试
logger.warning("达到速率限制,等待后重试...")
time.sleep(self.retry_delay * (attempt + 1))
continue
elif 500 <= response.status_code < 600:
# 服务器错误,需要重试
logger.warning(f"服务器错误 {response.status_code},等待后重试...")
time.sleep(self.retry_delay * (attempt + 1))
continue
else:
error_msg = self._handle_api_error(response)
raise Exception(f"API请求失败: {error_msg}")
except requests.exceptions.Timeout:
last_exception = Exception(f"请求超时 (尝试 {attempt + 1})")
logger.warning(f"请求超时,等待后重试...")
time.sleep(self.retry_delay * (attempt + 1))
except requests.exceptions.ConnectionError:
last_exception = Exception(f"网络连接错误 (尝试 {attempt + 1})")
logger.warning(f"网络连接错误,等待后重试...")
time.sleep(self.retry_delay * (attempt + 1))
except requests.exceptions.RequestException as e:
last_exception = Exception(f"请求异常: {str(e)}")
logger.warning(f"请求异常,等待后重试...")
time.sleep(self.retry_delay * (attempt + 1))
# 所有重试都失败
if last_exception:
raise last_exception
else:
raise Exception("API请求失败,未知错误")
def process_text(self,
prompt: str,
text: str,
system_prompt: Optional[str] = None,
model: str = "deepseek-chat",
temperature: float = 0.7,
max_tokens: int = 2000,
response_format: Optional[Dict] = None) -> str:
"""
处理文本的通用方法
Args:
prompt: 用户提示词
text: 需要处理的文本(约1万字符)
system_prompt: 系统提示词,如果为None则使用默认值
model: 使用的模型
temperature: 生成温度
max_tokens: 最大生成token数
Returns:
处理后的文本
Raises:
Exception: 当处理失败时抛出包含详细信息的异常
"""
# 输入验证
if not self.api_key:
raise Exception("API密钥未设置,请提供api_key或设置DEEPSEEK_API_KEY环境变量")
if not prompt or not text:
raise Exception("prompt和text不能为空")
# 检查文本长度(约1万字符)
if len(text) > 15000: # 留一些余量
logger.warning(f"输入文本长度({len(text)}字符)较长,可能会超过上下文限制")
# 准备系统提示词
system_content = system_prompt or self.default_system_prompt
# 构建消息
messages = [
{"role": "system", "content": system_content},
{"role": "user", "content": f"{prompt}\n\n文本内容:\n{text}"}
]
# 构建请求数据
payload = {
"model": model,
"messages": messages,
"temperature": temperature,
"max_tokens": max_tokens,
"stream": False
}
if response_format:
payload["response_format"] = response_format
try:
# 发送API请求
response_data = self._make_api_request(payload)
# 解析响应
if 'choices' in response_data and len(response_data['choices']) > 0:
result = response_data['choices'][0]['message']['content']
logger.info("文本处理成功完成")
return result.strip()
else:
raise Exception("API响应格式异常,未找到有效结果")
except Exception as e:
logger.error(f"文本处理失败: {str(e)}")
raise Exception(f"文本处理失败: {str(e)}")
def process_text_with_fallback(self,
prompt: str,
text: str,
system_prompt: Optional[str] = None,
**kwargs) -> str:
"""
带降级处理的文本处理方法
Args:
prompt: 用户提示词
text: 需要处理的文本
system_prompt: 系统提示词
**kwargs: 其他参数
Returns:
处理后的文本,如果API调用失败则返回降级结果
"""
try:
return self.process_text(prompt, text, system_prompt, **kwargs)
except Exception as e:
logger.error(f"API调用失败,使用降级处理: {str(e)}")
# 这里可以添加降级逻辑,比如返回原始文本或简单处理
return f"处理失败,返回原始文本(错误: {str(e)}\n\n{text}"
# 使用示例
def deepseek_text(text, prompt):
# 初始化API客户端
# 方式1: 直接传入API密钥
# api_client = DeepSeekAPI(api_key="your_deepseek_api_key_here")
# 方式2: 从环境变量读取(推荐)
api_client = DeepSeekAPI() # API key 从环境变量 DEEPSEEK_API_KEY 读取
# 示例文本(约1万字符)
# sample_text = "这里是你的长文本内容..." * 500 # 模拟长文本
# 自定义系统提示词(可选)
custom_system_prompt = "你是一个专业的文本分析助手,擅长根据提示词对长文本进行深入分析。"
try:
# 处理文本(使用 response_format 强制返回 JSON
result = api_client.process_text(
model=os.getenv('DEEPSEEK_MODEL', 'deepseek-chat'),
prompt=prompt,
text=text,
system_prompt=custom_system_prompt,
temperature=0.5,
max_tokens=20000,
response_format={"type": "json_object"}
)
#print("处理结果:")
#print(result)
return result
except Exception as e:
print(f"处理失败: {e}")
# 使用降级方法
fallback_result = api_client.process_text_with_fallback(
prompt=prompt,
text=text,
system_prompt=custom_system_prompt
)
#print("降级处理结果:")
#print(fallback_result)
return result
if __name__ == "__main__":
deepseek_text()
+27
View File
@@ -0,0 +1,27 @@
"""video 模块独立 .env 加载器 — 与 djapi/env_loader.py 功能一致但非 Django 依赖"""
import os
from pathlib import Path
def _load_dotenv():
"""从项目根目录 .env 加载环境变量(不覆盖已有)"""
# video/env.py → video/ → api/ → djapi/ (项目根)
base_dir = Path(__file__).resolve().parent.parent.parent
dotenv_path = base_dir / '.env'
if not dotenv_path.exists():
return
with open(dotenv_path) as f:
for line in f:
line = line.strip()
if not line or line.startswith('#') or '=' not in line:
continue
key, _, value = line.partition('=')
key = key.strip()
value = value.strip().strip('"').strip("'")
if key and key not in os.environ:
os.environ[key] = value
_load_dotenv()
+231
View File
@@ -0,0 +1,231 @@
import requests
from bs4 import BeautifulSoup
import re,os,subprocess
from datetime import timedelta, date
import yt_dlp
from audioRead import *
from newsProcess import news_to_db
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
def get_xwlb_video_link(url):
"""
从央视网新闻联播页面抓取历史完整版视频链接
"""
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
'Referer': 'https://tv.cctv.com/'
}
try:
response = requests.get(url, headers=headers, timeout=10)
response.encoding = 'utf-8'
if response.status_code != 200:
logger.error(f"请求失败,状态码: {response.status_code}")
#print(f"请求失败,状态码: {response.status_code}")
return []
except Exception as e:
logger.error(f"请求异常: {e}")
return []
soup = BeautifulSoup(response.text, 'html.parser')
video_links = []
# 查找所有包含“完整版《新闻联播》”的链接
# 方法1: 查找包含 <i class="sql0">完整版</i>《新闻联播》 的 a 标签
for a_tag in soup.find_all('a', href=True):
# 检查文本中是否包含“完整版”和“新闻联播”
title_text = a_tag.get_text(strip=True)
inner_html = str(a_tag)
# 判断是否是“完整版《新闻联播》”的链接
if ('完整版' in title_text and '新闻联播' in title_text) or \
(re.search(r'<i[^>]*>完整版</i>\s*《新闻联播》', inner_html)):
video_url = a_tag['href']
# 提取日期信息(从标题或链接中)
date_match = re.search(r'\d{8}', title_text)
if not date_match:
# 从链接中提取日期,如 /2025/09/25/...VID...250925.shtml
date_match = re.search(r'/(\d{4})/(\d{2})/(\d{2})/', video_url)
if date_match:
year, month, day = date_match.groups()
date_str = f"{year}{month}{day}"
else:
date_str = "未知日期"
else:
date_str = date_match.group()
video_links.append({
'date': date_str,
'title': title_text.strip(),
'url': video_url,
'page_url': url
})
logger.info(f"✅ 找到新闻联播完整版: {date_str} -> {video_url}")
return video_url
def xwlb_urls(start: str, end: str):
"""
start/end 格式 '20240925'
返回列表 ['https://tv.cctv.com/lm/xwlb/day/20240925.shtml', ...]
"""
d0 = date(int(start[:4]), int(start[4:6]), int(start[6:8]))
d1 = date(int(end[:4]), int(end[4:6]), int(end[6:8]))
urls = []
for n in range((d1 - d0).days + 1):
day = d0 + timedelta(days=n)
urls.append({"url": f"https://tv.cctv.com/lm/xwlb/day/{day:%Y%m%d}.shtml", "date": f"{day:%Y%m%d}"})
#print(urls)
return urls
def get_all_video_links(start: str, end: str):
base_urls=xwlb_urls(start,end)
#print(base_urls)
video_urls = []
for url in base_urls:
video=get_xwlb_video_link(url['url'])
video_urls.append({"url":video,"date":url['date']})
return video_urls
'''
get_xwlb_video_link() 方法获得的url
urls like: https://tv.cctv.com/2024/10/30/VIDEUlPz1Qusy41JFQj3LMLd241030.shtml
通过yt-dlp下载视频保存为mp4文件并用ffmpeg提取音频为mp3文件,文件名使用url 的日期部分如上面的url应保存为 20241030.mp4 20241030.mp3
文件保存路径为当前目录下的 xwlb_video 文件夹若不存在则创建
'''
def download_and_extract_audio(video_url,date_str,download_dir):
"""
使用yt-dlp下载视频并提取音频
"""
# 从URL中提取日期
os.makedirs(download_dir, exist_ok=True)
# 构建文件路径
mp4_path = os.path.join(download_dir, f"{date_str}.mp4")
mp3_path = os.path.join(download_dir, f"{date_str}.mp3")
try:
# 使用yt-dlp库下载视频
logger.info(f"📥 开始下载 {date_str} 的视频...")
# 配置yt-dlp选项
ydl_opts = {
'outtmpl': mp4_path,
'format': 'best[ext=mp4]/best',
'progress_hooks': [lambda d: print(f"\r📥 下载进度: {d.get('_percent_str', 'N/A').strip()} | {d.get('_speed_str', 'N/A').strip()} | 已下载: {d.get('_downloaded_bytes_str', 'N/A')}", end='') if d['status'] == 'downloading' else None],
}
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
ydl.download([video_url])
logger.info(f"📥 下载 {date_str} 完成")
# 使用ffmpeg提取音频
logger.info(f"🎵 开始提取 {date_str} 的音频...")
result = subprocess.run([
"ffmpeg",
"-i", mp4_path,
"-c:a", "libmp3lame", # 明确指定MP3编码器
"-q:a", "0",
"-map", "a",
mp3_path,
"-y" # 覆盖已存在文件
], check=True, stdout=None, stderr=None)
logger.info(f"🎵 提取 {date_str} 音频完成")
logger.info(f"✅ 成功处理 {date_str}: {mp4_path}, {mp3_path}")
except Exception as e:
logger.error(f"❌ 处理 {date_str} 时发生异常: {e}")
# 在get_all_video_links函数后添加调用代码
def process_videos(start_date, end_date):
"""
处理指定日期范围内的所有视频
"""
video_urls = get_all_video_links(start_date, end_date)
for sub_url in video_urls:
if sub_url: # 确保url不为空
"""
date_match = re.search(r'/(\d{4})/(\d{2})/(\d{2})/', url)
if not date_match:
print(f"❌ 无法从URL提取日期: {url}")
return
year, month, day = date_match.groups()
date_str = f"{year}{month}{day}"
"""
date_str= sub_url['date']
url = sub_url['url']
# 创建保存目录
download_dir = "/home/simon/myquant/djapi/api/video/xwlb_video"
download_and_extract_audio(url,date_str,download_dir)
print("=" * 80)
# MP3文件路径
mp3_path = os.path.join(download_dir, f"{date_str}.mp3")
# 分析提示词
# prompt = "请总结这段由中国中央电视台新闻联播音频转为文字的文本,理解其主要内容并提取其中的关键信息。"
# 输出文件夹
output_folder = "/home/simon/myquant/djapi/api/video/audio_processing"
# 处理长音频
try:
result = process_long_audio(mp3_path, output_folder,date_str)
news_to_db(date_str)
# 打印分析结果
print("分析结果:\n")
print(result)
except Exception as e:
print(f"处理失败: {e}")
print("=" * 80)
# ========================
# 主程序执行
# ========================
if __name__ == "__main__":
import sys
import re
from datetime import datetime
# 检查命令行参数
if len(sys.argv) < 2:
print("用法: python getVideo5.py <start_date> <end_date>")
print("日期格式: YYYYMMDD")
sys.exit(1)
start_date = sys.argv[1]
end_date = sys.argv[2] if len(sys.argv) > 2 and sys.argv[2] else start_date
# 检查日期格式
date_pattern = r'^\d{8}$'
if not re.match(date_pattern, start_date) or not re.match(date_pattern, end_date):
print("错误: 日期格式必须为 YYYYMMDD")
sys.exit(1)
# 检查日期有效性
try:
#start_dt = datetime.strptime(start_date, '%Y%m%d')
#end_dt = datetime.strptime(end_date, '%Y%m%d')
if start_date > end_date:
print(f"错误: start_date {start_date} 不能大于 end_date {end_date}")
sys.exit(1)
print("正在抓取央视《新闻联播》历史完整版视频链接...")
print("=" * 80)
process_videos(start_date, end_date)
exit()
except ValueError as e:
print(f"错误: 无效日期 - {e}")
sys.exit(1)
+10
View File
@@ -0,0 +1,10 @@
from getVideo5 import process_videos
from datetime import datetime
# 获取当日日期并格式化为yyyymmdd
today = datetime.now().strftime("%Y%m%d")
start_date = today
end_date = today
# 执行process_videos方法
process_videos(start_date, end_date)
+75
View File
@@ -0,0 +1,75 @@
"""
xwlb_daily 表结构如下
+--------------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+---------+------+-----+---------+----------------+
| nid | int(11) | NO | PRI | NULL | auto_increment |
| news_days | date | NO | | NULL | |
| daily_sub_id | int(11) | NO | | NULL | |
| news_raw | text | NO | | NULL | |
| news_improve | text | NO | | NULL | |
| news_title | text | NO | | NULL | |
+--------------+---------+------+-----+---------+----------------+
"""
from mysqlHandle import MySQLDB
from getVideo5 import process_videos
from datetime import datetime, timedelta
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
def get_missing_dates(start_date, end_date):
"""
给定日期范围查询xwlb_daily表中缺失的日期
"""
try:
# 连接数据库
db = MySQLDB()
# 查询指定日期范围内存在的所有日期
result = db.query_data(
table="xwlb_daily",
columns="DISTINCT(news_days) as news_days",
where="news_days BETWEEN %s AND %s order by news_days",
params=(start_date, end_date)
)
# 获取所有存在的日期
existing_dates = [row['news_days'] for row in result]
# 生成完整的日期范围
start = datetime.strptime(start_date, '%Y-%m-%d').date()
end = datetime.strptime(end_date, '%Y-%m-%d').date()
all_dates = []
current_date = start
while current_date <= end:
all_dates.append(current_date)
current_date = current_date + timedelta(days=1)
# 找出缺失的日期
existing_set = set(existing_dates)
missing_dates = [date.strftime('%Y%m%d') for date in all_dates if date not in existing_set]
logger.info(f"查询日期范围 {start_date}{end_date}")
logger.info(f"存在 {len(existing_dates)} 天数据,缺失 {len(missing_dates)} 天数据")
logger.info(f"缺失日期: {missing_dates}")
return missing_dates
except Exception as e:
logger.error(f"查询缺失日期时出错: {str(e)}")
return []
if __name__ == "__main__":
# 测试代码
start_date = "2025-01-01"
end_date = "2025-10-25"
missing_dates=get_missing_dates(start_date, end_date)
for date_str in missing_dates:
logger.info(f"正在处理缺失日期: {date_str}")
try:
process_videos(date_str,date_str)
logger.info(f"成功处理日期: {date_str}")
except Exception as e:
logger.error(f"处理日期 {date_str} 时出错: {str(e)}")
+5
View File
@@ -0,0 +1,5 @@
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from utils.mysql_handler import MySQLDB # noqa: F401, E402 — video 模块独立运行,sys.path 方式导入
+132
View File
@@ -0,0 +1,132 @@
"""
xwlb_daily 表结构如下
+--------------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+---------+------+-----+---------+----------------+
| nid | int(11) | NO | PRI | NULL | auto_increment |
| news_days | date | NO | | NULL | |
| daily_sub_id | int(11) | NO | | NULL | |
| news_raw | text | NO | | NULL | |
| news_improve | text | NO | | NULL | |
| news_title | text | NO | | NULL | |
+--------------+---------+------+-----+---------+----------------+
xwlb_daily_ext 表结构如下
+--------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+----------------+
| extid | int(11) | NO | PRI | NULL | auto_increment |
| news_date | date | NO | | NULL | |
| sub_id | tinyint(4) | NO | | NULL | |
| news_title | varchar(256) | NO | | NULL | |
| news_content | text | NO | | NULL | |
+--------------+--------------+------+-----+---------+----------------+
获取给定日期的所有news_improve字段内容以daily_sub_id 顺序拼接为一个字符串返回
调用mysqlHandler中的方法执行SQL查询
"""
from mysqlHandle import MySQLDB
from deepseek import deepseek_text
import json
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
def get_news_improve_by_date(target_date):
"""
获取指定日期的所有news_improve内容按daily_sub_id顺序拼接
Args:
target_date: 目标日期格式为'YYYY-MM-DD'
Returns:
str: 拼接后的字符串
"""
try:
# 创建数据库连接对象
db = MySQLDB()
# 查询目标日期在xwlb_daily_ext表中的记录数量
count_result = db.query_data(
table="xwlb_daily_ext",
columns="COUNT(*) as count",
where="news_date = %s",
params=(target_date,)
)
# 如果记录数量存在且大于5条,则返回空字符串
if count_result and count_result[0]['count'] > 5:
return None
# 重新创建数据库连接对象,因为每次查询都会关闭连接
db = MySQLDB()
# 查询目标日期在xwlb_daily表中的news_improve字段,按daily_sub_id升序排列
result = db.query_data(
table="xwlb_daily",
columns="news_improve",
where="news_days = %s order by daily_sub_id ASC",
params=(target_date,))
# 如果查询结果不为空
if result:
# 将每条记录的news_improve字段用换行符连接成字符串
combined_content = '\n'.join([row['news_improve'] for row in result])
# 返回拼接后的字符串
return combined_content
# 查询结果为空时返回空字符串
return None
except Exception as e:
#print(f"查询失败: {e}")
logger.error(f"查询失败: {e}")
return None
def news_to_db(target_date):
result = get_news_improve_by_date(target_date)
if result is None:
logger.warning(f"日期 {target_date} 没有新闻内容或者已经存在处理后的记录。跳过")
return None
logger.info(f"日期 {target_date} 的新闻内容长度:{len(result)} 字符")
prompt= "###请根据下面新闻内容的文本逻辑 \n - 帮我分割成各个独立的新闻内容(注意:不要修改新闻本身,仅分割文本),并给每个新闻总结一个标题; \n - 如果遇到'国内快讯''国际快讯''联播快讯',也请根据每个条快讯分割为一个新闻以及新闻标题; \n - 返回json格式。json格式包含:news_idnews_titlenews_content; news_id从1开始递增。"
try:
response = deepseek_text(result, prompt)
data = json.loads(response)
# DeepSeek json_object 模式返回的是 dict(如 {"news": [...]}),
# 普通模式返回的是纯数组 [...],这里做自适应提取
if isinstance(data, dict):
# 从 dict 中提取列表:找第一个 list 类型的 value
news_list = None
for v in data.values():
if isinstance(v, list):
news_list = v
break
if news_list is None:
# 所有 value 都不是 list,可能是 {"1": {...}, "2": {...}} 格式
news_list = list(data.values())
elif isinstance(data, list):
news_list = data
else:
raise ValueError(f"不支持 DeepSeek 响应格式: {type(data)}")
db = MySQLDB()
for news in news_list:
db.insert_data(
table="xwlb_daily_ext",
data={
"news_date": target_date,
"sub_id": news["news_id"],
"news_title": news["news_title"][:256], # 确保不超过varchar(256)限制
"news_content": news["news_content"]
}
)
logger.info(f"成功插入 {len(news_list)} 条新闻到数据库")
except json.JSONDecodeError as e:
logger.error(f"JSON解析失败: {e}")
logger.error(f"DeepSeek API返回内容: {response}")
except Exception as e:
logger.error(f"插入数据库失败: {e}")
#print(f"DeepSeek API返回结果:{response}")
if __name__ == "__main__":
import sys
from datetime import datetime
#提供日期参数,格式:YYYY-MM-DD
target_date = datetime.now().strftime('%Y-%m-%d')
news_to_db(target_date)
+112
View File
@@ -0,0 +1,112 @@
"""
newsRedo 手动重新执行新闻 AI 分割流程
用法
python newsRedo.py # 默认当天日期
python newsRedo.py 20250601 # yyyymmdd 格式
python newsRedo.py 2025-06-01 # yyyy-mm-dd 格式
流程
1. 检查 xwlb_daily_ext 是否已有 >5 已处理过正常跳过
2. 检查 xwlb_daily 是否有当天记录 无记录则先跑 getVideo5 全流程
3. 有记录但未处理 直接执行 news_to_db() AI 分割
"""
import sys
import re
import logging
from datetime import datetime
from mysqlHandle import MySQLDB
from newsProcess import news_to_db
from getVideo5 import process_videos
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
def _parse_date(date_str):
"""解析日期,返回 (yyyymmdd_str, yyyy_mm_dd_str),或报错退出"""
if not date_str:
today = datetime.now()
d8 = today.strftime('%Y%m%d')
d10 = today.strftime('%Y-%m-%d')
logger.info(f"未指定日期,使用当天: {d10}")
return d8, d10
if re.match(r'^\d{4}-\d{2}-\d{2}$', date_str):
try:
datetime.strptime(date_str, '%Y-%m-%d')
except ValueError:
print(f"无效日期: {date_str}")
sys.exit(1)
return date_str.replace('-', ''), date_str
if re.match(r'^\d{8}$', date_str):
try:
datetime.strptime(date_str, '%Y%m%d')
except ValueError:
print(f"无效日期: {date_str}")
sys.exit(1)
return date_str, f"{date_str[:4]}-{date_str[4:6]}-{date_str[6:8]}"
print("日期格式错误,请使用 yyyymmdd 或 yyyy-mm-dd 格式")
sys.exit(1)
def main():
date_str = sys.argv[1] if len(sys.argv) > 1 else None
date_d8, date_d10 = _parse_date(date_str)
db = MySQLDB()
# 1. 检查 xwlb_daily_ext 是否已处理过
try:
ext_count = db.query_data(
table="xwlb_daily_ext",
columns="COUNT(*) as count",
where="news_date = %s",
params=(date_d10,)
)
if ext_count and ext_count[0]['count'] > 5:
logger.info(f"日期 {date_d10} 已有 {ext_count[0]['count']} 条精编记录,无需重新处理。")
return
except Exception as e:
logger.error(f"查询 xwlb_daily_ext 失败: {e}")
finally:
db.close()
db = MySQLDB()
# 2. 检查 xwlb_daily 是否有当天数据
try:
daily_count = db.query_data(
table="xwlb_daily",
columns="COUNT(*) as count",
where="news_days = %s",
params=(date_d10,)
)
has_daily = daily_count and daily_count[0]['count'] > 0
except Exception as e:
logger.error(f"查询 xwlb_daily 失败: {e}")
has_daily = False
finally:
db.close()
# 3. 分支处理
if has_daily:
logger.info(f"日期 {date_d10} 在 xwlb_daily 中有记录,直接执行 AI 分割。")
try:
news_to_db(date_d10)
except Exception as e:
logger.error(f"news_to_db 执行出错: {e}")
sys.exit(1)
else:
logger.info(f"日期 {date_d10} 在 xwlb_daily 中无记录,重新执行视频下载全流程。")
try:
process_videos(date_d8, date_d8)
except Exception as e:
logger.error(f"process_videos 执行出错: {e}")
sys.exit(1)
if __name__ == "__main__":
main()
+10
View File
@@ -0,0 +1,10 @@
aiofiles==25.1.0
aiohttp==3.12.15
beautifulsoup4==4.14.2
dashscope==1.24.6
m3u8==6.0.0
mysql_connector_repackaged==0.3.1
playwright==1.55.0
pydub==0.25.1
Requests==2.32.5
yt_dlp==2025.11.12
+250
View File
@@ -0,0 +1,250 @@
import sys
from django.shortcuts import render
from django.http import HttpResponse
from rest_framework.decorators import api_view
from rest_framework.response import Response
from drf_spectacular.utils import extend_schema, OpenApiParameter, OpenApiTypes
from .stock.stock_utils import viewFunc_tsCodeAndDate, viewFunc_singleParam
from .stock.stock_basic import getStockBasic, getStockListByIndustry, getStockInfo
from .stock.getStockParam import getStockParam
from .stock.getStockEp import getStockEp_ttm, get_quarterly_eps
from .stock.getIndexs import get_index_daily_data, get_index_by_name
from .stock.stockMargin import getStockMargin, getDailyMargin
from .stock.getStockFina import get_finance_data_range
from .stock.getStockDiv2 import analyze_stock_dividend_and_price
from .stock.xwlbDaily import get_xwlb, get_xwlb_fine
from .serializers import (
StockDailySerializer, StockInfoSerializer, IndustryStockSerializer,
StockParamSerializer, StockEpSerializer, QuarterlyEpsSerializer,
IndexInfoSerializer, IndexDailySerializer, MarginDailySerializer,
StockMarginSerializer, FinanceDataSerializer, DividendSerializer,
XwlbNewsSerializer,
)
# === 通用参数定义(复用) ===
_PARAM_TSCODE = OpenApiParameter(name='tscode', type=str, default='000001.SZ',
description='股票代码,如 000001.SZ')
_PARAM_INDEX_CODE = OpenApiParameter(name='tscode', type=str, default='000001.SH',
description='指数代码,如 000001.SH=上证指数, 399001.SZ=深证成指, 399006.SZ=创业板指')
_PARAM_START = OpenApiParameter(name='start_date', type=str, default='20200101',
description='起始日期 yyyyMMdd')
_PARAM_END = OpenApiParameter(name='end_date', type=str, default='20251231',
description='结束日期 yyyyMMdd')
_PARAM_INDEX_NAME = OpenApiParameter(name='index_name', type=str, default='沪深300',
description='指数名称,如 沪深300、上证50')
_PARAM_INDUSTRY = OpenApiParameter(name='industry', type=str, required=True,
description='行业名称,如 银行、半导体')
_PARAM_TRADE_DATE = OpenApiParameter(name='trade_date', type=str,
description='交易日期 yyyyMMdd')
_PARAM_EXCHANGE_ID = OpenApiParameter(name='exchange_id', type=str,
description='交易所代码 SSE/SZSE')
@extend_schema(
responses={200: OpenApiTypes.STR},
description='返回服务器 Python 版本',
tags=['系统'],
)
@api_view(['GET'])
def python_version(request):
return HttpResponse(f"Python Version: {sys.version}")
@extend_schema(exclude=True)
@api_view(['GET'])
def home(request):
return render(request, 'home.html')
@extend_schema(
parameters=[_PARAM_TSCODE, _PARAM_START, _PARAM_END],
responses={200: StockDailySerializer(many=True)},
description='获取个股日线行情数据(开高低收、成交量、成交额)',
tags=['行情'],
)
@api_view(['GET'])
def stockbasic(request):
return viewFunc_tsCodeAndDate(request, getStockBasic)
@extend_schema(
parameters=[_PARAM_INDUSTRY],
responses={200: IndustryStockSerializer(many=True)},
description='按申万行业分类查询成分股列表',
tags=['基础数据'],
)
@api_view(['GET'])
def industrys(request):
return viewFunc_singleParam(request, getStockListByIndustry, param_name='industry')
@extend_schema(
parameters=[_PARAM_TSCODE],
responses={200: StockInfoSerializer()},
description='获取个股基本信息(名称、行业、上市日期、交易所等)',
tags=['基础数据'],
)
@api_view(['GET'])
def stockInfo(request):
return viewFunc_singleParam(request, getStockInfo, param_name='tscode')
@extend_schema(
parameters=[_PARAM_TSCODE, _PARAM_START, _PARAM_END],
responses={200: StockParamSerializer(many=True)},
description='获取个股每日参数(市值、PE/PB/PS、换手率等)',
tags=['行情'],
)
@api_view(['GET'])
def stockparam(request):
return viewFunc_tsCodeAndDate(request, getStockParam)
@extend_schema(
parameters=[_PARAM_TSCODE, _PARAM_START, _PARAM_END],
responses={200: StockEpSerializer(many=True)},
description='获取个股 TTM 每股收益(EPS',
tags=['财务'],
)
@api_view(['GET'])
def stockep(request):
return viewFunc_tsCodeAndDate(request, getStockEp_ttm)
@extend_schema(
parameters=[_PARAM_TSCODE, _PARAM_START, _PARAM_END],
responses={200: QuarterlyEpsSerializer(many=True)},
description='获取个股季度每股收益(EPS),按财报日期对齐',
tags=['财务'],
)
@api_view(['GET'])
def quarterlyEps(request):
return viewFunc_tsCodeAndDate(request, get_quarterly_eps)
@extend_schema(
parameters=[_PARAM_INDEX_NAME],
responses={200: IndexInfoSerializer()},
description='按名称模糊查询指数基本信息',
tags=['指数'],
)
@api_view(['GET'])
def indexByName(request):
index_name = request.GET.get('index_name', '沪深300')
if not index_name:
return Response({'error': '缺少 index_name 参数'}, status=400)
try:
data = get_index_by_name(index_name)
dict_data = data.to_dict(orient='records')
return Response(dict_data)
except ImportError:
return Response({'error': '模块不存在'}, status=500)
except Exception as e:
return Response({'error': str(e)}, status=500)
@extend_schema(
parameters=[_PARAM_INDEX_CODE, _PARAM_START, _PARAM_END],
responses={200: IndexDailySerializer(many=True)},
description='获取指数日线行情数据(含 PE/PB/市值/换手率等扩展指标)',
tags=['指数'],
)
@api_view(['GET'])
def indexDatas(request):
return viewFunc_tsCodeAndDate(request, get_index_daily_data)
@extend_schema(
parameters=[_PARAM_TSCODE, _PARAM_START, _PARAM_END],
responses={200: StockMarginSerializer(many=True)},
description='获取个股融资融券明细数据',
tags=['融资融券'],
)
@api_view(['GET'])
def stockMargin(request):
return viewFunc_tsCodeAndDate(request, getStockMargin)
@extend_schema(
parameters=[_PARAM_TRADE_DATE, _PARAM_START, _PARAM_END, _PARAM_EXCHANGE_ID],
responses={200: MarginDailySerializer(many=True)},
description='获取每日融资融券汇总数据(按交易所)',
tags=['融资融券'],
)
@api_view(['GET'])
def dailyMargin(request):
trade_date = request.GET.get('trade_date', None)
start_date = request.GET.get('start_date', None)
end_date = request.GET.get('end_date', None)
exchange_id = request.GET.get('exchange_id', None)
try:
data = getDailyMargin(trade_date=trade_date, start_date=start_date,
end_date=end_date, exchange_id=exchange_id)
dict_data = data.to_dict(orient='records')
return Response(dict_data)
except ImportError:
return Response({'error': '模块不存在'}, status=500)
except Exception as e:
return Response({'error': str(e)}, status=500)
@extend_schema(
parameters=[_PARAM_TSCODE, _PARAM_START, _PARAM_END],
responses={200: FinanceDataSerializer(many=True)},
description='获取个股财务报表分析数据(资产负债表+利润表+现金流,含运营/资产/负债/回报率指标)',
tags=['财务'],
)
@api_view(['GET'])
def getFinaData(request):
return viewFunc_tsCodeAndDate(request, get_finance_data_range)
@extend_schema(
parameters=[_PARAM_TSCODE, _PARAM_START, _PARAM_END],
responses={200: DividendSerializer(many=True)},
description='获取个股股息率数据(含 TTM 分红、收盘价、股息率)',
tags=['分红'],
)
@api_view(['GET'])
def getDivData(request):
return viewFunc_tsCodeAndDate(request, analyze_stock_dividend_and_price)
def _xwlb_view(request, data_func):
start_date = request.GET.get('start_date', None)
end_date = request.GET.get('end_date', None)
try:
data = data_func(start_date=start_date, end_date=end_date)
dict_data = data.to_dict(orient='records')
status = "success" if dict_data else "Error"
return Response({"status": status, "data": {"news": dict_data}})
except ImportError:
return Response({'error': '模块不存在'}, status=500)
except Exception as e:
return Response({'error': str(e)}, status=500)
@extend_schema(
parameters=[_PARAM_START, _PARAM_END],
responses={200: XwlbNewsSerializer(many=True)},
description='获取新闻联播原始识别文本(ASR 转写结果)',
tags=['新闻联播'],
)
@api_view(['GET'])
def xwlbNews(request):
return _xwlb_view(request, get_xwlb)
@extend_schema(
parameters=[_PARAM_START, _PARAM_END],
responses={200: XwlbNewsSerializer(many=True)},
description='获取新闻联播精编内容(AI 分割+标题提取后的独立新闻)',
tags=['新闻联播'],
)
@api_view(['GET'])
def xwlbFine(request):
return _xwlb_view(request, get_xwlb_fine)
+135
View File
@@ -0,0 +1,135 @@
# continuation.md
## 当前项目状态
djapi — Django 5.2 金融数据 API 项目,2026-06-17 已部署。
## Checkpoint 记录
| 日期 | 内容 |
|------|------|
| 2026-08-03 | 新增日报查询 API ×2news/reports/ + news/events/),基于 news_report/news_event 表,已部署 doorcome ✅ |
服务器:`simon@doorcome.cn`,路径 `/home/simon/myquant/djapi/`,虚拟环境 `/opt/miniconda/envs/django/`
## 已完成
### 1. 安全:密钥统一管理
- 所有密钥 → 环境变量,`.env` 统一管理
- `djapi/env_loader.py`Django 端)+ `api/video/env.py`video 端)双加载器
- 共享 `MySQLDB``api/utils/mysql_handler.py`
- `.env.example``.gitignore`
### 2. 代码质量
- `api/views.py`227 → ~130 行,消除重复
- `api/stock/stock_utils.py``viewFunc_singleParam()` 包装器
- `api/stock/config.py`:拆分为 config / strategy_config / scan_config
### 3. drf-spectacular 集成
- 14 端点 `@api_view` + `@extend_schema`8 tag 分组
- 13 SerializerSwagger `/api/docs/`
### 4. 股息率 API 优化
- **删除** `api/stock/getDivData_AK.py`akshare 版),`/api/getdivak/` 路由移除
- **优化** `api/stock/getStockDiv2.py`
- TTM 计算:`calculate_ttm_div` 行级循环 O(n²) → `rolling('360D').sum()` O(n)
- 删除向前填充逻辑(~30 行),避免与毛刺平滑冲突
- **修复** `api/stock/smoothBrush.py`if/elif 分支中 prev_valid/next_valid 赋值反了
### 5. video 模块重构与 Bug 修复
- 新增 `api/video/env.py` — .env 加载
- `newsRedo.py` 重写 — 三分支智能重处理
- **P0 修复**`getVideo5.py` 日期校验 bug`start_date > start_date``start_date > end_date`
- **P1 清理**:删除 `ai.py`(两个函数均为死代码),清理 `newsProcess.py` 冗余 import
- **P2 修复**`audioRead.py``transcribe_audio` 异常时返回 `['', '']` 统一类型;`analyze_and_correct_text` 防御 None
- **P3 修复**`newsProcess.py``news_to_db()` JSON 解析自适应 dict/listDeepSeek json_object 模式返回 dict 包装)
### 6. 文档与测试
- `CLAUDE.md``README.md``continuation.md`
- 18 个单元测试
## video 目录文件现状(10 个 .py)
| 文件 | 职责 |
|------|------|
| `env.py` | .env 加载 |
| `getVideo5.py` | 主流程:抓取→下载→ASR→入库 |
| `audioRead.py` | 音频转换、分割、ASR 识别、文本纠错 |
| `deepseek.py` | DeepSeek API 封装(类 + `deepseek_text` 函数,支持 `response_format` |
| `newsProcess.py` | AI 新闻分割+标题提取,JSON 自适应解析 |
| `newsRedo.py` | 手动重处理(三分支) |
| `main.py` | 定时任务入口(当天) |
| `main_videos.py` | 批量补缺(扫描缺失日期) |
| `mysqlHandle.py` | MySQLDB 重新导出 |
## 所有 API 端点(16 个)
| 端点 | 数据源 | 说明 |
|------|--------|------|
| `stockbasic/` | Tushare | 日线行情 |
| `stockinfo/` | Tushare | 个股基本信息 |
| `industrys/` | Tushare | 行业股票列表 |
| `stockparam/` | Tushare | 个股参数 |
| `stockep/` | Tushare | TTM EPS |
| `quarterlyEps/` | Tushare | 季度 EPS |
| `indexByName/` | Tushare | 指数查询 |
| `indexDatas/` | Tushare | 指数行情 |
| `dailymargin/` | Tushare | 每日融资融券汇总 |
| `stockmargin/` | Tushare | 个股融资融券 |
| `finance/` | Tushare | 财务报表分析 |
| `getdiv/` | Tushare | 股息率(TTM rolling + 毛刺平滑) |
| `xwlbNews/` | MySQL | 新闻联播原始文本 |
| `xwlbFine/` | MySQL | 新闻联播 AI 精编 |
| `news/reports/` | MySQL (news_) | 日报查询:默认最近 24h;传 id 返回详情含事件 |
| `news/events/` | MySQL (news_) | 重要事件聚合:最近 N 天 importance≥阈值 |
---
## 日报查询 API2026-08-03 新增)
### 模块
- 新增 `api/report/` 包(独立于 stock):`query.py`(连库+查询 SQL/ `views.py`2 视图)/ `serializers.py`OpenAPI/ `tests.py`17 个单测,mock 查询层)
- `api/urls.py` 注册 `news/reports/``news/events/``settings.py` SPECTACULAR TAGS 加「日报」
- 数据库:doorcome 本机 MariaDB `myquant``news_report`180 行)+ `news_event`4372 条),与现有 `MYSQL_*` 同库同用户
- 连接配置:服务器 `djapi/.env` 新增 `NEWS_DB_*`(复用 MYSQL_* 值,密码必填否则 500
- 文档:`docs/news_report_api.md`(使用手册,含线上地址/curl/真实样例);README API 概览表已加两行
### 部署(2026-08-03 完成)
- rsync 增量同步(**未用文档中的 --delete**,见下)→ 重启 uWSGI → 冒烟通过(列表/详情/聚合/400/404)
- 线上:`https://api.doorcome.cn/api/news/reports/``/api/news/events/`Swagger `/api/docs/`「日报」tag
### 已知事项
1. **服务器顶层历史平铺文件未清理**`/home/simon/myquant/djapi/` 顶层有 views.py/urls.py/smoothBrush.py/getStockDiv2.py/env_loader.py/akshare_data.py(历史 rsync 陷阱产物),`--delete` 会删除它们,但 `divSearch.py`(离线脚本)仍绝对导入顶层 getStockDiv2/smoothBrush → 本次增量同步保留;清理前需先修 divSearch.py 的导入
2. **既有失败测试**`api.tests.DateFormatCorrectionTest.test_empty_string`date_format_correction('') 期望 None 实得 ''),与本次无关
3. 冒烟曾发现 fetch_reports 列表 SQL 缺 `r.` 别名前缀(1052 ambiguous),已修复
4. 本地验证需绕过 macOS TCC`HOME=/tmp/djtest_home PYTHONPATH=/tmp/djtest_pkgs`tushare 写 ~/tk.csv 被拦 + quant 环境缺 mysql-connector-python
## 部署
```bash
# 全量同步
rsync -avz --delete \
--exclude='.env' --exclude='db.sqlite3' \
--exclude='*.log' --exclude='uwsgi.pid' \
--exclude='__pycache__/' --exclude='*.pyc' \
--exclude='xwlb_video/' --exclude='audio_processing/' \
/Users/summer/Downloads/cc-cursor/djapi/ \
simon@doorcome.cn:/home/simon/myquant/djapi/
# 单文件同步必须写完整路径
# 正确:rsync api/views.py simon@...:/.../djapi/api/views.py
# 重启
ssh simon@doorcome.cn "kill \$(lsof -ti:5004); sleep 2; /opt/miniconda/envs/django/bin/uwsgi --ini /home/simon/myquant/djapi/uwsgi.ini"
```
## 关键设计决策
- video 模块保护、向后兼容优先、不使用 python-dotenv
- rsync 陷阱:多文件源会展平路径
- `.env` 双加载:Django 端 `djapi/env_loader.py` + video 端 `api/video/env.py`
- 股息率 TTM 用 `rolling('360D').sum()` 向量化,不手动循环
- DeepSeek json_object 模式返回 dictnewsProcess 自适应提取 list
View File
+16
View File
@@ -0,0 +1,16 @@
"""
ASGI config for djapi project.
It exposes the ASGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/5.2/howto/deployment/asgi/
"""
import os
from django.core.asgi import get_asgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'djapi.settings')
application = get_asgi_application()
+51
View File
@@ -0,0 +1,51 @@
import os
from pathlib import Path
def _load_dotenv():
"""从项目根目录 .env 文件加载环境变量(不覆盖已有环境变量)"""
# 向上查找 .env:从当前文件位置 → djapi/ → 项目根目录
base_dir = Path(__file__).resolve().parent.parent
dotenv_path = base_dir / '.env'
if not dotenv_path.exists():
return
with open(dotenv_path) as f:
for line in f:
line = line.strip()
if not line or line.startswith('#') or '=' not in line:
continue
key, _, value = line.partition('=')
key = key.strip()
value = value.strip().strip('"').strip("'")
if key and key not in os.environ:
os.environ[key] = value
# 模块导入时自动加载 .env
_load_dotenv()
def get_env(key, default=None, required=False):
"""
从环境变量读取配置值
Args:
key: 环境变量名
default: 默认值
required: 是否必须若为 True 且变量不存在抛出 ValueError
Returns:
环境变量值或默认值
"""
value = os.getenv(key, default)
if required and (value is None or value == ''):
raise ValueError(f'缺少必须的环境变量: {key}')
return value
def get_env_bool(key, default=False):
"""读取布尔型环境变量"""
val = os.getenv(key, str(default)).lower()
return val in ('true', '1', 'yes')
+220
View File
@@ -0,0 +1,220 @@
"""
Django settings for djapi project.
Generated by 'django-admin startproject' using Django 5.2.1.
For more information on this file, see
https://docs.djangoproject.com/en/5.2/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/5.2/ref/settings/
"""
import os
from pathlib import Path
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
# 从 .env_loader 加载环境变量检测(本地开发可用 shell export 或 .env 文件)
from .env_loader import get_env, get_env_bool
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/5.2/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = get_env('DJANGO_SECRET_KEY', required=not get_env_bool('DJANGO_DEBUG', True))
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = get_env_bool('DJANGO_DEBUG', True)
ALLOWED_HOSTS = ['localhost', '127.0.0.1', 'uwsgi','api.doorcome.cn','doorcome.cn','echart.doorcome.cn']
CSRF_TRUSTED_ORIGINS = [
'https://api.doorcome.cn',
'https://echart.doorcome.cn',
]
#以下跨域名访问配置
CORS_ORIGIN_ALLOW_ALL = False
CORS_ALLOWED_ORIGINS = [
"https://api.doorcome.cn",
"https://echart.doorcome.cn",
]
#允许的 HTTP 方法和头部:
CORS_ALLOW_METHODS = [
"DELETE",
"GET",
"OPTIONS",
"PATCH",
"POST",
"PUT",
]
CORS_ALLOW_HEADERS = [
"accept",
"accept-encoding",
"authorization",
"content-type",
"dnt",
"origin",
"user-agent",
"x-csrftoken",
"x-requested-with",
'X-CSRFToken',
]
CORS_ALLOW_CREDENTIALS = True #允许携带凭据:
# 新增配置
CORS_EXPOSE_HEADERS = ['Content-Type', 'X-CSRFToken']
SESSION_COOKIE_DOMAIN = ".doorcome.cn" # 改为顶级域名共享cookie
CSRF_COOKIE_DOMAIN = ".doorcome.cn"
#SESSION_COOKIE_SAMESITE = 'Lax' #Lax模式会阻止跨域AJAX请求发送cookies
SESSION_COOKIE_SAMESITE = 'None' # 必须为None才能跨域传cookie
CSRF_COOKIE_SAMESITE = 'None' # 同步修改CSRF的SameSite
#以上跨域名访问配置
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
CSRF_COOKIE_SECURE = True # 如果使用 HTTPS 则设为 True
CSRF_COOKIE_HTTPONLY = False
SESSION_COOKIE_SECURE = True # 如果使用 HTTPS 则设为 True
#SESSION_COOKIE_DOMAIN = "api.doorcome.cn" # 可选,根据实际需求
# 允许所有域名进行跨域访问
CORS_ALLOW_ALL_ORIGINS = True
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'corsheaders', #跨域名访问 2025-07-08
'api', # my 1st application created 2025/5/23
'rest_framework',
'drf_spectacular', # 新增:drf-spectacular
]
# 配置 DRF 默认 schema 生成器
REST_FRAMEWORK = {
'DEFAULT_SCHEMA_CLASS': 'drf_spectacular.openapi.AutoSchema',
}
# 配置 spectacular
SPECTACULAR_SETTINGS = {
'TITLE': 'Finance API',
'DESCRIPTION': 'A 股金融数据 API — 行情、财务、分红、指数、融资融券、新闻联播',
'VERSION': '1.0.0',
'SERVE_INCLUDE_SCHEMA': False,
'TAGS': [
{'name': '行情', 'description': '个股日线行情、技术参数'},
{'name': '基础数据', 'description': '股票基本信息、行业分类'},
{'name': '财务', 'description': 'EPS、财务报表分析'},
{'name': '分红', 'description': '股息率、TTM 分红'},
{'name': '指数', 'description': '指数行情与查询'},
{'name': '融资融券', 'description': '融资融券明细与汇总'},
{'name': '新闻联播', 'description': '新闻联播 ASR 转写与 AI 精编'},
{'name': '日报', 'description': 'AI 财经日报查询(news_report / news_event'},
{'name': '系统', 'description': '系统信息'},
],
}
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'corsheaders.middleware.CorsMiddleware', #跨域名访问 2025-07-08
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
ROOT_URLCONF = 'djapi.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [BASE_DIR / 'templates'],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'djapi.wsgi.application'
# Database
# https://docs.djangoproject.com/en/5.2/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
}
}
# Password validation
# https://docs.djangoproject.com/en/5.2/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/5.2/topics/i18n/
#LANGUAGE_CODE = 'en-us'
LANGUAGE_CODE = 'zh-hans' #简体中文,影响管理界面?
#TIME_ZONE = 'UTC'
TIME_ZONE = 'Asia/Shanghai'
USE_I18N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/5.2/howto/static-files/
STATIC_URL = '/static/'
# 此时访问django的admin管理后台时,静态资源会调取失败。这时可以将该项目所有静态资源统一收集到一个文件夹下,然后由nginx统一去调取,真正做到动静分离(动的给uWSGI,静的由nginx直接调取)
STATIC_ROOT = BASE_DIR / 'static'
# Default primary key field type
# https://docs.djangoproject.com/en/5.2/ref/settings/#default-auto-field
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
CORS_DEBUG = True # 显示详细 CORS 错误
LOGGING = {
'version': 1,
'handlers': {
'console': {'class': 'logging.StreamHandler'},
},
'loggers': {
'corsheaders': {
'handlers': ['console'],
'level': 'DEBUG',
},
},
}
+34
View File
@@ -0,0 +1,34 @@
"""
URL configuration for djapi project.
The `urlpatterns` list routes URLs to views. For more information please see:
https://docs.djangoproject.com/en/5.2/topics/http/urls/
Examples:
Function views
1. Add an import: from my_app import views
2. Add a URL to urlpatterns: path('', views.home, name='home')
Class-based views
1. Add an import: from other_app.views import Home
2. Add a URL to urlpatterns: path('', Home.as_view(), name='home')
Including another URLconf
1. Import the include() function: from django.urls import include, path
2. Add a URL to urlpatterns: path('blog/', include('blog.urls'))
"""
from django.contrib import admin
from django.urls import path,include
from api import views # 导入应用的视图函数
from drf_spectacular.views import (
SpectacularAPIView,
SpectacularSwaggerView,
SpectacularRedocView,
)
urlpatterns = [
path('admin/', admin.site.urls),
path('', views.home, name='root_home'), # 根 URL 映射到主
path('api/', include('api.urls')), # 包含应用api的 URL 配置,表示为api.doorcome.cn/api/
path('api/schema/', SpectacularAPIView.as_view(), name='schema'), # 生成 OpenAPI schema
path('api/docs/', SpectacularSwaggerView.as_view(url_name='schema'), name='swagger-ui'), # Swagger 界面
path('api/redoc/', SpectacularRedocView.as_view(url_name='schema'), name='redoc'), # ReDoc 界面
]
+16
View File
@@ -0,0 +1,16 @@
"""
WSGI config for djapi project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/5.2/howto/deployment/wsgi/
"""
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'djapi.settings')
application = get_wsgi_application()
+111
View File
@@ -0,0 +1,111 @@
# 日报结构化入库:数据库表结构与数据契约
> 版本:v1.0 | 2026-08-03
> 用途:供 API / 前端对接读取日报数据。表位于 MySQL `myquant` 库,表前缀 `news_`
> 连接:`192.168.1.10:13306`pi 上 autossh 隧道 → doorcome.cn:3306 MariaDB 10.11),用户 `myquant`(密码在服务器 `.env``NEWS_DB_PASSWORD`)。
---
## 1. 表结构
### 1.1 news_report(日报主表,一行 = 一份日报)
| 字段 | 类型 | 说明 |
| --- | --- | --- |
| id | BIGINT UNSIGNED PK | 自增主键 |
| report_date | DATE | 日报日期 |
| report_type | VARCHAR(16) | `finance`=A 股日报 / `intl`=国际财经日报 |
| file_name | VARCHAR(160) | 历史文件源文件名;**新生成日报为空字符串 `""`** |
| generated_at | DATETIME | 生成时间 |
| ai_summary | TEXT | AI 摘要全文(含换行,按条目分行) |
| stats | JSON | 数据总览统计快照(见第 3 节),可为 NULL |
| created_at | DATETIME | 入库时间 |
唯一键:`(report_date, report_type, file_name)` —— 历史同一天多次生成(intl 一日 3 次)保留多行;新生成日报 `file_name=''` 每天每类型仅一行,重复生成覆盖。
### 1.2 news_event(日报事件明细,一行 = 一条事件)
| 字段 | 类型 | 说明 |
| --- | --- | --- |
| id | BIGINT UNSIGNED PK | 自增主键 |
| report_id | BIGINT UNSIGNED | FK → news_report.id |
| section | VARCHAR(16) | 板块:`xwlb`=新闻联播 / `news`=财经新闻 / `cninfo`=公告调研 / `intl`=国际重要事件 |
| rank | INT | 板块内序号(1 起) |
| importance | INT NULL | 重要度 1-5 |
| event_type | VARCHAR(64) NULL | 事件类型(如 宏观经济/地缘政治/新闻联播/公告) |
| title | VARCHAR(512) | 标题 |
| summary | TEXT NULL | 摘要/正文 |
| sentiment | VARCHAR(8) NULL | `positive` / `negative` / `neutral` |
| source | VARCHAR(64) NULL | 来源(如 `cls``investinglive.com` |
| url | VARCHAR(512) NULL | 原文链接(新闻联播为空) |
| created_at | DATETIME | 入库时间 |
索引:`idx_report_section (report_id, section)`
---
## 2. 数据契约
- **幂等语义**:同一 `(report_date, report_type, file_name)` 重复写入会覆盖主表并全量替换事件(DELETE + INSERT),不会产生重复行。
- **取最新**:同一天存在多份时(历史 intl 一日 3 次),前端按 `generated_at` 取最新;新日报 `file_name=''` 每天唯一。
- **板块差异**finance 日报含 `xwlb`+`news`+`cninfo` 三板块;intl 日报仅 `intl` 板块。前端按 `section` 过滤展示。
- **历史覆盖范围**2026-06-16 ~ 2026-08-03,共 177 行(finance 49 + intl 128finance 少 1 因为两个目录存在同名文件被幂等合并)。事件总计 4222 条。
---
## 3. stats JSON 结构
`news_report.stats` 为数据总览快照,前端自行解析。finance 与 intl 的 key 集合不同:
| key | finance | intl | 内容 |
| --- | --- | --- | --- |
| `pipeline` | ✅ | ✅ | M1→M6 管道各环节数量:`{label: 数量}` |
| `sources` | ✅ | — | 各新闻源文章数:`{源名: 数量}` |
| `news` | ✅ | — | 新闻统计:`{total, hi_threshold, sentiments, importances, event_types}` |
| `cninfo` | ✅ | — | 公告调研统计:`{total, hi_threshold, by_day, announcement, research, irm}` |
| `xwlb` | ✅ | — | 联播统计:`{total, date}`(有数据时才有) |
| `sentiment` | ✅ | ✅ | 情绪分布(历史文件为图例文本列表;新生成在 `news.sentiments` |
| `importance` | ✅ | ✅ | 重要度分布:`[{重要度, 数量}, ...]` |
| `event_types` | ✅ | ✅ | 事件类型 TOP:`[{事件类型, 数量}, ...]` |
| `source_dist` | — | ✅ | 文章来源分布:`[{来源, 文章数}, ...]` |
> 历史文件与新生成日报的 stats 结构存在差异(历史为 HTML 解析快照,新生成为结构化组装),前端建议按 key 防御性读取。
---
## 4. 常用查询示例(API 实现参考)
```sql
-- 某类型日报列表(取每天最新一份)
SELECT r.* FROM news_report r
JOIN (
SELECT report_date, report_type, MAX(generated_at) AS g
FROM news_report GROUP BY report_date, report_type
) t ON r.report_date = t.report_date AND r.report_type = t.report_type
AND r.generated_at = t.g
WHERE r.report_type = 'finance' AND r.report_date >= '2026-07-01'
ORDER BY r.report_date DESC;
-- 某日报的全部事件(按板块)
SELECT section, rank, importance, event_type, title, summary, sentiment, source, url
FROM news_event WHERE report_id = ? ORDER BY section, rank;
-- 最近 N 天重要事件聚合(跨日报检索)
SELECT e.* FROM news_event e
JOIN news_report r ON r.id = e.report_id
WHERE r.report_date >= DATE_SUB(CURDATE(), INTERVAL 7 DAY)
AND e.importance >= 4
ORDER BY e.importance DESC, r.report_date DESC;
```
---
## 5. 相关命令(数据生产侧)
```bash
uv run a-share report --date YYYYMMDD # 生成当日日报并入库(finance)
uv run a-share report-import # 历史 HTML 全量解析入库(幂等)
uv run a-share report-import --date YYYYMMDD --type intl
```
代码:`report_db/`(连接/写入)、`report_import/`(历史解析/导入)、`scheduler/reporter.py`(日报生成)。
+330
View File
@@ -0,0 +1,330 @@
# Milestone 10 后端实现逻辑:日报结构化入库
> 版本:v0.1(设计稿) | 2026-07
> 对应 project_plan.md「十八、Milestone 10」
> **范围**:本项目侧"后端"= 数据生产层(日报内容生成 + 结构化写入 MySQL)。
> 不包含 API 服务与前端页面(由用户另行实现),但表结构与数据契约以本文档为准,供 API/前端对接。
---
## 1. 定位
现有链路:`reporter.py` 收集数据 → `_render_html()` 渲染 HTML → scp 上传 doorcome。
改造后:`reporter.py` 收集数据 → 组装结构化 `ReportData` → 写入 MySQL`news_report` / `news_event`),不再产出 HTML。
另需:把 doorcome 上 178 份历史日报 HTML`finance_news_daily_*` ×50、`intl_news_daily_*` ×128)解析成同一 `ReportData` 结构入库。
---
## 2. 数据流总览
```
[历史 HTML ×178] [每日 pipeline]
doorcome:/var/www/html/echart/research/ crawler→extractor→dedup→llm→embed→qdrant
(一次性 scp 到 data/reports_history/
│ ▼
▼ reporter.generate_report()
report_import/parser.py │
BeautifulSoup 解析) ▼
▼ 组装 ReportData 组装 ReportData
report_import/importer.py │
│ (幂等 upsert
▼ │
┌────────────────────── MySQL (myquant 库) ──────────────────────┐
│ news_report(主表) news_event(事件明细) │
└────────────────────────────────────────────────────────────────┘
API / 前端(用户另行实现,只读)
```
---
## 3. 数据模型(Pydantic`report_db/models.py`
```python
class EventRow(BaseModel):
"""一条事件记录,对应 news_event 一行。"""
section: str # xwlb | news | cninfo | intl
rank: int # 板块内序号(从 1 开始)
importance: int | None = None
event_type: str | None = None
title: str
summary: str | None = None
sentiment: str | None = None # positive | negative | neutral | ''
source: str | None = None # 来源(如 cls / ForexLive
url: str | None = None
class ReportData(BaseModel):
"""一份完整日报,对应 news_report 一行 + news_event 多行。"""
report_date: date # 日报日期(YYYY-MM-DD
report_type: str # finance | intl
file_name: str # 源文件名(新生成时可为 "")
generated_at: datetime # 生成时间
ai_summary: str | None = None
stats: dict[str, Any] = Field(default_factory=dict) # 数据总览统计快照 → JSON 列
events: list[EventRow] = Field(default_factory=list)
```
---
## 4. 字段映射(核心契约)
### 4.1 事件 JSONdata/events/)→ news_event
现有事件文件结构与 news_event 字段对应关系(reporter 收集时直接转换):
| news_event 字段 | 事件 JSON 来源 |
| --- | --- |
| section | 来源判定:`source_id=="cninfo"``cninfo``source_id=="xwlb"``xwlb`;否则 `news`intl 解析固定 `intl` |
| importance | `event.importance` |
| event_type | `event.event_type` |
| title | `title` |
| summary | `event.summary` |
| sentiment | `event.sentiment` |
| source | `source_id` |
| url | `url`xwlb 为空) |
### 4.2 历史 HTML → ReportData
解析策略:**表头驱动列映射**。不同日报表格列集合不同:
| 板块 | 表格列(<th> | section |
| --- | --- | --- |
| 新闻联播(finance | `# / (空) / 标题 / 重要度 / 事件类型` | xwlb |
| 重要事件:新闻(finance) | `# / (空) / 标题 / 源 / 重要度 / 事件类型 / 摘要` | news |
| 重要事件:公告调研(finance | 同上 | cninfo |
| 重要事件(intl | `# / (空) / 标题 / 重要度 / 事件类型 / 摘要` | intl |
要点:
- 以表头文本定位列索引("标题""重要度""事件类型""摘要""源"),空 `<th>` 为情绪图标列(⚪/🔴/🟢 → neutral/negative/positive),**不要依赖列位置**。
- 情绪图标仅存在于有图标列的表;intl 表情绪列存在,finance 表情绪列存在(空 th 首列后)。
- intl 无"源"列时,尝试从标题尾部 `[来源]` 或摘要尾部提取,提取不到则 `source=None`
- 标题中的股票代码标注 `(600519, ...)` 与 ⭐(自选股标记)需剥除,只保留纯标题。
- AI 摘要:取 `h2`"一、AI 摘要")之后紧随的 `.ai-summary` 区块纯文本(保留换行)。
- 数据总览 → `stats` JSON:按 `h3` 标题映射 key(见 4.3),解析该 h3 后的首个 `<table>`,缺失的板块跳过、不报错。
- 容错:任一板块解析失败 → 记 WARNING 日志,该板块置空,不影响整份入库;整份文件解析失败 → 抛 `ReportParseError`(由 importer 捕获计数)。
### 4.3 数据总览 → stats JSON
| h3 标题(含板块名) | stats key |
| --- | --- |
| M1→M6 管道 / 管道 | `pipeline`(保留原始行) |
| 各源数据 | `sources` |
| 情绪分布 | `sentiment` |
| 重要度分布 | `importance` |
| 事件类型(TOP 10 / 分布) | `event_types` |
| 文章来源分布 | `source_dist` |
`stats` 存 MySQL `JSON` 列,前端自行解析展示。历史文件与未来新日报的 stats 结构可能不同(finance 与 intl 板块不同),一律按快照存储,不做跨版本规范化。
---
## 5. 模块设计
### 5.1 新包 `report_db/`DB 层)
```
report_db/
├── __init__.py # 导出 connect / init_schema / save_report
├── models.py # EventRow / ReportDataPydantic
├── schema.py # DDL 常量(news_report / news_event,见 project_plan.md 十八)
└── db.py # 连接、事务、写入
```
`db.py` 关键函数:
```python
def load_db_config() -> DbConfig:
"""从环境变量读取 NEWS_DB_HOST/PORT/USER/PASSWORD/NAME。
缺失 PASSWORD 时记 ERROR 并 raise,禁止默认密码。"""
def connect(cfg: DbConfig) -> Connection:
"""pymysql.connect(autocommit=False, charset="utf8mb4", cursorclass=DictCursor)。
失败时 logger.exception + raise。"""
def init_schema(conn: Connection) -> None:
"""执行 schema.py 中的 CREATE TABLE IF NOT EXISTS ×2。"""
def save_report(conn: Connection, report: ReportData) -> int:
"""事务内:
1. INSERT INTO news_report (...) VALUES (...) 或按 (report_date, report_type, file_name)
唯一键命中时 UPDATE(新生成日报重复执行 = 覆盖同 file_name/同日期,幂等);
2. 取 report_idDELETE 旧事件后批量 INSERT news_event(保证整份覆盖一致)。
返回 report_id。"""
def transaction(conn: Connection) -> contextmanager:
"""提交/回滚上下文管理器。"""
def fetch_report(conn: Connection, report_id: int) -> dict | None:
"""读侧辅助(联调/测试用),API 侧由用户自行实现。"""
```
要点:
- 所有 SQL 为 MySQL/MariaDB 方言(`JSON` 列、`ENGINE=InnoDB``COMMENT`),**不依赖 ORM**。
- 连接生命周期:每次 `save_report` 短连接(report 一天跑几次,量小,无需连接池;如未来加大再换)。
- 字符集 utf8mb4`SET NAMES utf8mb4` 由 pymysql charset 参数处理。
### 5.2 新包 `report_import/`(历史解析)
```
report_import/
├── __init__.py
├── parser.py # parse_finance_report / parse_intl_reportBeautifulSoup
└── importer.py # import_history(dir, date=None, type=None) -> ImportStats
```
`parser.py`
```python
class ReportParseError(Exception): ...
def parse_finance_report(html: str, file_name: str) -> ReportData: ...
def parse_intl_report(html: str, file_name: str) -> ReportData: ...
def parse_report(html: str, file_name: str) -> ReportData:
"""按文件名前缀分流:finance_news_daily_* / intl_news_daily_*。"""
```
- 依赖复用现有 `beautifulsoup4`(已在 pyproject 依赖),**不新增解析库**。
- `report_date` 从文件名解析(`*_daily_{YYYYMMDD}_*.html`),不信任目录名。
- `generated_at` 从文件名时间(`{HHMMSS}`)或 `<header>` 中"生成于"文本解析,解析不到用文件 mtime。
`importer.py`
```python
@dataclass
class ImportStats:
scanned: int = 0 # 扫描到的日报文件数
imported: int = 0 # 新入库
skipped: int = 0 # 已存在(幂等跳过)
failed: int = 0 # 解析失败
errors: list[str] = field(default_factory=list)
def import_history(report_dir: Path, date: str | None = None,
report_type: str | None = None) -> ImportStats:
"""遍历 {report_dir}/{YYYYMMDD}/*_news_daily_*.html
过滤 date / type,逐个 parse → save_report。"""
```
### 5.3 `scheduler/reporter.py` 改造(完全切换)
- 新增 `_build_report_data(news, cninfo, pipeline, ai_summary, day_str, xwlb) -> ReportData`
- 事件转换:`news["high"]``EventRow(section="news", ...)``cninfo["high"]``section="cninfo"``xwlb["items"]``section="xwlb"`
- `stats` 组装:`{"pipeline": pipeline, "sources": {...}, "sentiment": news["sentiments"], "importance": news["importances"], "event_types": news["event_types"], "cninfo": {...}}`
- 事件 `rank` 按板块内顺序编号。
- `generate_report(day_str, *, upload=True)` 改为:收集(逻辑不变)→ `_build_report_data``connect()` + `save_report()`;删除 `_render_html`/`_upload` 调用。
- `_render_*` 函数**保留但标记 deprecated**(注释说明"完全切换后不再调用"),不删除,保证最小改动、可回退。
- 返回值由 `Path | None` 改为 `report_id: int | None``scheduler/pipeline.py` 中 report 步骤仅判断非 None(实施时核实该处调用,保持兼容)。
- `stock_reporter.py` **不改动**(个股日报不在本期范围)。
### 5.4 `a_share_cli/main.py` 新增子命令
```
uv run a-share report-import [--dir data/reports_history] [--date YYYYMMDD] [--type finance|intl]
```
- 默认全量扫描 `REPORT_HISTORY_DIR`.env 可配,默认 `data/reports_history/`)。
- 输出 ImportStats 汇总(扫描/导入/跳过/失败)。
---
## 6. 关键流程
### 6.1 历史导入(一次执行,可重复)
```
1. scp -r doorcome:/var/www/html/echart/research/2026* → data/reports_history/
(一次手工操作,不进代码)
2. uv run a-share report-import
for each {date}/{file}:
report_type = 文件名前缀(finance|intl
ReportData = parse_report(html, file_name)
try: save_report(conn, ReportData) → imported += 1
except DuplicateKey: skipped += 1 # 已导入过
except ReportParseError as e: failed += 1; errors.append(str(e))
3. 校验: SELECT report_type, COUNT(*) FROM news_report GROUP BY report_type
期望 50 / 128
```
### 6.2 每日日报生成(pipeline 07:00 步骤)
```
generate_report(day_str):
news = _collect_news_events(day_str) # 不变
cninfo = _collect_cninfo_events(day_str) # 不变
xwlb = _collect_xwlb(day_str) # 不变
pipeline = _collect_pipeline_stats(day_str) # 不变
ai_summary = _generate_ai_summary(...) # 不变
report = _build_report_data(...) # 新增
save_report(connect(), report) # 新增(替代渲染+上传)
```
### 6.3 幂等策略
- 唯一键 `(report_date, report_type, file_name)`
- 历史导入:命中 → 跳过(或 `--force` 覆盖);
- 新日报:`file_name=""` 时唯一键退化为 `(report_date, report_type, "")`,同一天重复跑 → UPDATE 覆盖,事件表 DELETE+INSERT 全量替换,**不产生历史残留**。
---
## 7. 配置项(.env / .env.example
```env
# ---- 日报结构化入库 (M10) ----
NEWS_DB_HOST=127.0.0.1 # 开发走 ssh 隧道: ssh -L 13306:127.0.0.1:13306 pi
NEWS_DB_PORT=13306
NEWS_DB_USER=myquant
NEWS_DB_PASSWORD= # 填真实值,禁止写入源码/文档
NEWS_DB_NAME=myquant
REPORT_HISTORY_DIR=data/reports_history
```
---
## 8. 错误处理
| 场景 | 行为 |
| --- | --- |
| DB 不可达/凭据错误 | `connect()` 抛异常 → `generate_report` 记 ERROR 并返回 Nonepipeline 该步骤失败,其余步骤不受影响) |
| 单份历史文件解析失败 | 记 WARNING,`failed += 1`,继续下一份;结束输出失败清单 |
| 事件字段缺失(如无摘要列) | 对应字段留 None,不抛错 |
| 全部失败 | `report-import` 返回非 0 退出码,便于排查 |
---
## 9. 测试策略(tests/
| 文件 | 内容 |
| --- | --- |
| `tests/test_report_parser.py` | 用 fixtures(从 178 份中拷贝 finance/intl 各 1 份真实样例到 `tests/fixtures/`)断言:板块数、事件行数、字段映射、标题净化、幂等文件日期解析 |
| `tests/test_report_db.py` | 纯逻辑:`_build_report_data` 组装正确;SQL 层用 sqlite3 内存库建同构(简化 DDL)验证 upsert/覆盖语义 |
| `tests/test_report_import.py` | 临时目录构造 2-3 份假 HTML → 全流程导入 → 断言 ImportStats 计数与幂等 |
| 集成(`@pytest.mark.integration`,默认跳过) | 连真实 MySQLinit_schema + save_report + 查询回读 |
新增 pytest marker 说明:真实 DB 连接一律走 integration,**单元测试不得依赖生产库**。
---
## 10. 依赖变更
- `uv add pymysql`(纯 Python 驱动,唯一新增依赖)
- 解析复用现有 `beautifulsoup4`,不新增
---
## 11. 开放问题(沿自 project_plan.md 十八,不阻塞开发)
1. 生产连接:pi5 无法直连 `192.168.1.10:13306`(隧道仅绑 loopback)——需决定改 pi 的 autossh 绑定 / pi5 自建隧道。
2. intl 日报生成方不在本项目,未来 intl 新日报需按同一表结构写入(本项目仅负责解析历史 + finance 新日报)。
3. 个股日报(research 根目录文件)本期不处理。
---
## 12. 实施顺序(供开发排期)
1. `report_db/`models/schema/db+ `.env` 配置 + 建表验证
2. `report_import/parser.py` + fixtures + 单测
3. `report_import/importer.py` + CLI `report-import` + 178 份全量导入验收
4. `reporter.py` 改造(_build_report_data + save_report+ pipeline 兼容性验证
5. docs/db_schema.md 定稿(给 API/前端)、README / continuation.md 更新
+22
View File
@@ -0,0 +1,22 @@
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys
def main():
"""Run administrative tasks."""
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'djapi.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)
if __name__ == '__main__':
main()
+4
View File
@@ -0,0 +1,4 @@
# 项目级 Reasonix 配置覆盖(仅此工作区生效)
# 本机未安装 bubblewrap,为执行运维命令关闭 bash 沙箱(全局配置仍为 enforce)
[sandbox]
bash = "off"
+18
View File
@@ -0,0 +1,18 @@
aiofiles==25.1.0
aiohttp==3.12.15
akshare==1.17.91
beautifulsoup4==4.14.2
dashscope==1.24.6
Django==5.2.7
django-cors-headers==4.9.0
djangorestframework==3.16.1
drf_spectacular==0.28.0
m3u8==6.0.0
mysql-connector-python==9.3.0
numpy==2.3.4
pandas==2.3.3
playwright==1.55.0
pydub==0.25.1
Requests==2.32.5
tushare==1.4.24
yt_dlp==2025.10.22
+279
View File
@@ -0,0 +1,279 @@
select.admin-autocomplete {
width: 20em;
}
.select2-container--admin-autocomplete.select2-container {
min-height: 30px;
}
.select2-container--admin-autocomplete .select2-selection--single,
.select2-container--admin-autocomplete .select2-selection--multiple {
min-height: 30px;
padding: 0;
}
.select2-container--admin-autocomplete.select2-container--focus .select2-selection,
.select2-container--admin-autocomplete.select2-container--open .select2-selection {
border-color: var(--body-quiet-color);
min-height: 30px;
}
.select2-container--admin-autocomplete.select2-container--focus .select2-selection.select2-selection--single,
.select2-container--admin-autocomplete.select2-container--open .select2-selection.select2-selection--single {
padding: 0;
}
.select2-container--admin-autocomplete.select2-container--focus .select2-selection.select2-selection--multiple,
.select2-container--admin-autocomplete.select2-container--open .select2-selection.select2-selection--multiple {
padding: 0;
}
.select2-container--admin-autocomplete .select2-selection--single {
background-color: var(--body-bg);
border: 1px solid var(--border-color);
border-radius: 4px;
}
.select2-container--admin-autocomplete .select2-selection--single .select2-selection__rendered {
color: var(--body-fg);
line-height: 30px;
}
.select2-container--admin-autocomplete .select2-selection--single .select2-selection__clear {
cursor: pointer;
float: right;
font-weight: bold;
}
.select2-container--admin-autocomplete .select2-selection--single .select2-selection__placeholder {
color: var(--body-quiet-color);
}
.select2-container--admin-autocomplete .select2-selection--single .select2-selection__arrow {
height: 26px;
position: absolute;
top: 1px;
right: 1px;
width: 20px;
}
.select2-container--admin-autocomplete .select2-selection--single .select2-selection__arrow b {
border-color: #888 transparent transparent transparent;
border-style: solid;
border-width: 5px 4px 0 4px;
height: 0;
left: 50%;
margin-left: -4px;
margin-top: -2px;
position: absolute;
top: 50%;
width: 0;
}
.select2-container--admin-autocomplete[dir="rtl"] .select2-selection--single .select2-selection__clear {
float: left;
}
.select2-container--admin-autocomplete[dir="rtl"] .select2-selection--single .select2-selection__arrow {
left: 1px;
right: auto;
}
.select2-container--admin-autocomplete.select2-container--disabled .select2-selection--single {
background-color: var(--darkened-bg);
cursor: default;
}
.select2-container--admin-autocomplete.select2-container--disabled .select2-selection--single .select2-selection__clear {
display: none;
}
.select2-container--admin-autocomplete.select2-container--open .select2-selection--single .select2-selection__arrow b {
border-color: transparent transparent #888 transparent;
border-width: 0 4px 5px 4px;
}
.select2-container--admin-autocomplete .select2-selection--multiple {
background-color: var(--body-bg);
border: 1px solid var(--border-color);
border-radius: 4px;
cursor: text;
}
.select2-container--admin-autocomplete .select2-selection--multiple .select2-selection__rendered {
box-sizing: border-box;
list-style: none;
margin: 0;
padding: 0 10px 5px 5px;
width: 100%;
display: flex;
flex-wrap: wrap;
}
.select2-container--admin-autocomplete .select2-selection--multiple .select2-selection__rendered li {
list-style: none;
}
.select2-container--admin-autocomplete .select2-selection--multiple .select2-selection__placeholder {
color: var(--body-quiet-color);
margin-top: 5px;
float: left;
}
.select2-container--admin-autocomplete .select2-selection--multiple .select2-selection__clear {
cursor: pointer;
float: right;
font-weight: bold;
margin: 5px;
position: absolute;
right: 0;
}
.select2-container--admin-autocomplete .select2-selection--multiple .select2-selection__choice {
background-color: var(--darkened-bg);
border: 1px solid var(--border-color);
border-radius: 4px;
cursor: default;
float: left;
margin-right: 5px;
margin-top: 5px;
padding: 0 5px;
}
.select2-container--admin-autocomplete .select2-selection--multiple .select2-selection__choice__remove {
color: var(--body-quiet-color);
cursor: pointer;
display: inline-block;
font-weight: bold;
margin-right: 2px;
}
.select2-container--admin-autocomplete .select2-selection--multiple .select2-selection__choice__remove:hover {
color: var(--body-fg);
}
.select2-container--admin-autocomplete[dir="rtl"] .select2-selection--multiple .select2-selection__choice, .select2-container--admin-autocomplete[dir="rtl"] .select2-selection--multiple .select2-selection__placeholder, .select2-container--admin-autocomplete[dir="rtl"] .select2-selection--multiple .select2-search--inline {
float: right;
}
.select2-container--admin-autocomplete[dir="rtl"] .select2-selection--multiple .select2-selection__choice {
margin-left: 5px;
margin-right: auto;
}
.select2-container--admin-autocomplete[dir="rtl"] .select2-selection--multiple .select2-selection__choice__remove {
margin-left: 2px;
margin-right: auto;
}
.select2-container--admin-autocomplete.select2-container--focus .select2-selection--multiple {
border: solid var(--body-quiet-color) 1px;
outline: 0;
}
.select2-container--admin-autocomplete.select2-container--disabled .select2-selection--multiple {
background-color: var(--darkened-bg);
cursor: default;
}
.select2-container--admin-autocomplete.select2-container--disabled .select2-selection__choice__remove {
display: none;
}
.select2-container--admin-autocomplete.select2-container--open.select2-container--above .select2-selection--single, .select2-container--admin-autocomplete.select2-container--open.select2-container--above .select2-selection--multiple {
border-top-left-radius: 0;
border-top-right-radius: 0;
}
.select2-container--admin-autocomplete.select2-container--open.select2-container--below .select2-selection--single, .select2-container--admin-autocomplete.select2-container--open.select2-container--below .select2-selection--multiple {
border-bottom-left-radius: 0;
border-bottom-right-radius: 0;
}
.select2-container--admin-autocomplete .select2-search--dropdown {
background: var(--darkened-bg);
}
.select2-container--admin-autocomplete .select2-search--dropdown .select2-search__field {
background: var(--body-bg);
color: var(--body-fg);
border: 1px solid var(--border-color);
border-radius: 4px;
}
.select2-container--admin-autocomplete .select2-search--inline .select2-search__field {
background: transparent;
color: var(--body-fg);
border: none;
outline: 0;
box-shadow: none;
-webkit-appearance: textfield;
}
.select2-container--admin-autocomplete .select2-results > .select2-results__options {
max-height: 200px;
overflow-y: auto;
color: var(--body-fg);
background: var(--body-bg);
}
.select2-container--admin-autocomplete .select2-results__option[role=group] {
padding: 0;
}
.select2-container--admin-autocomplete .select2-results__option[aria-disabled=true] {
color: var(--body-quiet-color);
}
.select2-container--admin-autocomplete .select2-results__option[aria-selected=true] {
background-color: var(--selected-bg);
color: var(--body-fg);
}
.select2-container--admin-autocomplete .select2-results__option .select2-results__option {
padding-left: 1em;
}
.select2-container--admin-autocomplete .select2-results__option .select2-results__option .select2-results__group {
padding-left: 0;
}
.select2-container--admin-autocomplete .select2-results__option .select2-results__option .select2-results__option {
margin-left: -1em;
padding-left: 2em;
}
.select2-container--admin-autocomplete .select2-results__option .select2-results__option .select2-results__option .select2-results__option {
margin-left: -2em;
padding-left: 3em;
}
.select2-container--admin-autocomplete .select2-results__option .select2-results__option .select2-results__option .select2-results__option .select2-results__option {
margin-left: -3em;
padding-left: 4em;
}
.select2-container--admin-autocomplete .select2-results__option .select2-results__option .select2-results__option .select2-results__option .select2-results__option .select2-results__option {
margin-left: -4em;
padding-left: 5em;
}
.select2-container--admin-autocomplete .select2-results__option .select2-results__option .select2-results__option .select2-results__option .select2-results__option .select2-results__option .select2-results__option {
margin-left: -5em;
padding-left: 6em;
}
.select2-container--admin-autocomplete .select2-results__option--highlighted[aria-selected] {
background-color: var(--primary);
color: var(--primary-fg);
}
.select2-container--admin-autocomplete .select2-results__group {
cursor: default;
display: block;
padding: 6px;
}
.errors .select2-selection {
border: 1px solid var(--error-fg);
}
File diff suppressed because it is too large Load Diff
+343
View File
@@ -0,0 +1,343 @@
/* CHANGELISTS */
#changelist {
display: flex;
align-items: flex-start;
justify-content: space-between;
}
#changelist .changelist-form-container {
flex: 1 1 auto;
min-width: 0;
}
#changelist table {
width: 100%;
}
.change-list .hiddenfields { display:none; }
.change-list .filtered table {
border-right: none;
}
.change-list .filtered {
min-height: 400px;
}
.change-list .filtered .results, .change-list .filtered .paginator,
.filtered #toolbar, .filtered div.xfull {
width: auto;
}
.change-list .filtered table tbody th {
padding-right: 1em;
}
#changelist-form .results {
overflow-x: auto;
width: 100%;
}
#changelist .toplinks {
border-bottom: 1px solid var(--hairline-color);
}
#changelist .paginator {
color: var(--body-quiet-color);
border-bottom: 1px solid var(--hairline-color);
background: var(--body-bg);
overflow: hidden;
}
/* CHANGELIST TABLES */
#changelist table thead th {
padding: 0;
white-space: nowrap;
vertical-align: middle;
}
#changelist table thead th.action-checkbox-column {
width: 1.5em;
text-align: center;
}
#changelist table tbody td.action-checkbox {
text-align: center;
}
#changelist table tfoot {
color: var(--body-quiet-color);
}
/* TOOLBAR */
#toolbar {
padding: 8px 10px;
margin-bottom: 15px;
border-top: 1px solid var(--hairline-color);
border-bottom: 1px solid var(--hairline-color);
background: var(--darkened-bg);
color: var(--body-quiet-color);
}
#toolbar form input {
border-radius: 4px;
font-size: 0.875rem;
padding: 5px;
color: var(--body-fg);
}
#toolbar #searchbar {
height: 1.1875rem;
border: 1px solid var(--border-color);
padding: 2px 5px;
margin: 0;
vertical-align: top;
font-size: 0.8125rem;
max-width: 100%;
}
#toolbar #searchbar:focus {
border-color: var(--body-quiet-color);
}
#toolbar form input[type="submit"] {
border: 1px solid var(--border-color);
font-size: 0.8125rem;
padding: 4px 8px;
margin: 0;
vertical-align: middle;
background: var(--body-bg);
box-shadow: 0 -15px 20px -10px rgba(0, 0, 0, 0.15) inset;
cursor: pointer;
color: var(--body-fg);
}
#toolbar form input[type="submit"]:focus,
#toolbar form input[type="submit"]:hover {
border-color: var(--body-quiet-color);
}
#changelist-search img {
vertical-align: middle;
margin-right: 4px;
}
#changelist-search .help {
word-break: break-word;
}
/* FILTER COLUMN */
#changelist-filter {
flex: 0 0 240px;
order: 1;
background: var(--darkened-bg);
border-left: none;
margin: 0 0 0 30px;
}
@media (forced-colors: active) {
#changelist-filter {
border: 1px solid;
}
}
#changelist-filter h2 {
font-size: 0.875rem;
text-transform: uppercase;
letter-spacing: 0.5px;
padding: 5px 15px;
margin-bottom: 12px;
border-bottom: none;
}
#changelist-filter h3,
#changelist-filter details summary {
font-weight: 400;
padding: 0 15px;
margin-bottom: 10px;
}
#changelist-filter details summary > * {
display: inline;
}
#changelist-filter details > summary {
list-style-type: none;
}
#changelist-filter details > summary::-webkit-details-marker {
display: none;
}
#changelist-filter details > summary::before {
content: '→';
font-weight: bold;
color: var(--link-hover-color);
}
#changelist-filter details[open] > summary::before {
content: '↓';
}
#changelist-filter ul {
margin: 5px 0;
padding: 0 15px 15px;
border-bottom: 1px solid var(--hairline-color);
}
#changelist-filter ul:last-child {
border-bottom: none;
}
#changelist-filter li {
list-style-type: none;
margin-left: 0;
padding-left: 0;
}
#changelist-filter a {
display: block;
color: var(--body-quiet-color);
word-break: break-word;
}
#changelist-filter li.selected {
border-left: 5px solid var(--hairline-color);
padding-left: 10px;
margin-left: -15px;
}
#changelist-filter li.selected a {
color: var(--link-selected-fg);
}
#changelist-filter a:focus, #changelist-filter a:hover,
#changelist-filter li.selected a:focus,
#changelist-filter li.selected a:hover {
color: var(--link-hover-color);
}
#changelist-filter #changelist-filter-extra-actions {
font-size: 0.8125rem;
margin-bottom: 10px;
border-bottom: 1px solid var(--hairline-color);
}
/* DATE DRILLDOWN */
.change-list .toplinks {
display: flex;
padding-bottom: 5px;
flex-wrap: wrap;
gap: 3px 17px;
font-weight: bold;
}
.change-list .toplinks a {
font-size: 0.8125rem;
}
.change-list .toplinks .date-back {
color: var(--body-quiet-color);
}
.change-list .toplinks .date-back:focus,
.change-list .toplinks .date-back:hover {
color: var(--link-hover-color);
}
/* ACTIONS */
.filtered .actions {
border-right: none;
}
#changelist table input {
margin: 0;
vertical-align: baseline;
}
/* Once the :has() pseudo-class is supported by all browsers, the tr.selected
selector and the JS adding the class can be removed. */
#changelist tbody tr.selected {
background-color: var(--selected-row);
}
#changelist tbody tr:has(.action-select:checked) {
background-color: var(--selected-row);
}
@media (forced-colors: active) {
#changelist tbody tr.selected {
background-color: SelectedItem;
}
#changelist tbody tr:has(.action-select:checked) {
background-color: SelectedItem;
}
}
#changelist .actions {
padding: 10px;
background: var(--body-bg);
border-top: none;
border-bottom: none;
line-height: 1.5rem;
color: var(--body-quiet-color);
width: 100%;
}
#changelist .actions span.all,
#changelist .actions span.action-counter,
#changelist .actions span.clear,
#changelist .actions span.question {
font-size: 0.8125rem;
margin: 0 0.5em;
}
#changelist .actions:last-child {
border-bottom: none;
}
#changelist .actions select {
vertical-align: top;
height: 1.5rem;
color: var(--body-fg);
border: 1px solid var(--border-color);
border-radius: 4px;
font-size: 0.875rem;
padding: 0 0 0 4px;
margin: 0;
margin-left: 10px;
}
#changelist .actions select:focus {
border-color: var(--body-quiet-color);
}
#changelist .actions label {
display: inline-block;
vertical-align: middle;
font-size: 0.8125rem;
}
#changelist .actions .button {
font-size: 0.8125rem;
border: 1px solid var(--border-color);
border-radius: 4px;
background: var(--body-bg);
box-shadow: 0 -15px 20px -10px rgba(0, 0, 0, 0.15) inset;
cursor: pointer;
height: 1.5rem;
line-height: 1;
padding: 4px 8px;
margin: 0;
color: var(--body-fg);
}
#changelist .actions .button:focus, #changelist .actions .button:hover {
border-color: var(--body-quiet-color);
}
+130
View File
@@ -0,0 +1,130 @@
@media (prefers-color-scheme: dark) {
:root {
--primary: #264b5d;
--primary-fg: #f7f7f7;
--body-fg: #eeeeee;
--body-bg: #121212;
--body-quiet-color: #d0d0d0;
--body-medium-color: #e0e0e0;
--body-loud-color: #ffffff;
--breadcrumbs-link-fg: #e0e0e0;
--breadcrumbs-bg: var(--primary);
--link-fg: #81d4fa;
--link-hover-color: #4ac1f7;
--link-selected-fg: #6f94c6;
--hairline-color: #272727;
--border-color: #353535;
--error-fg: #e35f5f;
--message-success-bg: #006b1b;
--message-warning-bg: #583305;
--message-error-bg: #570808;
--darkened-bg: #212121;
--selected-bg: #1b1b1b;
--selected-row: #00363a;
--close-button-bg: #333333;
--close-button-hover-bg: #666666;
color-scheme: dark;
}
}
html[data-theme="dark"] {
--primary: #264b5d;
--primary-fg: #f7f7f7;
--body-fg: #eeeeee;
--body-bg: #121212;
--body-quiet-color: #d0d0d0;
--body-medium-color: #e0e0e0;
--body-loud-color: #ffffff;
--breadcrumbs-link-fg: #e0e0e0;
--breadcrumbs-bg: var(--primary);
--link-fg: #81d4fa;
--link-hover-color: #4ac1f7;
--link-selected-fg: #6f94c6;
--hairline-color: #272727;
--border-color: #353535;
--error-fg: #e35f5f;
--message-success-bg: #006b1b;
--message-warning-bg: #583305;
--message-error-bg: #570808;
--darkened-bg: #212121;
--selected-bg: #1b1b1b;
--selected-row: #00363a;
--close-button-bg: #333333;
--close-button-hover-bg: #666666;
color-scheme: dark;
}
/* THEME SWITCH */
.theme-toggle {
cursor: pointer;
border: none;
padding: 0;
background: transparent;
vertical-align: middle;
margin-inline-start: 5px;
margin-top: -1px;
}
.theme-toggle svg {
vertical-align: middle;
height: 1.5rem;
width: 1.5rem;
display: none;
}
/*
Fully hide screen reader text so we only show the one matching the current
theme.
*/
.theme-toggle .visually-hidden {
display: none;
}
html[data-theme="auto"] .theme-toggle .theme-label-when-auto {
display: block;
}
html[data-theme="dark"] .theme-toggle .theme-label-when-dark {
display: block;
}
html[data-theme="light"] .theme-toggle .theme-label-when-light {
display: block;
}
/* ICONS */
.theme-toggle svg.theme-icon-when-auto,
.theme-toggle svg.theme-icon-when-dark,
.theme-toggle svg.theme-icon-when-light {
fill: var(--header-link-color);
color: var(--header-bg);
}
html[data-theme="auto"] .theme-toggle svg.theme-icon-when-auto {
display: block;
}
html[data-theme="dark"] .theme-toggle svg.theme-icon-when-dark {
display: block;
}
html[data-theme="light"] .theme-toggle svg.theme-icon-when-light {
display: block;
}
+29
View File
@@ -0,0 +1,29 @@
/* DASHBOARD */
.dashboard td, .dashboard th {
word-break: break-word;
}
.dashboard .module table th {
width: 100%;
}
.dashboard .module table td {
white-space: nowrap;
}
.dashboard .module table td a {
display: block;
padding-right: .6em;
}
/* RECENT ACTIONS MODULE */
.module ul.actionlist {
margin-left: 0;
}
ul.actionlist li {
list-style-type: none;
overflow: hidden;
text-overflow: ellipsis;
}
+498
View File
@@ -0,0 +1,498 @@
@import url('widgets.css');
/* FORM ROWS */
.form-row {
overflow: hidden;
padding: 10px;
font-size: 0.8125rem;
border-bottom: 1px solid var(--hairline-color);
}
.form-row img, .form-row input {
vertical-align: middle;
}
.form-row label input[type="checkbox"] {
margin-top: 0;
vertical-align: 0;
}
form .form-row p {
padding-left: 0;
}
.flex-container {
display: flex;
}
.form-multiline {
flex-wrap: wrap;
}
.form-multiline > div {
padding-bottom: 10px;
}
/* FORM LABELS */
label {
font-weight: normal;
color: var(--body-quiet-color);
font-size: 0.8125rem;
}
.required label, label.required {
font-weight: bold;
}
/* RADIO BUTTONS */
form div.radiolist div {
padding-right: 7px;
}
form div.radiolist.inline div {
display: inline-block;
}
form div.radiolist label {
width: auto;
}
form div.radiolist input[type="radio"] {
margin: -2px 4px 0 0;
padding: 0;
}
form ul.inline {
margin-left: 0;
padding: 0;
}
form ul.inline li {
float: left;
padding-right: 7px;
}
/* FIELDSETS */
fieldset .fieldset-heading,
fieldset .inline-heading,
:not(.inline-related) .collapse summary {
border: 1px solid var(--header-bg);
margin: 0;
padding: 8px;
font-weight: 400;
font-size: 0.8125rem;
background: var(--header-bg);
color: var(--header-link-color);
}
/* ALIGNED FIELDSETS */
.aligned label {
display: block;
padding: 4px 10px 0 0;
min-width: 160px;
width: 160px;
word-wrap: break-word;
}
.aligned label:not(.vCheckboxLabel):after {
content: '';
display: inline-block;
vertical-align: middle;
}
.aligned label + p, .aligned .checkbox-row + div.help, .aligned label + div.readonly {
padding: 6px 0;
margin-top: 0;
margin-bottom: 0;
margin-left: 0;
overflow-wrap: break-word;
}
.aligned ul label {
display: inline;
float: none;
width: auto;
}
.aligned .form-row input {
margin-bottom: 0;
}
.colMS .aligned .vLargeTextField, .colMS .aligned .vXMLLargeTextField {
width: 350px;
}
form .aligned ul {
margin-left: 160px;
padding-left: 10px;
}
form .aligned div.radiolist {
display: inline-block;
margin: 0;
padding: 0;
}
form .aligned p.help,
form .aligned div.help {
margin-top: 0;
margin-left: 160px;
padding-left: 10px;
}
form .aligned p.date div.help.timezonewarning,
form .aligned p.datetime div.help.timezonewarning,
form .aligned p.time div.help.timezonewarning {
margin-left: 0;
padding-left: 0;
font-weight: normal;
}
form .aligned p.help:last-child,
form .aligned div.help:last-child {
margin-bottom: 0;
padding-bottom: 0;
}
form .aligned input + p.help,
form .aligned textarea + p.help,
form .aligned select + p.help,
form .aligned input + div.help,
form .aligned textarea + div.help,
form .aligned select + div.help {
margin-left: 160px;
padding-left: 10px;
}
form .aligned select option:checked {
background-color: var(--selected-row);
}
form .aligned ul li {
list-style: none;
}
form .aligned table p {
margin-left: 0;
padding-left: 0;
}
.aligned .vCheckboxLabel {
padding: 1px 0 0 5px;
}
.aligned .vCheckboxLabel + p.help,
.aligned .vCheckboxLabel + div.help {
margin-top: -4px;
}
.colM .aligned .vLargeTextField, .colM .aligned .vXMLLargeTextField {
width: 610px;
}
fieldset .fieldBox {
margin-right: 20px;
}
/* WIDE FIELDSETS */
.wide label {
width: 200px;
}
form .wide p.help,
form .wide ul.errorlist,
form .wide div.help {
padding-left: 50px;
}
form div.help ul {
padding-left: 0;
margin-left: 0;
}
.colM fieldset.wide .vLargeTextField, .colM fieldset.wide .vXMLLargeTextField {
width: 450px;
}
/* COLLAPSIBLE FIELDSETS */
.collapse summary .fieldset-heading,
.collapse summary .inline-heading {
background: transparent;
border: none;
color: currentColor;
display: inline;
margin: 0;
padding: 0;
}
/* MONOSPACE TEXTAREAS */
fieldset.monospace textarea {
font-family: var(--font-family-monospace);
}
/* SUBMIT ROW */
.submit-row {
padding: 12px 14px 12px;
margin: 0 0 20px;
background: var(--darkened-bg);
border: 1px solid var(--hairline-color);
border-radius: 4px;
overflow: hidden;
display: flex;
gap: 10px;
flex-wrap: wrap;
}
body.popup .submit-row {
overflow: auto;
}
.submit-row input {
height: 2.1875rem;
line-height: 0.9375rem;
}
.submit-row input, .submit-row a {
margin: 0;
}
.submit-row input.default {
text-transform: uppercase;
}
.submit-row a.deletelink {
margin-left: auto;
}
.submit-row a.deletelink {
display: block;
background: var(--delete-button-bg);
border-radius: 4px;
padding: 0.625rem 0.9375rem;
height: 0.9375rem;
line-height: 0.9375rem;
color: var(--button-fg);
}
.submit-row a.closelink {
display: inline-block;
background: var(--close-button-bg);
border-radius: 4px;
padding: 10px 15px;
height: 0.9375rem;
line-height: 0.9375rem;
color: var(--button-fg);
}
.submit-row a.deletelink:focus,
.submit-row a.deletelink:hover,
.submit-row a.deletelink:active {
background: var(--delete-button-hover-bg);
text-decoration: none;
}
.submit-row a.closelink:focus,
.submit-row a.closelink:hover,
.submit-row a.closelink:active {
background: var(--close-button-hover-bg);
text-decoration: none;
}
/* CUSTOM FORM FIELDS */
.vSelectMultipleField {
vertical-align: top;
}
.vCheckboxField {
border: none;
}
.vDateField, .vTimeField {
margin-right: 2px;
margin-bottom: 4px;
}
.vDateField {
min-width: 6.85em;
}
.vTimeField {
min-width: 4.7em;
}
.vURLField {
width: 30em;
}
.vLargeTextField, .vXMLLargeTextField {
width: 48em;
}
.flatpages-flatpage #id_content {
height: 40.2em;
}
.module table .vPositiveSmallIntegerField {
width: 2.2em;
}
.vIntegerField {
width: 5em;
}
.vBigIntegerField {
width: 10em;
}
.vForeignKeyRawIdAdminField {
width: 5em;
}
.vTextField, .vUUIDField {
width: 20em;
}
/* INLINES */
.inline-group {
padding: 0;
margin: 0 0 30px;
}
.inline-group thead th {
padding: 8px 10px;
}
.inline-group .aligned label {
width: 160px;
}
.inline-related {
position: relative;
}
.inline-related h4,
.inline-related:not(.tabular) .collapse summary {
margin: 0;
color: var(--body-medium-color);
padding: 5px;
font-size: 0.8125rem;
background: var(--darkened-bg);
border: 1px solid var(--hairline-color);
border-left-color: var(--darkened-bg);
border-right-color: var(--darkened-bg);
}
.inline-related h3 span.delete {
float: right;
}
.inline-related h3 span.delete label {
margin-left: 2px;
font-size: 0.6875rem;
}
.inline-related fieldset {
margin: 0;
background: var(--body-bg);
border: none;
width: 100%;
}
.inline-group .tabular fieldset.module {
border: none;
}
.inline-related.tabular fieldset.module table {
width: 100%;
overflow-x: scroll;
}
.last-related fieldset {
border: none;
}
.inline-group .tabular tr.has_original td {
padding-top: 2em;
}
.inline-group .tabular tr td.original {
padding: 2px 0 0 0;
width: 0;
_position: relative;
}
.inline-group .tabular th.original {
width: 0px;
padding: 0;
}
.inline-group .tabular td.original p {
position: absolute;
left: 0;
height: 1.1em;
padding: 2px 9px;
overflow: hidden;
font-size: 0.5625rem;
font-weight: bold;
color: var(--body-quiet-color);
_width: 700px;
}
.inline-group div.add-row,
.inline-group .tabular tr.add-row td {
color: var(--body-quiet-color);
background: var(--darkened-bg);
padding: 8px 10px;
border-bottom: 1px solid var(--hairline-color);
}
.inline-group .tabular tr.add-row td {
padding: 8px 10px;
border-bottom: 1px solid var(--hairline-color);
}
.inline-group div.add-row a,
.inline-group .tabular tr.add-row td a {
font-size: 0.75rem;
}
.empty-form {
display: none;
}
/* RELATED FIELD ADD ONE / LOOKUP */
.related-lookup {
margin-left: 5px;
display: inline-block;
vertical-align: middle;
background-repeat: no-repeat;
background-size: 14px;
}
.related-lookup {
width: 1rem;
height: 1rem;
background-image: url(../img/search.svg);
}
form .related-widget-wrapper ul {
display: inline-block;
margin-left: 0;
padding-left: 0;
}
.clearable-file-input input {
margin-top: 0;
}
+61
View File
@@ -0,0 +1,61 @@
/* LOGIN FORM */
.login {
background: var(--darkened-bg);
height: auto;
}
.login #header {
height: auto;
padding: 15px 16px;
justify-content: center;
}
.login #header h1 {
font-size: 1.125rem;
margin: 0;
}
.login #header h1 a {
color: var(--header-link-color);
}
.login #content {
padding: 20px;
}
.login #container {
background: var(--body-bg);
border: 1px solid var(--hairline-color);
border-radius: 4px;
overflow: hidden;
width: 28em;
min-width: 300px;
margin: 100px auto;
height: auto;
}
.login .form-row {
padding: 4px 0;
}
.login .form-row label {
display: block;
line-height: 2em;
}
.login .form-row #id_username, .login .form-row #id_password {
padding: 8px;
width: 100%;
box-sizing: border-box;
}
.login .submit-row {
padding: 1em 0 0 0;
margin: 0;
text-align: center;
}
.login .password-reset-link {
text-align: center;
}
+150
View File
@@ -0,0 +1,150 @@
.sticky {
position: sticky;
top: 0;
max-height: 100vh;
}
.toggle-nav-sidebar {
z-index: 20;
left: 0;
display: flex;
align-items: center;
justify-content: center;
flex: 0 0 23px;
width: 23px;
border: 0;
border-right: 1px solid var(--hairline-color);
background-color: var(--body-bg);
cursor: pointer;
font-size: 1.25rem;
color: var(--link-fg);
padding: 0;
}
[dir="rtl"] .toggle-nav-sidebar {
border-left: 1px solid var(--hairline-color);
border-right: 0;
}
.toggle-nav-sidebar:hover,
.toggle-nav-sidebar:focus {
background-color: var(--darkened-bg);
}
#nav-sidebar {
z-index: 15;
flex: 0 0 275px;
left: -276px;
margin-left: -276px;
border-top: 1px solid transparent;
border-right: 1px solid var(--hairline-color);
background-color: var(--body-bg);
overflow: auto;
}
[dir="rtl"] #nav-sidebar {
border-left: 1px solid var(--hairline-color);
border-right: 0;
left: 0;
margin-left: 0;
right: -276px;
margin-right: -276px;
}
.toggle-nav-sidebar::before {
content: '\00BB';
}
.main.shifted .toggle-nav-sidebar::before {
content: '\00AB';
}
.main > #nav-sidebar {
visibility: hidden;
}
.main.shifted > #nav-sidebar {
margin-left: 0;
visibility: visible;
}
[dir="rtl"] .main.shifted > #nav-sidebar {
margin-right: 0;
}
#nav-sidebar .module th {
width: 100%;
overflow-wrap: anywhere;
}
#nav-sidebar .module th,
#nav-sidebar .module caption {
padding-left: 16px;
}
#nav-sidebar .module td {
white-space: nowrap;
}
[dir="rtl"] #nav-sidebar .module th,
[dir="rtl"] #nav-sidebar .module caption {
padding-left: 8px;
padding-right: 16px;
}
#nav-sidebar .current-app .section:link,
#nav-sidebar .current-app .section:visited {
color: var(--header-color);
font-weight: bold;
}
#nav-sidebar .current-model {
background: var(--selected-row);
}
@media (forced-colors: active) {
#nav-sidebar .current-model {
background-color: SelectedItem;
}
}
.main > #nav-sidebar + .content {
max-width: calc(100% - 23px);
}
.main.shifted > #nav-sidebar + .content {
max-width: calc(100% - 299px);
}
@media (max-width: 767px) {
#nav-sidebar, #toggle-nav-sidebar {
display: none;
}
.main > #nav-sidebar + .content,
.main.shifted > #nav-sidebar + .content {
max-width: 100%;
}
}
#nav-filter {
width: 100%;
box-sizing: border-box;
padding: 2px 5px;
margin: 5px 0;
border: 1px solid var(--border-color);
background-color: var(--darkened-bg);
color: var(--body-fg);
}
#nav-filter:focus {
border-color: var(--body-quiet-color);
}
#nav-filter.no-results {
background: var(--message-error-bg);
}
#nav-sidebar table {
width: 100%;
}
+904
View File
@@ -0,0 +1,904 @@
/* Tablets */
input[type="submit"], button {
-webkit-appearance: none;
appearance: none;
}
@media (max-width: 1024px) {
/* Basic */
html {
-webkit-text-size-adjust: 100%;
}
td, th {
padding: 10px;
font-size: 0.875rem;
}
.small {
font-size: 0.75rem;
}
/* Layout */
#container {
min-width: 0;
}
#content {
padding: 15px 20px 20px;
}
div.breadcrumbs {
padding: 10px 30px;
}
/* Header */
#header {
flex-direction: column;
padding: 15px 30px;
justify-content: flex-start;
}
#site-name {
margin: 0 0 8px;
line-height: 1.2;
}
#user-tools {
margin: 0;
font-weight: 400;
line-height: 1.85;
text-align: left;
}
#user-tools a {
display: inline-block;
line-height: 1.4;
}
/* Dashboard */
.dashboard #content {
width: auto;
}
#content-related {
margin-right: -290px;
}
.colSM #content-related {
margin-left: -290px;
}
.colMS {
margin-right: 290px;
}
.colSM {
margin-left: 290px;
}
.dashboard .module table td a {
padding-right: 0;
}
td .changelink, td .addlink {
font-size: 0.8125rem;
}
/* Changelist */
#toolbar {
border: none;
padding: 15px;
}
#changelist-search > div {
display: flex;
flex-wrap: nowrap;
max-width: 480px;
}
#changelist-search label {
line-height: 1.375rem;
}
#toolbar form #searchbar {
flex: 1 0 auto;
width: 0;
height: 1.375rem;
margin: 0 10px 0 6px;
}
#toolbar form input[type=submit] {
flex: 0 1 auto;
}
#changelist-search .quiet {
width: 0;
flex: 1 0 auto;
margin: 5px 0 0 25px;
}
#changelist .actions {
display: flex;
flex-wrap: wrap;
padding: 15px 0;
}
#changelist .actions label {
display: flex;
}
#changelist .actions select {
background: var(--body-bg);
}
#changelist .actions .button {
min-width: 48px;
margin: 0 10px;
}
#changelist .actions span.all,
#changelist .actions span.clear,
#changelist .actions span.question,
#changelist .actions span.action-counter {
font-size: 0.6875rem;
margin: 0 10px 0 0;
}
#changelist-filter {
flex-basis: 200px;
}
.change-list .filtered .results,
.change-list .filtered .paginator,
.filtered #toolbar,
.filtered .actions,
#changelist .paginator {
border-top-color: var(--hairline-color); /* XXX Is this used at all? */
}
#changelist .results + .paginator {
border-top: none;
}
/* Forms */
label {
font-size: 1rem;
}
/*
Minifiers remove the default (text) "type" attribute from "input" HTML
tags. Add input:not([type]) to make the CSS stylesheet work the same.
*/
.form-row input:not([type]),
.form-row input[type=text],
.form-row input[type=password],
.form-row input[type=email],
.form-row input[type=url],
.form-row input[type=tel],
.form-row input[type=number],
.form-row textarea,
.form-row select,
.form-row .vTextField {
box-sizing: border-box;
margin: 0;
padding: 6px 8px;
min-height: 2.25rem;
font-size: 1rem;
}
.form-row select {
height: 2.25rem;
}
.form-row select[multiple] {
height: auto;
min-height: 0;
}
fieldset .fieldBox + .fieldBox {
margin-top: 10px;
padding-top: 10px;
border-top: 1px solid var(--hairline-color);
}
textarea {
max-width: 100%;
max-height: 120px;
}
.aligned label {
padding-top: 6px;
}
.aligned .related-lookup,
.aligned .datetimeshortcuts,
.aligned .related-lookup + strong {
align-self: center;
margin-left: 15px;
}
form .aligned div.radiolist {
margin-left: 2px;
}
.submit-row {
padding: 8px;
}
.submit-row a.deletelink {
padding: 10px 7px;
}
.button, input[type=submit], input[type=button], .submit-row input, a.button {
padding: 7px;
}
/* Selector */
.selector {
display: flex;
width: 100%;
}
.selector .selector-filter {
display: flex;
align-items: center;
}
.selector .selector-filter input {
width: 100%;
min-height: 0;
flex: 1 1;
}
.selector-available, .selector-chosen {
width: auto;
flex: 1 1;
display: flex;
flex-direction: column;
}
.selector select {
width: 100%;
flex: 1 0 auto;
margin-bottom: 5px;
}
.selector-chooseall, .selector-clearall {
align-self: center;
}
.stacked {
flex-direction: column;
max-width: 480px;
}
.stacked > * {
flex: 0 1 auto;
}
.stacked select {
margin-bottom: 0;
}
.stacked .selector-available, .stacked .selector-chosen {
width: auto;
}
.stacked ul.selector-chooser {
padding: 0 2px;
transform: none;
}
.stacked .selector-chooser li {
padding: 3px;
}
.help-tooltip, .selector .help-icon {
display: none;
}
.datetime input {
width: 50%;
max-width: 120px;
}
.datetime span {
font-size: 0.8125rem;
}
.datetime .timezonewarning {
display: block;
font-size: 0.6875rem;
color: var(--body-quiet-color);
}
.datetimeshortcuts {
color: var(--border-color); /* XXX Redundant, .datetime span also sets #ccc */
}
.form-row .datetime input.vDateField, .form-row .datetime input.vTimeField {
width: 75%;
}
.inline-group {
overflow: auto;
}
/* Messages */
ul.messagelist li {
padding-left: 55px;
background-position: 30px 12px;
}
ul.messagelist li.error {
background-position: 30px 12px;
}
ul.messagelist li.warning {
background-position: 30px 14px;
}
/* Login */
.login #header {
padding: 15px 20px;
}
.login #site-name {
margin: 0;
}
/* GIS */
div.olMap {
max-width: calc(100vw - 30px);
max-height: 300px;
}
.olMap + .clear_features {
display: block;
margin-top: 10px;
}
/* Docs */
.module table.xfull {
width: 100%;
}
pre.literal-block {
overflow: auto;
}
}
/* Mobile */
@media (max-width: 767px) {
/* Layout */
#header, #content {
padding: 15px;
}
div.breadcrumbs {
padding: 10px 15px;
}
/* Dashboard */
.colMS, .colSM {
margin: 0;
}
#content-related, .colSM #content-related {
width: 100%;
margin: 0;
}
#content-related .module {
margin-bottom: 0;
}
#content-related .module h2 {
padding: 10px 15px;
font-size: 1rem;
}
/* Changelist */
#changelist {
align-items: stretch;
flex-direction: column;
}
#toolbar {
padding: 10px;
}
#changelist-filter {
margin-left: 0;
}
#changelist .actions label {
flex: 1 1;
}
#changelist .actions select {
flex: 1 0;
width: 100%;
}
#changelist .actions span {
flex: 1 0 100%;
}
#changelist-filter {
position: static;
width: auto;
margin-top: 30px;
}
.object-tools {
float: none;
margin: 0 0 15px;
padding: 0;
overflow: hidden;
}
.object-tools li {
height: auto;
margin-left: 0;
}
.object-tools li + li {
margin-left: 15px;
}
/* Forms */
.form-row {
padding: 15px 0;
}
.aligned .form-row,
.aligned .form-row > div {
max-width: 100vw;
}
.aligned .form-row > div {
width: calc(100vw - 30px);
}
.flex-container {
flex-flow: column;
}
.flex-container.checkbox-row {
flex-flow: row;
}
textarea {
max-width: none;
}
.vURLField {
width: auto;
}
fieldset .fieldBox + .fieldBox {
margin-top: 15px;
padding-top: 15px;
}
.aligned label {
width: 100%;
min-width: auto;
padding: 0 0 10px;
}
.aligned label:after {
max-height: 0;
}
.aligned .form-row input,
.aligned .form-row select,
.aligned .form-row textarea {
flex: 1 1 auto;
max-width: 100%;
}
.aligned .checkbox-row input {
flex: 0 1 auto;
margin: 0;
}
.aligned .vCheckboxLabel {
flex: 1 0;
padding: 1px 0 0 5px;
}
.aligned label + p,
.aligned label + div.help,
.aligned label + div.readonly {
padding: 0;
margin-left: 0;
}
.aligned p.file-upload {
font-size: 0.8125rem;
}
span.clearable-file-input {
margin-left: 15px;
}
span.clearable-file-input label {
font-size: 0.8125rem;
padding-bottom: 0;
}
.aligned .timezonewarning {
flex: 1 0 100%;
margin-top: 5px;
}
form .aligned .form-row div.help {
width: 100%;
margin: 5px 0 0;
padding: 0;
}
form .aligned ul,
form .aligned ul.errorlist {
margin-left: 0;
padding-left: 0;
}
form .aligned div.radiolist {
margin-top: 5px;
margin-right: 15px;
margin-bottom: -3px;
}
form .aligned div.radiolist:not(.inline) div + div {
margin-top: 5px;
}
/* Related widget */
.related-widget-wrapper {
width: 100%;
display: flex;
align-items: flex-start;
}
.related-widget-wrapper .selector {
order: 1;
flex: 1 0 auto;
}
.related-widget-wrapper > a {
order: 2;
}
.related-widget-wrapper .radiolist ~ a {
align-self: flex-end;
}
.related-widget-wrapper > select ~ a {
align-self: center;
}
/* Selector */
.selector {
flex-direction: column;
gap: 10px 0;
}
.selector-available, .selector-chosen {
flex: 1 1 auto;
}
.selector select {
max-height: 96px;
}
.selector ul.selector-chooser {
display: flex;
width: 60px;
height: 30px;
padding: 0 2px;
transform: none;
}
.selector ul.selector-chooser li {
float: left;
}
.selector-remove {
background-position: 0 0;
}
:enabled.selector-remove:focus, :enabled.selector-remove:hover {
background-position: 0 -24px;
}
.selector-add {
background-position: 0 -48px;
}
:enabled.selector-add:focus, :enabled.selector-add:hover {
background-position: 0 -72px;
}
/* Inlines */
.inline-group[data-inline-type="stacked"] .inline-related {
border: 1px solid var(--hairline-color);
border-radius: 4px;
margin-top: 15px;
overflow: auto;
}
.inline-group[data-inline-type="stacked"] .inline-related > * {
box-sizing: border-box;
}
.inline-group[data-inline-type="stacked"] .inline-related .module {
padding: 0 10px;
}
.inline-group[data-inline-type="stacked"] .inline-related .module .form-row {
border-top: 1px solid var(--hairline-color);
border-bottom: none;
}
.inline-group[data-inline-type="stacked"] .inline-related .module .form-row:first-child {
border-top: none;
}
.inline-group[data-inline-type="stacked"] .inline-related h3 {
padding: 10px;
border-top-width: 0;
border-bottom-width: 2px;
display: flex;
flex-wrap: wrap;
align-items: center;
}
.inline-group[data-inline-type="stacked"] .inline-related h3 .inline_label {
margin-right: auto;
}
.inline-group[data-inline-type="stacked"] .inline-related h3 span.delete {
float: none;
flex: 1 1 100%;
margin-top: 5px;
}
.inline-group[data-inline-type="stacked"] .aligned .form-row > div:not([class]) {
width: 100%;
}
.inline-group[data-inline-type="stacked"] .aligned label {
width: 100%;
}
.inline-group[data-inline-type="stacked"] div.add-row {
margin-top: 15px;
border: 1px solid var(--hairline-color);
border-radius: 4px;
}
.inline-group div.add-row,
.inline-group .tabular tr.add-row td {
padding: 0;
}
.inline-group div.add-row a,
.inline-group .tabular tr.add-row td a {
display: block;
padding: 8px 10px 8px 26px;
background-position: 8px 9px;
}
/* Submit row */
.submit-row {
padding: 10px;
margin: 0 0 15px;
flex-direction: column;
gap: 8px;
}
.submit-row input, .submit-row input.default, .submit-row a {
text-align: center;
}
.submit-row a.closelink {
padding: 10px 0;
text-align: center;
}
.submit-row a.deletelink {
margin: 0;
}
/* Messages */
ul.messagelist li {
padding-left: 40px;
background-position: 15px 12px;
}
ul.messagelist li.error {
background-position: 15px 12px;
}
ul.messagelist li.warning {
background-position: 15px 14px;
}
/* Paginator */
.paginator .this-page, .paginator a:link, .paginator a:visited {
padding: 4px 10px;
}
/* Login */
body.login {
padding: 0 15px;
}
.login #container {
width: auto;
max-width: 480px;
margin: 50px auto;
}
.login #header,
.login #content {
padding: 15px;
}
.login #content-main {
float: none;
}
.login .form-row {
padding: 0;
}
.login .form-row + .form-row {
margin-top: 15px;
}
.login .form-row label {
margin: 0 0 5px;
line-height: 1.2;
}
.login .submit-row {
padding: 15px 0 0;
}
.login br {
display: none;
}
.login .submit-row input {
margin: 0;
text-transform: uppercase;
}
.errornote {
margin: 0 0 20px;
padding: 8px 12px;
font-size: 0.8125rem;
}
/* Calendar and clock */
.calendarbox, .clockbox {
position: fixed !important;
top: 50% !important;
left: 50% !important;
transform: translate(-50%, -50%);
margin: 0;
border: none;
overflow: visible;
}
.calendarbox:before, .clockbox:before {
content: '';
position: fixed;
top: 50%;
left: 50%;
width: 100vw;
height: 100vh;
background: rgba(0, 0, 0, 0.75);
transform: translate(-50%, -50%);
}
.calendarbox > *, .clockbox > * {
position: relative;
z-index: 1;
}
.calendarbox > div:first-child {
z-index: 2;
}
.calendarbox .calendar, .clockbox h2 {
border-radius: 4px 4px 0 0;
overflow: hidden;
}
.calendarbox .calendar-cancel, .clockbox .calendar-cancel {
border-radius: 0 0 4px 4px;
overflow: hidden;
}
.calendar-shortcuts {
padding: 10px 0;
font-size: 0.75rem;
line-height: 0.75rem;
}
.calendar-shortcuts a {
margin: 0 4px;
}
.timelist a {
background: var(--body-bg);
padding: 4px;
}
.calendar-cancel {
padding: 8px 10px;
}
.clockbox h2 {
padding: 8px 15px;
}
.calendar caption {
padding: 10px;
}
.calendarbox .calendarnav-previous, .calendarbox .calendarnav-next {
z-index: 1;
top: 10px;
}
/* History */
table#change-history tbody th, table#change-history tbody td {
font-size: 0.8125rem;
word-break: break-word;
}
table#change-history tbody th {
width: auto;
}
/* Docs */
table.model tbody th, table.model tbody td {
font-size: 0.8125rem;
word-break: break-word;
}
}
+89
View File
@@ -0,0 +1,89 @@
/* TABLETS */
@media (max-width: 1024px) {
[dir="rtl"] .colMS {
margin-right: 0;
}
[dir="rtl"] #user-tools {
text-align: right;
}
[dir="rtl"] #changelist .actions label {
padding-left: 10px;
padding-right: 0;
}
[dir="rtl"] #changelist .actions select {
margin-left: 0;
margin-right: 15px;
}
[dir="rtl"] .change-list .filtered .results,
[dir="rtl"] .change-list .filtered .paginator,
[dir="rtl"] .filtered #toolbar,
[dir="rtl"] .filtered div.xfull,
[dir="rtl"] .filtered .actions,
[dir="rtl"] #changelist-filter {
margin-left: 0;
}
[dir="rtl"] .inline-group div.add-row a,
[dir="rtl"] .inline-group .tabular tr.add-row td a {
padding: 8px 26px 8px 10px;
background-position: calc(100% - 8px) 9px;
}
[dir="rtl"] .object-tools li {
float: right;
}
[dir="rtl"] .object-tools li + li {
margin-left: 0;
margin-right: 15px;
}
[dir="rtl"] .dashboard .module table td a {
padding-left: 0;
padding-right: 16px;
}
}
/* MOBILE */
@media (max-width: 767px) {
[dir="rtl"] .aligned .related-lookup,
[dir="rtl"] .aligned .datetimeshortcuts {
margin-left: 0;
margin-right: 15px;
}
[dir="rtl"] .aligned ul,
[dir="rtl"] form .aligned ul.errorlist {
margin-right: 0;
}
[dir="rtl"] #changelist-filter {
margin-left: 0;
margin-right: 0;
}
[dir="rtl"] .aligned .vCheckboxLabel {
padding: 1px 5px 0 0;
}
[dir="rtl"] .selector-remove {
background-position: 0 0;
}
[dir="rtl"] :enabled.selector-remove:focus, :enabled.selector-remove:hover {
background-position: 0 -24px;
}
[dir="rtl"] .selector-add {
background-position: 0 -48px;
}
[dir="rtl"] :enabled.selector-add:focus, :enabled.selector-add:hover {
background-position: 0 -72px;
}
}
+293
View File
@@ -0,0 +1,293 @@
/* GLOBAL */
th {
text-align: right;
}
.module h2, .module caption {
text-align: right;
}
.module ul, .module ol {
margin-left: 0;
margin-right: 1.5em;
}
.viewlink, .addlink, .changelink, .hidelink {
padding-left: 0;
padding-right: 16px;
background-position: 100% 1px;
}
.deletelink {
padding-left: 0;
padding-right: 16px;
background-position: 100% 1px;
}
.object-tools {
float: left;
}
thead th:first-child,
tfoot td:first-child {
border-left: none;
}
/* LAYOUT */
#user-tools {
right: auto;
left: 0;
text-align: left;
}
div.breadcrumbs {
text-align: right;
}
#content-main {
float: right;
}
#content-related {
float: left;
margin-left: -300px;
margin-right: auto;
}
.colMS {
margin-left: 300px;
margin-right: 0;
}
/* SORTABLE TABLES */
table thead th.sorted .sortoptions {
float: left;
}
thead th.sorted .text {
padding-right: 0;
padding-left: 42px;
}
/* dashboard styles */
.dashboard .module table td a {
padding-left: .6em;
padding-right: 16px;
}
/* changelists styles */
.change-list .filtered table {
border-left: none;
border-right: 0px none;
}
#changelist-filter {
border-left: none;
border-right: none;
margin-left: 0;
margin-right: 30px;
}
#changelist-filter li.selected {
border-left: none;
padding-left: 10px;
margin-left: 0;
border-right: 5px solid var(--hairline-color);
padding-right: 10px;
margin-right: -15px;
}
#changelist table tbody td:first-child, #changelist table tbody th:first-child {
border-right: none;
border-left: none;
}
.paginator .end {
margin-left: 6px;
margin-right: 0;
}
.paginator input {
margin-left: 0;
margin-right: auto;
}
/* FORMS */
.aligned label {
padding: 0 0 3px 1em;
}
.submit-row a.deletelink {
margin-left: 0;
margin-right: auto;
}
.vDateField, .vTimeField {
margin-left: 2px;
}
.aligned .form-row input {
margin-left: 5px;
}
form .aligned ul {
margin-right: 163px;
padding-right: 10px;
margin-left: 0;
padding-left: 0;
}
form ul.inline li {
float: right;
padding-right: 0;
padding-left: 7px;
}
form .aligned p.help,
form .aligned div.help {
margin-left: 0;
margin-right: 160px;
padding-right: 10px;
}
form div.help ul,
form .aligned .checkbox-row + .help,
form .aligned p.date div.help.timezonewarning,
form .aligned p.datetime div.help.timezonewarning,
form .aligned p.time div.help.timezonewarning {
margin-right: 0;
padding-right: 0;
}
form .wide p.help,
form .wide ul.errorlist,
form .wide div.help {
padding-left: 0;
padding-right: 50px;
}
.submit-row {
text-align: right;
}
fieldset .fieldBox {
margin-left: 20px;
margin-right: 0;
}
.errorlist li {
background-position: 100% 12px;
padding: 0;
}
.errornote {
background-position: 100% 12px;
padding: 10px 12px;
}
/* WIDGETS */
.calendarnav-previous {
top: 0;
left: auto;
right: 10px;
background: url(../img/calendar-icons.svg) 0 -15px no-repeat;
}
.calendarnav-next {
top: 0;
right: auto;
left: 10px;
background: url(../img/calendar-icons.svg) 0 0 no-repeat;
}
.calendar caption, .calendarbox h2 {
text-align: center;
}
.selector {
float: right;
}
.selector .selector-filter {
text-align: right;
}
.selector-add {
background: url(../img/selector-icons.svg) 0 -96px no-repeat;
background-size: 24px auto;
}
:enabled.selector-add:focus, :enabled.selector-add:hover {
background-position: 0 -120px;
}
.selector-remove {
background: url(../img/selector-icons.svg) 0 -144px no-repeat;
background-size: 24px auto;
}
:enabled.selector-remove:focus, :enabled.selector-remove:hover {
background-position: 0 -168px;
}
.selector-chooseall {
background: url(../img/selector-icons.svg) right -128px no-repeat;
}
:enabled.selector-chooseall:focus, :enabled.selector-chooseall:hover {
background-position: 100% -144px;
}
.selector-clearall {
background: url(../img/selector-icons.svg) 0 -160px no-repeat;
}
:enabled.selector-clearall:focus, :enabled.selector-clearall:hover {
background-position: 0 -176px;
}
.inline-deletelink {
float: left;
}
form .form-row p.datetime {
overflow: hidden;
}
.related-widget-wrapper {
float: right;
}
/* MISC */
.inline-related h2, .inline-group h2 {
text-align: right
}
.inline-related h3 span.delete {
padding-right: 20px;
padding-left: inherit;
left: 10px;
right: inherit;
float:left;
}
.inline-related h3 span.delete label {
margin-left: inherit;
margin-right: 2px;
}
.inline-group .tabular td.original p {
right: 0;
}
.selector .selector-chooser {
margin: 0;
}
@@ -0,0 +1,19 @@
/* Hide warnings fields if usable password is selected */
form:has(#id_usable_password input[value="true"]:checked) .messagelist {
display: none;
}
/* Hide password fields if unusable password is selected */
form:has(#id_usable_password input[value="false"]:checked) .field-password1,
form:has(#id_usable_password input[value="false"]:checked) .field-password2 {
display: none;
}
/* Select appropriate submit button */
form:has(#id_usable_password input[value="true"]:checked) input[type="submit"].unset-password {
display: none;
}
form:has(#id_usable_password input[value="false"]:checked) input[type="submit"].set-password {
display: none;
}
@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2012-2017 Kevin Brown, Igor Vaynberg, and Select2 contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
+481
View File
@@ -0,0 +1,481 @@
.select2-container {
box-sizing: border-box;
display: inline-block;
margin: 0;
position: relative;
vertical-align: middle; }
.select2-container .select2-selection--single {
box-sizing: border-box;
cursor: pointer;
display: block;
height: 28px;
user-select: none;
-webkit-user-select: none; }
.select2-container .select2-selection--single .select2-selection__rendered {
display: block;
padding-left: 8px;
padding-right: 20px;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap; }
.select2-container .select2-selection--single .select2-selection__clear {
position: relative; }
.select2-container[dir="rtl"] .select2-selection--single .select2-selection__rendered {
padding-right: 8px;
padding-left: 20px; }
.select2-container .select2-selection--multiple {
box-sizing: border-box;
cursor: pointer;
display: block;
min-height: 32px;
user-select: none;
-webkit-user-select: none; }
.select2-container .select2-selection--multiple .select2-selection__rendered {
display: inline-block;
overflow: hidden;
padding-left: 8px;
text-overflow: ellipsis;
white-space: nowrap; }
.select2-container .select2-search--inline {
float: left; }
.select2-container .select2-search--inline .select2-search__field {
box-sizing: border-box;
border: none;
font-size: 100%;
margin-top: 5px;
padding: 0; }
.select2-container .select2-search--inline .select2-search__field::-webkit-search-cancel-button {
-webkit-appearance: none; }
.select2-dropdown {
background-color: white;
border: 1px solid #aaa;
border-radius: 4px;
box-sizing: border-box;
display: block;
position: absolute;
left: -100000px;
width: 100%;
z-index: 1051; }
.select2-results {
display: block; }
.select2-results__options {
list-style: none;
margin: 0;
padding: 0; }
.select2-results__option {
padding: 6px;
user-select: none;
-webkit-user-select: none; }
.select2-results__option[aria-selected] {
cursor: pointer; }
.select2-container--open .select2-dropdown {
left: 0; }
.select2-container--open .select2-dropdown--above {
border-bottom: none;
border-bottom-left-radius: 0;
border-bottom-right-radius: 0; }
.select2-container--open .select2-dropdown--below {
border-top: none;
border-top-left-radius: 0;
border-top-right-radius: 0; }
.select2-search--dropdown {
display: block;
padding: 4px; }
.select2-search--dropdown .select2-search__field {
padding: 4px;
width: 100%;
box-sizing: border-box; }
.select2-search--dropdown .select2-search__field::-webkit-search-cancel-button {
-webkit-appearance: none; }
.select2-search--dropdown.select2-search--hide {
display: none; }
.select2-close-mask {
border: 0;
margin: 0;
padding: 0;
display: block;
position: fixed;
left: 0;
top: 0;
min-height: 100%;
min-width: 100%;
height: auto;
width: auto;
opacity: 0;
z-index: 99;
background-color: #fff;
filter: alpha(opacity=0); }
.select2-hidden-accessible {
border: 0 !important;
clip: rect(0 0 0 0) !important;
-webkit-clip-path: inset(50%) !important;
clip-path: inset(50%) !important;
height: 1px !important;
overflow: hidden !important;
padding: 0 !important;
position: absolute !important;
width: 1px !important;
white-space: nowrap !important; }
.select2-container--default .select2-selection--single {
background-color: #fff;
border: 1px solid #aaa;
border-radius: 4px; }
.select2-container--default .select2-selection--single .select2-selection__rendered {
color: #444;
line-height: 28px; }
.select2-container--default .select2-selection--single .select2-selection__clear {
cursor: pointer;
float: right;
font-weight: bold; }
.select2-container--default .select2-selection--single .select2-selection__placeholder {
color: #999; }
.select2-container--default .select2-selection--single .select2-selection__arrow {
height: 26px;
position: absolute;
top: 1px;
right: 1px;
width: 20px; }
.select2-container--default .select2-selection--single .select2-selection__arrow b {
border-color: #888 transparent transparent transparent;
border-style: solid;
border-width: 5px 4px 0 4px;
height: 0;
left: 50%;
margin-left: -4px;
margin-top: -2px;
position: absolute;
top: 50%;
width: 0; }
.select2-container--default[dir="rtl"] .select2-selection--single .select2-selection__clear {
float: left; }
.select2-container--default[dir="rtl"] .select2-selection--single .select2-selection__arrow {
left: 1px;
right: auto; }
.select2-container--default.select2-container--disabled .select2-selection--single {
background-color: #eee;
cursor: default; }
.select2-container--default.select2-container--disabled .select2-selection--single .select2-selection__clear {
display: none; }
.select2-container--default.select2-container--open .select2-selection--single .select2-selection__arrow b {
border-color: transparent transparent #888 transparent;
border-width: 0 4px 5px 4px; }
.select2-container--default .select2-selection--multiple {
background-color: white;
border: 1px solid #aaa;
border-radius: 4px;
cursor: text; }
.select2-container--default .select2-selection--multiple .select2-selection__rendered {
box-sizing: border-box;
list-style: none;
margin: 0;
padding: 0 5px;
width: 100%; }
.select2-container--default .select2-selection--multiple .select2-selection__rendered li {
list-style: none; }
.select2-container--default .select2-selection--multiple .select2-selection__clear {
cursor: pointer;
float: right;
font-weight: bold;
margin-top: 5px;
margin-right: 10px;
padding: 1px; }
.select2-container--default .select2-selection--multiple .select2-selection__choice {
background-color: #e4e4e4;
border: 1px solid #aaa;
border-radius: 4px;
cursor: default;
float: left;
margin-right: 5px;
margin-top: 5px;
padding: 0 5px; }
.select2-container--default .select2-selection--multiple .select2-selection__choice__remove {
color: #999;
cursor: pointer;
display: inline-block;
font-weight: bold;
margin-right: 2px; }
.select2-container--default .select2-selection--multiple .select2-selection__choice__remove:hover {
color: #333; }
.select2-container--default[dir="rtl"] .select2-selection--multiple .select2-selection__choice, .select2-container--default[dir="rtl"] .select2-selection--multiple .select2-search--inline {
float: right; }
.select2-container--default[dir="rtl"] .select2-selection--multiple .select2-selection__choice {
margin-left: 5px;
margin-right: auto; }
.select2-container--default[dir="rtl"] .select2-selection--multiple .select2-selection__choice__remove {
margin-left: 2px;
margin-right: auto; }
.select2-container--default.select2-container--focus .select2-selection--multiple {
border: solid black 1px;
outline: 0; }
.select2-container--default.select2-container--disabled .select2-selection--multiple {
background-color: #eee;
cursor: default; }
.select2-container--default.select2-container--disabled .select2-selection__choice__remove {
display: none; }
.select2-container--default.select2-container--open.select2-container--above .select2-selection--single, .select2-container--default.select2-container--open.select2-container--above .select2-selection--multiple {
border-top-left-radius: 0;
border-top-right-radius: 0; }
.select2-container--default.select2-container--open.select2-container--below .select2-selection--single, .select2-container--default.select2-container--open.select2-container--below .select2-selection--multiple {
border-bottom-left-radius: 0;
border-bottom-right-radius: 0; }
.select2-container--default .select2-search--dropdown .select2-search__field {
border: 1px solid #aaa; }
.select2-container--default .select2-search--inline .select2-search__field {
background: transparent;
border: none;
outline: 0;
box-shadow: none;
-webkit-appearance: textfield; }
.select2-container--default .select2-results > .select2-results__options {
max-height: 200px;
overflow-y: auto; }
.select2-container--default .select2-results__option[role=group] {
padding: 0; }
.select2-container--default .select2-results__option[aria-disabled=true] {
color: #999; }
.select2-container--default .select2-results__option[aria-selected=true] {
background-color: #ddd; }
.select2-container--default .select2-results__option .select2-results__option {
padding-left: 1em; }
.select2-container--default .select2-results__option .select2-results__option .select2-results__group {
padding-left: 0; }
.select2-container--default .select2-results__option .select2-results__option .select2-results__option {
margin-left: -1em;
padding-left: 2em; }
.select2-container--default .select2-results__option .select2-results__option .select2-results__option .select2-results__option {
margin-left: -2em;
padding-left: 3em; }
.select2-container--default .select2-results__option .select2-results__option .select2-results__option .select2-results__option .select2-results__option {
margin-left: -3em;
padding-left: 4em; }
.select2-container--default .select2-results__option .select2-results__option .select2-results__option .select2-results__option .select2-results__option .select2-results__option {
margin-left: -4em;
padding-left: 5em; }
.select2-container--default .select2-results__option .select2-results__option .select2-results__option .select2-results__option .select2-results__option .select2-results__option .select2-results__option {
margin-left: -5em;
padding-left: 6em; }
.select2-container--default .select2-results__option--highlighted[aria-selected] {
background-color: #5897fb;
color: white; }
.select2-container--default .select2-results__group {
cursor: default;
display: block;
padding: 6px; }
.select2-container--classic .select2-selection--single {
background-color: #f7f7f7;
border: 1px solid #aaa;
border-radius: 4px;
outline: 0;
background-image: -webkit-linear-gradient(top, white 50%, #eeeeee 100%);
background-image: -o-linear-gradient(top, white 50%, #eeeeee 100%);
background-image: linear-gradient(to bottom, white 50%, #eeeeee 100%);
background-repeat: repeat-x;
filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#FFFFFFFF', endColorstr='#FFEEEEEE', GradientType=0); }
.select2-container--classic .select2-selection--single:focus {
border: 1px solid #5897fb; }
.select2-container--classic .select2-selection--single .select2-selection__rendered {
color: #444;
line-height: 28px; }
.select2-container--classic .select2-selection--single .select2-selection__clear {
cursor: pointer;
float: right;
font-weight: bold;
margin-right: 10px; }
.select2-container--classic .select2-selection--single .select2-selection__placeholder {
color: #999; }
.select2-container--classic .select2-selection--single .select2-selection__arrow {
background-color: #ddd;
border: none;
border-left: 1px solid #aaa;
border-top-right-radius: 4px;
border-bottom-right-radius: 4px;
height: 26px;
position: absolute;
top: 1px;
right: 1px;
width: 20px;
background-image: -webkit-linear-gradient(top, #eeeeee 50%, #cccccc 100%);
background-image: -o-linear-gradient(top, #eeeeee 50%, #cccccc 100%);
background-image: linear-gradient(to bottom, #eeeeee 50%, #cccccc 100%);
background-repeat: repeat-x;
filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#FFEEEEEE', endColorstr='#FFCCCCCC', GradientType=0); }
.select2-container--classic .select2-selection--single .select2-selection__arrow b {
border-color: #888 transparent transparent transparent;
border-style: solid;
border-width: 5px 4px 0 4px;
height: 0;
left: 50%;
margin-left: -4px;
margin-top: -2px;
position: absolute;
top: 50%;
width: 0; }
.select2-container--classic[dir="rtl"] .select2-selection--single .select2-selection__clear {
float: left; }
.select2-container--classic[dir="rtl"] .select2-selection--single .select2-selection__arrow {
border: none;
border-right: 1px solid #aaa;
border-radius: 0;
border-top-left-radius: 4px;
border-bottom-left-radius: 4px;
left: 1px;
right: auto; }
.select2-container--classic.select2-container--open .select2-selection--single {
border: 1px solid #5897fb; }
.select2-container--classic.select2-container--open .select2-selection--single .select2-selection__arrow {
background: transparent;
border: none; }
.select2-container--classic.select2-container--open .select2-selection--single .select2-selection__arrow b {
border-color: transparent transparent #888 transparent;
border-width: 0 4px 5px 4px; }
.select2-container--classic.select2-container--open.select2-container--above .select2-selection--single {
border-top: none;
border-top-left-radius: 0;
border-top-right-radius: 0;
background-image: -webkit-linear-gradient(top, white 0%, #eeeeee 50%);
background-image: -o-linear-gradient(top, white 0%, #eeeeee 50%);
background-image: linear-gradient(to bottom, white 0%, #eeeeee 50%);
background-repeat: repeat-x;
filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#FFFFFFFF', endColorstr='#FFEEEEEE', GradientType=0); }
.select2-container--classic.select2-container--open.select2-container--below .select2-selection--single {
border-bottom: none;
border-bottom-left-radius: 0;
border-bottom-right-radius: 0;
background-image: -webkit-linear-gradient(top, #eeeeee 50%, white 100%);
background-image: -o-linear-gradient(top, #eeeeee 50%, white 100%);
background-image: linear-gradient(to bottom, #eeeeee 50%, white 100%);
background-repeat: repeat-x;
filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#FFEEEEEE', endColorstr='#FFFFFFFF', GradientType=0); }
.select2-container--classic .select2-selection--multiple {
background-color: white;
border: 1px solid #aaa;
border-radius: 4px;
cursor: text;
outline: 0; }
.select2-container--classic .select2-selection--multiple:focus {
border: 1px solid #5897fb; }
.select2-container--classic .select2-selection--multiple .select2-selection__rendered {
list-style: none;
margin: 0;
padding: 0 5px; }
.select2-container--classic .select2-selection--multiple .select2-selection__clear {
display: none; }
.select2-container--classic .select2-selection--multiple .select2-selection__choice {
background-color: #e4e4e4;
border: 1px solid #aaa;
border-radius: 4px;
cursor: default;
float: left;
margin-right: 5px;
margin-top: 5px;
padding: 0 5px; }
.select2-container--classic .select2-selection--multiple .select2-selection__choice__remove {
color: #888;
cursor: pointer;
display: inline-block;
font-weight: bold;
margin-right: 2px; }
.select2-container--classic .select2-selection--multiple .select2-selection__choice__remove:hover {
color: #555; }
.select2-container--classic[dir="rtl"] .select2-selection--multiple .select2-selection__choice {
float: right;
margin-left: 5px;
margin-right: auto; }
.select2-container--classic[dir="rtl"] .select2-selection--multiple .select2-selection__choice__remove {
margin-left: 2px;
margin-right: auto; }
.select2-container--classic.select2-container--open .select2-selection--multiple {
border: 1px solid #5897fb; }
.select2-container--classic.select2-container--open.select2-container--above .select2-selection--multiple {
border-top: none;
border-top-left-radius: 0;
border-top-right-radius: 0; }
.select2-container--classic.select2-container--open.select2-container--below .select2-selection--multiple {
border-bottom: none;
border-bottom-left-radius: 0;
border-bottom-right-radius: 0; }
.select2-container--classic .select2-search--dropdown .select2-search__field {
border: 1px solid #aaa;
outline: 0; }
.select2-container--classic .select2-search--inline .select2-search__field {
outline: 0;
box-shadow: none; }
.select2-container--classic .select2-dropdown {
background-color: white;
border: 1px solid transparent; }
.select2-container--classic .select2-dropdown--above {
border-bottom: none; }
.select2-container--classic .select2-dropdown--below {
border-top: none; }
.select2-container--classic .select2-results > .select2-results__options {
max-height: 200px;
overflow-y: auto; }
.select2-container--classic .select2-results__option[role=group] {
padding: 0; }
.select2-container--classic .select2-results__option[aria-disabled=true] {
color: grey; }
.select2-container--classic .select2-results__option--highlighted[aria-selected] {
background-color: #3875d7;
color: white; }
.select2-container--classic .select2-results__group {
cursor: default;
display: block;
padding: 6px; }
.select2-container--classic.select2-container--open .select2-dropdown {
border-color: #5897fb; }
File diff suppressed because one or more lines are too long
+613
View File
@@ -0,0 +1,613 @@
/* SELECTOR (FILTER INTERFACE) */
.selector {
display: flex;
flex: 1;
gap: 0 10px;
}
.selector select {
height: 17.2em;
flex: 1 0 auto;
overflow: scroll;
width: 100%;
}
.selector-available, .selector-chosen {
display: flex;
flex-direction: column;
flex: 1 1;
}
.selector-available-title, .selector-chosen-title {
border: 1px solid var(--border-color);
border-radius: 4px 4px 0 0;
}
.selector .helptext {
font-size: 0.6875rem;
}
.selector-chosen .list-footer-display {
border: 1px solid var(--border-color);
border-top: none;
border-radius: 0 0 4px 4px;
margin: 0 0 10px;
padding: 8px;
text-align: center;
background: var(--primary);
color: var(--header-link-color);
cursor: pointer;
}
.selector-chosen .list-footer-display__clear {
color: var(--breadcrumbs-fg);
}
.selector-chosen-title {
background: var(--secondary);
color: var(--header-link-color);
padding: 8px;
}
.aligned .selector-chosen-title label {
color: var(--header-link-color);
width: 100%;
}
.selector-available-title {
background: var(--darkened-bg);
color: var(--body-quiet-color);
padding: 8px;
}
.aligned .selector-available-title label {
width: 100%;
}
.selector .selector-filter {
border: 1px solid var(--border-color);
border-width: 0 1px;
padding: 8px;
color: var(--body-quiet-color);
font-size: 0.625rem;
margin: 0;
text-align: left;
display: flex;
gap: 8px;
}
.selector .selector-filter label,
.inline-group .aligned .selector .selector-filter label {
float: left;
margin: 7px 0 0;
width: 18px;
height: 18px;
padding: 0;
overflow: hidden;
line-height: 1;
min-width: auto;
}
.selector-filter input {
flex-grow: 1;
}
.selector ul.selector-chooser {
align-self: center;
width: 30px;
background-color: var(--selected-bg);
border-radius: 10px;
margin: 0;
padding: 0;
transform: translateY(-17px);
}
.selector-chooser li {
margin: 0;
padding: 3px;
list-style-type: none;
}
.selector select {
padding: 0 10px;
margin: 0 0 10px;
border-radius: 0 0 4px 4px;
}
.selector .selector-chosen--with-filtered select {
margin: 0;
border-radius: 0;
height: 14em;
}
.selector .selector-chosen:not(.selector-chosen--with-filtered) .list-footer-display {
display: none;
}
.selector-add, .selector-remove {
width: 24px;
height: 24px;
display: block;
text-indent: -3000px;
overflow: hidden;
cursor: default;
opacity: 0.55;
border: none;
}
:enabled.selector-add, :enabled.selector-remove {
opacity: 1;
}
:enabled.selector-add:hover, :enabled.selector-remove:hover {
cursor: pointer;
}
.selector-add {
background: url(../img/selector-icons.svg) 0 -144px no-repeat;
background-size: 24px auto;
}
:enabled.selector-add:focus, :enabled.selector-add:hover {
background-position: 0 -168px;
}
.selector-remove {
background: url(../img/selector-icons.svg) 0 -96px no-repeat;
background-size: 24px auto;
}
:enabled.selector-remove:focus, :enabled.selector-remove:hover {
background-position: 0 -120px;
}
.selector-chooseall, .selector-clearall {
display: inline-block;
height: 16px;
text-align: left;
margin: 0 auto;
overflow: hidden;
font-weight: bold;
line-height: 16px;
color: var(--body-quiet-color);
text-decoration: none;
opacity: 0.55;
border: none;
}
:enabled.selector-chooseall:focus, :enabled.selector-clearall:focus,
:enabled.selector-chooseall:hover, :enabled.selector-clearall:hover {
color: var(--link-fg);
}
:enabled.selector-chooseall, :enabled.selector-clearall {
opacity: 1;
}
:enabled.selector-chooseall:hover, :enabled.selector-clearall:hover {
cursor: pointer;
}
.selector-chooseall {
padding: 0 18px 0 0;
background: url(../img/selector-icons.svg) right -160px no-repeat;
cursor: default;
}
:enabled.selector-chooseall:focus, :enabled.selector-chooseall:hover {
background-position: 100% -176px;
}
.selector-clearall {
padding: 0 0 0 18px;
background: url(../img/selector-icons.svg) 0 -128px no-repeat;
cursor: default;
}
:enabled.selector-clearall:focus, :enabled.selector-clearall:hover {
background-position: 0 -144px;
}
/* STACKED SELECTORS */
.stacked {
float: left;
width: 490px;
display: block;
}
.stacked select {
width: 480px;
height: 10.1em;
}
.stacked .selector-available, .stacked .selector-chosen {
width: 480px;
}
.stacked .selector-available {
margin-bottom: 0;
}
.stacked .selector-available input {
width: 422px;
}
.stacked ul.selector-chooser {
display: flex;
height: 30px;
width: 64px;
margin: 0 0 10px 40%;
background-color: #eee;
border-radius: 10px;
transform: none;
}
.stacked .selector-chooser li {
float: left;
padding: 3px 3px 3px 5px;
}
.stacked .selector-chooseall, .stacked .selector-clearall {
display: none;
}
.stacked .selector-add {
background: url(../img/selector-icons.svg) 0 -48px no-repeat;
background-size: 24px auto;
cursor: default;
}
.stacked :enabled.selector-add {
background-position: 0 -48px;
cursor: pointer;
}
.stacked :enabled.selector-add:focus, .stacked :enabled.selector-add:hover {
background-position: 0 -72px;
cursor: pointer;
}
.stacked .selector-remove {
background: url(../img/selector-icons.svg) 0 0 no-repeat;
background-size: 24px auto;
cursor: default;
}
.stacked :enabled.selector-remove {
background-position: 0 0px;
cursor: pointer;
}
.stacked :enabled.selector-remove:focus, .stacked :enabled.selector-remove:hover {
background-position: 0 -24px;
cursor: pointer;
}
.selector .help-icon {
background: url(../img/icon-unknown.svg) 0 0 no-repeat;
display: inline-block;
vertical-align: middle;
margin: -2px 0 0 2px;
width: 13px;
height: 13px;
}
.selector .selector-chosen .help-icon {
background: url(../img/icon-unknown-alt.svg) 0 0 no-repeat;
}
.selector .search-label-icon {
background: url(../img/search.svg) 0 0 no-repeat;
display: inline-block;
height: 1.125rem;
width: 1.125rem;
}
/* DATE AND TIME */
p.datetime {
line-height: 20px;
margin: 0;
padding: 0;
color: var(--body-quiet-color);
font-weight: bold;
}
.datetime span {
white-space: nowrap;
font-weight: normal;
font-size: 0.6875rem;
color: var(--body-quiet-color);
}
.datetime input, .form-row .datetime input.vDateField, .form-row .datetime input.vTimeField {
margin-left: 5px;
margin-bottom: 4px;
}
table p.datetime {
font-size: 0.6875rem;
margin-left: 0;
padding-left: 0;
}
.datetimeshortcuts .clock-icon, .datetimeshortcuts .date-icon {
position: relative;
display: inline-block;
vertical-align: middle;
height: 24px;
width: 24px;
overflow: hidden;
}
.datetimeshortcuts .clock-icon {
background: url(../img/icon-clock.svg) 0 0 no-repeat;
background-size: 24px auto;
}
.datetimeshortcuts a:focus .clock-icon,
.datetimeshortcuts a:hover .clock-icon {
background-position: 0 -24px;
}
.datetimeshortcuts .date-icon {
background: url(../img/icon-calendar.svg) 0 0 no-repeat;
background-size: 24px auto;
top: -1px;
}
.datetimeshortcuts a:focus .date-icon,
.datetimeshortcuts a:hover .date-icon {
background-position: 0 -24px;
}
.timezonewarning {
font-size: 0.6875rem;
color: var(--body-quiet-color);
}
/* URL */
p.url {
line-height: 20px;
margin: 0;
padding: 0;
color: var(--body-quiet-color);
font-size: 0.6875rem;
font-weight: bold;
}
.url a {
font-weight: normal;
}
/* FILE UPLOADS */
p.file-upload {
line-height: 20px;
margin: 0;
padding: 0;
color: var(--body-quiet-color);
font-size: 0.6875rem;
font-weight: bold;
}
.file-upload a {
font-weight: normal;
}
.file-upload .deletelink {
margin-left: 5px;
}
span.clearable-file-input label {
color: var(--body-fg);
font-size: 0.6875rem;
display: inline;
float: none;
}
/* CALENDARS & CLOCKS */
.calendarbox, .clockbox {
margin: 5px auto;
font-size: 0.75rem;
width: 19em;
text-align: center;
background: var(--body-bg);
color: var(--body-fg);
border: 1px solid var(--hairline-color);
border-radius: 4px;
box-shadow: 0 2px 4px rgba(0, 0, 0, 0.15);
overflow: hidden;
position: relative;
}
.clockbox {
width: auto;
}
.calendar {
margin: 0;
padding: 0;
}
.calendar table {
margin: 0;
padding: 0;
border-collapse: collapse;
background: white;
width: 100%;
}
.calendar caption, .calendarbox h2 {
margin: 0;
text-align: center;
border-top: none;
font-weight: 700;
font-size: 0.75rem;
color: #333;
background: var(--accent);
}
.calendar th {
padding: 8px 5px;
background: var(--darkened-bg);
border-bottom: 1px solid var(--border-color);
font-weight: 400;
font-size: 0.75rem;
text-align: center;
color: var(--body-quiet-color);
}
.calendar td {
font-weight: 400;
font-size: 0.75rem;
text-align: center;
padding: 0;
border-top: 1px solid var(--hairline-color);
border-bottom: none;
}
.calendar td.selected a {
background: var(--secondary);
color: var(--button-fg);
}
.calendar td.nonday {
background: var(--darkened-bg);
}
.calendar td.today a {
font-weight: 700;
}
.calendar td a, .timelist a {
display: block;
font-weight: 400;
padding: 6px;
text-decoration: none;
color: var(--body-quiet-color);
}
.calendar td a:focus, .timelist a:focus,
.calendar td a:hover, .timelist a:hover {
background: var(--primary);
color: white;
}
.calendar td a:active, .timelist a:active {
background: var(--header-bg);
color: white;
}
.calendarnav {
font-size: 0.625rem;
text-align: center;
color: #ccc;
margin: 0;
padding: 1px 3px;
}
.calendarnav a:link, #calendarnav a:visited,
#calendarnav a:focus, #calendarnav a:hover {
color: var(--body-quiet-color);
}
.calendar-shortcuts {
background: var(--body-bg);
color: var(--body-quiet-color);
font-size: 0.6875rem;
line-height: 0.6875rem;
border-top: 1px solid var(--hairline-color);
padding: 8px 0;
}
.calendarbox .calendarnav-previous, .calendarbox .calendarnav-next {
display: block;
position: absolute;
top: 8px;
width: 15px;
height: 15px;
text-indent: -9999px;
padding: 0;
}
.calendarnav-previous {
left: 10px;
background: url(../img/calendar-icons.svg) 0 0 no-repeat;
}
.calendarnav-next {
right: 10px;
background: url(../img/calendar-icons.svg) 0 -15px no-repeat;
}
.calendar-cancel {
margin: 0;
padding: 4px 0;
font-size: 0.75rem;
background: var(--close-button-bg);
border-top: 1px solid var(--border-color);
color: var(--button-fg);
}
.calendar-cancel:focus, .calendar-cancel:hover {
background: var(--close-button-hover-bg);
}
.calendar-cancel a {
color: var(--button-fg);
display: block;
}
ul.timelist, .timelist li {
list-style-type: none;
margin: 0;
padding: 0;
}
.timelist a {
padding: 2px;
}
/* EDIT INLINE */
.inline-deletelink {
float: right;
text-indent: -9999px;
background: url(../img/inline-delete.svg) 0 0 no-repeat;
width: 1.5rem;
height: 1.5rem;
border: 0px none;
margin-bottom: .25rem;
}
.inline-deletelink:focus, .inline-deletelink:hover {
cursor: pointer;
}
/* RELATED WIDGET WRAPPER */
.related-widget-wrapper {
display: flex;
gap: 0 10px;
flex-grow: 1;
flex-wrap: wrap;
margin-bottom: 5px;
}
.related-widget-wrapper-link {
opacity: .6;
filter: grayscale(1);
}
.related-widget-wrapper-link:link {
opacity: 1;
filter: grayscale(0);
}
/* GIS MAPS */
.dj_map {
width: 600px;
height: 400px;
}
+20
View File
@@ -0,0 +1,20 @@
The MIT License (MIT)
Copyright (c) 2014 Code Charm Ltd
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Some files were not shown because too many files have changed in this diff Show More