fix: 修复 xwlb 日期错位与重复行问题(方案 A)
- run_xwlb: 新增 --date(处理日)参数,落盘改为处理日目录,下游 M2-M6 打通
- run_xwlb: 幂等落盘(按 url_hash 去重,重写 index 替代裸追加)
- pipeline: xwlb 步骤传 --date {date}
- run_extractor: xwlb publish_time 从 url 兜底解析真实播出日
- 新增 tests/test_run_xwlb.py 9 个测试
- 清理 60 天历史死数据(用户决策 A1)
- docs: 文档重构收尾(删除 deployment.md,已在服务器直接修改)
This commit is contained in:
@@ -62,7 +62,7 @@ STEP_TIMEOUTS: dict[str, int] = {
|
||||
# crawler 不支持 --date,固定写当天目录; dedup 不加 --reset 以保持增量
|
||||
STEP_COMMANDS: dict[str, list[str]] = {
|
||||
"crawler": ["uv", "run", "python", "-m", "scripts.run_crawler"], # 无 --date
|
||||
"xwlb": ["uv", "run", "python", "-m", "scripts.run_xwlb"],
|
||||
"xwlb": ["uv", "run", "python", "-m", "scripts.run_xwlb", "--date", "{date}"], # 处理日目录
|
||||
"extractor": ["uv", "run", "python", "-m", "scripts.run_extractor", "--date", "{date}"],
|
||||
"dedup": ["uv", "run", "python", "-m", "scripts.run_dedup", "--date", "{date}"],
|
||||
"llm": ["uv", "run", "python", "-m", "scripts.run_event_extraction", "--date", "{date}"],
|
||||
|
||||
Reference in New Issue
Block a user