simon
65ead54b4f
- 删除 5 个过时/残留文档(project_plan/agent_prompt/optimization_plan/report_db_design/deploy/README) - 新建 docs/architecture.md(项目架构:11 包职责+数据模型+配置+产物) - 重写 docs/user-guide.md(CLI 全量+增量/断点续跑+MCP+FAQ) - 重写 README.md(精简入口+文档索引) - 更新 continuation.md(追加本次记录) - 更新 .gitignore(排除 data/* 运行产物)
33 lines
744 B
Python
33 lines
744 B
Python
"""新闻抓取模块 (M1)。
|
|
|
|
公共 API:
|
|
- load_crawler_config: 加载 sources.yaml
|
|
- crawl_all: 一键抓取所有启用的源
|
|
- crawl_source: 抓取单个源(供测试/调试)
|
|
- CrawlerConfig / SourceConfig / CrawlResult: 数据模型
|
|
"""
|
|
|
|
from .config import load_crawler_config
|
|
from .engine import crawl_all, crawl_source, extract_article_links
|
|
from .models import (
|
|
ArticleLink,
|
|
CrawlerConfig,
|
|
CrawlerSettings,
|
|
CrawlResult,
|
|
CrawlStage,
|
|
SourceConfig,
|
|
)
|
|
|
|
__all__ = [
|
|
"ArticleLink",
|
|
"CrawlResult",
|
|
"CrawlStage",
|
|
"CrawlerConfig",
|
|
"CrawlerSettings",
|
|
"SourceConfig",
|
|
"crawl_all",
|
|
"crawl_source",
|
|
"extract_article_links",
|
|
"load_crawler_config",
|
|
]
|