feat: 大模型使用场景化配置与去重多源记录
- 新增 configs/llm_models.yaml: 4 个场景(event_extraction/daily_report/stock_report/embedding)
可独立配置 provider/model/api_key_env/base_url_env/temperature 等,含用途与模型要求说明
- 新增 configs/loader.py: YAML 场景加载器(优先级: CLI 参数 > YAML > .env > 内置默认)
- llm/client.py: load_llm_config 支持 scene 参数,LLMConfig 增加 max_attempts
- embedding/factory+remote+local: provider/model/batch_limit 支持场景覆盖
- scheduler/reporter+stock_reporter: 日报/个股摘要接入场景配置
- dedup: Fingerprint.source_ids 多源记录 + 旧库自动迁移 + DedupResult 多源字段
- scripts/run_dedup: uniques JSON 的 sources 字段 + data/deduped/{day}/sources.json 汇总
- scripts/run_event_extraction: 接入 event_extraction 场景
- 补充测试: 场景优先级/零值、多源合并、旧库迁移、embedding 场景覆盖
This commit is contained in:
+5
-1
@@ -21,6 +21,8 @@ from .models import EmbeddingError
|
||||
if TYPE_CHECKING:
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
from configs.loader import load_scene_config
|
||||
|
||||
LOCAL_DEFAULT_MODEL = "BAAI/bge-m3"
|
||||
LOCAL_DEFAULT_DIM = 1024
|
||||
|
||||
@@ -56,10 +58,12 @@ class LocalBGEEmbeddingProvider(EmbeddingProvider):
|
||||
normalize: bool = True,
|
||||
) -> None:
|
||||
st_cls = _try_import_st()
|
||||
# 模型优先级:CLI/参数 > LOCAL_EMBEDDING_MODEL > 默认值
|
||||
# 模型优先级:CLI/参数 > YAML scenes.embedding > LOCAL_EMBEDDING_MODEL > 默认值
|
||||
# 不读全局 EMBEDDING_MODEL,避免与 DashScope 冲突
|
||||
scene_model = load_scene_config("embedding").get("model")
|
||||
self.model = (
|
||||
model
|
||||
or scene_model
|
||||
or _read_env("LOCAL_EMBEDDING_MODEL", LOCAL_DEFAULT_MODEL)
|
||||
or LOCAL_DEFAULT_MODEL
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user