Compare commits
42 Commits
291aa0b4b7
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
| ba73b30a04 | |||
| c377264610 | |||
| 4d6f499508 | |||
| 966f2e1270 | |||
| e3eab5ade1 | |||
| aa31243bae | |||
| f5729d5a18 | |||
| 543c3ff906 | |||
| 74df53889c | |||
| bb8ea931b5 | |||
| f93a6c5c68 | |||
| 9a3a34c58b | |||
| 7c49df94fc | |||
| 8f4727a772 | |||
| eb02fd32a5 | |||
| f0ce8dc37e | |||
| 14c20a6123 | |||
| bdc7bd5069 | |||
| f50dc1aed0 | |||
| 4b81b83fd8 | |||
| 67370657db | |||
| 8ee0131d47 | |||
| 65bf2346a3 | |||
| 03a22b0c1c | |||
| 7d92b3f1fe | |||
| b51ed0175b | |||
| 0812489f9d | |||
| 4c380f1157 | |||
| 2c09c61165 | |||
| f9f175ee79 | |||
| 32fa374e86 | |||
| 6ae54e3edd | |||
| 841986740c | |||
| 84b3b7652a | |||
| a677220f4d | |||
| 758b1c4f77 | |||
| 0ae2db1808 | |||
| e786bd5bf0 | |||
| a22c527daf | |||
| bd81c6dae6 | |||
| b9fbf357a4 | |||
| 1f68bd5ca8 |
@@ -96,3 +96,4 @@ run_profiles/*.yml
|
||||
_runtime/
|
||||
test_results/
|
||||
*.code-workspace
|
||||
runtime/
|
||||
@@ -2,6 +2,42 @@
|
||||
|
||||
可作为命令行工具运行,也可作为 `pip install -e .` 的本地可编辑安装库接入其他项目。
|
||||
|
||||
## 0. 10 分钟上手
|
||||
|
||||
如果你是第一次进入这个仓库,先不要从所有文档开始读,先确认环境和最小工具链可用。
|
||||
|
||||
推荐顺序:
|
||||
1. 准备 Python 3.9+ 环境(当前仓库已在 `pyproject.toml` 中声明 `requires-python >= 3.9`)。
|
||||
2. 在仓库根目录创建并激活虚拟环境。
|
||||
3. 以可编辑模式安装本项目。
|
||||
4. 先跑配置校验,再跑一个轻量集成测试,确认本地环境可用。
|
||||
5. 只在确认入口可运行后,再去看架构与状态机文档。
|
||||
|
||||
最小命令链:
|
||||
|
||||
```bash
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -e .
|
||||
python run.py --help
|
||||
python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml
|
||||
pytest tests/integration/test_toolchain_config_and_graph.py -q
|
||||
```
|
||||
|
||||
如果你只是想直接运行仓库内现成配置,优先用仓库入口:
|
||||
|
||||
```bash
|
||||
python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
|
||||
```
|
||||
|
||||
如果你已经完成 `pip install -e .`,也可以使用安装后的命令行入口:
|
||||
|
||||
```bash
|
||||
ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
|
||||
```
|
||||
|
||||
面向第一次接触仓库的更完整说明见:`docs/getting_started.md`
|
||||
|
||||
## 1. 项目简介
|
||||
|
||||
本项目用于实现异构系统之间的数据推送:将项目侧管理平台的十余种业务数据,稳定推送到集团侧管理平台。
|
||||
@@ -18,6 +54,7 @@
|
||||
项目采用:`schema` 严格约束 + 状态机 + 节点模型 + 分层架构 + 注册式 `domain` + 数据源与同步系统解耦。
|
||||
|
||||
架构详见:
|
||||
- `docs/getting_started.md`
|
||||
- `docs/architecture.md`
|
||||
- `docs/node_state_definition.md`
|
||||
- `docs/state_machine.md`
|
||||
@@ -58,15 +95,26 @@
|
||||
|
||||
## 5. 使用方法
|
||||
|
||||
先区分两种常见使用方式:
|
||||
- 仓库内开发与调试:优先使用 `python run.py ...`,因为它不依赖 shell 中已经存在 `ecm-sync-run`。
|
||||
- 作为库或已安装命令使用:先执行 `pip install -e .`,再使用 `ecm-sync-run ...`。
|
||||
|
||||
运行配置遵循一个约定:
|
||||
- 跟踪在仓库里的 `run_profiles/*.yaml` 与 `run_profiles/preset/*.default.yaml` 只写必要的覆盖项。
|
||||
- pipeline 启动时会打印并写入一份 **Resolved Full Config**,把省略的默认值全部补齐,便于排障和审阅。
|
||||
- 字段含义与全量示例见:`docs/runtime_config_guide.md`
|
||||
|
||||
### JSONL 同步
|
||||
- 基线配置(优先):`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
|
||||
- 兼容旧入口:`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
|
||||
- 使用自定义配置:先从 `run_profiles/preset/jsonl_to_jsonl.default.yaml` 复制到 `run_profiles/jsonl_to_jsonl.local.yaml`,按需修改后运行:`ecm-sync-run --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
|
||||
- 仓库内直接运行:`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
|
||||
- 已安装命令行入口:`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
|
||||
- 多项目 JSONL 示例:`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.multi_project.default.yaml`
|
||||
- 当前已经支持多 project 运行;标准写法是直接使用多项目编排配置文件,或在支持 implicit multi-project 的 profile 中由配置本身决定运行模式。
|
||||
- 自定义配置:从 `run_profiles/preset/` 复制一份到 `run_profiles/*.local.yaml`,只改有差异的字段即可。
|
||||
|
||||
### API 同步
|
||||
- 基线配置:`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_api.default.yaml`
|
||||
- 兼容旧入口:`python run.py --config_path=run_profiles/preset/jsonl_to_api.default.yaml`
|
||||
- 使用自定义配置:先从 `run_profiles/preset/jsonl_to_api.default.yaml` 复制到 `run_profiles/jsonl_to_api.local.yaml`,按需修改后运行:`ecm-sync-run --config_path=run_profiles/jsonl_to_api.local.yaml`
|
||||
- 仓库内直接运行:`python run.py --config_path=run_profiles/preset/jsonl_to_api.default.yaml`
|
||||
- 已安装命令行入口:`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_api.default.yaml`
|
||||
- 如果是单项目联调,优先参考仓库内现成示例:`run_profiles/jsonl_to_api.yaml`
|
||||
|
||||
### 库模式使用
|
||||
- 本地可编辑安装:`pip install -e .`
|
||||
@@ -75,19 +123,34 @@
|
||||
### 配置与校验
|
||||
- 模板配置:`run_profiles/preset/*.default.yaml`
|
||||
- 建议先跑 default,再在 `run_profiles/*.local.yaml` 做覆盖(该目录 YAML 已默认忽略)
|
||||
- 跟踪中的模板与示例配置都采用“只写必要字段”的风格;启动日志里的 `Resolved Full Config` 才是最终生效配置。
|
||||
- 状态机配置校验:`python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml`
|
||||
- 状态机图生成:`python tools/render_graph.py --format mermaid --out docs/state_machine.mmd`
|
||||
|
||||
### 测试
|
||||
- 全量:`pytest tests -q`
|
||||
- 说明:`tests/README.md`
|
||||
- 远程环境初始化工具:`python tools/remote_env.py --config tools/remote_env.example.yaml init|run|stop|clear`
|
||||
- 测试总览:`docs/testing_guide.md`
|
||||
|
||||
### 其他工具
|
||||
- UI 调试:`python -m ui.main --host 127.0.0.1 --port 8765`
|
||||
- 运维与排障:`docs/operations.md`
|
||||
- 远程测试环境:`tools/remote_env.py`
|
||||
- 脚本化回归与数据集构建:`scripts/README.md`
|
||||
|
||||
## 文档导航
|
||||
|
||||
如果你是按任务查文档,建议按下面的顺序进入:
|
||||
- 第一次接手仓库:`docs/getting_started.md`
|
||||
- 想理解整体分层:`docs/architecture.md`
|
||||
- 想理解状态与迁移:`docs/node_state_definition.md` + `docs/state_machine.md`
|
||||
- 想调运行配置:`docs/runtime_config_guide.md` + `run_profiles/README.md`(包含多项目配置写法)
|
||||
- 想接新数据源或改 handler:`sync_state_machine/datasource/README.md` + `docs/library_integration_guide.md`
|
||||
- 想排障:`docs/operations.md`
|
||||
- 想补测试或跑联调:`docs/testing_guide.md` + `tests/README.md`
|
||||
|
||||
- 新人上手:`docs/getting_started.md`
|
||||
- 总体架构:`docs/architecture.md`
|
||||
- 状态定义:`docs/node_state_definition.md`
|
||||
- 状态机规则:`docs/state_machine.md`
|
||||
@@ -96,5 +159,6 @@
|
||||
- 库化与接入:`docs/library_integration_guide.md`
|
||||
- Collection 查询与升级:`docs/collection_query_and_upgrade_plan.md`
|
||||
- 运维迭代:`docs/operations.md`
|
||||
- 测试指南:`docs/testing_guide.md`
|
||||
|
||||
历史资料已归档到 `docs/archive/`。
|
||||
|
||||
+14
-14
@@ -3,11 +3,11 @@
|
||||
## 1. 分层职责(pipeline + strategy + datasource + domain)
|
||||
|
||||
### pipeline(编排层)
|
||||
- 入口:`pipeline/full_sync_pipeline.py`
|
||||
- 入口:`sync_state_machine/pipeline/full_sync_pipeline.py`
|
||||
- 负责组装并驱动整条流程:加载持久化、执行 bind/create/update、调用 datasource 同步、持久化结果。
|
||||
- reset 语义:
|
||||
- 加载阶段仅恢复持久化状态(不做状态重置);
|
||||
- 加载后通过 `BaseSyncStrategy.run_reset()` 触发 `E01`:
|
||||
- 加载后通过 `run_phase2_cleanup()` 触发 `E01`:
|
||||
- `create_zombie=true` -> `S15`(随后删除);
|
||||
- `create_zombie=false` -> `S00`(归并为 `UNCHECKED/NONE/PENDING`,并清空 `error`)。
|
||||
- 若持久化核心枚举非法(`action/status/binding_status`),节点进入加载异常隔离态 `S17`:
|
||||
@@ -24,29 +24,29 @@
|
||||
- 非僵尸节点统一进入 `S00`,再进入本轮 bind/create/update 判定。
|
||||
|
||||
### strategy(决策组织层)
|
||||
- 外部入口:`sync_system/strategy.py`
|
||||
- 具体实现下沉:`sync_system/strategy_ops/`
|
||||
- `bind_ops.py`: `run_bind()` + `phase_core_state()/phase_dependency_check()/phase_auto_binding()`
|
||||
- `create_ops.py`: `run_create()`
|
||||
- `update_ops.py`: `run_update()`
|
||||
- 外部入口:`sync_state_machine/sync_system/strategy.py`
|
||||
- 具体实现下沉:`sync_state_machine/sync_system/strategy_ops/`
|
||||
- `bind_ops.py`: bind 阶段静态函数与纯工具
|
||||
- `create_ops.py`: create 阶段静态函数与纯工具
|
||||
- `update_ops.py`: update 阶段静态函数与纯工具
|
||||
- `delete_ops.py`: `run_delete()`
|
||||
- `reset_ops.py`: `get_phase1_reset_defaults()/run_phase2_cleanup()`
|
||||
- strategy 负责收集上下文并调用状态机事件函数,不直接硬编码状态值。
|
||||
- `reset_ops.py`: `run_phase2_cleanup()`
|
||||
- strategy 负责 bind/create/update 的相位内部编排,并调用状态机事件函数,不直接硬编码状态值。
|
||||
|
||||
### datasource(执行与回写层)
|
||||
- 入口:`datasource/datasource.py`
|
||||
- 入口:`sync_state_machine/datasource/datasource.py`
|
||||
- 执行 handler(create/update/delete),并结合 `E40` 将执行结果落地到节点状态。
|
||||
|
||||
### domain(业务适配层)
|
||||
- 目录:`domain/*`
|
||||
- 目录:`sync_state_machine/domain/*`
|
||||
- 负责业务字段映射、handler 细节与 schema 约束,不承担状态迁移判定。
|
||||
|
||||
## 2. 运行时契约(Runtime Contract)
|
||||
|
||||
### 真源
|
||||
- 状态机配置真源:`config/node_state_machine.yaml`
|
||||
- 事件执行入口:`engine/events.py`
|
||||
- 状态落地入口:`engine/state_apply.py`
|
||||
- 状态机配置真源:`sync_state_machine/config/node_state_machine.yaml`
|
||||
- 事件执行入口:`sync_state_machine/engine/events.py`
|
||||
- 状态落地入口:`sync_state_machine/engine/state_apply.py`
|
||||
|
||||
### 执行规则
|
||||
1. 业务层构造 `context`。
|
||||
|
||||
@@ -38,7 +38,7 @@ JSONL 数据源对应实现:
|
||||
- 从目录中按 `{node_type}_N.jsonl` 规则发现并加载数据
|
||||
- 批量 create/update/delete 时,先改内存缓存,再统一刷盘
|
||||
- 支持 `read_only`,在只读模式下禁止写回
|
||||
- 支持按项目过滤(`target_project_ids`)在 load 阶段预过滤
|
||||
- 支持按项目过滤(`handler_configs.project.data_id_filter`)在 load 阶段预过滤
|
||||
|
||||
### 2.2 行为特点
|
||||
|
||||
@@ -53,7 +53,7 @@ JSONL 数据源对应实现:
|
||||
- `type: jsonl`
|
||||
- `jsonl_dir`: 数据目录
|
||||
- `read_only`: 是否只读
|
||||
- `target_project_ids`: 项目过滤列表
|
||||
- `handler_configs.project.data_id_filter`: 项目过滤列表
|
||||
- `handler_configs`: 按节点类型的 handler 参数
|
||||
|
||||
---
|
||||
@@ -131,8 +131,8 @@ API 数据源中,校验分为三层:
|
||||
- 例如 `ContractCreate.model_validate(...)`、`ContractUpdate.model_validate(...)`
|
||||
- load 响应也会在解析后转为领域 schema
|
||||
2. 通用更新过滤与差异识别
|
||||
- `_filter_update_data`:仅在 schema 相关字段有变化时提交更新
|
||||
- `_get_schema_diff`:用于差异字段识别与调试输出
|
||||
- `build_update_request_data`:update 入口,仅在 schema 相关字段有变化时提交更新
|
||||
- `get_update_schema_diff`:update 入口,用于差异字段识别与调试输出
|
||||
3. 轮询接口输入校验
|
||||
- `PushIdsSchema` 校验 push_ids 格式后再请求 `/push/result`
|
||||
|
||||
@@ -159,8 +159,9 @@ local_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
|
||||
read_only: false
|
||||
target_project_ids: []
|
||||
handler_configs: {}
|
||||
handler_configs:
|
||||
project:
|
||||
data_id_filter: []
|
||||
```
|
||||
|
||||
### 4.2 API 数据源配置示例(含限流)
|
||||
@@ -181,9 +182,10 @@ remote_datasource:
|
||||
api_rate_limit_max_requests: 10
|
||||
api_rate_limit_window_seconds: 10.0
|
||||
|
||||
target_project_ids:
|
||||
- "project-id-1"
|
||||
handler_configs: {}
|
||||
handler_configs:
|
||||
project:
|
||||
data_id_filter:
|
||||
- "project-id-1"
|
||||
```
|
||||
|
||||
### 4.3 配置模型位置
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
# ECM Sync Follow-up Plan
|
||||
|
||||
## 目标
|
||||
|
||||
本轮先落两件事:
|
||||
|
||||
1. 项目聚合读取接口,降低同步读取请求数。
|
||||
2. 增强现有 validate,让 strategy 可以直接复用 schema diff validator,并输出聚合日志。
|
||||
|
||||
第三件事作为下一步:在 backend 里为 data mapping 增加静态样本测试,绕开 pipeline,直接验证本地样本和远程样本在 sync schema 层是否足够一致。
|
||||
|
||||
## 当前结论
|
||||
|
||||
### 1. 项目聚合读取接口
|
||||
|
||||
现有按业务拆散的读取方式请求数过高,不适合大规模同步。
|
||||
|
||||
本轮改为提供统一项目聚合接口,最小聚合范围:
|
||||
|
||||
- `project`
|
||||
- `project_detail_data`
|
||||
- `preparation`
|
||||
- `production`
|
||||
- `contract`
|
||||
|
||||
目标:
|
||||
|
||||
- 以项目为同步分片单位。
|
||||
- 减少绝大多数项目级 fan-out 请求。
|
||||
- 保持写入侧业务流程不变,只优化读取侧接口粒度。
|
||||
|
||||
接口返回的数据仍然基于现有 service 组装,不新增业务含义。
|
||||
|
||||
### 2. validate 增强方案
|
||||
|
||||
现有 validate 主要停留在 schema 校验,缺少可读的聚合 diff 输出。
|
||||
|
||||
本轮新增 schema diff validator,原则:
|
||||
|
||||
- 仍以现有 sync schema 为比较中心。
|
||||
- 支持按 schema 聚合。
|
||||
- 支持忽略字段配置。
|
||||
- 支持输出字段级聚合统计。
|
||||
- 可以被 strategy 直接引用,也可以脱离 pipeline 单独调用。
|
||||
|
||||
建议输出格式:
|
||||
|
||||
- `field_name`
|
||||
- `mismatch_count / total_count`
|
||||
- `schema_refs`
|
||||
- `sample local/remote`
|
||||
|
||||
例如:
|
||||
|
||||
```text
|
||||
ProjectDiff
|
||||
code 12/30 [ProjectResponseBase, ProjectBaseInfoUpdate] sample[project-1]=""/"012A"
|
||||
```
|
||||
|
||||
### 3. data mapping 静态测试
|
||||
|
||||
下一步在 depm backend 中做静态 mapping 测试,目标不是验证整条 pipeline,而是验证 mapping 本身。
|
||||
|
||||
测试输入:
|
||||
|
||||
- 本地数据静态文件
|
||||
- 远程数据静态文件
|
||||
|
||||
要求:
|
||||
|
||||
- 本地样本通过本地 `data_mapping` 转成 sync schema。
|
||||
- 远程样本直接按 sync schema 校验。
|
||||
- 忽略掉少量必须动态生成或动态获取的字段。
|
||||
- 通过统一 validator 输出字段级 diff 报告。
|
||||
|
||||
这类测试应当绕开推送系统,通过手工配对样本来验证 mapping,而不是混在 pipeline 里排查。
|
||||
|
||||
## 已落地项
|
||||
|
||||
### 项目聚合接口
|
||||
|
||||
backend 中新增项目同步聚合读取接口,按项目统一返回:
|
||||
|
||||
- `project`
|
||||
- `project_detail_data`
|
||||
- `preparation`
|
||||
- `production`
|
||||
- `contract`
|
||||
|
||||
### schema diff validator
|
||||
|
||||
同步引擎新增可独立调用的 schema diff validator:
|
||||
|
||||
- 支持忽略顶层字段
|
||||
- 支持忽略列表元素内特定子字段
|
||||
- 支持输出字段级 mismatch 聚合统计
|
||||
- strategy 在 update diff 阶段直接复用该 validator
|
||||
|
||||
## 下一步
|
||||
|
||||
### 先做
|
||||
|
||||
1. 为 `project` 建第一批静态样本。
|
||||
2. 用统一 validator 对本地样本和远程样本做 schema 层比较。
|
||||
3. 只在 validator 配置里维护少量 ignore 字段,不额外引入 contract 文件。
|
||||
|
||||
### 暂不做
|
||||
|
||||
1. 不引入 schema 外的独立契约系统。
|
||||
2. 不为所有 domain 一次性补齐静态样本。
|
||||
3. 不把副作用字段强行纳入完全一致比较。
|
||||
@@ -0,0 +1,258 @@
|
||||
# Construction 领域重构指南
|
||||
|
||||
## 1. 范围
|
||||
|
||||
本文档应覆盖 construction 领域中两部分字段的并集:
|
||||
|
||||
- 集团侧 `enum_config` 中 construction 相关的全部字段
|
||||
- depm 中所有已经使用代码 enum、代码中的初始化列表、硬编码 dict、类 enum、`config` 表、`dict` 表承载,并且会影响施工初始化、手动创建、手动编辑、接口展示或业务校验的相关字段
|
||||
|
||||
当前这份文档先给出 construction 领域的总范围和主线判断,再把本轮优先落地对象收敛到一条主线:
|
||||
|
||||
- `construction.section_type_code`
|
||||
|
||||
以下复杂问题本轮先不展开具体迁移方案,但仍属于 construction 领域后续必须补齐的梳理范围:
|
||||
|
||||
- `task_id`
|
||||
- `task_type`
|
||||
- 其它施工节点层级和映射规则特别复杂的问题
|
||||
|
||||
也就是说,施工文档不是永久只看 section,而是本轮先把 section 这一层讲清楚,同时把 task 相关复杂问题明确记为后续梳理项,而不是视为领域外问题。
|
||||
|
||||
## 2. 当前现状
|
||||
|
||||
construction 领域当前至少有两类配置来源:
|
||||
|
||||
### 2.1 `construction_section_config`
|
||||
|
||||
当前按项目类型配置 section 信息,例如光伏下的:
|
||||
|
||||
- `code`
|
||||
- `name`
|
||||
- `weight`
|
||||
- `project_root_nodes`
|
||||
|
||||
这份配置同时承担:
|
||||
|
||||
- section 候选项来源
|
||||
- 施工初始化入口信息来源
|
||||
|
||||
### 2.2 `construction_node_config`
|
||||
|
||||
当前按项目类型配置节点信息,例如:
|
||||
|
||||
- `task_code`
|
||||
- `type_id`
|
||||
- `type_name`
|
||||
- `unit`
|
||||
- `weight`
|
||||
- `is_primary`
|
||||
- `section`
|
||||
|
||||
这份配置描述的是节点补充信息,不是字段级配置。
|
||||
|
||||
## 3. 重构目标
|
||||
|
||||
construction 领域本轮的目标不是重写全部施工配置,而是:
|
||||
|
||||
- 把 section 候选项迁到新的字段级配置
|
||||
- 保留 `construction_node_config` 作为自由业务配置
|
||||
- 让施工初始化和手动校验可以逐步复用统一的字段级入口
|
||||
|
||||
## 4. `construction.section_type_code` 的新结构
|
||||
|
||||
`construction_section_config` 应改造成新的字段级 key:
|
||||
|
||||
- `construction.section_type_code`
|
||||
|
||||
它的 JSON 中必须包含:
|
||||
|
||||
- `field_config`
|
||||
|
||||
并且可以额外包含业务自定义字段,例如:
|
||||
|
||||
- `description`
|
||||
- `init_config`
|
||||
|
||||
推荐结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"description": "施工区字段配置",
|
||||
"field_config": [
|
||||
{"value": "GF", "label": "光伏场区", "weight": 1.4},
|
||||
{"value": "SY", "label": "升压站", "weight": 1.4},
|
||||
{"value": "SC", "label": "送出(自建)", "weight": 1.2},
|
||||
{"value": "SCDW", "label": "送出(电网建设)", "weight": 1.0}
|
||||
],
|
||||
"init_config": {
|
||||
"root_nodes_by_value": {
|
||||
"GF": [
|
||||
{"type_id": "01", "type_code": "01", "type_name": "光伏区土建工程"},
|
||||
{"type_id": "03", "type_code": "03", "type_name": "光伏区安装工程"}
|
||||
],
|
||||
"SY": [
|
||||
{"type_id": "02", "type_code": "02", "type_name": "升压站土建工程"}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
这里的原则是:
|
||||
|
||||
- `field_config` 负责 section 候选项
|
||||
- `init_config` 负责 section 相关初始化附属数据
|
||||
- 两者可以同处一个 JSON 中,不需要再拆第二份 section 配置
|
||||
|
||||
## 5. `construction_node_config` 的定位
|
||||
|
||||
`construction_node_config` 不迁到字段级配置。
|
||||
|
||||
它仍然是自由业务配置,负责描述:
|
||||
|
||||
- 可能节点有哪些种类
|
||||
- 哪些节点是主节点
|
||||
- 节点单位、权重等补充信息
|
||||
|
||||
建议只做一项统一性改造:
|
||||
|
||||
- 顶层补 `description`
|
||||
|
||||
## 6. 需要迁移到 `field_config` 的功能
|
||||
|
||||
construction 领域应重点梳理下面两类功能:
|
||||
|
||||
### 6.1 项目初始化链路
|
||||
|
||||
需要明确:
|
||||
|
||||
- 初始化 section 时,候选项是否改由 `get_field_options(project_type, "construction.section_type_code")` 提供
|
||||
- 初始化 root node 时,是否从同一配置对象中的业务自定义字段读取附属数据
|
||||
|
||||
也就是说,初始化链路应逐步从“直接解释旧 `construction_section_config`”迁移到“先读取字段级配置,再读取其业务附属字段”。
|
||||
|
||||
### 6.2 手动创建 / 编辑链路
|
||||
|
||||
需要明确:
|
||||
|
||||
- 如果某接口允许手动传 `section_type_code`
|
||||
- 是否应改为调用 `validate_field_value(project_type, "construction.section_type_code", value)`
|
||||
|
||||
这样可以保证手动写入与初始化使用同一套候选项来源。
|
||||
|
||||
## 7. 当前迁移示例
|
||||
|
||||
### 示例 1:section 候选项读取
|
||||
|
||||
当前:
|
||||
|
||||
- 通过 construction 私有 helper 直接读取并解释 `construction_section_config`
|
||||
|
||||
目标:
|
||||
|
||||
- 通过 `ConfigService.get_field_options(project_type, "construction.section_type_code")` 获取 section 列表
|
||||
|
||||
### 示例 2:section 合法值校验
|
||||
|
||||
当前:
|
||||
|
||||
- 可能由业务代码自行判断,或者根本没有统一校验入口
|
||||
|
||||
目标:
|
||||
|
||||
- 统一走 `ConfigService.validate_field_value(...)`
|
||||
|
||||
### 示例 3:初始化附属数据读取
|
||||
|
||||
当前:
|
||||
|
||||
- 直接从旧 `construction_section_config` 读取 `project_root_nodes`
|
||||
|
||||
目标:
|
||||
|
||||
- 从 `construction.section_type_code` 的完整配置对象中读取业务自定义字段
|
||||
|
||||
## 8. 梳理清单
|
||||
|
||||
construction 领域后续梳理时,至少要回答下面这些问题:
|
||||
|
||||
1. 哪些字段属于集团侧 `enum_config` 已覆盖或可对应的字段。
|
||||
2. 哪些字段在初始化时使用候选项,哪些字段在手动创建时也要校验。
|
||||
3. 哪些逻辑仍在直接解释旧 JSON,而不是通过统一 helper 读取。
|
||||
4. 哪些逻辑仍在使用硬编码白名单。
|
||||
5. `construction_node_config` 中哪些数据只是节点补充信息,哪些其实已经在承担字段候选项职责。
|
||||
|
||||
如果发现某字段既参与初始化又参与手动创建,就应优先考虑迁到字段级配置。
|
||||
|
||||
## 9. 按本轮要求的改造落点
|
||||
|
||||
construction 领域这轮不是泛泛地“改配置”,而是明确围绕 3 条链路收口:
|
||||
|
||||
- 初始化
|
||||
- create 校验
|
||||
- 下拉列表
|
||||
|
||||
### 9.1 初始化
|
||||
|
||||
当前施工初始化本来就不是 dict 驱动,而是:
|
||||
|
||||
- `construction_section_config`
|
||||
- `ConstructionTypeTree`
|
||||
- `ConstructionTypeMapping`
|
||||
|
||||
因此本轮初始化改造的重点不是“把 dict 搬到 config”,而是:
|
||||
|
||||
- 把 `section_type_code` 的候选项读取统一收口到新的 config 工具
|
||||
- 保留类型树和 root node 等结构化数据,但不再让 section 候选项散落在私有 helper 里
|
||||
|
||||
### 9.2 Create 校验
|
||||
|
||||
当前 `ConstructionService.create()` / `create_for_scdw()` 已经在 service 开头做了较强校验:
|
||||
|
||||
- `section_type_code` 来自 `construction_section_config`
|
||||
- `section_sub_type_code` 来自 section config 和类型树映射
|
||||
- `type_id` 来自施工类型树或 SCDW 专用配置
|
||||
|
||||
本轮目标是:
|
||||
|
||||
- `section_type_code` 统一改成通过 `ConfigService.validate_field_value(project_type, "construction.section_type_code", value)` 校验
|
||||
- `section_sub_type_code`、`type_id` 继续走配置/类型树校验,不做降级
|
||||
|
||||
### 9.3 下拉列表
|
||||
|
||||
当前施工下拉接口已经基本同源:
|
||||
|
||||
- `/construction/section/type/list`
|
||||
- `/construction/section/sub/type/list`
|
||||
- `/construction/task/type/list`
|
||||
|
||||
它们当前都来自 `construction_section_config + 类型树`。
|
||||
|
||||
本轮目标不是重做接口,而是:
|
||||
|
||||
- 让 section 级选项由新的 config 工具统一返回
|
||||
- 保证下拉、初始化、create 校验继续同源
|
||||
|
||||
### 9.4 清理项
|
||||
|
||||
construction 领域这轮优先清理的是“section 候选项读取分散”问题,不是一次性重写全部施工结构。
|
||||
|
||||
完成后应清理:
|
||||
|
||||
- 只负责 section 候选项翻译的旧 helper 直接解析逻辑
|
||||
- 不再适合作为 section 真源的旧硬编码映射
|
||||
|
||||
但下面这些不属于本轮删除目标:
|
||||
|
||||
- `ConstructionTypeTree`
|
||||
- `ConstructionTypeMapping`
|
||||
- 与节点树生成强绑定的结构化配置
|
||||
|
||||
## 10. 测试关注点
|
||||
|
||||
construction 领域 service 层集成测试至少覆盖:
|
||||
|
||||
1. 初始化:项目初始化后生成的 section / root node / 主施工任务是否来自目标 config。
|
||||
2. 下拉:section / sub type / task type 列表是否和初始化使用同源配置。
|
||||
3. Create:合法 `section_type_code`、`section_sub_type_code`、`type_id` 可以创建,非法值明确报错。
|
||||
@@ -0,0 +1,108 @@
|
||||
# Construction Task 领域重构指南
|
||||
|
||||
## 1. 范围
|
||||
|
||||
本文档对应 backend 的 `construction` 主表,也就是施工任务本体。
|
||||
|
||||
当前优先关注:
|
||||
|
||||
- `construction.section_type_code`
|
||||
- `construction.task_type`
|
||||
- `construction.status`
|
||||
|
||||
## 2. 字段盘点
|
||||
|
||||
### 2.1 `section_type_code`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 本地有硬编码 `section_type_id_dict`
|
||||
- 也有 `construction_section_config` 作为配置来源
|
||||
- 模型里是 `String`
|
||||
- 创建 schema 里是 `str`
|
||||
- 同步 schema 对应字段名是 `construction_section`,类型是 `str`
|
||||
|
||||
当前使用:
|
||||
|
||||
- 是施工初始化和施工任务分区的关键字段
|
||||
- construction 服务和 config helper 都直接依赖它
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 本地明确值:`GF`、`SY`、`SC`、`SCDW`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 当前没看到集团侧 `enum_config` 有独立的 Construction 顶层段
|
||||
- 同步 schema 里有对应字段,但仍是普通字符串
|
||||
|
||||
结论:
|
||||
|
||||
- 应迁到 `field_config`
|
||||
- 继续沿用 `construction.section_type_code`
|
||||
|
||||
### 2.2 `task_type`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `String`
|
||||
- 创建 schema 里是 `str`
|
||||
- 本地只有一个层级 enum `ConstructionCategory`,值是 `01`、`02`、`0201`、`03`、`0301`、`04`、`05`
|
||||
|
||||
当前使用:
|
||||
|
||||
- 施工树和类型树逻辑会使用它
|
||||
- 但业务任务实际取值往往来自施工树节点,而不只是 `ConstructionCategory`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 如果按层级 category 看,明确值是:`01`、`02`、`0201`、`03`、`0301`、`04`、`05`
|
||||
- 如果按真实任务类型看,当前无法从单一代码 enum 列出完整集合,因为还受施工树/`construction_node_config` 影响
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 同步 schema 里 `task_type` 仍是 `str`
|
||||
- 当前没有看到集团侧 `enum_config` 对这个字段给出统一候选项表
|
||||
|
||||
结论:
|
||||
|
||||
- 不能简单做成单一代码 enum
|
||||
- 更适合拆成“层级 category”与“真实任务类型”两层定义
|
||||
|
||||
### 2.3 `status`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `SmallInteger`
|
||||
- 同步 schema 里是 `Optional[int]`
|
||||
- 本地有 `progress_status_dict`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- `-1`
|
||||
- `0`
|
||||
- `1`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 当前没看到集团侧 `enum_config` 独立列出 Construction.status
|
||||
- 同步 schema 仍是 `int`
|
||||
|
||||
结论:
|
||||
|
||||
- 保留数值状态字段
|
||||
- 需要统一是走代码常量、dict 映射还是后续字段配置
|
||||
|
||||
## 3. 重点结论
|
||||
|
||||
- `section_type_code` 是典型的字段级配置候选。
|
||||
- `task_type` 当前最复杂,不能直接当成一个静态 enum 处理。
|
||||
- `status` 目前是简单数值状态,但也需要统一正式定义源。
|
||||
|
||||
## 4. 待修改项清单
|
||||
|
||||
1. 将 `section_type_code` 正式迁到 `field_config`。
|
||||
2. 清理 `section_type_code` 的硬编码 dict 与 config helper 双来源。
|
||||
3. 拆分 `task_type` 的“层级 category”与“真实任务类型”语义。
|
||||
4. 明确哪些 `task_type` 值来自施工树,哪些来自配置表。
|
||||
5. 为 `status` 建立统一常量定义,避免散落数值字面量。
|
||||
@@ -0,0 +1,192 @@
|
||||
# Contract 领域重构指南
|
||||
|
||||
## 1. 范围
|
||||
|
||||
本文档只做 Contract 领域的项目侧梳理。
|
||||
|
||||
当前优先关注:
|
||||
|
||||
- `contract.contract_type`
|
||||
- `contract.code`
|
||||
- `contract.currency`
|
||||
|
||||
## 2. 字段盘点
|
||||
|
||||
### 2.1 `contract_type`
|
||||
|
||||
当前实现:
|
||||
|
||||
- `app/modules/contract/enums.py` 定义了 `ContractType`
|
||||
- 取值:`product`、`service`、`project`
|
||||
- `Contract` 模型里是 `String`
|
||||
- `ContractCreate` 里是 `str | None`
|
||||
- 同步 schema `schemas/contract/contract.py` 里也是 `str`
|
||||
|
||||
当前使用:
|
||||
|
||||
- 是合同主表核心分类字段
|
||||
- 合同结算服务里会按 `contract_type == "project"` 分支处理工程类合同
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- `product`
|
||||
- `service`
|
||||
- `project`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧 `enum_config.Contract.contract_type` 当前只有 `product`、`service`
|
||||
- 项目侧比集团侧多一个 `project`
|
||||
- 同步 schema 当前没有把它收紧成 enum
|
||||
|
||||
结论:
|
||||
|
||||
- 保留为代码 enum
|
||||
- 先统一“是否正式保留 `project`”
|
||||
|
||||
### 2.2 `code`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是普通 `String`
|
||||
- 没有本地正式 enum
|
||||
- 初始化侧实际依赖 `contract_config` / `contract_code_dict` / `supply_chain_group`
|
||||
|
||||
当前使用:
|
||||
|
||||
- 初始化合同时,`code` 承担业务子类型编码职责,例如 `EPC`、`PC`、`KY`、`JL`
|
||||
- 不是单纯的“自由合同编号”
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前无法从单一代码 enum 列出权威全集
|
||||
- 本地已明确出现的典型值包括:`EPC`、`PC`、`PF`、`CG`、`OSC`、`KY`、`JL`、`E`、`SCYW`、`ZJ`、`TY`、`NB`、`XB`、`ZB`、`GS`、`CN`、`QT`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧 `enum_config.Contract.code` 已明确把这类值当成正式候选项,并且按项目类型变化
|
||||
- 同步 schema 里的 `code` 仍是普通 `str`
|
||||
- 项目侧现在没有把这组值沉成统一的字段配置或正式 enum
|
||||
|
||||
结论:
|
||||
|
||||
- 不建议继续当自由字符串处理
|
||||
- 更适合迁到 `field_config`,或至少统一走配置表来源
|
||||
|
||||
### 2.3 `currency`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `String`
|
||||
- 默认值 `CNY`
|
||||
- schema 里是普通字符串
|
||||
|
||||
当前使用:
|
||||
|
||||
- 当前主要是合同基础信息字段
|
||||
- 没看到复杂分支逻辑
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前代码里唯一明确值是默认值 `CNY`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 当前没看到集团侧 `enum_config` 对这个字段给出正式候选项
|
||||
- 同步 schema 里也是 `str`
|
||||
|
||||
结论:
|
||||
|
||||
- 先保持字符串字段
|
||||
- 如果未来真的出现多币种,再决定是否需要正式枚举
|
||||
|
||||
## 3. 重点结论
|
||||
|
||||
- `contract_type` 应保留为代码 enum,但要先明确是否保留 `project`。
|
||||
- `code` 当前实际上承担合同业务子类型职责,不应继续当自由字符串处理。
|
||||
- `currency` 当前没有足够证据说明要做枚举化。
|
||||
|
||||
## 4. 待修改项清单
|
||||
|
||||
1. 统一 `ContractType` 的正式定义源。
|
||||
2. 确认 `contract_type=project` 是否继续作为正式值保留。
|
||||
3. 为 `contract_type` 在 schema 层补 enum 校验。
|
||||
4. 把 `contract.code` 从“自由字符串”改成统一配置来源。
|
||||
5. 评估 `contract.code` 是否直接迁到 `field_config`。
|
||||
6. 区分清楚 `contract.code` 与“合同编号/contract_sn”的职责,避免一个字段同时承担两层语义。
|
||||
|
||||
## 5. 按本轮要求的改造落点
|
||||
|
||||
contract 领域本轮要直接收口 3 条链路:
|
||||
|
||||
- 初始化
|
||||
- create 校验
|
||||
- 下拉列表
|
||||
|
||||
### 5.1 初始化
|
||||
|
||||
当前合同初始化已经走 `contract_config`。
|
||||
|
||||
因此这轮初始化改造的核心不是重做合同初始化,而是:
|
||||
|
||||
- 把 `contract.code` 明确成正式字段级配置来源
|
||||
- 后续让合同初始化、物资初始化、前端下拉都围绕这套配置工作
|
||||
|
||||
`contract_type` 继续保留为代码 enum,不迁到 `field_config`。
|
||||
|
||||
### 5.2 Create 校验
|
||||
|
||||
当前 `ContractService.create()` 的校验主要依赖:
|
||||
|
||||
- `contarct_type_dict`
|
||||
- `contract_code_dict`
|
||||
- service / product 的硬编码白名单
|
||||
|
||||
这和初始化使用的 `contract_config` 不是同一真源。
|
||||
|
||||
本轮目标是:
|
||||
|
||||
- `contract_type` 继续按代码 enum 校验
|
||||
- `contract.code` 改成通过新的 config 工具校验
|
||||
- 避免继续新增 `if contract_type == ... and code in [...]` 这种硬编码白名单
|
||||
|
||||
### 5.3 下拉列表
|
||||
|
||||
当前合同下拉接口:
|
||||
|
||||
- `/contract/type/list`
|
||||
- `/contract/sub-type/list`
|
||||
|
||||
其中:
|
||||
|
||||
- `type/list` 直接读代码字典
|
||||
- `sub-type/list` 先写死过滤列表,再读代码字典
|
||||
|
||||
这和初始化 `contract_config` 不一致。
|
||||
|
||||
本轮目标是:
|
||||
|
||||
- 合同类型列表继续由代码 enum 提供
|
||||
- 合同子类型列表改为动态读取 config
|
||||
- 保证前端看到的 code 列表和初始化、create 校验同源
|
||||
|
||||
### 5.4 清理项
|
||||
|
||||
本轮 contract 领域改造完成后,应清理:
|
||||
|
||||
- 只用于 `contract.code` 下拉和校验的旧硬编码 code 白名单
|
||||
- 只承担旧合同子类型翻译的代码字典入口
|
||||
|
||||
但下面这些不应误删:
|
||||
|
||||
- `ContractType` enum
|
||||
- 其它真正稳定的固定语义 enum
|
||||
|
||||
## 6. 测试关注点
|
||||
|
||||
contract 领域 service 层集成测试至少覆盖:
|
||||
|
||||
1. 初始化:`contract_config` 能正确初始化合同,并继续驱动下游结算/物资链路。
|
||||
2. 下拉:合同子类型下拉是否来自目标 config,且与初始化 code 集一致。
|
||||
3. Create:合法 `contract_type + code` 创建成功,非法 code 明确报错。
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
# Contract Settlement 领域重构指南
|
||||
|
||||
## 1. 范围
|
||||
|
||||
本文档对应 backend 的 `contract_settlement` 主表。
|
||||
|
||||
当前优先关注:
|
||||
|
||||
- `contract_settlement.contract_type`
|
||||
- `contract_settlement.status`
|
||||
- `contract_settlement.payment_method`
|
||||
|
||||
## 2. 字段盘点
|
||||
|
||||
### 2.1 `contract_type`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `String`
|
||||
- 没有单独 enum 类型绑定到模型
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前主要跟随关联合同的 `contract_type`
|
||||
- 代码现状至少包含:`product`、`service`、`project`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧 `enum_config.ContractSettlement.contract_type` 当前只有 `product`、`service`
|
||||
- 项目侧如果继续沿用 `project`,会比集团侧多一个值
|
||||
- 当前仓库里没有 standalone 的同步 schema 来收紧这个字段
|
||||
|
||||
结论:
|
||||
|
||||
- 先与合同主表 `contract_type` 保持一致
|
||||
- 再确认结算领域是否允许 `project`
|
||||
|
||||
### 2.2 `status`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `SmallInteger`
|
||||
- 默认值 `0`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前没有独立 enum 类
|
||||
- 从集团侧和现有语义看,应至少覆盖 `-1`、`0`、`1`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧 `enum_config.ContractSettlement.status` 明确给出 `-1`、`0`、`1`
|
||||
- 项目侧模型没有沉成统一常量或 enum
|
||||
|
||||
结论:
|
||||
|
||||
- 应补统一状态定义源
|
||||
|
||||
### 2.3 `payment_method`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 本地 `PaymentMethod` enum 有两个值:`node_payment`、`progress_payment`
|
||||
- 模型里仍是 `String`
|
||||
- `SettlePaymentMethodsSelect` 手动校验合法值
|
||||
|
||||
当前使用:
|
||||
|
||||
- 合同结算服务会按 `payment_method` 分支校验和处理
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- `node_payment`
|
||||
- `progress_payment`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 当前没看到集团侧 `enum_config` 为该字段给出正式候选项
|
||||
- 当前仓库里也没有 standalone 的同步 schema 覆盖该字段
|
||||
|
||||
结论:
|
||||
|
||||
- 保留为本地代码 enum
|
||||
- 需要把“手动 validator”提升为统一 schema / service 约束
|
||||
|
||||
## 3. 重点结论
|
||||
|
||||
- `contract_settlement` 里最稳定的枚举字段是 `payment_method`,应继续保留为代码 enum。
|
||||
- `contract_type` 不能单独拍板,必须跟合同主表一起决定是否保留 `project`。
|
||||
- `status` 当前只是模型整数,没有正式定义源,需要补齐。
|
||||
|
||||
## 4. 待修改项清单
|
||||
|
||||
1. 明确 `contract_settlement.contract_type` 是否允许 `project`。
|
||||
2. 为 `status` 补正式常量或 enum 定义。
|
||||
3. 将 `payment_method` 的合法值校验统一沉到正式 schema/enum,而不是只在个别 schema 手动校验。
|
||||
4. 明确结算主表与合同主表在 `contract_type` 上是否必须同源。
|
||||
|
||||
## 5. 按本轮要求的改造落点
|
||||
|
||||
contract_settlement 领域不属于项目初始化第一批重点,但它直接涉及“写死下拉改动态”这件事。
|
||||
|
||||
### 5.1 初始化
|
||||
|
||||
本领域当前没有独立的项目初始化候选项链路。
|
||||
|
||||
`contract_type` 主要跟随合同主表,`payment_method` 主要出现在结算业务选择和后续分支处理中。
|
||||
|
||||
### 5.2 Create / 业务校验
|
||||
|
||||
当前 `payment_method` 的合法值主要靠:
|
||||
|
||||
- `PaymentMethod` enum
|
||||
- `SettlePaymentMethodsSelect` schema 里的手动 validator
|
||||
|
||||
而 service `payment_methods_select()` 本身没有再做额外校验。
|
||||
|
||||
本轮目标是:
|
||||
|
||||
- 保留 `payment_method` 的固定语义 enum 属性
|
||||
- 但在业务层入口也要能通过统一 config 工具或统一 enum 工具做显式校验
|
||||
- 不把校验只留在 schema 里
|
||||
|
||||
### 5.3 下拉列表
|
||||
|
||||
当前支付方式只有选择接口,没有正式的动态下拉接口。
|
||||
|
||||
因此这轮要补的不是“替换已有动态接口”,而是:
|
||||
|
||||
- 把现有写死支付方式候选项明确收口
|
||||
- 如果前端需要单独下拉接口,应改成动态返回,而不是继续依赖前端或 schema 写死
|
||||
|
||||
同理,`payment_type`、`advance_type` 这类当前靠本地 enum 固定的候选项,也应按同样思路处理。
|
||||
|
||||
### 5.4 清理项
|
||||
|
||||
本轮 contract_settlement 领域完成后,应避免继续保留:
|
||||
|
||||
- 只存在于 schema validator、但没有统一出口的支付方式候选项定义
|
||||
- 前端和后端各自写死一份支付方式列表
|
||||
|
||||
## 6. 测试关注点
|
||||
|
||||
contract_settlement 领域 service 层集成测试至少覆盖:
|
||||
|
||||
1. `payment_method` 合法值更新成功,非法值明确报错。
|
||||
2. 如果补了下拉接口,下拉返回是否与 service 校验同源。
|
||||
3. `contract_type` 跟随合同主表时,不出现与合同主表不一致的脏数据。
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
# 项目侧字段分流总表草案
|
||||
|
||||
## 1. 目的
|
||||
|
||||
这份总表只回答一件事:
|
||||
|
||||
- 哪些字段继续保留为代码 enum / 代码常量。
|
||||
- 哪些字段应迁到 `field_config`。
|
||||
- 哪些字段暂时不能直接收口,必须先补领域边界或映射关系。
|
||||
|
||||
## 2. 分流原则
|
||||
|
||||
优先按三类处理:
|
||||
|
||||
- 保留代码 enum:值域固定、语义稳定、业务分支已直接依赖。
|
||||
- 迁到 `field_config`:值域跟 `project_type`、初始化模板或业务配置强相关。
|
||||
- 暂缓收口:当前 backend 与 sync schema 映射不完整,或领域边界本身还没定。
|
||||
|
||||
## 3. 总表
|
||||
|
||||
| 领域 | 字段 | 当前主要来源 | 建议归类 | 备注 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| project | `project_type` | 代码 enum + 初始化字典 | 保留代码 enum | 核心路由字段 |
|
||||
| project | `project_subtype` | 自由字符串 | 迁到 `field_config` | 建议 key:`project.project_subtype` |
|
||||
| project | `syxs` | 自由字符串 | 迁到 `field_config` | 建议 key:`project.syxs` |
|
||||
| project | `status` | 代码 enum + 服务推导 | 保留代码 enum | 正式集合含 `early`,并清理历史 `NULL` |
|
||||
| project | `progress_type` | 代码 enum + DB enum | 保留代码 enum | 已与 sync schema 对齐 |
|
||||
| project | `key_constraints` | 代码 enum 分叉 + DB enum | 保留代码 enum | 项目侧保留 `OTHER`,备注集团侧使用 `QT` |
|
||||
| contract | `contract_type` | 代码 enum | 保留代码 enum | 先确认是否保留 `project` |
|
||||
| contract | `code` | 初始化字典 / config / 自由字符串 | 迁到 `field_config` | 建议 key:`contract.code` |
|
||||
| contract | `currency` | 自由字符串 | 暂不处理 | 当前仅见 `CNY` |
|
||||
| construction_task | `section_type_code` | 硬编码 dict + config | 迁到 `field_config` | 建议 key:`construction.section_type_code` |
|
||||
| construction_task | `task_type` | 类型树 + 字符串字段 | 暂缓收口 | 先拆 category 与真实任务类型 |
|
||||
| construction_task | `status` | 数值字段 + dict | 保留代码常量 | 先统一常量定义源 |
|
||||
| material | `material_type` | 本地列表 + 自由字符串 | 迁到 `field_config` | 建议 key:`material.material_type` |
|
||||
| material | `status` | 数值字段 | 暂缓收口 | 先和 `supply_status` 划边界 |
|
||||
| material | `supply_status` | 代码 enum | 保留代码常量 | 先确认是否为正式推送字段 |
|
||||
| material | `approval_party` | 本地候选项 | 暂不处理 | 当前无集团侧对应 |
|
||||
| contract_settlement | `contract_type` | 跟随合同主表 | 暂缓收口 | 先确认是否允许 `project` |
|
||||
| contract_settlement | `status` | 数值字段 | 保留代码常量 | 需补统一定义源 |
|
||||
| contract_settlement | `payment_method` | 代码 enum | 保留代码 enum | 不建议迁 `field_config` |
|
||||
| personnel | `personnel_type` | 本地代码常量 | 保留代码 enum | 本地真实领域是人员资质 |
|
||||
| personnel | `task_type` | 本地候选项常量 | 保留代码常量 | 先补 schema 白名单 |
|
||||
| personnel | `is_duty` | 数值常量 | 保留代码常量 | 与集团侧 `status` 不是同一字段 |
|
||||
| preparation | `code` | 初始化字典 + 自由字符串 | 迁到 `field_config` | 建议 key:`preparation.code` |
|
||||
| preparation | `approval_status` | 通用审批体系 | 接入通用审批定义 | 不单独建 `field_config` |
|
||||
| preparation | `is_exist` | 布尔字段 | 暂不处理 | 不属于候选项字段 |
|
||||
| production | `code` | 初始化字典 + 自由字符串 | 迁到 `field_config` | 建议 key:`production.code` |
|
||||
| production | `approval_status` | 通用审批体系 | 接入通用审批定义 | 不单独建 `field_config` |
|
||||
| production | `is_exist` | 布尔字段 | 暂不处理 | 不属于候选项字段 |
|
||||
|
||||
## 4. 第一批建议落地项
|
||||
|
||||
优先进入 `field_config` 的字段:
|
||||
|
||||
- `project.project_subtype`
|
||||
- `project.syxs`
|
||||
- `contract.code`
|
||||
- `construction.section_type_code`
|
||||
- `material.material_type`
|
||||
- `preparation.code`
|
||||
- `production.code`
|
||||
|
||||
优先保留并收紧 schema 的代码 enum 字段:
|
||||
|
||||
- `project.project_type`
|
||||
- `project.status`
|
||||
- `project.progress_type`
|
||||
- `project.key_constraints`
|
||||
- `contract.contract_type`
|
||||
- `contract_settlement.payment_method`
|
||||
|
||||
优先先做边界确认、暂不直接配置化的字段:
|
||||
|
||||
- `construction.task_type`
|
||||
- `material.status`
|
||||
- `contract_settlement.contract_type`
|
||||
|
||||
## 5. 使用方式
|
||||
|
||||
这份表可以直接作为后续代码改造顺序的输入:
|
||||
|
||||
1. 先处理“迁到 `field_config`”的一组字段。
|
||||
2. 再处理“保留代码 enum,但 schema 还没收口”的一组字段。
|
||||
3. 最后处理“暂缓收口”的结构性问题字段。
|
||||
|
||||
## 6. 按本轮目标拆出的第一批字段
|
||||
|
||||
这次不是单纯做字段分类,而是要直接服务于 3 条链路:
|
||||
|
||||
- 项目初始化
|
||||
- create 业务校验
|
||||
- 前端下拉列表
|
||||
|
||||
因此第一批字段不能只看“适不适合进 `field_config`”,还要看它能不能尽快把这 3 条链路收口到同一真源。
|
||||
|
||||
### 6.1 第一批优先收口字段
|
||||
|
||||
建议第一批优先处理:
|
||||
|
||||
- `contract.code`
|
||||
- `material.material_type`
|
||||
- `construction.section_type_code`
|
||||
- `preparation.code`
|
||||
- `production.code`
|
||||
- `power.code`
|
||||
|
||||
原因:
|
||||
|
||||
- 都属于 code 类字段
|
||||
- 都直接影响初始化、创建或下拉其中至少两条链路
|
||||
- 当前都存在 config / dict / 代码常量混用问题,或者已具备明显收口价值
|
||||
|
||||
### 6.2 每个字段要同时回答的 3 个问题
|
||||
|
||||
对上面每个字段,后续文档和代码改造都应同时回答:
|
||||
|
||||
1. 初始化时从哪里拿候选项。
|
||||
2. create service 开头用哪个 config key 做校验。
|
||||
3. 前端下拉接口用哪个 config key 返回选项。
|
||||
|
||||
如果这 3 个问题的答案不是同一个 key,就说明还没有真正收口。
|
||||
|
||||
### 6.3 本轮不优先收口但要避免误迁的字段
|
||||
|
||||
下面这些字段虽然也和 enum 有关,但本轮不应抢先塞进字段级配置:
|
||||
|
||||
- `project.status`
|
||||
- `project.progress_type`
|
||||
- `project.key_constraints`
|
||||
- `contract.contract_type`
|
||||
- `contract_settlement.payment_method`
|
||||
|
||||
原因是:
|
||||
|
||||
- 这些字段语义更固定
|
||||
- 业务分支已经直接依赖
|
||||
- 本轮重点是先把“按项目类型变化的 code 类字段”收口,而不是把固定状态类枚举全部配置化
|
||||
|
||||
### 6.4 对物资字段的特别说明
|
||||
|
||||
`material.material_type` 虽然在表里只是一行,但它实际上牵扯 4 套来源:
|
||||
|
||||
- 初始化合同候选项:`contract_config`
|
||||
- 物资初始化补充:`material_contract_*`
|
||||
- 前端下拉:`material_types`
|
||||
- create 内部默认展示/单位:`contract_product_type_list`
|
||||
|
||||
所以 `material.material_type` 不是简单迁一个 key,而是本轮最需要先统一真源的一项。
|
||||
|
||||
### 6.5 本轮完成后的字段判定标准
|
||||
|
||||
一个字段只有同时满足下面 3 条,才算本轮真正完成收口:
|
||||
|
||||
- 初始化使用它对应的 field config
|
||||
- create 校验使用它对应的 field config
|
||||
- 下拉列表返回也使用它对应的 field config
|
||||
@@ -0,0 +1,757 @@
|
||||
# init_project_data 现状分析
|
||||
|
||||
## 1. 范围
|
||||
|
||||
本文档只分析 backend 里的 `ProjectService.init_project_data)` 当前到底做了什么。
|
||||
|
||||
重点回答 4 个问题:
|
||||
|
||||
- 这个函数当前创建了哪些业务数据。
|
||||
- 它直接读取了哪些 `config` 和 `dict`。
|
||||
- 哪些 `dict` 是真的在负责初始化,哪些只是历史残留。
|
||||
- 如果把这里 `dict` 负责的部分迁到 `config`,复杂度到底高不高。
|
||||
|
||||
## 2. 触发时机
|
||||
|
||||
`init_project_data)` 在项目审批通过后执行。
|
||||
|
||||
调用链是:
|
||||
|
||||
- `ProjectService.audit_project()`
|
||||
- `ProjectService._audit_approval()`
|
||||
- `ProjectService.init_project_data)`
|
||||
|
||||
也就是说,这不是“创建项目主表时立即执行”的逻辑,而是“审批通过后批量灌初始化数据”的逻辑。
|
||||
|
||||
## 3. 函数当前做的 6 件事
|
||||
|
||||
按执行顺序,当前函数会做下面 6 件事:
|
||||
|
||||
1. 读取当前 `project_type` 整个 category 下的全部 config。
|
||||
2. 读取 6 组 dict 数据。
|
||||
3. 初始化 `preparation` 里程碑节点。
|
||||
4. 初始化 `production` 投产节点。
|
||||
5. 初始化 `contract` 合同,并在合同初始化内部继续初始化 `contract_settlement` 和 `material`。
|
||||
6. 初始化 `power` 力能配置、`construction` 施工任务、`archive` 电子档案。
|
||||
|
||||
注意:
|
||||
|
||||
- 这个函数表面上不长,但它本质上是一个“项目初始化编排器”。
|
||||
- 真正复杂的地方不只在它自己,而在它调用出去的各个 `init_from_*`。
|
||||
|
||||
## 4. 直接读取了什么
|
||||
|
||||
### 4.1 直接读取的 config
|
||||
|
||||
函数一开始会调用:
|
||||
|
||||
- `config_service.get_category_config(category=project.project_type)`
|
||||
|
||||
这一步不是只读某一个 key,而是把当前 `project_type` 这个 category 下的全部 config 都读出来,再做:
|
||||
|
||||
- `configs = {config_key: decoded_json}`
|
||||
|
||||
但在 `init_project_data)` 自己内部,真正直接用到的 config 只有 1 个:
|
||||
|
||||
- `contract_config`
|
||||
|
||||
也就是说:
|
||||
|
||||
- 直接读取:整类 config
|
||||
- 直接消费:只有 `contract_config`
|
||||
- 其余 config 在这个函数里只是被一并装进了 `configs`,并没有直接使用
|
||||
|
||||
### 4.2 直接读取的 dict
|
||||
|
||||
函数会一次性读取 6 个 dict key:
|
||||
|
||||
- `preparation_{project_type}`
|
||||
- `production_{project_type}`
|
||||
- `construction_contract_{project_type}`
|
||||
- `service_contract_{project_type}`
|
||||
- `material_contract_{project_type}`
|
||||
- `power_{project_type}`
|
||||
|
||||
但这 6 个 key 不是都真正被当前函数消费了。
|
||||
|
||||
## 5. 哪些数据是怎么创建出来的
|
||||
|
||||
### 5.1 Preparation
|
||||
|
||||
创建来源:
|
||||
|
||||
- 直接使用 dict:`preparation_{project_type}`
|
||||
|
||||
创建方式:
|
||||
|
||||
- 读取 dict 项的 `code`、`name`
|
||||
- 先删项目下旧的 `Preparation`
|
||||
- 再批量插入新的 `Preparation`
|
||||
|
||||
涉及配置数量:
|
||||
|
||||
- `config`: 0
|
||||
- `dict`: 1 组
|
||||
|
||||
当前种子数据条目数:
|
||||
|
||||
- `preparation_photovoltaic`: 12 条
|
||||
- `preparation_onshore_wind`: 13 条
|
||||
- `preparation_offshore_wind`: 14 条
|
||||
|
||||
结论:
|
||||
|
||||
- 这是最典型的“dict 直接承担初始化模板”场景。
|
||||
- 结构非常简单,迁到 `config` 不复杂。
|
||||
|
||||
### 5.2 Production
|
||||
|
||||
创建来源:
|
||||
|
||||
- 直接使用 dict:`production_{project_type}`
|
||||
|
||||
创建方式:
|
||||
|
||||
- 读取 dict 项的 `code`、`name`
|
||||
- 先删项目下旧的 `Production`
|
||||
- 再批量插入新的 `Production`
|
||||
|
||||
涉及配置数量:
|
||||
|
||||
- `config`: 0
|
||||
- `dict`: 1 组
|
||||
|
||||
当前种子数据条目数:
|
||||
|
||||
- `production_photovoltaic`: 9 条
|
||||
- `production_onshore_wind`: 9 条
|
||||
- `production_offshore_wind`: 9 条
|
||||
|
||||
结论:
|
||||
|
||||
- 和 `preparation` 一样,是平铺型 code/name 列表。
|
||||
- 迁到 `config` 也不复杂。
|
||||
|
||||
### 5.3 Contract
|
||||
|
||||
创建来源:
|
||||
|
||||
- 直接使用 config:`contract_config`
|
||||
|
||||
创建方式:
|
||||
|
||||
- `init_project_data)` 从 `configs["contract_config"]` 取出列表
|
||||
- `ContractService.init_from_project()` 根据每项的 `code`、`name`、`contract_type` 创建合同
|
||||
- 先删项目下旧合同,再批量插入新合同
|
||||
|
||||
后续联动:
|
||||
|
||||
- 初始化 `contract_settlement`
|
||||
- 初始化 `material`
|
||||
|
||||
涉及配置数量:
|
||||
|
||||
- `config`: 1 个 key
|
||||
- `dict`: 0 组直接参与合同创建
|
||||
|
||||
当前 `contract_config` 条目数:
|
||||
|
||||
- `photovoltaic`: 21 条
|
||||
- `onshore_wind`: 20 条
|
||||
- `offshore_wind`: 20 条
|
||||
|
||||
结论:
|
||||
|
||||
- 合同初始化本体已经是 config 驱动,不是 dict 驱动。
|
||||
- 所以这里真正要迁移的不是“合同本身”,而是合同初始化后还依赖的那些配套 dict。
|
||||
|
||||
### 5.4 Material
|
||||
|
||||
创建来源:
|
||||
|
||||
- 不是 `init_project_data)` 直接创建。
|
||||
- 它是 `contract.init_from_project()` 内部继续调用 `material.init_from_contract()` 创建。
|
||||
|
||||
当前来源拆分:
|
||||
|
||||
- 合同列表来源:`contract_config`
|
||||
- 物资单位来源:`material_contract_{project_type}` dict
|
||||
|
||||
也就是说,当前 `material` 初始化其实是两段拼起来的:
|
||||
|
||||
- 先由 `contract_config` 决定会创建哪些货物类合同
|
||||
- 再由 `material_contract_*` dict 给这些物资补 `unit`
|
||||
|
||||
涉及配置数量:
|
||||
|
||||
- `config`: 1 个上游 key `contract_config`
|
||||
- `dict`: 1 组 `material_contract_{project_type}`
|
||||
|
||||
当前种子数据条目数:
|
||||
|
||||
- `material_contract_photovoltaic`: 14 条
|
||||
- `material_contract_onshore_wind`: 12 条
|
||||
- `material_contract_offshore_wind`: 7 条
|
||||
|
||||
这里有一个关键现状:
|
||||
|
||||
- `material` 实际创建多少条,取决于 `contract_config` 里有多少 `product` 合同。
|
||||
- `material_contract_*` dict 只负责补充单位和名称,不负责决定是否创建。
|
||||
- 因此这两套来源当前不是同一个真源。
|
||||
|
||||
已看到的分叉:
|
||||
|
||||
- 光伏 `contract_config` 里的货物类候选项和 `material_contract_photovoltaic` 并不完全一致。
|
||||
- 陆风也存在类似分叉。
|
||||
- 海风更明显:当前能看到 `contract_config` 已有较完整货物类合同,但 `dict` 侧条目明显更少。
|
||||
|
||||
结论:
|
||||
|
||||
- `material` 不是简单的“把一个 dict 搬到 config”。
|
||||
- 真正要做的是先合并“合同货物候选项”和“物资单位/展示信息”两套来源。
|
||||
- 这块复杂度中等,不是最高,但已经不是平铺搬运。
|
||||
|
||||
### 5.5 Power
|
||||
|
||||
创建来源:
|
||||
|
||||
- 直接使用 dict:`power_{project_type}`
|
||||
|
||||
创建方式:
|
||||
|
||||
- 读取 dict 项的 `code`、`name`
|
||||
- 统一绑定到 `contract_data.get("EPC")`
|
||||
- 先删项目下旧 `Power`
|
||||
- 再批量插入新 `Power`
|
||||
|
||||
涉及配置数量:
|
||||
|
||||
- `config`: 0
|
||||
- `dict`: 1 组
|
||||
|
||||
当前种子数据条目数:
|
||||
|
||||
- `power_photovoltaic`: 1 条
|
||||
- `power_onshore_wind`: 2 条
|
||||
- `power_offshore_wind`: 2 条
|
||||
|
||||
结论:
|
||||
|
||||
- 也是平铺型初始化模板,迁到 `config` 不复杂。
|
||||
- 但它和合同存在一条强耦合:默认绑定 `EPC` 合同。
|
||||
|
||||
### 5.6 Construction
|
||||
|
||||
创建来源:
|
||||
|
||||
- `init_project_data)` 自己没有直接给施工传 dict 或 config 数据
|
||||
- 只调用:`create_constructions_for_project(tenant_id, project_id)`
|
||||
|
||||
但施工初始化内部会继续读取:
|
||||
|
||||
- `construction_section_config`
|
||||
|
||||
并依赖:
|
||||
|
||||
- `BuildMethod`
|
||||
- `ConstructionTypeTree`
|
||||
- `ConstructionTypeMapping`
|
||||
|
||||
实际创建内容:
|
||||
|
||||
- 按 `construction_section_config.sections[*].project_root_nodes` 创建项目根单位工程
|
||||
- 基于这些根节点再创建主施工任务
|
||||
- 对质量验收类施工继续创建施工树
|
||||
- 对主施工继续自动补次级施工
|
||||
|
||||
涉及配置数量:
|
||||
|
||||
- `config`: 至少 1 个 key `construction_section_config`
|
||||
- `dict`: 0
|
||||
- 另外强依赖数据库里的施工类型树,不是简单 JSON 配置
|
||||
|
||||
结论:
|
||||
|
||||
- 施工初始化本来就不是 dict 驱动,已经是 config + 类型树驱动。
|
||||
- 这里真正复杂的不是“dict 要不要迁 config”,而是施工本身就已经是结构化初始化。
|
||||
|
||||
### 5.7 Archive
|
||||
|
||||
创建来源:
|
||||
|
||||
- 不依赖 dict
|
||||
- 也不依赖 `init_project_data)` 读出来的 config
|
||||
|
||||
当前方式:
|
||||
|
||||
- 按 `template/{project_type}_archive.xlsx` 模板文件初始化电子档案树
|
||||
|
||||
涉及配置数量:
|
||||
|
||||
- `config`: 0(对 `_init_project_data` 而言)
|
||||
- `dict`: 0
|
||||
- 外部模板文件: 1 份 Excel
|
||||
|
||||
结论:
|
||||
|
||||
- 电子档案初始化不属于这次 dict/config 收敛的核心问题。
|
||||
|
||||
## 6. 这 6 组 dict 里,哪些真的在负责事情
|
||||
|
||||
### 6.1 直接被 `init_project_data)` 消费的 dict
|
||||
|
||||
- `preparation_{project_type}`
|
||||
- `production_{project_type}`
|
||||
- `power_{project_type}`
|
||||
|
||||
这 3 组 dict 是真的在当前函数里直接承担初始化模板职责。
|
||||
|
||||
### 6.2 被 `init_project_data)` 取了,但当前函数里没有直接消费的 dict
|
||||
|
||||
- `construction_contract_{project_type}`
|
||||
- `service_contract_{project_type}`
|
||||
- `material_contract_{project_type}`
|
||||
|
||||
其中:
|
||||
|
||||
- `construction_contract_*`:当前只看到被 `init_project_data)` 取出,但没有实际使用。
|
||||
- `service_contract_*`:当前也只看到被 `init_project_data)` 取出,但没有实际使用。
|
||||
- `material_contract_*`:当前函数本身没用,但会在 `MaterialService.init_from_contract()` 里被重新读取一次,用于补单位。
|
||||
|
||||
这说明当前 `init_project_data)` 至少存在两类历史包袱:
|
||||
|
||||
- 取了但没用的 dict key
|
||||
- 同一套语义在上游合同 config 和下游 material dict 之间分叉
|
||||
|
||||
## 7. 把 dict 迁到 config,会不会太复杂
|
||||
|
||||
### 7.1 不复杂的部分
|
||||
|
||||
下面 3 组迁移到 config,复杂度不高:
|
||||
|
||||
- `preparation_{project_type}`
|
||||
- `production_{project_type}`
|
||||
- `power_{project_type}`
|
||||
|
||||
原因很简单:
|
||||
|
||||
- 它们当前就是平铺的 `code/name` 列表
|
||||
- 服务接口已经接受 `list[dict]`
|
||||
- 初始化逻辑只做“删旧 + 批量重建”
|
||||
- 不依赖复杂树结构,也不依赖额外业务联动
|
||||
|
||||
如果迁移:
|
||||
|
||||
- `preparation`、`production` 很适合直接改成 `field_config`
|
||||
- `power` 更像一个小型初始化模板,也可以直接放到 config,但不一定要走 `field_config` 这个名字
|
||||
|
||||
### 7.2 中等复杂的部分
|
||||
|
||||
`material_contract_{project_type}` 迁移到 `config`,复杂度中等。
|
||||
|
||||
原因不是数据量大,而是来源分叉:
|
||||
|
||||
- 哪些货物类记录会被创建,取决于 `contract_config`
|
||||
- 每种物资的显示名、单位,又取决于 `material_contract_*` dict
|
||||
|
||||
所以这块不是“dict 搬家”,而是“先统一真源”。
|
||||
|
||||
更稳妥的做法是:
|
||||
|
||||
- 先把 `contract_config` 和 `material_contract_*` 合并成一套正式结构
|
||||
- 再决定是否把单位、显示名、parts 一并并到 `material.field_config` 或其它 config key
|
||||
|
||||
### 7.3 高复杂的部分其实不在 dict
|
||||
|
||||
真正复杂的初始化不是 dict,而是施工:
|
||||
|
||||
- 它已经依赖 `construction_section_config`
|
||||
- 又依赖 `ConstructionTypeTree`
|
||||
- 又依赖 `ConstructionTypeMapping`
|
||||
- 还会继续生成施工树和次级施工
|
||||
|
||||
所以如果目标是“降低 `init_project_data` 初始化复杂度”,优先清理 dict 只能解决一部分表层问题,不能解决施工初始化本身的结构复杂度。
|
||||
|
||||
## 8. 额外现状问题
|
||||
|
||||
### 8.1 `init_project_data)` 读了整类 config,但只直接用了 `contract_config`
|
||||
|
||||
这会导致:
|
||||
|
||||
- 函数看起来像“config 驱动”
|
||||
- 实际上只有合同初始化是直接 config 驱动
|
||||
|
||||
### 8.2 有 3 个 dict key 在当前函数里不是实际输入
|
||||
|
||||
- `construction_contract_*`
|
||||
- `service_contract_*`
|
||||
- `material_contract_*`
|
||||
|
||||
其中前 2 个当前更像历史残留。
|
||||
|
||||
### 8.3 当前只明显覆盖了 3 类项目类型的初始化种子
|
||||
|
||||
从现有 `dict.csv` / `config.csv` 看,当前明确有初始化种子的主要是:
|
||||
|
||||
- `photovoltaic`
|
||||
- `onshore_wind`
|
||||
- `offshore_wind`
|
||||
|
||||
而 `project_type` 枚举里其实还有:
|
||||
|
||||
- `pumped_storage`
|
||||
- `hydropower`
|
||||
- `thermal_power`
|
||||
- `gas_turbine`
|
||||
|
||||
也就是说,`project_type` 枚举范围和初始化种子覆盖范围并不一致。
|
||||
|
||||
### 8.4 海风配置目前看起来不完整
|
||||
|
||||
当前能直接看到:
|
||||
|
||||
- `offshore_wind` 有 `contract_config`
|
||||
- 也有 `preparation_*`、`production_*`、`power_*`、`material_contract_*` dict
|
||||
|
||||
但当前 `config.csv` 里没看到海风的:
|
||||
|
||||
- `construction_section_config`
|
||||
- `construction_node_config`
|
||||
- `material_types`
|
||||
- `supply_chain_group`
|
||||
|
||||
这意味着海风初始化链路当前很可能并不完整,至少配置层不齐。
|
||||
|
||||
## 9. 结论
|
||||
|
||||
可以把 `init_project_data)` 当前的配置来源,拆成 4 类理解:
|
||||
|
||||
1. 纯 dict 初始化:`preparation`、`production`、`power`
|
||||
2. 纯 config 初始化:`contract`
|
||||
3. config + dict 混合初始化:`material`
|
||||
4. config + 类型树 + 模板文件初始化:`construction`、`archive`
|
||||
|
||||
所以“把这里 dict 负责的东西迁到 config 会不会太复杂”的答案不是一个统一结论:
|
||||
|
||||
- `preparation`、`production`、`power`:不复杂,可以迁。
|
||||
- `material_contract`:中等复杂,先统一真源再迁。
|
||||
- `construction`:复杂点不在 dict,本来就不是 dict 驱动。
|
||||
|
||||
## 10. 建议落地顺序
|
||||
|
||||
1. 先把 `init_project_data)` 里“取了但没直接用”的 `construction_contract_*`、`service_contract_*` 从函数里清出来,确认是否还能删掉。
|
||||
2. 再把 `preparation_{project_type}`、`production_{project_type}`、`power_{project_type}` 迁到 config。
|
||||
3. 再处理 `material_contract_*`,但前提是先统一 `contract_config` 和 material 初始化所需的单位/展示信息来源。
|
||||
4. 施工初始化不要和这批 dict 迁移混做,单独处理。
|
||||
|
||||
## 10.1 按本轮改造目标,初始化链路具体要改什么
|
||||
|
||||
如果把这次目标收敛成“初始化统一走配置工具”,那么 `init_project_data)` 不应该再只是分析对象,而是明确要往下面的形态改。
|
||||
|
||||
### A. 保留并整理的 config
|
||||
|
||||
初始化链路应继续保留并整理这几类 config:
|
||||
|
||||
- `contract_config`:合同初始化候选项
|
||||
- `construction_section_config`:施工区域 / 单位工程 / 任务入口配置
|
||||
- `material_types`:物资类型、展示名、单位、parts 等字段配置
|
||||
- 后续新增的字段级配置 key,例如:
|
||||
- `preparation.code`
|
||||
- `production.code`
|
||||
- `power.code`
|
||||
|
||||
这里的关键不是 config 数量多少,而是把“初始化真的要用的候选项”收口到 config 工具统一读取。
|
||||
|
||||
### B. 应从 dict 迁出的初始化候选项
|
||||
|
||||
下面这几类当前属于平铺型初始化模板,适合迁出 dict:
|
||||
|
||||
- `preparation_{project_type}`
|
||||
- `production_{project_type}`
|
||||
- `power_{project_type}`
|
||||
|
||||
迁移后的目标形态是:
|
||||
|
||||
- `init_project_data)` 不再直接读这些 dict key
|
||||
- 改由 `ConfigService.get_field_options()` 或等价工具读取字段候选项
|
||||
- service 仍只负责“删旧 + 批量重建”,不再关心候选项来自 dict 还是 config
|
||||
|
||||
### C. 应去掉的不适当 dict 用法
|
||||
|
||||
本轮最典型的不适当 dict 用法是:
|
||||
|
||||
- `material_contract_{project_type}` 用 `remark` 存物资单位
|
||||
|
||||
这类做法的问题不是“它现在不能工作”,而是:
|
||||
|
||||
- 含义不显式
|
||||
- 初始化、创建、下拉三条链路都没法自然共用
|
||||
- 后续导出 CSV / 校验时也不容易发现结构错误
|
||||
|
||||
迁移方向应是:
|
||||
|
||||
- 把 `unit`、`label`、`parts` 等字段并回正式 config 结构
|
||||
- 不再让单位这类业务字段挂在 dict `remark` 上
|
||||
|
||||
### D. `init_project_data)` 迁移后的读取原则
|
||||
|
||||
迁移后,`init_project_data)` 本身应只做 2 件事:
|
||||
|
||||
- 找到当前项目类型下所需的候选项配置
|
||||
- 把候选项列表传给各领域初始化 service
|
||||
|
||||
它不应该继续承担:
|
||||
|
||||
- 现场拼 JSON 结构
|
||||
- 自己手工区分 dict / config / remark
|
||||
- 同时取一堆历史 key 再由下游各自二次分叉读取
|
||||
|
||||
### E. 初始化改造完成后的判定标准
|
||||
|
||||
初始化侧改造完成后,至少应满足:
|
||||
|
||||
- `preparation`、`production`、`power` 初始化不再依赖 dict
|
||||
- `material` 初始化不再依赖 dict `remark` 补单位
|
||||
- 初始化候选项与前端下拉、create 校验使用同一套 config 真源
|
||||
- `init_project_data)` 中不再保留只读取不用的 dict key
|
||||
|
||||
## 11. 从接口创建视角反看这些字段是不是已经“主要 config 化”
|
||||
|
||||
只看 `init_project_data)`,容易得出“系统已经比较 config 驱动”的印象。
|
||||
|
||||
但如果再抽样看接口创建路径,会发现当前真实情况是:
|
||||
|
||||
- 初始化层里,确实有一部分已经走 `config`
|
||||
- 但创建接口里,很多 enum-like 字段仍然主要靠代码常量或 service 层校验
|
||||
- schema 层大多还是普通 `str`,并没有把这些候选值正式收口到统一配置来源
|
||||
|
||||
### 11.1 Contract 创建
|
||||
|
||||
接口入参:
|
||||
|
||||
- `ContractCreate` 里的 `contract_type`、`code` 都只是普通字符串
|
||||
|
||||
创建时字段来源:
|
||||
|
||||
- `contract_type`、`code` 直接来自接口传参
|
||||
- 合同名称不是前端传,而是 service 根据 `contract_code_dict` 动态拼出来
|
||||
|
||||
当前校验方式:
|
||||
|
||||
- schema 层:只有长度约束,没有枚举白名单
|
||||
- service 层:
|
||||
- `code` 必须在 `contract_code_dict`
|
||||
- `contract_type` 必须在 `contarct_type_dict`
|
||||
- `service` 合同再限制为 `JL`、`KY`、`E`、`SCYW`
|
||||
- `product` 合同再限制为一组硬编码产品 code
|
||||
|
||||
结论:
|
||||
|
||||
- 合同初始化是 `contract_config` 驱动
|
||||
- 但合同接口创建仍然主要依赖代码字典和硬编码白名单
|
||||
- 这说明“初始化层已 config 化”并不等于“合同域已经完成 config 收口”
|
||||
|
||||
### 11.2 Material 创建
|
||||
|
||||
接口入参:
|
||||
|
||||
- `MaterialCreate.material_type`、`approval_party` 也都是普通字符串
|
||||
|
||||
创建时字段来源:
|
||||
|
||||
- `material_type` 直接来自接口传参
|
||||
- `show_name`、`unit` 不是前端传,而是 `MaterialService.get_material_info_by_code()` 从 `contract_product_type_list` 里反查
|
||||
- 创建物资时还会同步创建一个 `product` 合同,合同 code 直接复用 `material_type`
|
||||
|
||||
当前校验方式:
|
||||
|
||||
- schema 层:
|
||||
- `material_type` 只是示例值来自 `contract_product_type_list`
|
||||
- `approval_party` 只有示例,没有正式白名单
|
||||
- service 层:
|
||||
- 强校验的是 `contract_quantity > 0`、项目存在、tenant 存在
|
||||
- `material_type` 本身没有显式抛错白名单
|
||||
- 如果 code 不在 `contract_product_type_list`,`get_material_info_by_code()` 会退回默认值 `支架/组`
|
||||
- 随后又会去调用合同创建,而合同创建对 `code` 有一层更严格的白名单校验
|
||||
|
||||
结论:
|
||||
|
||||
- material 创建不是 config 驱动,而是代码常量驱动
|
||||
- 而且 `material_type` 的直接校验位置并不清晰,存在“本服务默认值兜底,合同服务再拦截”的现状
|
||||
- 这正是还需要整理 config/真源的证据,不适合直接认为已经收口完成
|
||||
|
||||
### 11.3 Construction 创建
|
||||
|
||||
接口入参:
|
||||
|
||||
- `ConstructionCreate.section_type_code`、`section_sub_type_code`、`type_id` 都是普通字符串
|
||||
|
||||
创建时字段来源:
|
||||
|
||||
- `section_type_code`、`section_sub_type_code`、`type_id` 直接来自接口传参
|
||||
- `section_type_name`、`section_sub_type_name`、`task_code`、`task_type` 则由 service 根据配置和类型树反查生成
|
||||
|
||||
当前校验方式:
|
||||
|
||||
- schema 层:只有字符串长度和数值范围,没有枚举白名单
|
||||
- service 层:
|
||||
- `section_type_code` 必须存在于 `construction_section_config`
|
||||
- `section_sub_type_code` 必须能映射到统一类型节点
|
||||
- `type_id` 必须能映射到施工类型树节点
|
||||
- `SCDW` 分支还会单独校验专用任务编码列表
|
||||
|
||||
结论:
|
||||
|
||||
- 施工创建是这 3 个抽样域里最接近“配置/类型树驱动”的
|
||||
- 但即便如此,接口 schema 本身仍未把候选值正式声明出来,真正校验仍落在 service 里
|
||||
|
||||
### 11.4 总判断
|
||||
|
||||
如果只问“初始化层面能不能看出 enum_config 类字段怎么来的”,答案是:
|
||||
|
||||
- 能看出一部分。
|
||||
- `preparation`、`production`、`power` 明显还来自 dict。
|
||||
- `contract` 初始化候选项明显来自 `contract_config`。
|
||||
- `material` 初始化候选项来自 `contract_config`,单位补充来自 `material_contract_*` dict。
|
||||
- `construction` 初始化候选项来自 config + 类型树。
|
||||
|
||||
但如果进一步问“那是不是已经主要用 config 写了,所以没必要再整理 config”,答案是否定的。
|
||||
|
||||
原因是当前系统仍然存在 3 个没有收口的问题:
|
||||
|
||||
- 初始化来源和接口创建来源不是同一套真源。
|
||||
- schema 层没有把这些字段正式建模成受控候选值。
|
||||
- service 层仍保留大量代码字典、硬编码白名单和默认兜底。
|
||||
|
||||
所以现在做 config 整理仍然有必要,而且必要性不在“把所有东西都搬到 config”,而在:
|
||||
|
||||
- 明确每个字段的唯一真源
|
||||
- 消除 dict/config/代码常量三头并存
|
||||
- 让初始化和接口创建共用同一套候选值与校验逻辑
|
||||
|
||||
## 12. 前端下拉选项当前是通过什么路径返回的
|
||||
|
||||
除了创建接口本身,前端通常会先调一些 list 接口拿可选项。
|
||||
|
||||
这部分如果和初始化来源不一致,就会出现:
|
||||
|
||||
- 初始化能创建出来的值,不一定和前端可选项完全一致
|
||||
- 前端下拉看到的值,也不一定就是创建 service 实际校验使用的那套真源
|
||||
|
||||
下面按合同、物资、施工 3 类抽样。
|
||||
|
||||
### 12.1 Contract 的前端下拉路径
|
||||
|
||||
当前相关接口主要有:
|
||||
|
||||
- `/contract/type/list`
|
||||
- `/contract/sub-type/list`
|
||||
|
||||
返回路径:
|
||||
|
||||
- `router -> ContractService.get_contract_type_list()`
|
||||
- `router -> ContractService.get_sub_type_list(contract_type)`
|
||||
|
||||
数据来源:
|
||||
|
||||
- `get_contract_type_list()` 直接读代码里的 `contarct_type_dict`
|
||||
- `get_sub_type_list()` 先根据传入 `contract_type` 选一组硬编码 `filter_key_list`
|
||||
- 再去代码里的 `contract_code_dict` 组装返回值
|
||||
|
||||
和初始化是否一致:
|
||||
|
||||
- 不一致。
|
||||
- 初始化合同候选项来自 `contract_config`。
|
||||
- 但前端拿到的合同类型/子类型下拉,来自代码字典和硬编码列表。
|
||||
|
||||
这意味着:
|
||||
|
||||
- 前端创建合同时看到的 code/type 候选,不是直接从初始化同源配置拿出来的。
|
||||
- 如果未来 `contract_config` 改了,但代码字典没改,就会分叉。
|
||||
|
||||
### 12.2 Material 的前端下拉路径
|
||||
|
||||
当前至少有两条相关路径:
|
||||
|
||||
- `/material/material-types`
|
||||
- `/material-detail/contract-product-type/list`
|
||||
|
||||
第一条路径:`/material/material-types`
|
||||
|
||||
- `router` 先根据 `project_id` 查项目类型
|
||||
- 再调用 `config_service.get_config_json(project.project_type, "material_types")`
|
||||
- 返回 `material_types` 里的 `label/value`
|
||||
|
||||
它的数据来源是:
|
||||
|
||||
- config:`material_types`
|
||||
|
||||
第二条路径:`/material-detail/contract-product-type/list`
|
||||
|
||||
- `router -> MaterialDetailService.get_material_part_options()`
|
||||
- service 内部同样会先读项目的 `material_types` config
|
||||
- 再按当前 material 的 `material_type` 找到对应 kit 和 parts
|
||||
|
||||
它的数据来源也是:
|
||||
|
||||
- config:`material_types`
|
||||
|
||||
但同时,物资模块里还保留了老的代码常量路径:
|
||||
|
||||
- `MaterialService.get_material_info_by_code()` 直接读 `contract_product_type_list`
|
||||
- `MaterialDetailService.get_kit_options()` 也直接读 `contract_product_type_list`
|
||||
|
||||
和初始化是否一致:
|
||||
|
||||
- 也不一致。
|
||||
- 前端下拉主要已经走 `material_types` config。
|
||||
- 但初始化物资时,候选项来源还是 `contract_config`,单位来源还是 `material_contract_*` dict。
|
||||
- 而创建 service 内部补 `show_name/unit` 时,又还在读代码常量 `contract_product_type_list`。
|
||||
|
||||
这是当前分叉最明显的一类:
|
||||
|
||||
- 前端选项:config
|
||||
- 初始化来源:config + dict
|
||||
- 创建补充逻辑:代码常量
|
||||
|
||||
### 12.3 Construction 的前端下拉路径
|
||||
|
||||
当前相关接口主要有:
|
||||
|
||||
- `/construction/section/type/list`
|
||||
- `/construction/section/sub/type/list`
|
||||
- `/construction/task/type/list`
|
||||
|
||||
返回路径:
|
||||
|
||||
- `router -> ConstructionService.get_section_type_list(project_type)`
|
||||
- `router -> ConstructionService.get_section_sub_type_list(section_type_code, project_type)`
|
||||
- `router -> ConstructionService.get_task_type_list(section_type_code, section_sub_type_code, project_type)`
|
||||
|
||||
数据来源:
|
||||
|
||||
- `section_type_list` 通过 `config_helper.section_type_id_dict()` 读取 `construction_section_config`
|
||||
- `section_sub_type_list` 通过 `config_helper.section_type_project_root_node_mapping()` 读取 `construction_section_config.project_root_nodes`
|
||||
- `task_type_list` 则继续基于 `construction_section_config` 和 `ConstructionTypeTree` 生成
|
||||
|
||||
和初始化是否一致:
|
||||
|
||||
- 基本一致。
|
||||
- 施工初始化本来就是 `construction_section_config + 类型树` 驱动。
|
||||
- 前端下拉接口也是同一条配置/类型树路径。
|
||||
|
||||
所以施工是目前 3 类里最接近“初始化与前端下拉同源”的。
|
||||
|
||||
### 12.4 总结
|
||||
|
||||
从前端下拉接口角度看,当前 3 类字段的真实状态可以概括成:
|
||||
|
||||
1. Contract:前端下拉走代码字典,和初始化 `contract_config` 不一致。
|
||||
2. Material:前端下拉主要走 `material_types` config,但初始化和创建内部仍分别依赖 `contract_config`、`material_contract_*` dict、`contract_product_type_list` 代码常量,分叉最严重。
|
||||
3. Construction:前端下拉和初始化都主要走 `construction_section_config + 类型树`,相对一致。
|
||||
|
||||
所以如果后续要做收口,优先级也比较明确:
|
||||
|
||||
- 合同:先统一 `contract_config` 与前端 `type/sub-type list`。
|
||||
- 物资:先统一 `material_types`、`contract_config`、`material_contract_*`、`contract_product_type_list` 四套来源。
|
||||
- 施工:优先补 schema/创建校验与配置同源,配置来源本身相对已经收敛。
|
||||
@@ -0,0 +1,201 @@
|
||||
# Material 领域重构指南
|
||||
|
||||
## 1. 范围
|
||||
|
||||
本文档只做 Material 主表梳理。
|
||||
|
||||
当前优先关注:
|
||||
|
||||
- `material.material_type`
|
||||
- `material.status`
|
||||
- `material.supply_status`
|
||||
- `material.approval_party`
|
||||
|
||||
## 2. 字段盘点
|
||||
|
||||
### 2.1 `material_type`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 本地没有独立 enum 类
|
||||
- 主要来源是 `contract_product_type_list`
|
||||
- 模型里是 `String`
|
||||
- 创建 schema 里是 `str`
|
||||
- 同步 schema 里也是 `str`
|
||||
|
||||
当前使用:
|
||||
|
||||
- 是物资主类型字段
|
||||
- schema 示例和业务逻辑都直接依赖 `contract_product_type_list`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前本地明确值:`TY`、`ZJ`、`NB`、`XB`、`ZB`、`GS`、`CN`、`DL`、`XT`、`CBJDB`、`SX`、`ECXT`、`QT`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧 `enum_config.Material.material_type` 是按项目类型变化的
|
||||
- 光伏下有:`ZJ`、`TY`、`NB`、`XB`、`ZB`、`GS`、`CN`、`QT`
|
||||
- 陆风/海风下有:`FJ`、`TT`、`XB`、`ZB`、`GS`、`CN`、`QT`
|
||||
- 项目侧当前使用的本地列表明显偏光伏,且代码里还存在命名差异:例如本地用 `XT` / `CBJDB` / `SX` / `ECXT`,集团侧常见的是 `35KV` / `CYJD` / `SVG` / `ECPG`
|
||||
- 同步 schema 里 `material_type` 仍是 `str`
|
||||
|
||||
结论:
|
||||
|
||||
- 应迁到 `field_config`
|
||||
- 不能继续只靠本地 `contract_product_type_list`
|
||||
|
||||
### 2.2 `status`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `SmallInteger`
|
||||
- 同步 schema 里是 `Optional[int]`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前未看到单独 enum 类
|
||||
- 现有注释和使用都表明它是数值状态字段
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧 `enum_config.Material.status` 给的是 `-1`、`0`、`1`
|
||||
- 项目侧没有在 material 模块里沉成统一常量
|
||||
|
||||
结论:
|
||||
|
||||
- 应统一正式状态定义源
|
||||
|
||||
### 2.3 `supply_status`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `SmallInteger`
|
||||
- 本地有 `SupplyStatus` enum:`-1`、`0`、`1`、`None`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- `-1`
|
||||
- `0`
|
||||
- `1`
|
||||
- `None`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧 Material.status 也是按 `-1`、`0`、`1` 表达进度状态
|
||||
- 项目侧存在 `status` 和 `supply_status` 两个相近字段,语义有重叠
|
||||
|
||||
结论:
|
||||
|
||||
- 需要明确 `status` 和 `supply_status` 哪个才是正式推送字段
|
||||
|
||||
### 2.4 `approval_party`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `String`
|
||||
- schema 里是 `str`
|
||||
- 当前示例值是 `jiagong`、`yigong`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- `jiagong`
|
||||
- `yigong`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 当前没看到集团侧 `enum_config` 对这个字段给出正式候选项
|
||||
- 同步 schema 里也没有这个字段
|
||||
|
||||
结论:
|
||||
|
||||
- 先保留本地代码级候选项
|
||||
- 后续如需推送再决定是否外提
|
||||
|
||||
## 3. 重点结论
|
||||
|
||||
- `material_type` 是标准的 `field_config` 候选字段,不能再继续绑死在本地光伏型列表上。
|
||||
- `status` 与 `supply_status` 当前语义重叠,必须先定正式推送字段。
|
||||
- `approval_party` 暂时还是本地专有字段,不需要现在就硬并到集团侧。
|
||||
|
||||
## 4. 待修改项清单
|
||||
|
||||
1. 将 `material_type` 正式迁到 `field_config`。
|
||||
2. 按项目类型重建 `material_type` 候选项,不再只用本地光伏型列表。
|
||||
3. 统一 `35KV/XT`、`CYJD/CBJDB`、`SVG/SX`、`ECPG/ECXT` 这类命名差异。
|
||||
4. 明确 `status` 与 `supply_status` 的职责边界。
|
||||
5. 为 `material_type` 在 schema 层增加统一校验入口。
|
||||
|
||||
## 5. 按本轮要求的改造落点
|
||||
|
||||
material 是本轮最需要针对性改造的领域之一,因为它当前同时存在:
|
||||
|
||||
- 初始化来源分叉
|
||||
- create 校验分叉
|
||||
- 下拉来源分叉
|
||||
|
||||
### 5.1 初始化
|
||||
|
||||
当前 material 初始化不是单源:
|
||||
|
||||
- 候选项来自 `contract_config` 里的 product 合同
|
||||
- 单位和部分展示信息来自 `material_contract_{project_type}` dict
|
||||
|
||||
而且 `unit` 现在还挂在 dict `remark` 上。
|
||||
|
||||
本轮目标是:
|
||||
|
||||
- 明确 `material.material_type` 的正式配置来源
|
||||
- 把 `label`、`unit`、`parts` 等附属字段纳入正式 config 结构
|
||||
- 去掉“从 dict remark 补单位”这种不适当用法
|
||||
|
||||
### 5.2 Create 校验
|
||||
|
||||
当前 create 路径里:
|
||||
|
||||
- `material_type` 入参本身没有正式 config 校验
|
||||
- `show_name`、`unit` 由 `contract_product_type_list` 反查
|
||||
- 随后又会调用合同创建,由合同 service 再做一层 code 白名单校验
|
||||
|
||||
这说明 material create 现在不是单一入口校验。
|
||||
|
||||
本轮目标是:
|
||||
|
||||
- 在 `MaterialService.create()` 开头基于 config 校验 `material_type`
|
||||
- 用 `get_field_option_map()` 之类的方法直接拿 `label`、`unit`、`parts`
|
||||
- 不再依赖 `contract_product_type_list` 和默认兜底值
|
||||
|
||||
### 5.3 下拉列表
|
||||
|
||||
当前物资下拉已经部分走 config:
|
||||
|
||||
- `/material/material-types` 读 `material_types`
|
||||
- `/material-detail/contract-product-type/list` 也读 `material_types`
|
||||
|
||||
但模块里仍保留了旧的硬编码路径:
|
||||
|
||||
- `get_material_info_by_code()`
|
||||
- `get_kit_options()`
|
||||
- `contract_product_type_list`
|
||||
|
||||
本轮目标是:
|
||||
|
||||
- 让物资类型和部件列表统一从新的 config 工具返回
|
||||
- 下拉、初始化、create 校验全部围绕同一套 `material_type` 配置
|
||||
|
||||
### 5.4 清理项
|
||||
|
||||
material 领域这轮改造完成后,应优先清理:
|
||||
|
||||
- `material_contract_*` 中用 `remark` 承载单位的旧逻辑
|
||||
- `contract_product_type_list` 里只服务于 `material_type` 真源的旧定义
|
||||
- `get_material_info_by_code()` 这类旧反查入口
|
||||
|
||||
## 6. 测试关注点
|
||||
|
||||
material 领域 service 层集成测试至少覆盖:
|
||||
|
||||
1. 初始化:项目初始化后 material 的 `material_type`、`show_name`、`unit` 是否来自目标 config。
|
||||
2. 下拉:物资类型和部件列表是否来自同一 config,且与初始化同源。
|
||||
3. Create:合法 `material_type` 能创建,非法 `material_type` 明确报错,不再依赖默认兜底。
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
# Personnel 领域重构指南
|
||||
|
||||
## 1. 范围
|
||||
|
||||
当前仓库里没有独立的 `personnel` 项目业务模块。
|
||||
|
||||
最接近的本地实现是:
|
||||
|
||||
- `app/modules/safety/models/safety_personnel_qualification.py`
|
||||
- `app/modules/safety/schemas/personnel_qualification.py`
|
||||
|
||||
因此本文档先按“人员资质”这一真实落点梳理,而不是假设仓库里已经有独立 `personnel` 主表。
|
||||
|
||||
当前优先关注:
|
||||
|
||||
- `personnel_type`
|
||||
- `task_type`
|
||||
- `is_duty`
|
||||
|
||||
## 2. 字段盘点
|
||||
|
||||
### 2.1 `personnel_type`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里通过类常量定义两个值:`safe`、`special`
|
||||
- 模型字段本身是 `String`
|
||||
- schema 里仍是 `str`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- `safe`
|
||||
- `special`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧 `enum_config.Personnel` 里并没有 `personnel_type`
|
||||
- 集团侧更像是在描述“人员/吊装设备”等资源配置编码,而不是人员资质类型
|
||||
- 当前仓库里也没有 standalone 的 sync schema 对这个领域建模
|
||||
|
||||
结论:
|
||||
|
||||
- 这是本地代码级 enum 字段
|
||||
- 可保留为代码 enum
|
||||
|
||||
### 2.2 `task_type`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里通过 `PERSONNEL_TASK_TYPE` 类常量给出候选项
|
||||
- 字段本身是 `String`
|
||||
- schema 里仍是 `str`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- `电工`
|
||||
- `焊工`
|
||||
- `高空作业人员`
|
||||
- `无人机操作员`
|
||||
- `其他`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧 `enum_config.Personnel` 没有这个字段
|
||||
- 当前仓库里也没有 sync schema 对应
|
||||
|
||||
结论:
|
||||
|
||||
- 这是本地代码级候选项字段
|
||||
- 可以保留为代码常量,但要补 schema 校验
|
||||
|
||||
### 2.3 `is_duty`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里通过类常量定义:`1` 在场,`2` 退场
|
||||
- 字段本身是 `SmallInteger`
|
||||
- schema 里是 `int`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- `1`
|
||||
- `2`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧 `enum_config.Personnel.status` 给的是 `-1`、`0`、`1`
|
||||
- 这更像资源进度状态,不是本地 `is_duty` 语义
|
||||
- 两边不是同一字段,不应强行对齐
|
||||
|
||||
结论:
|
||||
|
||||
- 保留为本地数值常量字段
|
||||
|
||||
## 3. 重点结论
|
||||
|
||||
- 当前所谓 `personnel` 领域,在本地真实落点是“人员资质”,不是集团侧 `Personnel` 那套资源配置模型。
|
||||
- 因此这里最重要的不是照抄集团侧,而是先承认两个系统的领域边界不一致。
|
||||
|
||||
## 4. 待修改项清单
|
||||
|
||||
1. 明确 `personnel` 在本地的正式领域名是否就是 `safety_personnel_qualification`。
|
||||
2. 为 `personnel_type` 补正式 enum/schema 校验。
|
||||
3. 为 `task_type` 补统一白名单校验。
|
||||
4. 不要把集团侧 `Personnel.code/status` 直接映射到本地 `personnel_type/is_duty`。
|
||||
|
||||
## 5. 按本轮要求的改造落点
|
||||
|
||||
personnel 领域不是这轮“项目初始化 + 项目类型 field_config”第一批重点,但仍需要按统一口径写清楚 create、下拉和清理边界。
|
||||
|
||||
### 5.1 初始化
|
||||
|
||||
本地人员资质领域当前没有项目初始化批量灌数链路。
|
||||
|
||||
因此本轮不把它纳入 `init_project_data)` 第一批改造范围。
|
||||
|
||||
### 5.2 Create 校验
|
||||
|
||||
当前 `PersonnelQualificationService.create()` 已直接在 service 层校验:
|
||||
|
||||
- `personnel_type`
|
||||
- `task_type`
|
||||
|
||||
校验来源是本地模型类常量:
|
||||
|
||||
- `PERSONNEL_TYPE_SAFE`
|
||||
- `PERSONNEL_TYPE_SPECIAL`
|
||||
- `PERSONNEL_TASK_TYPE`
|
||||
|
||||
这类字段当前更像固定语义本地枚举,不是按 `project_type` 变化的字段级配置候选项。
|
||||
|
||||
本轮目标是:
|
||||
|
||||
- 继续把它们视为本地固定候选项
|
||||
- 统一 create / update / list 使用同一套定义源
|
||||
- 不误迁到项目类型 field_config
|
||||
|
||||
### 5.3 下拉列表
|
||||
|
||||
当前已有:
|
||||
|
||||
- `/safety/personnel/task_type/list`
|
||||
|
||||
它直接返回 `PERSONNEL_TASK_TYPE`。
|
||||
|
||||
本轮这里不要求强行改成 `project_type` 相关 config,而是要求:
|
||||
|
||||
- 明确它是本地固定候选项
|
||||
- 前端下拉和 create 校验继续同源
|
||||
|
||||
### 5.4 清理项
|
||||
|
||||
personnel 领域本轮的清理重点不是删 enum,而是避免错误地把集团侧 `Personnel` 语义强行映射进来。
|
||||
|
||||
应避免:
|
||||
|
||||
- 把本地 `personnel_type`、`task_type` 误改造成和集团侧 `code/status` 一一对应
|
||||
- 同一字段在 service 和下拉各自维护两套值
|
||||
|
||||
## 6. 测试关注点
|
||||
|
||||
personnel 领域 service 层集成测试至少覆盖:
|
||||
|
||||
1. `personnel_type` 合法值和非法值创建校验。
|
||||
2. `special` 人员必须校验 `task_type`。
|
||||
3. `task_type/list` 返回值与 create 校验使用同源常量。
|
||||
|
||||
@@ -0,0 +1,697 @@
|
||||
# 项目侧字段级配置重构计划
|
||||
|
||||
## 1. 目标
|
||||
|
||||
本次重构的目标,不是先统一项目侧全部 config / enum / dict 的历史实现,而是先补一套项目侧自己的“字段级配置总表”和配套工具,使下面这类问题有统一解法:
|
||||
|
||||
- 某个字段在某个 `project_type` 下有哪些可选项。
|
||||
- 这些可选项的中文名是什么。
|
||||
- 业务代码如何方便地做手动校验。
|
||||
- 初始化流程如何从同一套字段级配置中读取候选项。
|
||||
- 哪些字段应该继续在代码里用 enum 处理。
|
||||
- 哪些字段应该迁到配置文件处理。
|
||||
|
||||
这套能力对齐集团侧 `enum_config.json` 的思路,但项目侧不照搬集团侧的存储形式,而是基于现有 `config` 表和 CSV 数据维护方式落地。
|
||||
|
||||
因此,本次重构除了建设字段级配置总表本身,还需要同步完成一件事:
|
||||
|
||||
- 梳理字段边界,明确哪些字段属于“固定语义、固定值集合、适合保留在代码 enum 中”,哪些字段属于“随 `project_type` 或初始化流程变化、应迁到字段级配置中”。
|
||||
|
||||
也就是说,这次不是单纯补一个配置表,而是要同时建立一套可执行的字段分流规则:
|
||||
|
||||
- enum 字段:继续留在代码中,由 enum 表达固定约束。
|
||||
- field_config 字段:迁到配置中,由字段级配置表达候选项。
|
||||
|
||||
## 1.1 本轮实际要改的内容
|
||||
|
||||
这次文档和后续代码改造,不再停留在“抽象上应该统一”的层面,而是直接围绕下面 5 件事展开。
|
||||
|
||||
### 0. 通用能力
|
||||
|
||||
要补两类公共能力:
|
||||
|
||||
- 增强现有配置类,在 `ConfigService` 上新增字段级配置读取、映射、校验工具。
|
||||
- 增强基于 JSON 导出 CSV 的工具,使字段级配置和现有 config 维护流程能直接联动。
|
||||
|
||||
### 1. 优化项目初始化过程
|
||||
|
||||
目标不是只分析 `init_project_data)`,而是把初始化链路改成统一从配置工具取候选项:
|
||||
|
||||
- 整理项目初始化当前实际使用到的 config。
|
||||
- 把适合迁移的初始化候选项改为通过新的 config 工具获取后创建。
|
||||
- 去掉不适当的 dict 用法,例如把单位塞在 dict `remark` 里。
|
||||
|
||||
### 2. 优化创建数据的验证
|
||||
|
||||
这次不改 schema 目录类型系统,但要改 service 层入口。
|
||||
|
||||
要求是:
|
||||
|
||||
- 在 create 类接口对应的 service 业务入口开头,基于 config 工具校验 code 类字段。
|
||||
- 不再继续新增代码字典白名单、硬编码 if/else 白名单。
|
||||
|
||||
### 3. 统一下拉列表来源
|
||||
|
||||
这次不仅要梳理下拉列表,还要把它们收口到新的 config 工具:
|
||||
|
||||
- 梳理所有 code 类下拉列表当前返回路径。
|
||||
- 能迁移的下拉统一改成通过 config 工具返回。
|
||||
- 少量现有写死的下拉列表,例如合同结算类型列表,也要改成动态获取。
|
||||
|
||||
### 4. 删除废弃 enum / 硬编码字典
|
||||
|
||||
完成上述迁移后,要清掉已经不再适合作为真源的:
|
||||
|
||||
- 废弃 enum
|
||||
- 废弃硬编码 dict
|
||||
- 只用于旧下拉或旧创建校验的代码常量
|
||||
|
||||
### 5. 自动化测试脚本
|
||||
|
||||
这次不是只改文档和代码,还要补自动化测试,覆盖至少 3 类集成场景:
|
||||
|
||||
- 初始化脚本 / 初始化 service 集成测试
|
||||
- 各业务下拉列表获取集成测试
|
||||
- 各业务 create service 集成测试
|
||||
|
||||
测试脚本写在 `tests` 下,不写到项目根目录。
|
||||
|
||||
## 2. 重构范围
|
||||
|
||||
本次重构只覆盖一类字段:
|
||||
|
||||
- 对应集团侧 `enum_config.json` 相应字段的项目侧字段。
|
||||
- 和 `project_type` 相关。
|
||||
- 和项目初始化过程相关。
|
||||
|
||||
也就是说,本次优先处理的是“初始化候选项字段”,而不是全部字符串字段。
|
||||
|
||||
优先关注的业务范围包括:
|
||||
|
||||
- project 初始化时需要的字段候选项
|
||||
- contract 初始化相关字段
|
||||
- construction 初始化相关字段
|
||||
- material 初始化相关字段
|
||||
- power / preparation / production 中与项目类型初始化直接相关的字段
|
||||
|
||||
从现状代码看,这些初始化来源当前混杂在:
|
||||
|
||||
- Python enum
|
||||
- Python 硬编码 dict
|
||||
- `dict_data`
|
||||
- `config` 表中的自由 JSON
|
||||
- 外部业务大表或类型树
|
||||
|
||||
本次要做的不是一次性清理所有来源,而是为这些字段补一个统一的字段级配置出口。
|
||||
|
||||
对本轮来说,这里的“统一出口”不是泛指,而是明确服务于 3 条业务链路:
|
||||
|
||||
- 初始化:通过统一配置工具取候选项并创建数据
|
||||
- 创建:通过统一配置工具校验 code 类字段
|
||||
- 下拉:通过统一配置工具返回选项列表
|
||||
|
||||
对本轮来说,这里的“统一出口”不是泛指,而是明确服务于 3 条业务链路:
|
||||
|
||||
- 初始化:通过统一配置工具取候选项并创建数据
|
||||
- 创建:通过统一配置工具校验 code 类字段
|
||||
- 下拉:通过统一配置工具返回选项列表
|
||||
|
||||
## 3. 不在本次范围内的内容
|
||||
|
||||
以下内容明确不在本次重构范围:
|
||||
|
||||
- 接口返回层的翻译统一
|
||||
- `dict` 体系的整体重构
|
||||
- 所有历史自由 config 的一次性收敛
|
||||
- 所有外部树和业务大表的一次性字段化
|
||||
- schema 目录下的类型系统改造
|
||||
- 运行时直接读取 JSON 文件作为配置源
|
||||
|
||||
这里补充两条边界:
|
||||
|
||||
- 不允许为了兼容旧逻辑继续新增 dict `remark` 之类的隐式字段承载方式。
|
||||
- 不允许为了减少改动,在新工具落地后继续保留“前端下拉走 config、初始化走 dict、create 校验走代码字典”的三头并存状态。
|
||||
|
||||
这里补充两条边界:
|
||||
|
||||
- 不允许为了兼容旧逻辑继续新增 dict `remark` 之类的隐式字段承载方式。
|
||||
- 不允许为了减少改动,在新工具落地后继续保留“前端下拉走 config、初始化走 dict、create 校验走代码字典”的三头并存状态。
|
||||
|
||||
因此,本次允许字段级配置中的 `value / label` 与现有 dict 翻译短期重复。后续如有需要,可再通过脚本生成 dict 或做统一翻译收敛。
|
||||
|
||||
## 4. 核心设计
|
||||
|
||||
### 4.1 新增“字段级配置总表”概念
|
||||
|
||||
项目侧需要新增一层固定格式的字段级配置。
|
||||
|
||||
这层配置的职责只有一个:
|
||||
|
||||
- 对外统一回答“某字段在某项目类型下有哪些可选项”。
|
||||
|
||||
它不是为了取代所有业务 config,而是为了给业务代码、初始化流程、推送系统提供统一的读取接口。
|
||||
|
||||
### 4.2 存储方式
|
||||
|
||||
字段级配置继续存放在现有 `config` 表中,并通过现有 CSV 数据维护。
|
||||
|
||||
建议约束如下:
|
||||
|
||||
- `category = project_type`
|
||||
- `config_type = json`
|
||||
- `config_key` 不强制按规则自动拼接
|
||||
- 代码里仍允许手动传入 key 查询
|
||||
|
||||
虽然推荐把 key 命名为 `model_name.field_name`,例如:
|
||||
|
||||
- `construction.section_type_code`
|
||||
- `material.material_type`
|
||||
- `contract.code`
|
||||
|
||||
但这只是推荐命名,不作为框架自动推导前提。
|
||||
|
||||
也就是说:
|
||||
|
||||
- 本次不建设“根据 model + field 自动拼 config key”的机制
|
||||
- 只建设“拿到 key 后,按统一 schema 读取字段级配置”的机制
|
||||
|
||||
### 4.3 固定 JSON 结构
|
||||
|
||||
字段级配置的 `default_value` / `config_value` 必须是一个 JSON object。
|
||||
|
||||
对于字段级配置,只强约束一件事:
|
||||
|
||||
- 顶层必须有 `field_config`
|
||||
|
||||
其中:
|
||||
|
||||
- `field_config` 必须是列表
|
||||
- 列表中每个元素至少有 `value`、`label`
|
||||
|
||||
推荐最小结构如下:
|
||||
|
||||
```json
|
||||
{
|
||||
"field_config": [
|
||||
{"value": "GF", "label": "光伏场区", "weight": 1.4},
|
||||
{"value": "SY", "label": "升压站", "weight": 1.4}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
除 `field_config` 外,其它顶层字段不做统一规定,由各业务自己定义。
|
||||
|
||||
例如,某些字段配置可能会额外包含:
|
||||
|
||||
- `description`
|
||||
- `init_config`
|
||||
- 其他与该字段强相关的附属配置
|
||||
|
||||
这些字段允许存在,但不纳入统一字段级 schema 强约束。
|
||||
|
||||
### 4.4 非字段级 config 的统一要求
|
||||
|
||||
对于仍然保留的自由业务 config,不要求包含 `field_config`。
|
||||
|
||||
但建议默认增加一个顶层字段:
|
||||
|
||||
- `description`
|
||||
|
||||
这样至少能保证所有 config 在文档和工具层面具备最低限度的语义说明。
|
||||
|
||||
## 5. `field_config` Schema 设计
|
||||
|
||||
### 5.1 字段候选项结构
|
||||
|
||||
字段级配置中的 `field_config` 列表项,最少包含:
|
||||
|
||||
- `value: str`
|
||||
- `label: str`
|
||||
|
||||
允许附加字段,例如:
|
||||
|
||||
- `weight`
|
||||
- `unit`
|
||||
- `section`
|
||||
- `legacy`
|
||||
- `flag`
|
||||
- `skip_init`
|
||||
- `skip_create`
|
||||
|
||||
这些附加字段不要求在所有字段中统一,但它们必须和值本身强相关。
|
||||
|
||||
### 5.2 建议的 Pydantic 模型
|
||||
|
||||
建议新增专门的 Pydantic schema,用于字段级配置校验。
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
|
||||
class FieldOptionItem(BaseModel):
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
value: str = Field(..., description="候选值")
|
||||
label: str = Field(..., description="显示名")
|
||||
```
|
||||
|
||||
说明:
|
||||
|
||||
- 只需要识别 `field_config` 这个字段
|
||||
- 只需要校验 `field_config` 是 `list[FieldOptionItem]`
|
||||
- 其它顶层字段不在统一 schema 里写死
|
||||
|
||||
## 6. `ConfigService` 统一承载
|
||||
|
||||
字段级配置相关的辅助方法统一放到 `ConfigService`,不再额外拆新的独立 service。
|
||||
|
||||
也就是说,这次应增强的是:
|
||||
|
||||
- `app/modules/system/services/config.py`
|
||||
|
||||
而不是在别处再起一套并行 helper。
|
||||
|
||||
### 6.1 建议新增的方法
|
||||
|
||||
```python
|
||||
async def get_field_config(project_type: str, key: str, tenant_id: str | None = None) -> dict[str, Any]:
|
||||
...
|
||||
|
||||
|
||||
async def get_field_options(project_type: str, key: str, tenant_id: str | None = None) -> list[dict[str, Any]]:
|
||||
...
|
||||
|
||||
|
||||
async def get_field_option_map(project_type: str, key: str, tenant_id: str | None = None) -> dict[str, dict[str, Any]]:
|
||||
...
|
||||
|
||||
|
||||
async def get_field_label(project_type: str, key: str, value: str, tenant_id: str | None = None) -> str | None:
|
||||
...
|
||||
|
||||
|
||||
async def validate_field_value(
|
||||
project_type: str,
|
||||
key: str,
|
||||
value: str,
|
||||
tenant_id: str | None = None,
|
||||
*,
|
||||
error_prefix: str | None = None,
|
||||
) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
如果后续需要覆盖“批量校验多个字段”或“直接返回下拉结构”的场景,可以继续补:
|
||||
|
||||
```python
|
||||
async def get_field_options_for_frontend(project_type: str, key: str, tenant_id: str | None = None) -> list[dict[str, Any]]:
|
||||
...
|
||||
|
||||
|
||||
async def validate_field_values(project_type: str, items: list[tuple[str, str]], tenant_id: str | None = None) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
如果后续需要覆盖“批量校验多个字段”或“直接返回下拉结构”的场景,可以继续补:
|
||||
|
||||
```python
|
||||
async def get_field_options_for_frontend(project_type: str, key: str, tenant_id: str | None = None) -> list[dict[str, Any]]:
|
||||
...
|
||||
|
||||
|
||||
async def validate_field_values(project_type: str, items: list[tuple[str, str]], tenant_id: str | None = None) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
### 6.2 各接口职责
|
||||
|
||||
`get_field_config()`:
|
||||
|
||||
- 读取指定 `project_type + key` 的 JSON 配置
|
||||
- 找出顶层 `field_config`
|
||||
- 校验 `field_config` 是否满足 `list[FieldOptionItem]`
|
||||
- 返回完整配置对象
|
||||
|
||||
`get_field_options()`:
|
||||
|
||||
- 返回 `field_config` 列表
|
||||
- 供初始化、下拉、候选项获取直接使用
|
||||
|
||||
`get_field_option_map()`:
|
||||
|
||||
- 返回 `{value: option}` 的映射
|
||||
- 供快速查找、补充属性读取使用
|
||||
|
||||
`get_field_label()`:
|
||||
|
||||
- 根据 `value` 返回 `label`
|
||||
- 本次主要服务于业务内部使用,不承担接口翻译统一职责
|
||||
|
||||
`validate_field_value()`:
|
||||
|
||||
- 手动调用
|
||||
- 校验给定值是否在当前项目类型该字段的候选项中
|
||||
- 校验失败时抛出业务异常
|
||||
|
||||
另外,本轮会直接用到两种典型调用方式:
|
||||
|
||||
- 初始化时:`get_field_options()` / `get_field_option_map()`
|
||||
- 创建和编辑时:`validate_field_value()`
|
||||
- 下拉接口时:`get_field_options_for_frontend()` 或 `get_field_options()`
|
||||
|
||||
另外,本轮会直接用到两种典型调用方式:
|
||||
|
||||
- 初始化时:`get_field_options()` / `get_field_option_map()`
|
||||
- 创建和编辑时:`validate_field_value()`
|
||||
- 下拉接口时:`get_field_options_for_frontend()` 或 `get_field_options()`
|
||||
|
||||
这里特别强调:
|
||||
|
||||
- `validate` 是手动调用的
|
||||
- 本次不做自动注入所有模型或 schema 的统一校验框架
|
||||
- 目标是让业务层“方便调用”,不是一次性接管所有校验入口
|
||||
|
||||
### 6.3 `ConfigService` 需要补齐的基础能力
|
||||
|
||||
`ConfigService` 当前除了读取外,还需要补齐两类基础能力:
|
||||
|
||||
#### 1. JSON 序列化 / 反序列化统一承载
|
||||
|
||||
不应继续让调用方自己到处 `msgspec.json.decode(...)`。
|
||||
|
||||
`ConfigService` 应统一负责:
|
||||
|
||||
- 读取 JSON 配置时反序列化
|
||||
- 更新 JSON 配置时序列化
|
||||
- 在字段级配置方法中复用这一套逻辑
|
||||
|
||||
也就是说,配置的 JSON 处理应沉到 `ConfigService` 本身,而不是散落在各业务模块里。
|
||||
|
||||
#### 2. 关键读取方法加缓存
|
||||
|
||||
`ConfigService` 中的关键读取方法应增加缓存。
|
||||
|
||||
优先包括:
|
||||
|
||||
- `get_config_json()`
|
||||
- `get_field_config()`
|
||||
- `get_field_options()`
|
||||
- `get_field_option_map()`
|
||||
- `get_field_label()`
|
||||
|
||||
要求:
|
||||
|
||||
- 读接口走缓存
|
||||
- 更新配置后显式失效相关缓存
|
||||
- 不允许缓存让旧配置长期滞留
|
||||
|
||||
`validate_field_value()` 自身不需要单独缓存,但应复用已经缓存过的字段选项读取结果。
|
||||
|
||||
### 6.4 实现要求
|
||||
|
||||
工具类内部应统一处理:
|
||||
|
||||
- 读取 `config` 表 JSON
|
||||
- 用 Pydantic 校验 `field_config`
|
||||
- 构建 options / map / label
|
||||
- 输出清晰错误,便于发现配置格式问题
|
||||
|
||||
不允许的处理方式:
|
||||
|
||||
- 遇到 schema 错误时静默降级
|
||||
- 遇到缺字段时自动猜测兼容成其他结构
|
||||
|
||||
如果配置不满足 `field_config` schema,应直接报错。
|
||||
|
||||
## 7. JSON / CSV 工具重整
|
||||
|
||||
本次不要求运行时直接读取 JSON 文件。
|
||||
|
||||
但为了方便调试和维护,应把现有 `scripts/tool/csv_data_tools/` 按新的字段级配置目标重新整理。
|
||||
|
||||
目标是:
|
||||
|
||||
- 一个 key 对应一个 JSON 文件
|
||||
- JSON 文件仍然采用 config 行列表格式
|
||||
- 工具负责 JSON 和 `config.csv`、数据库之间的转换
|
||||
|
||||
这块和本轮目标直接对应:
|
||||
|
||||
- 字段级配置调整后,可以直接导出为 `config.csv`
|
||||
- 初始化改造前后,可以快速对比 config 差异
|
||||
- 下拉列表和创建校验依赖的 config,可以单独导出检查,不再手工翻长 CSV
|
||||
|
||||
这块和本轮目标直接对应:
|
||||
|
||||
- 字段级配置调整后,可以直接导出为 `config.csv`
|
||||
- 初始化改造前后,可以快速对比 config 差异
|
||||
- 下拉列表和创建校验依赖的 config,可以单独导出检查,不再手工翻长 CSV
|
||||
|
||||
建议目录:
|
||||
|
||||
- `scripts/initdata/json_config/`
|
||||
|
||||
建议能力:
|
||||
|
||||
- `validate-json-config`: 校验 JSON 文件是否满足 config 行结构,以及 `field_config` 约束
|
||||
- `json-to-csv`: 从 `json_config` 目录批量生成 `config.csv`
|
||||
- `json-to-db`: 从 `json_config` 目录批量导入数据库
|
||||
- `csv-to-json`: 从 `config.csv` 按 key 导出 JSON 文件
|
||||
- `db-to-json`: 从数据库导出 JSON 文件用于调试
|
||||
|
||||
当前已有的 `scripts/tool/csv_data_tools/csv_data_tools.py` 可以复用思路,但需要围绕“字段级配置总表 + 一 key 一文件”的目标重新整理。
|
||||
|
||||
## 8. 领域重构指南
|
||||
|
||||
主计划只定义系统级方案。具体领域的梳理应拆到独立文件,例如:
|
||||
|
||||
- [construction.md](/Users/zyl/my_projects/ecm_sync_system/docs/enum_refactor/construction.md)
|
||||
- [project.md](/Users/zyl/my_projects/ecm_sync_system/docs/enum_refactor/project.md)
|
||||
|
||||
后续 contract / material 等领域也应按同样方式分别整理。
|
||||
|
||||
每个领域文档都不是只看“集团侧已有字段”或者“准备迁到 `field_config` 的字段”。
|
||||
|
||||
每个领域都必须同时覆盖两部分字段:
|
||||
|
||||
- depm 中所有已经使用以下来源之一的字段:代码 enum、代码中的初始化列表、硬编码 dict、类 enum、`config` 表、`dict` 表
|
||||
- 集团侧 `enum_config` 中该领域涉及的全部字段
|
||||
|
||||
两部分做并集,不做交集。
|
||||
|
||||
也就是说:
|
||||
|
||||
- `enum_config` 中出现的字段,必须全部纳入领域文档
|
||||
- 即使某字段暂时没有出现在 `enum_config` 中,只要 depm 已经在用代码 enum / 类 enum / 初始化列表 / 硬编码 dict / `config` 表 / `dict` 表承载它,并且它影响初始化、手动创建、编辑、校验、展示,也必须纳入领域文档
|
||||
|
||||
这里要特别避免一种错误写法:
|
||||
|
||||
- 不能把领域范围写成“只讨论和集团侧 `enum_config` 直接对应的字段”
|
||||
- 不能因为某字段暂时只有 depm 本地 enum / 类 enum,而集团侧还没有对应项,就把它排除在领域文档之外
|
||||
|
||||
每个领域文档都应至少覆盖以下内容:
|
||||
|
||||
### 8.1 先做字段总表
|
||||
|
||||
每个领域文档开头都应先列出字段总表,至少回答:
|
||||
|
||||
- 集团侧 `enum_config` 覆盖了哪些字段
|
||||
- depm 侧当前还额外维护了哪些相关字段
|
||||
- 每个字段当前使用的来源是什么:代码 enum、初始化列表、硬编码 dict、类 enum、`config` 表、`dict` 表,或它们的混用
|
||||
- 每个字段分别影响哪些链路:项目初始化、手动创建、手动编辑、接口展示、历史数据兼容、业务校验
|
||||
|
||||
如果同一个字段在多个来源重复定义,也必须明确记出来,而不是只记录“最终准备保留哪个”。
|
||||
|
||||
### 8.2 整理不查表的 enum 类数据
|
||||
|
||||
不是只有查表型数据才需要整理。
|
||||
|
||||
如果某领域存在不依赖字典表、也不依赖 `config` 表的纯 enum / 纯硬编码候选项,同样要梳理:
|
||||
|
||||
- 当前定义位置在哪
|
||||
- 是否重复定义
|
||||
- 是否和集团侧 `enum_config` 对应字段有关
|
||||
- 是否参与初始化、手动创建、手动编辑、展示翻译或历史校验
|
||||
- 是否需要继续保留为代码 enum
|
||||
- 是否应该迁到新的 `field_config`
|
||||
|
||||
也就是说,纯 enum 数据如果和初始化字段、手动创建字段有关,也在本轮视野内。
|
||||
|
||||
### 8.3 梳理哪些功能需要迁移到新的 `field_config`
|
||||
|
||||
每个领域都需要明确列出:
|
||||
|
||||
- 哪些初始化功能依赖字段候选项
|
||||
- 哪些手动创建 / 编辑功能依赖字段候选项
|
||||
- 哪些历史白名单校验需要迁移到 `validate_field_value()`
|
||||
- 哪些直接遍历旧 JSON 的逻辑需要迁移到 `get_field_options()` / `get_field_option_map()`
|
||||
- 哪些字段虽然目前由 `dict` 表或代码 enum 承载,但实际上已经在承担“按项目类型变化的候选项字段”职责
|
||||
|
||||
并且必须给出示例。
|
||||
|
||||
示例类型包括:
|
||||
|
||||
- 初始化时从旧 config 直接遍历候选项,改为从 `field_config` 读取
|
||||
- 手动创建时从硬编码 dict 判断合法值,改为调用 `validate_field_value()`
|
||||
- 展示名称时从字段选项里直接拿 `label`
|
||||
|
||||
对本轮来说,每个领域文档都至少要把下面 3 条链路写清楚:
|
||||
|
||||
- 初始化:这个字段初始化时从哪里取候选项,迁移后改成哪个 config key
|
||||
- 创建:这个字段 create service 当前怎么校验,迁移后改成哪个 `validate_field_value()` 调用
|
||||
- 下拉:这个字段前端当前通过哪个接口取值,迁移后改成哪个 config key 统一返回
|
||||
|
||||
对本轮来说,每个领域文档都至少要把下面 3 条链路写清楚:
|
||||
|
||||
- 初始化:这个字段初始化时从哪里取候选项,迁移后改成哪个 config key
|
||||
- 创建:这个字段 create service 当前怎么校验,迁移后改成哪个 `validate_field_value()` 调用
|
||||
- 下拉:这个字段前端当前通过哪个接口取值,迁移后改成哪个 config key 统一返回
|
||||
|
||||
### 8.4 必须完整对齐集团侧 `enum_config`
|
||||
|
||||
领域梳理时,不能只挑 depm 里已经处理过的字段,也不能只挑“看起来最容易改”的字段。
|
||||
|
||||
每个领域都必须把集团侧 `enum_config` 中该领域涉及的字段逐一对齐到 depm,明确说明它现在处于哪一种状态:
|
||||
|
||||
- depm 已有同名或同语义字段,并且来源清晰
|
||||
- depm 有近似字段,但命名、语义、来源或取值范围不完全一致
|
||||
- depm 还没有正式落点,只能先记录缺口
|
||||
|
||||
对齐 `enum_config` 时,每个字段都应继续检查两条链路:
|
||||
|
||||
- 项目初始化时,取值从哪里来,如何展开
|
||||
- 手动创建 / 更新时,取值如何校验,是否和初始化使用同一来源
|
||||
|
||||
如果两条链路来源不一致,应明确记录并给出迁移方案。
|
||||
|
||||
## 9. 实施步骤
|
||||
|
||||
### 第一阶段:公共能力
|
||||
### 第一阶段:公共能力
|
||||
|
||||
- 明确字段级配置固定 JSON 结构
|
||||
- 新增 `FieldOptionItem`
|
||||
- 在 `ConfigService` 中新增字段级配置方法
|
||||
- 给关键读取方法加缓存
|
||||
- 补齐 JSON 序列化 / 反序列化统一处理
|
||||
- 明确错误处理规则:配置结构不合法时直接报错
|
||||
- 增强 JSON / CSV 工具,支持字段级配置导入导出
|
||||
|
||||
### 第二阶段:初始化改造
|
||||
|
||||
- 逐项整理 `init_project_data)` 当前真实依赖的 config / dict
|
||||
- 把 `preparation`、`production`、`power` 这类平铺型初始化模板迁到 config
|
||||
- 整理 `material` 初始化,把单位等附属属性从不适当的 dict 备注迁出
|
||||
- 清理 `init_project_data)` 中取了但不该继续作为真源的 dict 依赖
|
||||
|
||||
### 第三阶段:创建校验改造
|
||||
|
||||
- 为 contract / material / construction 等 create service 入口补基于 config 的 code 校验
|
||||
- 把当前分散在 service 内的硬编码白名单逐步迁到 config 工具
|
||||
- 保留真正应该继续存在的代码 enum,不把固定语义状态误迁到 config
|
||||
|
||||
### 第四阶段:下拉列表改造
|
||||
### 第四阶段:下拉列表改造
|
||||
|
||||
- 梳理 code 类下拉接口
|
||||
- 下拉统一改成通过 config 工具返回
|
||||
- 少量仍写死的下拉改为动态获取
|
||||
- 保证前端下拉、初始化候选项、create 校验三者同源
|
||||
|
||||
### 第五阶段:清理和测试
|
||||
|
||||
- 删除废弃 enum 和废弃代码字典
|
||||
- 删除不再适合作为真源的 dict 依赖
|
||||
- 补 service 层集成测试和初始化测试
|
||||
- 梳理 code 类下拉接口
|
||||
- 下拉统一改成通过 config 工具返回
|
||||
- 少量仍写死的下拉改为动态获取
|
||||
- 保证前端下拉、初始化候选项、create 校验三者同源
|
||||
|
||||
### 第五阶段:清理和测试
|
||||
|
||||
- 删除废弃 enum 和废弃代码字典
|
||||
- 删除不再适合作为真源的 dict 依赖
|
||||
- 补 service 层集成测试和初始化测试
|
||||
|
||||
## 10. 产出要求
|
||||
|
||||
本次重构最终至少要产出:
|
||||
|
||||
- 一套明确的字段级配置 schema
|
||||
- 一套明确的 `ConfigService` 字段级配置接口
|
||||
- 一套明确的缓存与缓存失效规则
|
||||
- 一套 JSON / CSV / DB 三者之间的转换与维护规范
|
||||
- 一组领域级重构文档
|
||||
- 一组初始化 / 下拉 / create 的 service 层集成测试
|
||||
- 一组初始化 / 下拉 / create 的 service 层集成测试
|
||||
|
||||
判断本次重构是否成功的标准不是“历史实现全部统一”,而是:
|
||||
|
||||
- 后续新增一个与项目类型初始化相关的字段时,知道应优先写入字段级配置总表
|
||||
- 业务代码可以通过 `ConfigService` 统一获取 options / map / label / validate
|
||||
- 初始化流程和手动创建校验可以逐步收敛到同一来源
|
||||
- 前端下拉、初始化、create 校验不再各自读不同真源
|
||||
|
||||
## 11. 自动化测试要求
|
||||
|
||||
这次测试不是只补 unit test,而是以 service 层集成测试为主。
|
||||
|
||||
至少覆盖下面 3 组:
|
||||
|
||||
### 11.1 初始化测试
|
||||
|
||||
- 覆盖项目初始化入口
|
||||
- 校验初始化创建出的 preparation / production / contract / material / power / construction 数据是否来自目标 config
|
||||
- 校验错误配置时直接报错,不做静默降级
|
||||
|
||||
### 11.2 下拉列表测试
|
||||
|
||||
- 覆盖 contract / material / construction 等核心下拉接口或对应 service
|
||||
- 校验返回项是否来自预期 config key
|
||||
- 校验下拉返回和初始化候选项是否一致
|
||||
|
||||
### 11.3 Create 校验测试
|
||||
|
||||
- 直接走 service 层 create
|
||||
- 对合法 code 校验通过
|
||||
- 对非法 code 明确抛错
|
||||
- 校验不再依赖旧硬编码字典或 dict `remark`
|
||||
|
||||
### 11.4 测试组织方式
|
||||
|
||||
- 测试代码写在 `tests/` 下
|
||||
- 优先写 service 层集成测试,不在项目根目录放临时脚本
|
||||
- 如果需要初始化测试数据,应复用现有测试体系,不改 `schemas/`、不改 `filtered_datasource/`
|
||||
- 前端下拉、初始化、create 校验不再各自读不同真源
|
||||
|
||||
## 11. 自动化测试要求
|
||||
|
||||
这次测试不是只补 unit test,而是以 service 层集成测试为主。
|
||||
|
||||
至少覆盖下面 3 组:
|
||||
|
||||
### 11.1 初始化测试
|
||||
|
||||
- 覆盖项目初始化入口
|
||||
- 校验初始化创建出的 preparation / production / contract / material / power / construction 数据是否来自目标 config
|
||||
- 校验错误配置时直接报错,不做静默降级
|
||||
|
||||
### 11.2 下拉列表测试
|
||||
|
||||
- 覆盖 contract / material / construction 等核心下拉接口或对应 service
|
||||
- 校验返回项是否来自预期 config key
|
||||
- 校验下拉返回和初始化候选项是否一致
|
||||
|
||||
### 11.3 Create 校验测试
|
||||
|
||||
- 直接走 service 层 create
|
||||
- 对合法 code 校验通过
|
||||
- 对非法 code 明确抛错
|
||||
- 校验不再依赖旧硬编码字典或 dict `remark`
|
||||
|
||||
### 11.4 测试组织方式
|
||||
|
||||
- 测试代码写在 `tests/` 下
|
||||
- 优先写 service 层集成测试,不在项目根目录放临时脚本
|
||||
- 如果需要初始化测试数据,应复用现有测试体系,不改 `schemas/`、不改 `filtered_datasource/`
|
||||
@@ -0,0 +1,133 @@
|
||||
# Preparation 领域重构指南
|
||||
|
||||
## 1. 范围
|
||||
|
||||
本文档对应 backend 的 `preparation` 里程碑节点表。
|
||||
|
||||
当前优先关注:
|
||||
|
||||
- `preparation.code`
|
||||
- `preparation.approval_status`
|
||||
- `preparation.is_exist`
|
||||
|
||||
## 2. 字段盘点
|
||||
|
||||
### 2.1 `code`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `String`
|
||||
- 没有本地 enum 类
|
||||
- 项目初始化时,节点集合来自 `preparation_{project_type}` 字典数据
|
||||
- 同步 schema `project_extensions/preparation.py` 里也是 `str`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前无法从单一代码 enum 列出完整范围
|
||||
- 已知典型值包括:`JSZB`、`HZBA`、`KYNS`、`TZJC`、`YDYS`、`LCPF`、`YDPF`、`JRPF`、`HPPF`、`SGXK`、`SBPF`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧 `enum_config.Preparation.code` 已明确把该字段作为按项目类型变化的正式候选项
|
||||
- 本地当前仍主要靠初始化字典/字符串承载
|
||||
|
||||
结论:
|
||||
|
||||
- 更适合迁到 `field_config`
|
||||
- 不应继续只靠初始化字典和自由字符串
|
||||
|
||||
### 2.2 `approval_status`
|
||||
|
||||
当前实现:
|
||||
|
||||
- backend 依赖 `ApprovalMixin`
|
||||
- 本地 `PreparationApproval` schema 仍使用 int
|
||||
- 同步 schema `PreparationDetail` 里是 `Optional[str]`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前本地未在 preparation 模块中沉成独立 enum
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 同步 schema 有字段,但没有正式 enum
|
||||
|
||||
结论:
|
||||
|
||||
- 应统一接入通用审批状态体系
|
||||
|
||||
### 2.3 `is_exist`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `Boolean`
|
||||
- backend 和 sync schema 均有该字段
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- `true`
|
||||
- `false`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 当前没看到 enum_config 对这个字段给出候选项
|
||||
|
||||
结论:
|
||||
|
||||
- 保持布尔字段即可
|
||||
|
||||
## 3. 重点结论
|
||||
|
||||
- `preparation.code` 是标准的初始化候选项字段,最适合迁到 `field_config`。
|
||||
- `approval_status` 不需要在本领域单独发明枚举,应直接接入通用审批体系。
|
||||
- `is_exist` 是布尔存在性字段,不在本轮字段级配置重点内。
|
||||
|
||||
## 4. 待修改项清单
|
||||
|
||||
1. 将 `preparation.code` 正式迁到 `field_config`。
|
||||
2. 区分“节点编码范围”与“审批状态/存在性”这两类字段,不要混在一个配置源里。
|
||||
3. 将 `approval_status` 接入通用审批状态定义。
|
||||
|
||||
## 5. 按本轮要求的改造落点
|
||||
|
||||
preparation 属于本轮最适合先收口的一类:
|
||||
|
||||
- 初始化平铺
|
||||
- code 字段明确
|
||||
- 旧代码里已经出现多份硬编码映射
|
||||
|
||||
### 5.1 初始化
|
||||
|
||||
当前 preparation 初始化直接来自 `preparation_{project_type}` dict。
|
||||
|
||||
本轮目标是:
|
||||
|
||||
- 把 `preparation.code` 迁到 field config
|
||||
- 初始化改成通过 config 工具读取候选项再批量创建
|
||||
- 不再依赖 dict 作为初始化真源
|
||||
|
||||
### 5.2 Create 校验
|
||||
|
||||
当前 `PreparationCreate` / `PreparationService` 没有统一的 code 白名单校验入口。
|
||||
|
||||
本轮目标是:
|
||||
|
||||
- 在 create / update 对应的 service 入口开头,基于 config 校验 `preparation.code`
|
||||
- 不再新增本地 code 白名单或名称反查表来兜底
|
||||
|
||||
### 5.3 下拉列表
|
||||
|
||||
preparation 当前没有单独的“节点编码下拉接口”,前端更多拿的是项目实例列表 `/preparation/list`。
|
||||
|
||||
这轮文档上的要求是:
|
||||
|
||||
- 如果后续需要独立 code 候选项下拉,应直接从 `preparation.code` field config 返回
|
||||
|
||||
## 6. 测试关注点
|
||||
|
||||
preparation 领域 service 层集成测试至少覆盖:
|
||||
|
||||
1. 初始化:项目初始化时生成的节点是否来自目标 field config。
|
||||
2. Create / Update:合法 `code` 可以写入,非法 `code` 明确报错。
|
||||
3. 列表:项目实例列表顺序是否来自配置顺序,而不是硬编码 `sort_code`。
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
# Production 领域重构指南
|
||||
|
||||
## 1. 范围
|
||||
|
||||
本文档对应 backend 的 `production` 投产节点表。
|
||||
|
||||
当前优先关注:
|
||||
|
||||
- `production.code`
|
||||
- `production.approval_status`
|
||||
- `production.is_exist`
|
||||
|
||||
## 2. 字段盘点
|
||||
|
||||
### 2.1 `code`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `String`
|
||||
- 没有本地 enum 类
|
||||
- 项目初始化时,节点集合来自 `production_{project_type}` 字典数据
|
||||
- 同步 schema `project_extensions/production.py` 里也是 `str`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前无法从单一 enum 文件列出完整范围
|
||||
- 已知典型值包括:`SYZDSD`、`SPBW`、`QBBW`、`KHWC240`、`YJSC`、`DBTC`、`CQZS`、`WGJS`、`JGJS`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧 `enum_config.Production.code` 已把该字段按项目类型列成正式候选项
|
||||
- 本地仍主要靠初始化字典和自由字符串
|
||||
|
||||
结论:
|
||||
|
||||
- 更适合迁到 `field_config`
|
||||
|
||||
### 2.2 `approval_status`
|
||||
|
||||
当前实现:
|
||||
|
||||
- backend 依赖 `ApprovalMixin`
|
||||
- 本地 `ProductionApproval` schema 仍使用 int
|
||||
- 同步 schema `ProductionDetail` 里是 `Optional[str]`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前本地未在 production 模块中沉成独立 enum
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 同步 schema 有字段,但没有正式 enum
|
||||
|
||||
结论:
|
||||
|
||||
- 应统一接入通用审批状态体系
|
||||
|
||||
### 2.3 `is_exist`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `Boolean`
|
||||
- backend 和 sync schema 都有
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- `true`
|
||||
- `false`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 当前没看到 enum_config 对该字段给候选项
|
||||
|
||||
结论:
|
||||
|
||||
- 保持布尔字段即可
|
||||
|
||||
## 3. 重点结论
|
||||
|
||||
- `production.code` 和 `preparation.code` 一样,属于典型的初始化节点编码字段,应迁到 `field_config`。
|
||||
- `approval_status` 仍应走通用审批体系,而不是 production 自己维护一套值域。
|
||||
- `is_exist` 不属于本轮需要抽成字段候选表的重点字段。
|
||||
|
||||
## 4. 待修改项清单
|
||||
|
||||
1. 将 `production.code` 正式迁到 `field_config`。
|
||||
2. 将项目初始化里对 `production_{project_type}` 的依赖逐步收敛到统一字段配置入口。
|
||||
3. 将 `approval_status` 接入通用审批状态定义。
|
||||
|
||||
## 5. 按本轮要求的改造落点
|
||||
|
||||
production 和 preparation 一样,属于本轮适合优先收口的平铺型节点编码领域。
|
||||
|
||||
### 5.1 初始化
|
||||
|
||||
当前 production 初始化直接来自 `production_{project_type}` dict。
|
||||
|
||||
本轮目标是:
|
||||
|
||||
- 把 `production.code` 迁到 field config
|
||||
- 初始化改成通过 config 工具读取候选项后批量创建
|
||||
- 不再依赖 dict 作为初始化真源
|
||||
∏
|
||||
### 5.2 Create 校验
|
||||
|
||||
当前 production 领域也没有统一的 code 候选项校验入口。
|
||||
|
||||
本轮目标是:
|
||||
|
||||
- 在 create / update 的 service 入口开头,基于 config 校验 `production.code`
|
||||
- 不再继续用本地硬编码列表承担合法值白名单职责
|
||||
|
||||
### 5.3 下拉列表
|
||||
|
||||
production 当前也没有单独的 code 候选项下拉接口,更多是返回项目实例列表 `/production/list`。
|
||||
|
||||
本轮要求是:
|
||||
|
||||
- 如果需要单独 code 候选项下拉,应直接从 `production.code` field config 返回
|
||||
|
||||
## 6. 测试关注点
|
||||
|
||||
production 领域 service 层集成测试至少覆盖:
|
||||
|
||||
1. 初始化:项目初始化时生成的投产节点是否来自目标 field config。
|
||||
2. Create / Update:合法 `code` 可以写入,非法 `code` 明确报错。
|
||||
3. 列表:项目实例列表顺序是否来自配置顺序,而不是硬编码 `sort_code`。
|
||||
|
||||
@@ -0,0 +1,399 @@
|
||||
# Project 领域重构指南
|
||||
|
||||
## 1. 范围
|
||||
|
||||
本文档只做 Project 领域的项目侧梳理。
|
||||
|
||||
必须覆盖两类字段:
|
||||
|
||||
- 集团侧 `enum_config` 对应到 Project 的字段:`project_type`、`project_subtype`、`syxs`、`status`
|
||||
- depm 本地已经做成 enum / 类 enum,或者已经承担初始化、创建校验、业务分支职责的字段:`progress_type`、`key_constraints`、`build_methods`
|
||||
|
||||
本文重点只回答四件事:
|
||||
|
||||
- 当前有几种实现
|
||||
- 模型层怎么存
|
||||
- 是否真的用到了
|
||||
- 创建、校验、初始化时怎么取值,以及下一步怎么改
|
||||
|
||||
## 2. 字段盘点
|
||||
|
||||
### 2.1 `project_type`
|
||||
|
||||
当前实现:
|
||||
|
||||
- `app/common/enums.py` 有一套 `ProjectType`
|
||||
- `app/modules/project/enums.py` 又有一套 `ProjectType`
|
||||
- `Project` 模型里是 `String`
|
||||
- `ProjectCreate` / `ProjectUpdate` 里是 `str`
|
||||
|
||||
当前使用:
|
||||
|
||||
- `ProjectService.init_project_data)` 用它作为初始化路由键,去取 `preparation_{project_type}`、`production_{project_type}`、`power_{project_type}` 等配置
|
||||
- construction 模块的部分接口和工具也直接复用 `ProjectType`
|
||||
|
||||
当前问题:
|
||||
|
||||
- 定义重复
|
||||
- 模型层不是 enum
|
||||
- 创建 / 更新入口没有真正按 enum 校验
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前代码里的正式候选项是:`photovoltaic`、`onshore_wind`、`offshore_wind`、`pumped_storage`、`hydropower`、`thermal_power`、`gas_turbine`
|
||||
- `app/common/enums.py`、`app/modules/project/enums.py`、初始化 `dict_data.csv` 当前三处是一致的
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 当前未看到和项目侧不同的取值项
|
||||
- 这个字段目前可视为两边一致
|
||||
- 但同步 schema 里的 `project_type` 字段当前仍写成 `str`,并没有直接收紧成 `ProjectType`
|
||||
|
||||
结论:
|
||||
|
||||
- 保留为代码 enum
|
||||
- 它是核心路由字段,不迁到 `field_config`
|
||||
|
||||
### 2.2 `project_subtype`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `String`
|
||||
- `ProjectCreate` / `ProjectUpdate` 里是 `str`
|
||||
- 没看到本地正式 enum
|
||||
|
||||
当前使用:
|
||||
|
||||
- 当前主要是项目主表字段
|
||||
- 在 Project 模块内没看到稳定初始化使用点
|
||||
|
||||
当前问题:
|
||||
|
||||
- 没有统一候选项来源
|
||||
- 没有统一校验
|
||||
- 现在只是“字符串字段”
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 目前列不出权威范围
|
||||
- 当前项目侧没有正式 enum、没有 `field_config`、没有 schema 白名单
|
||||
- 从代码看,它现在理论上就是任意字符串
|
||||
|
||||
项目侧可举例范围:
|
||||
|
||||
- 目前只能借集团侧样例来猜测,例如光伏下可能是 `centralized`、`distributed`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧至少给出了光伏下的两个候选项:`centralized`、`distributed`
|
||||
- 项目侧目前一个正式值都没有落下来
|
||||
- 其它项目类型在集团侧也基本还是空占位,所以当前只能确认“项目侧缺正式范围,集团侧只给了局部样例”
|
||||
- 同步 schema 对应字段名是 `sub_type`,当前类型是 `Optional[str]`,也还没有 enum 化
|
||||
|
||||
困难:
|
||||
|
||||
- 项目侧没有正式定义源,无法从代码中反推出完整候选项集合
|
||||
- 集团侧除光伏外也没有给出有效值,导致现在只能列出局部范围,不能写成全项目类型完整表
|
||||
|
||||
结论:
|
||||
|
||||
- 改为 `field_config`
|
||||
- 建议 key:`project.project_subtype`
|
||||
|
||||
### 2.3 `syxs`
|
||||
|
||||
当前实现:
|
||||
|
||||
- 模型里是 `String`
|
||||
- 当前 `ProjectCreate` / `ProjectUpdate` 里没有这个字段
|
||||
- 没看到本地正式 enum
|
||||
|
||||
当前使用:
|
||||
|
||||
- 在 Project 模块内暂时没看到明确创建、初始化或校验使用点
|
||||
- 当前更像是“模型里已有字段,但项目侧链路尚未收敛”的状态
|
||||
|
||||
当前问题:
|
||||
|
||||
- 有字段,但没有统一入口
|
||||
- 没有统一候选项来源
|
||||
- 没有统一校验
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 目前列不出权威范围
|
||||
- 项目模型里有这个字段,但 Project 创建 schema 里没有该字段
|
||||
- 代码中也没找到项目侧正式 enum 或统一校验入口
|
||||
|
||||
项目侧可举例范围:
|
||||
|
||||
- 当前只能参考集团侧海风场景,例如 `hssy`、`lssy`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧至少给出了海风下的两个候选项:`hssy`、`lssy`
|
||||
- 项目侧目前还没有正式范围、正式入口和正式校验
|
||||
- 其它项目类型在集团侧也基本还是空占位,所以当前差异主要是“集团侧有局部定义,项目侧还没落下来”
|
||||
- 同步 schema 中 `syxs` 当前也是 `Optional[str]`,还没有 enum 化
|
||||
|
||||
困难:
|
||||
|
||||
- 项目侧没有正式输入入口,也没有权威候选项定义
|
||||
- 集团侧只有海风场景给了有效值,因此暂时只能列局部范围
|
||||
|
||||
结论:
|
||||
|
||||
- 改为 `field_config`
|
||||
- 建议 key:`project.syxs`
|
||||
|
||||
### 2.4 `status`
|
||||
|
||||
当前实现:
|
||||
|
||||
- `app/modules/project/enums.py` 里有 `ProjectStatus`
|
||||
- `app/common/enums.py` 里也有一套 `ProjectStatus`
|
||||
- 模型里是 `String`
|
||||
- 响应 schema 里是 `str`
|
||||
|
||||
当前使用:
|
||||
|
||||
- `ProjectService.change_project_status()` 会根据 `investment_decision_date`、`actual_construction_date`、`actual_production_date`、`actual_full_production_date` 计算并写回 `status`
|
||||
|
||||
当前问题:
|
||||
|
||||
- 定义重复
|
||||
- 模型层不是 enum
|
||||
- 当前状态值由服务逻辑推导,但接口层没有统一约束
|
||||
- 当前正式状态集合必须补上 `early`
|
||||
- 当前库里还存在 `NULL`,这说明状态机和历史数据都还没收口干净
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 目标正式值应为:`early`、`preparation`、`started`、`partially_connected`、`completed`
|
||||
- 当前代码定义和服务推导还没有完整覆盖这 5 个值
|
||||
- 当前实际数据里还存在 `NULL`,这应视为待清理异常值,而不是正式取值
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 项目侧最终也要纳入 `early`
|
||||
- 因此这个字段的目标差异不是“是否要 early”,而是“项目侧何时补齐 early 并清掉 NULL”
|
||||
- 同步 schema 里的 `status` 当前也是 `Optional[str]`,没有正式 enum 类型
|
||||
|
||||
结论:
|
||||
|
||||
- 保留为代码 enum
|
||||
- 正式状态集合按 `early`、`preparation`、`started`、`partially_connected`、`completed` 收口
|
||||
- `NULL` 不是正式值,必须通过数据修复和状态回填消除
|
||||
|
||||
### 2.5 `progress_type`
|
||||
|
||||
当前实现:
|
||||
|
||||
- `app/common/enums.py` 和 `app/modules/project/enums.py` 都定义了 `ProgressType`
|
||||
- 模型里是 `SQLEnum(ProgressType)`
|
||||
- `ProjectInfoResponse` 里使用 `ProgressType`
|
||||
- `ProjectCreate` 里仍是 `str`
|
||||
|
||||
当前使用:
|
||||
|
||||
- 是 Project 主表正式字段
|
||||
- `project_detail_data` 初始化时会写入 `progress_type`
|
||||
- 列表筛选也会直接按它查询
|
||||
|
||||
当前问题:
|
||||
|
||||
- 定义重复
|
||||
- 模型层已收紧,但创建入口还没收紧
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前正式值是:`A`、`B`、`C`
|
||||
- `app/common/enums.py`、`app/modules/project/enums.py`、初始化 `dict_data.csv` 当前是一致的
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧同步 schema 中,`progress_type` 也是正式 enum
|
||||
- 同步 schema 的取值项是:`A`、`B`、`C`
|
||||
- 这一点和项目侧代码 enum 一致
|
||||
- 当前真正的差异不在取值范围,而在项目侧创建入口仍是 `str`,还没有像同步 schema 一样收紧到 enum 类型
|
||||
|
||||
困难:
|
||||
|
||||
- 这个字段属于 depm 本地已枚举化字段,不是集团侧 `Project` 段直出字段
|
||||
- 因此这里只能写“项目侧权威范围明确,集团侧暂无直连字段”
|
||||
|
||||
结论:
|
||||
|
||||
- 保留为代码 enum
|
||||
- 不迁到 `field_config`
|
||||
|
||||
### 2.6 `key_constraints`
|
||||
|
||||
当前实现:
|
||||
|
||||
- `app/common/enums.py` 和 `app/modules/project/enums.py` 都定义了 `KeyConstraints`
|
||||
- 两套定义已分叉:common 里有 `OTHER`,project 模块里没有
|
||||
- 模型里是 `SQLEnum(KeyConstraints)`
|
||||
- `ProjectInfoResponse` 里使用 `KeyConstraints`
|
||||
- `ProjectSquareUpdate` 里还是 `str`
|
||||
|
||||
当前使用:
|
||||
|
||||
- 是 Project 主表正式字段
|
||||
- `project_detail_data` 初始化时会和 `progress_type` 一起写入
|
||||
- `project_detail_data` 审批回写时也会更新 Project 主表
|
||||
|
||||
当前问题:
|
||||
|
||||
- 定义分叉
|
||||
- 主表是 enum,但部分接口入参仍是字符串
|
||||
- 项目侧正式口径应保留 `OTHER`
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 项目侧正式值应为:`NONE`、`LAND`、`TRANSMISSION`、`BOTH`、`OTHER`
|
||||
- 当前 `app/common/enums.py` 和初始化 `dict_data.csv` 已包含这 5 个值
|
||||
- 当前 `app/modules/project/enums.py` 还少 `OTHER`
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧同步 schema 中,`key_constraints` 也是正式 enum
|
||||
- 同步 schema 的取值项是:`NONE`、`LAND`、`TRANSMISSION`、`BOTH`、`QT`
|
||||
- 项目侧正式口径保留 `OTHER`
|
||||
- 这里需要备注的差异只有一条:集团侧用 `QT`,项目侧用 `OTHER`
|
||||
|
||||
结论:
|
||||
|
||||
- 保留为代码 enum
|
||||
- 项目侧按 `OTHER` 收口
|
||||
- 在同步和推送链路上单独备注 `OTHER/QT` 差异
|
||||
- 同时补齐 project 模块那套缺失的 `OTHER`
|
||||
|
||||
### 2.7 `build_methods`
|
||||
|
||||
当前实现:
|
||||
|
||||
- `app/modules/project/enums.py` 里有 `BuildMethod`
|
||||
- 模型里是 `String`
|
||||
- `ProjectCreate` 里是 `str`
|
||||
- 响应 schema 通过 `enum_to_build_method_name` 做名称映射
|
||||
|
||||
当前使用:
|
||||
|
||||
- construction 服务里确实使用了 `project.build_methods`
|
||||
- 它会影响施工分支判断,例如 `power`、`mixed`、`none`、`self`
|
||||
|
||||
当前问题:
|
||||
|
||||
- 模型层不是 enum
|
||||
- 创建入口不是 enum
|
||||
- 已经参与业务分支,但没有统一校验
|
||||
|
||||
项目侧取值范围:
|
||||
|
||||
- 当前代码 enum 给出的值是:`self`、`power`、`mixed`、`none`
|
||||
- construction 分支判断当前也确实在使用这 4 个值
|
||||
|
||||
集团侧差异:
|
||||
|
||||
- 集团侧同步 schema 里没有 `build_methods` 这个名字,对应字段是 `scgc_type`
|
||||
- `scgc_type` 当前在同步 schema 里是 `Optional[str]`
|
||||
- 目前没有看到同步 schema 中为 `scgc_type` 定义正式 enum
|
||||
|
||||
也就是说:
|
||||
|
||||
- 语义上两边大概率是在说同一个字段
|
||||
- 但集团侧当前还没有把它收紧成正式 enum
|
||||
- 因此这个字段更像“项目侧本地先有 enum,集团侧 schema 还停留在字符串字段”
|
||||
|
||||
困难:
|
||||
|
||||
- 这是项目侧本地业务字段,不是集团侧 `Project` 段直出字段
|
||||
- 项目侧虽然已有 enum,但模型和 schema 还没完全收紧
|
||||
|
||||
结论:
|
||||
|
||||
- 保留为代码 enum
|
||||
- 不迁到 `field_config`
|
||||
- 需要把模型层和 schema 层一起收紧
|
||||
|
||||
## 3. 重点结论
|
||||
|
||||
- `project_type` 是初始化路由键,必须保留为代码 enum。
|
||||
- `project_subtype`、`syxs` 是典型的按项目类型变化字段,应迁到 `field_config`。
|
||||
- `progress_type`、`key_constraints`、`build_methods` 都已经在项目侧真实使用,不是可忽略字段;它们应保留为代码 enum,但当前实现没有收紧干净。
|
||||
- `status` 也应保留为代码 enum,正式集合必须包含 `early`,并且要清掉历史 `NULL`。
|
||||
- `key_constraints` 项目侧正式口径保留 `OTHER`,只需要在文档和同步链路上备注集团侧使用 `QT`。
|
||||
|
||||
## 4. 待修改项清单
|
||||
|
||||
1. 统一 `ProjectType` 定义,只保留一套正式 enum。
|
||||
2. 统一 `ProjectStatus` 定义,并把 `early` 纳入正式状态集合。
|
||||
3. 统一 `ProgressType` 定义,只保留一套正式 enum。
|
||||
4. 统一 `KeyConstraints` 定义,项目侧正式保留 `OTHER`。
|
||||
5. 将 `project.project_subtype` 改为字段级配置,放入 `field_config`。
|
||||
6. 将 `project.syxs` 改为字段级配置,放入 `field_config`。
|
||||
7. 为 `project_type` 在 schema 层增加正式 enum 校验,不再使用裸 `str`。
|
||||
8. 为 `progress_type` 在创建入口增加正式 enum 校验。
|
||||
9. 为 `build_methods` 在 schema 层增加正式 enum 校验。
|
||||
10. 为 `key_constraints` 在仍使用字符串入参的接口上补 enum 校验。
|
||||
11. 清理 `Project.status` 的历史 `NULL` 数据,并补状态回填逻辑,确保 `NULL` 不再出现。
|
||||
12. 评估 `Project` 模型中的 `project_type`、`status`、`build_methods` 是否要收紧为数据库 enum;如果暂不收紧,也要先统一 schema 校验。
|
||||
13. 为 `project_subtype`、`syxs` 增加统一读取入口:`get_field_options()` / `validate_field_value()`。
|
||||
14. 明确 `status` 的唯一来源:是继续由 `change_project_status()` 推导,还是允许手动写入。
|
||||
|
||||
## 5. 推荐落地顺序
|
||||
|
||||
1. 先统一 `ProjectType`、`ProgressType`、`KeyConstraints`、`ProjectStatus` 的定义源。
|
||||
2. 再补 schema 层校验,把 `project_type`、`progress_type`、`build_methods`、`key_constraints` 的裸字符串入口收紧。
|
||||
3. 先补 `status=early` 和历史 `NULL` 清理,再把 `project_subtype`、`syxs` 迁到 `field_config`。
|
||||
|
||||
## 6. 按本轮要求的改造落点
|
||||
|
||||
project 领域这轮要区分两类字段:
|
||||
|
||||
- 固定语义枚举:继续保留代码 enum
|
||||
- 按项目类型变化的候选项:迁到 field config
|
||||
|
||||
### 6.1 初始化
|
||||
|
||||
Project 本身不直接批量初始化子表候选项,但 `project_type` 是整个初始化流程的路由键。
|
||||
|
||||
因此本轮在 project 领域的初始化要求是:
|
||||
|
||||
- `project_type` 继续保留代码 enum,作为 config 路由键
|
||||
- `project_subtype`、`syxs` 如果进入初始化或默认值链路,应直接从 field config 读取
|
||||
|
||||
### 6.2 Create / Update 校验
|
||||
|
||||
这轮 project 领域要分开处理:
|
||||
|
||||
- `project_type`、`status`、`progress_type`、`key_constraints`、`build_methods`:继续按代码 enum 收紧
|
||||
- `project_subtype`、`syxs`:通过新的 config 工具校验
|
||||
|
||||
也就是说,project 领域不是“全部配置化”,而是“固定 enum 和 field config 分流后分别收紧”。
|
||||
|
||||
### 6.3 下拉列表
|
||||
|
||||
当前 project 领域里:
|
||||
|
||||
- `project_type`、`progress_type`、`key_constraints`、`build_methods` 更适合继续由代码 enum 提供候选项
|
||||
- `project_subtype`、`syxs` 后续如果提供前端下拉,应直接走 field config
|
||||
|
||||
本轮目标是明确边界,避免把本地固定枚举误迁到 config,也避免把按项目类型变化字段继续留成自由字符串。
|
||||
|
||||
### 6.4 清理项
|
||||
|
||||
project 领域本轮改造完成后,应清理:
|
||||
|
||||
- 重复的 enum 定义源
|
||||
- 仍以裸字符串接收固定 enum 字段的入口
|
||||
- `status` 历史 `NULL` 值
|
||||
|
||||
## 7. 测试关注点
|
||||
|
||||
project 领域 service 层集成测试至少覆盖:
|
||||
|
||||
1. `project_type`、`progress_type`、`key_constraints`、`build_methods` 的合法/非法值校验。
|
||||
2. `project_subtype`、`syxs` 的 field config 读取与校验。
|
||||
3. `status` 状态推导补齐 `early` 且不再产生 `NULL`。
|
||||
@@ -93,7 +93,6 @@
|
||||
- endpoint 创建 handler
|
||||
- endpoint 统一注入:
|
||||
- `handler_config`
|
||||
- `target_project_ids`
|
||||
- datasource-specific runtime object(如 api client)
|
||||
|
||||
最终让 `factory` 只负责:
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
# 快速上手
|
||||
|
||||
这份文档只服务第一次接手 `ecm_sync_system` 的开发者。目标不是完整讲架构,而是让你先跑起来、知道从哪里入手,再去看更细的说明。
|
||||
|
||||
## 1. 先做什么
|
||||
|
||||
第一次进入仓库时,建议按下面顺序执行:
|
||||
|
||||
1. 准备 Python 3.9+。
|
||||
2. 在仓库根目录创建并激活虚拟环境。
|
||||
3. 执行 `pip install -e .` 安装当前项目。
|
||||
4. 跑状态机配置校验,确认基础工具链正常。
|
||||
5. 跑一条轻量测试,确认测试与渲染工具链可用。
|
||||
6. 再去看架构文档和状态机文档。
|
||||
|
||||
推荐命令:
|
||||
|
||||
```bash
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -e .
|
||||
python run.py --help
|
||||
python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml
|
||||
pytest tests/integration/test_toolchain_config_and_graph.py -q
|
||||
```
|
||||
|
||||
## 2. 仓库里最重要的目录
|
||||
|
||||
- `sync_state_machine/`
|
||||
- 真正的引擎实现目录。大多数“核心代码入口”都在这里。
|
||||
- `run_profiles/`
|
||||
- 运行配置与模板。
|
||||
- `tests/`
|
||||
- 单元测试与集成测试。
|
||||
- `tools/`
|
||||
- 状态机校验、图渲染、远程环境等工具。
|
||||
- `scripts/`
|
||||
- 自动化测试编排和数据集相关脚本。
|
||||
- `docs/`
|
||||
- 面向开发者的设计、运行和排障文档。
|
||||
|
||||
## 3. 先读哪几篇文档
|
||||
|
||||
按任务来看:
|
||||
|
||||
- 想知道系统整体怎么分层:`docs/architecture.md`
|
||||
- 想知道状态机怎么工作:`docs/node_state_definition.md` 和 `docs/state_machine.md`
|
||||
- 想知道一次同步是怎么跑的:`docs/sync_flow.md`
|
||||
- 想调运行配置:`docs/runtime_config_guide.md`(包含单项目、多项目、`persist.url`、`sqlalchemy`、业务侧注册 datasource 的当前写法)
|
||||
- 想接新 datasource 或 handler:`sync_state_machine/datasource/README.md` 和 `docs/library_integration_guide.md`
|
||||
- 想看测试体系:`docs/testing_guide.md` 和 `tests/README.md`
|
||||
- 想做排障:`docs/operations.md`
|
||||
|
||||
## 4. 新人最容易踩的坑
|
||||
|
||||
### 4.1 先用仓库入口,不要默认 shell 里已经有命令
|
||||
|
||||
如果你还没执行过 `pip install -e .`,那么 `ecm-sync-run` 不一定可用。
|
||||
|
||||
首次调试建议优先用:
|
||||
|
||||
```bash
|
||||
python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
|
||||
```
|
||||
|
||||
安装完成后,再切换到:
|
||||
|
||||
```bash
|
||||
ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
|
||||
```
|
||||
|
||||
### 4.2 核心实现路径以 `sync_state_machine/` 为准
|
||||
|
||||
阅读历史文档时,如果看到类似下面的写法:
|
||||
|
||||
- `pipeline/full_sync_pipeline.py`
|
||||
- `sync_system/strategy.py`
|
||||
- `datasource/datasource.py`
|
||||
- `engine/events.py`
|
||||
|
||||
请优先理解为它们在当前仓库中的真实位置分别是:
|
||||
|
||||
- `sync_state_machine/pipeline/full_sync_pipeline.py`
|
||||
- `sync_state_machine/sync_system/strategy.py`
|
||||
- `sync_state_machine/datasource/datasource.py`
|
||||
- `sync_state_machine/engine/events.py`
|
||||
|
||||
## 5. 如果你要开始改代码
|
||||
|
||||
### 5.1 改 domain 行为
|
||||
|
||||
优先查看:
|
||||
|
||||
- `sync_state_machine/domain/<node_type>/`
|
||||
- `sync_state_machine/sync_system/strategy_ops/`
|
||||
- `tests/unit/` 中对应测试
|
||||
|
||||
### 5.2 改状态机语义
|
||||
|
||||
优先查看:
|
||||
|
||||
- `sync_state_machine/config/node_state_machine.yaml`
|
||||
- `sync_state_machine/engine/events.py`
|
||||
- `sync_state_machine/engine/state_apply.py`
|
||||
|
||||
改完建议至少执行:
|
||||
|
||||
```bash
|
||||
python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml
|
||||
pytest tests/integration/test_toolchain_config_and_graph.py -q
|
||||
```
|
||||
|
||||
### 5.3 改运行配置或 datasource
|
||||
|
||||
优先查看:
|
||||
|
||||
- `docs/runtime_config_guide.md`
|
||||
- `run_profiles/README.md`
|
||||
- `sync_state_machine/datasource/README.md`
|
||||
|
||||
## 6. 出错先看哪里
|
||||
|
||||
推荐排查顺序:
|
||||
|
||||
1. 先看运行日志中的 `Resolved Full Config`。
|
||||
2. 再看节点 `sync_log / status / error`。
|
||||
3. 再对照 `docs/operations.md` 和 `docs/state_machine.md`。
|
||||
4. 如果是配置问题,先让校验报错,不要绕过 schema 或状态机校验。
|
||||
@@ -11,7 +11,7 @@
|
||||
推荐接入方式:
|
||||
|
||||
- `sync_state_machine` 负责同步引擎、状态机、pipeline、datasource 抽象。
|
||||
- 业务系统负责提供自己的数据源适配层,不要反向让 `sync_state_machine` 依赖业务系统。
|
||||
- 业务系统负责提供自己的 datasource / handler 适配层,不要反向让 `sync_state_machine` 依赖业务系统。
|
||||
|
||||
---
|
||||
|
||||
@@ -46,13 +46,18 @@ ecm-sync-run --config_path=run_profiles/preset/jsonl_to_api.default.yaml
|
||||
|
||||
- `sync_state_machine.build_config`
|
||||
- `sync_state_machine.build_default_config`
|
||||
- `sync_state_machine.run_pipeline_from_config`
|
||||
- `sync_state_machine.create_pipeline_from_config`
|
||||
- `sync_state_machine.build_config_from_file`
|
||||
- `sync_state_machine.PipelineRunConfig`
|
||||
- `sync_state_machine.JsonlDataSourceConfig`
|
||||
- `sync_state_machine.ApiDataSourceConfig`
|
||||
- `sync_state_machine.DomainRegistry`
|
||||
|
||||
配置加载分两步:
|
||||
|
||||
1. 先构造或读取 `PipelineRunConfig`
|
||||
2. 再交给 `create_pipeline_from_config()` 创建 pipeline
|
||||
|
||||
内置 domain 注册会在导入包时自动完成。
|
||||
|
||||
---
|
||||
@@ -200,7 +205,7 @@ class DemoJsonlHandler(BaseJsonlHandler):
|
||||
|
||||
---
|
||||
|
||||
### 6.2 如果 backend 是数据库:建议写“数据库 handler / adapter”,不要让引擎直接依赖 ORM
|
||||
### 6.2 如果 backend 是数据库:建议写 handler / adapter,不要让引擎直接依赖 ORM
|
||||
|
||||
推荐结构:
|
||||
|
||||
@@ -234,6 +239,7 @@ backend/
|
||||
|
||||
- 不要在 `sync_state_machine` 包里 import backend 的 model/repository
|
||||
- 不要让 `DomainRegistry` 注册逻辑散落在 backend 业务层之外
|
||||
- 不要在 datasource 之外做额外的 project 过滤;项目范围应通过 `handler_configs.project.data_id_filter` 下沉到具体 handler 解释
|
||||
|
||||
---
|
||||
|
||||
@@ -268,6 +274,7 @@ backend/
|
||||
- `strategy_class`
|
||||
- `jsonl_handler_class`
|
||||
- `api_handler_class`
|
||||
- 以及任意通过 `register_handler()` 追加的 datasource handler
|
||||
|
||||
示例:
|
||||
|
||||
@@ -282,9 +289,11 @@ DomainRegistry.register(
|
||||
jsonl_handler_class=DemoJsonlHandler,
|
||||
api_handler_class=DemoApiHandler,
|
||||
)
|
||||
|
||||
DomainRegistry.register_handler("demo", "custom_remote", DemoCustomHandler)
|
||||
```
|
||||
|
||||
如果是 backend 接入,建议在 backend 的 integration 启动阶段集中注册,而不是分散在 controller / service 里。
|
||||
如果是 backend 接入,建议在启动入口集中注册,而不是分散在 controller / service 里。
|
||||
|
||||
---
|
||||
|
||||
@@ -311,7 +320,9 @@ async def run_demo_sync() -> None:
|
||||
"api_base_url": "https://example.com",
|
||||
"api_uid": "xxx",
|
||||
"api_secret": "xxx",
|
||||
"target_project_ids": ["demo-project"],
|
||||
"handler_configs": {
|
||||
"project": {"data_id_filter": ["demo-project"]}
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
@@ -320,7 +331,7 @@ async def run_demo_sync() -> None:
|
||||
await pipeline.run()
|
||||
```
|
||||
|
||||
如果 backend 自己提供本地数据库数据,不要把 backend model 直接灌进 pipeline;先经过 handler / mapper 转成 schema 再进入引擎。
|
||||
如果 backend 自己提供本地数据库数据,不要把 backend model 直接灌进 pipeline;先经过 handler / mapper 转成 schema 再进入引擎。
|
||||
|
||||
---
|
||||
|
||||
@@ -353,8 +364,9 @@ async def run_demo_sync() -> None:
|
||||
## 11. 相关文件
|
||||
|
||||
- 包入口:[sync_state_machine/__init__.py](../sync_state_machine/__init__.py)
|
||||
- CLI:[sync_state_machine/cli.py](../sync_state_machine/cli.py)
|
||||
- CLI / 运行入口:[sync_state_machine/cli.py](../sync_state_machine/cli.py)
|
||||
- 配置工厂:[sync_state_machine/config](../sync_state_machine/config)
|
||||
- datasource 基础层:[sync_state_machine/datasource](../sync_state_machine/datasource)
|
||||
- pipeline 工厂:[sync_state_machine/pipeline/factory.py](../sync_state_machine/pipeline/factory.py)
|
||||
- domain 注册:[sync_state_machine/common/registry.py](../sync_state_machine/common/registry.py)
|
||||
- backend 集成入口:[backend/app/thirdparty/ecm_sync/services/sync_runner.py](../backend/app/thirdparty/ecm_sync/services/sync_runner.py)
|
||||
@@ -0,0 +1,627 @@
|
||||
# 多项目同步重构方案
|
||||
|
||||
## 1. 背景和目标
|
||||
|
||||
### 背景
|
||||
|
||||
当前 `multi_project_pipeline` 的主要问题:
|
||||
|
||||
1. 顶层概念过重,在单项目同步之上又叠了一层特殊编排。
|
||||
2. 通过共享快照传递 `collection` / `binding` / 运行态,边界不清晰。
|
||||
3. 配置入口过多,存在 `global_config` / `default_project_config` / `config_path` 多层模型。
|
||||
4. 项目完成后哪些对象可以释放,没有明确生命周期。
|
||||
|
||||
### 目标
|
||||
|
||||
新方案只解决四件事:
|
||||
|
||||
1. 顶层仍然是 pipeline,但直接写新的 `ProjectBatchPipeline`。
|
||||
2. `global` 数据只同步一次,不复制到每个项目,不在每个项目中重复校验。
|
||||
3. 改动尽量限制在 `persistence / collection / binding_manager / datasource / pipeline` 这些基础层。
|
||||
4. 项目级运行态支持 `load -> sync -> persist -> unload -> close`。
|
||||
|
||||
### 一个明确结论
|
||||
|
||||
这里不建议先做复杂的 session 框架。
|
||||
|
||||
第一阶段最合适的方式是:
|
||||
|
||||
1. 在 `sync_project(...)` 里创建一个很薄的 `ProjectScopeRuntime`。
|
||||
2. 它只是把“当前项目 scope 下的一组对象”收拢起来。
|
||||
3. 如果你临时不想单独建类,也可以先写成函数内局部变量。
|
||||
|
||||
但从清理顺序和可读性考虑,推荐保留 `ProjectScopeRuntime` 这个薄对象。
|
||||
|
||||
它不是复杂 session 系统,本质上就是“单个项目的一次运行上下文”。
|
||||
|
||||
## 2. 新的 Batch Project Pipeline 和伪代码
|
||||
|
||||
### 结构结论
|
||||
|
||||
顶层对象建议是:`ProjectBatchPipeline`
|
||||
|
||||
它负责:
|
||||
|
||||
1. 跑一次 global phase。
|
||||
2. 并发调度所有项目。
|
||||
3. 管理 batch 级共享对象。
|
||||
4. 管理项目级对象的创建和清理。
|
||||
|
||||
它不负责:
|
||||
|
||||
1. 业务字段映射。
|
||||
2. handler 细节。
|
||||
3. strategy 逻辑。
|
||||
|
||||
### 共享对象和项目级对象
|
||||
|
||||
#### 整个 batch 共享的对象
|
||||
|
||||
1. `BatchConfig`
|
||||
2. `StateMachineRuntime`
|
||||
3. `Logger`
|
||||
4. `PersistenceService`
|
||||
5. `global` 只读运行态
|
||||
6. datasource 的共享静态资源,例如 API client、认证信息、限流器、handler registry
|
||||
7. semaphore 和任务调度器
|
||||
|
||||
#### 每个项目内创建的对象
|
||||
|
||||
1. `ProjectScopeRuntime`
|
||||
2. `local_collection`
|
||||
3. `remote_collection`
|
||||
4. `binding_manager`
|
||||
5. 当前项目 scope 对应的 persistence view
|
||||
6. 当前项目的 local datasource runtime
|
||||
7. 当前项目的 remote datasource runtime
|
||||
8. 当前项目统计对象
|
||||
|
||||
### Global 数据处理原则
|
||||
|
||||
这点直接定死:
|
||||
|
||||
1. `global` 数据不复制到每个 project scope。
|
||||
2. `global` 数据不在每个项目里重复 bootstrap。
|
||||
3. `global` 数据不在每个项目里重复 post-check。
|
||||
|
||||
项目运行时如果需要共享节点或绑定:
|
||||
|
||||
1. 先查当前 project scope。
|
||||
2. 未命中再回退到 `global` 只读运行态。
|
||||
|
||||
### 伪代码
|
||||
|
||||
```python
|
||||
class ProjectBatchPipeline:
|
||||
def __init__(self, config: BatchConfig):
|
||||
self.config = config
|
||||
self.runtime = build_state_machine_runtime()
|
||||
self.logger = build_logger(config.logging)
|
||||
self.persistence = PersistenceService(config.persist)
|
||||
self.global_runtime = None
|
||||
self.shared_local_ds = build_shared_local_datasource_resources(config.local_datasource)
|
||||
self.shared_remote_ds = build_shared_remote_datasource_resources(config.remote_datasource)
|
||||
|
||||
async def run(self) -> None:
|
||||
self.global_runtime = await self.sync_global()
|
||||
await self.dispatch_projects()
|
||||
|
||||
async def sync_global(self) -> "GlobalReadonlyRuntime":
|
||||
...
|
||||
|
||||
async def dispatch_projects(self) -> None:
|
||||
...
|
||||
|
||||
async def sync_project(self, local_project_id: str, remote_project_id: str) -> None:
|
||||
scope_runtime = await self.create_project_scope_runtime(local_project_id, remote_project_id)
|
||||
try:
|
||||
await scope_runtime.load()
|
||||
await run_project_sync(scope_runtime, self.config.project_node_types)
|
||||
await scope_runtime.persist()
|
||||
finally:
|
||||
await scope_runtime.unload()
|
||||
await scope_runtime.close()
|
||||
```
|
||||
|
||||
```python
|
||||
class ProjectScopeRuntime:
|
||||
def __init__(self, scope: str, persistence_view, global_runtime, shared_local_ds, shared_remote_ds):
|
||||
self.scope = scope
|
||||
self.persistence_view = persistence_view
|
||||
self.local_collection = DataCollection(...)
|
||||
self.remote_collection = DataCollection(...)
|
||||
self.binding_manager = BindingManager(...)
|
||||
self.local_datasource = build_local_datasource_runtime(shared_local_ds, self.local_collection)
|
||||
self.remote_datasource = build_remote_datasource_runtime(shared_remote_ds, self.remote_collection)
|
||||
self.global_runtime = global_runtime
|
||||
|
||||
async def load(self) -> None: ...
|
||||
async def persist(self) -> None: ...
|
||||
async def unload(self) -> None: ...
|
||||
async def close(self) -> None: ...
|
||||
```
|
||||
|
||||
## 3. 组件说明
|
||||
|
||||
### 3.1 Persistence
|
||||
|
||||
#### 角色
|
||||
|
||||
`PersistenceService` 是 batch 共享对象。
|
||||
|
||||
它共享的是:
|
||||
|
||||
1. 后端类型。
|
||||
2. DB 路径或连接串。
|
||||
3. 连接池或底层连接能力。
|
||||
|
||||
它不共享的是:
|
||||
|
||||
1. 当前项目的内存缓存。
|
||||
2. 当前 scope 的脏状态。
|
||||
|
||||
#### 关键语义
|
||||
|
||||
建议把 persistence 理解成两层:
|
||||
|
||||
1. `PersistenceService`
|
||||
- 共享后端服务。
|
||||
2. `ScopedPersistenceView`
|
||||
- 绑定某个 `scope` 的轻量 view。
|
||||
|
||||
所以“scope 属于 persistence 的 scope view”的意思是:
|
||||
|
||||
1. 整个 batch 共享一个 persistence service。
|
||||
2. 每个项目拿一个绑定了当前 `scope` 的轻量操作视图。
|
||||
|
||||
不是说每个项目都新建一套 persistence backend。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class PersistenceService:
|
||||
async def initialize(self) -> None: ...
|
||||
def view(self, scope: str) -> ScopedPersistenceView: ...
|
||||
async def close(self) -> None: ...
|
||||
|
||||
|
||||
class ScopedPersistenceView:
|
||||
async def load_nodes(self, collection_id: str) -> list[dict]: ...
|
||||
async def save_nodes(self, collection_id: str, nodes: list[SyncNode]) -> None: ...
|
||||
async def delete_nodes(self, collection_id: str, node_ids: list[str]) -> None: ...
|
||||
async def load_bindings(self, node_type: str) -> list[dict]: ...
|
||||
async def save_bindings(self, node_type: str, bindings: dict[str, str | None]) -> None: ...
|
||||
```
|
||||
|
||||
### 3.2 Collection
|
||||
|
||||
#### 角色
|
||||
|
||||
`Collection` 必须是 scope 级对象,不能做单例。
|
||||
|
||||
原因:
|
||||
|
||||
1. 它内部天然持有当前 scope 的节点集合。
|
||||
2. 它持有 data_id 索引。
|
||||
3. 它持有删除集合和脏状态。
|
||||
|
||||
#### 关键语义
|
||||
|
||||
project scope 的 `Collection` 只存项目域数据。
|
||||
|
||||
对于共享数据,不复制 global,而是支持只读回退:
|
||||
|
||||
1. 先查当前 scope。
|
||||
2. 未命中再查 global 只读 collection。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class DataCollection:
|
||||
async def load_from_persistence(self) -> None: ...
|
||||
async def persist(self) -> None: ...
|
||||
async def unload(self) -> None: ...
|
||||
|
||||
def get(self, node_id: str) -> SyncNode | None: ...
|
||||
|
||||
def get_by_data_id(self, node_type: str, data_id: str) -> SyncNode | None:
|
||||
node = self._get_local(node_type, data_id)
|
||||
if node is not None:
|
||||
return node
|
||||
return self._get_global_fallback(node_type, data_id)
|
||||
```
|
||||
|
||||
### 3.3 BindingManager
|
||||
|
||||
#### 角色
|
||||
|
||||
`BindingManager` 也是 scope 级对象,不应共享。
|
||||
|
||||
它负责:
|
||||
|
||||
1. 当前 scope 的绑定运行态。
|
||||
2. 当前 scope 的绑定持久化。
|
||||
3. 当前 scope 未命中时对 global 只读绑定做回退查询。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class BindingManager:
|
||||
async def load_from_persistence(self, node_type: str) -> None: ...
|
||||
async def persist(self) -> None: ...
|
||||
async def unload(self) -> None: ...
|
||||
|
||||
async def get_remote_id(self, node_type: str, local_id: str) -> str | None:
|
||||
remote_id = self._get_local_remote_id(node_type, local_id)
|
||||
if remote_id is not None:
|
||||
return remote_id
|
||||
return await self._get_global_remote_id(node_type, local_id)
|
||||
```
|
||||
|
||||
### 3.4 DataSource
|
||||
|
||||
#### 角色
|
||||
|
||||
`local_datasource / remote_datasource / logging` 不应按项目变化,这一点成立。
|
||||
|
||||
但按当前实现,`BaseDataSource` 不适合直接做单例,因为它内部持有:
|
||||
|
||||
1. `_collection`
|
||||
2. `_stats`
|
||||
3. `_action_summary`
|
||||
4. `_action_by_node_id`
|
||||
|
||||
#### 推荐的最小改法
|
||||
|
||||
第一阶段不强行拆成两个正式类,而是采用下面的语义:
|
||||
|
||||
1. datasource 的静态配置和共享 client 是 batch 级共享的。
|
||||
2. datasource 的运行态对象仍然是项目级创建的。
|
||||
|
||||
也就是说:
|
||||
|
||||
1. 配置不按项目变化。
|
||||
2. runtime 仍然按项目隔离。
|
||||
|
||||
这样改动范围最小,也更不容易污染现有 handler 行为。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class ScopedDataSourceRuntime(BaseDataSource):
|
||||
def set_collection(self, collection: DataCollection) -> None: ...
|
||||
async def load_all(self, order: list[str]) -> None: ...
|
||||
async def sync_all(self, order: list[str]) -> None: ...
|
||||
def reset_runtime_state(self) -> None: ...
|
||||
async def close(self) -> None: ...
|
||||
```
|
||||
|
||||
## 4. 改进路线图
|
||||
|
||||
### 第一阶段:先收敛架构边界
|
||||
|
||||
只改基础层,不动业务层:
|
||||
|
||||
1. 新建 `ProjectBatchPipeline`。
|
||||
2. 明确 `global` 和 `project_id:<local_project_id>` 语义。
|
||||
3. 引入 `ProjectScopeRuntime` 这个薄的项目运行上下文。
|
||||
4. 给 `Collection` 和 `BindingManager` 增加 global fallback 语义。
|
||||
5. persistence 侧增加共享 service + scope view 语义。
|
||||
|
||||
### 第二阶段:清理生命周期
|
||||
|
||||
把项目级清理流程固定下来:
|
||||
|
||||
1. `load`
|
||||
2. `sync`
|
||||
3. `persist`
|
||||
4. `unload`
|
||||
5. `close`
|
||||
|
||||
### 第三阶段:再决定 datasource 是否正式拆层
|
||||
|
||||
如果第一阶段跑通,再决定是否把 datasource 正式拆成:
|
||||
|
||||
1. 共享静态层。
|
||||
2. scope runtime 层。
|
||||
|
||||
这一步不是第一批必须完成。
|
||||
|
||||
### 最终落点
|
||||
|
||||
最终希望达到的状态:
|
||||
|
||||
1. global 数据只同步一次,只读共享。
|
||||
2. project 数据按 scope 独立运行。
|
||||
3. 项目跑完即可卸载运行态。
|
||||
4. 大部分改动都限制在基础设施层。
|
||||
8. 当前项目统计对象
|
||||
|
||||
### 2.3 为什么推荐 `ProjectScopeRuntime`
|
||||
|
||||
你问的是:要不要做 session 对象,还是做子 pipeline,还是都放函数里。
|
||||
|
||||
建议是:
|
||||
|
||||
1. 不做新的复杂 session 框架。
|
||||
2. 不做 `FullSyncPipeline` 风格的子 pipeline 继承树。
|
||||
3. 做一个很薄的 `ProjectScopeRuntime`。
|
||||
|
||||
它的作用只有两个:
|
||||
|
||||
1. 把当前项目的一组 scope 对象收拢到一起。
|
||||
2. 让清理顺序明确,不要把资源散在 `sync_project(...)` 的局部变量里。
|
||||
|
||||
如果你坚持更简单,也可以第一版直接写在 `sync_project(...)` 里;但文档层面仍建议用 `ProjectScopeRuntime` 表达这个层次。
|
||||
|
||||
### 2.4 伪代码
|
||||
|
||||
```python
|
||||
class ProjectBatchPipeline:
|
||||
def __init__(self, config: BatchConfig):
|
||||
self.config = config
|
||||
self.runtime = build_state_machine_runtime()
|
||||
self.logger = build_logger(config.logging)
|
||||
self.persistence = PersistenceService(config.persist)
|
||||
self.global_runtime = None
|
||||
self.shared_local_ds = build_shared_local_datasource_resources(config.local_datasource)
|
||||
self.shared_remote_ds = build_shared_remote_datasource_resources(config.remote_datasource)
|
||||
|
||||
async def run(self) -> None:
|
||||
self.global_runtime = await self.sync_global()
|
||||
await self.dispatch_projects()
|
||||
|
||||
async def sync_global(self) -> "GlobalReadonlyRuntime":
|
||||
# 只同步 shared_node_types
|
||||
...
|
||||
|
||||
async def dispatch_projects(self) -> None:
|
||||
# semaphore 控制并发
|
||||
...
|
||||
|
||||
async def sync_project(self, local_project_id: str, remote_project_id: str) -> None:
|
||||
scope_runtime = await self.create_project_scope_runtime(local_project_id, remote_project_id)
|
||||
try:
|
||||
await scope_runtime.load()
|
||||
await run_project_sync(scope_runtime, self.config.project_node_types)
|
||||
await scope_runtime.persist()
|
||||
finally:
|
||||
await scope_runtime.unload()
|
||||
await scope_runtime.close()
|
||||
```
|
||||
|
||||
```python
|
||||
class ProjectScopeRuntime:
|
||||
def __init__(self, scope: str, persistence_view, global_runtime, shared_local_ds, shared_remote_ds):
|
||||
self.scope = scope
|
||||
self.persistence_view = persistence_view
|
||||
self.local_collection = DataCollection(...)
|
||||
self.remote_collection = DataCollection(...)
|
||||
self.binding_manager = BindingManager(...)
|
||||
self.local_datasource = build_local_datasource_runtime(shared_local_ds, self.local_collection)
|
||||
self.remote_datasource = build_remote_datasource_runtime(shared_remote_ds, self.remote_collection)
|
||||
self.global_runtime = global_runtime
|
||||
|
||||
async def load(self) -> None: ...
|
||||
async def persist(self) -> None: ...
|
||||
async def unload(self) -> None: ...
|
||||
async def close(self) -> None: ...
|
||||
```
|
||||
|
||||
### 2.5 Global 数据的处理原则
|
||||
|
||||
这点现在可以直接定死:
|
||||
|
||||
1. `global` 数据不复制到每个 project scope。
|
||||
2. `global` 数据不在每个项目里重复 bootstrap。
|
||||
3. `global` 数据不在每个项目里重复 post-check。
|
||||
|
||||
项目级运行时如果需要共享节点或绑定,采用:
|
||||
|
||||
1. 当前 project scope 先查本地运行态。
|
||||
2. 未命中时回退到 `global` 只读运行态。
|
||||
|
||||
这样可以避免复制大体量 global 数据。
|
||||
|
||||
2. `preload_shared_state(...)` 把项目 pipeline 变成了“半冷启动、半热注入”的特殊运行态。
|
||||
3. `ephemeral_node_types` / `bootstrap_binding_node_types` 这些参数开始承担原本不属于它们的职责。
|
||||
## 3. 组件说明
|
||||
|
||||
### 3.1 Persistence
|
||||
|
||||
#### 角色
|
||||
|
||||
`PersistenceService` 是 batch 共享对象。
|
||||
|
||||
它共享的是:
|
||||
|
||||
1. 后端类型。
|
||||
2. DB 路径或连接串。
|
||||
3. 连接池或底层连接能力。
|
||||
|
||||
它不共享的是:
|
||||
|
||||
1. 当前项目的内存缓存。
|
||||
2. 当前 scope 的脏状态。
|
||||
|
||||
#### 关键语义
|
||||
|
||||
建议把 `persistence` 理解成两层:
|
||||
|
||||
1. `PersistenceService`
|
||||
- 共享后端服务。
|
||||
2. `ScopedPersistenceView`
|
||||
- 绑定某个 `scope` 的轻量 view。
|
||||
|
||||
所以“scope 属于 persistence 的 scope view”这句话的意思是:
|
||||
|
||||
1. 整个 batch 共享一个 persistence service。
|
||||
2. 每个项目拿一个绑定了当前 `scope` 的轻量操作视图。
|
||||
|
||||
不是说每个项目都新建一套 persistence backend。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class PersistenceService:
|
||||
async def initialize(self) -> None: ...
|
||||
def view(self, scope: str) -> ScopedPersistenceView: ...
|
||||
async def close(self) -> None: ...
|
||||
|
||||
|
||||
class ScopedPersistenceView:
|
||||
async def load_nodes(self, collection_id: str) -> list[dict]: ...
|
||||
async def save_nodes(self, collection_id: str, nodes: list[SyncNode]) -> None: ...
|
||||
async def delete_nodes(self, collection_id: str, node_ids: list[str]) -> None: ...
|
||||
async def load_bindings(self, node_type: str) -> list[dict]: ...
|
||||
async def save_bindings(self, node_type: str, bindings: dict[str, str | None]) -> None: ...
|
||||
```
|
||||
|
||||
### 3.2 Collection
|
||||
|
||||
#### 角色
|
||||
|
||||
`Collection` 必须是 scope 级对象,不能做单例。
|
||||
|
||||
原因很简单:
|
||||
|
||||
1. 它内部天然持有当前 scope 的节点集合。
|
||||
2. 它持有 data_id 索引。
|
||||
3. 它持有删除集合和脏状态。
|
||||
|
||||
#### 关键语义
|
||||
|
||||
project scope 的 `Collection` 只存项目域数据。
|
||||
|
||||
对于共享数据,不复制 global,而是支持只读回退:
|
||||
|
||||
1. 先查当前 scope。
|
||||
2. 未命中再查 global 只读 collection。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class DataCollection:
|
||||
async def load_from_persistence(self) -> None: ...
|
||||
async def persist(self) -> None: ...
|
||||
async def unload(self) -> None: ...
|
||||
|
||||
def get(self, node_id: str) -> SyncNode | None: ...
|
||||
def get_by_data_id(self, node_type: str, data_id: str) -> SyncNode | None:
|
||||
node = self._get_local(node_type, data_id)
|
||||
if node is not None:
|
||||
return node
|
||||
return self._get_global_fallback(node_type, data_id)
|
||||
```
|
||||
|
||||
### 3.3 BindingManager
|
||||
|
||||
#### 角色
|
||||
|
||||
`BindingManager` 也是 scope 级对象,不应共享。
|
||||
|
||||
它负责:
|
||||
|
||||
1. 当前 scope 的绑定运行态。
|
||||
2. 当前 scope 的绑定持久化。
|
||||
3. 当前 scope 未命中时对 global 只读绑定做回退查询。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class BindingManager:
|
||||
async def load_from_persistence(self, node_type: str) -> None: ...
|
||||
async def persist(self) -> None: ...
|
||||
async def unload(self) -> None: ...
|
||||
|
||||
async def get_remote_id(self, node_type: str, local_id: str) -> str | None:
|
||||
remote_id = self._get_local_remote_id(node_type, local_id)
|
||||
if remote_id is not None:
|
||||
return remote_id
|
||||
return await self._get_global_remote_id(node_type, local_id)
|
||||
```
|
||||
|
||||
### 3.4 DataSource
|
||||
|
||||
#### 角色
|
||||
|
||||
`local_datasource / remote_datasource / logging` 不应按项目变化,这一点是对的。
|
||||
|
||||
但按当前实现,`BaseDataSource` 也不适合直接做单例,因为它内部持有:
|
||||
|
||||
1. `_collection`
|
||||
2. `_stats`
|
||||
3. `_action_summary`
|
||||
4. `_action_by_node_id`
|
||||
|
||||
所以第一阶段不建议“直接把当前 `BaseDataSource` 改成单例 + 注入 scope”。
|
||||
|
||||
#### 推荐的最小改法
|
||||
|
||||
第一阶段先不强行拆成两个正式类,而是采用下面的语义:
|
||||
|
||||
1. datasource 的静态配置和共享 client 是 batch 级共享的。
|
||||
2. datasource 的运行态对象仍然是项目级创建的。
|
||||
|
||||
也就是说:
|
||||
|
||||
1. 配置不按项目变化。
|
||||
2. runtime 仍然按项目隔离。
|
||||
|
||||
这样改动范围最小,也更不容易污染现有 handler 行为。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class ScopedDataSourceRuntime(BaseDataSource):
|
||||
def set_collection(self, collection: DataCollection) -> None: ...
|
||||
async def load_all(self, order: list[str]) -> None: ...
|
||||
async def sync_all(self, order: list[str]) -> None: ...
|
||||
def reset_runtime_state(self) -> None: ...
|
||||
async def close(self) -> None: ...
|
||||
```
|
||||
|
||||
当前多项目配置要求:
|
||||
|
||||
1. `global_config`
|
||||
### 第二阶段:清理生命周期
|
||||
|
||||
把项目级清理流程固定下来:
|
||||
|
||||
1. `load`
|
||||
2. `sync`
|
||||
3. `persist`
|
||||
4. `unload`
|
||||
5. `close`
|
||||
|
||||
### 第三阶段:再决定 datasource 是否正式拆层
|
||||
|
||||
如果第一阶段跑通,可以再决定是否把 datasource 正式拆成:
|
||||
|
||||
1. 共享静态层。
|
||||
2. scope runtime 层。
|
||||
|
||||
这一步不是必须第一批完成。
|
||||
|
||||
### 最终落点
|
||||
|
||||
最终希望达到的状态是:
|
||||
|
||||
1. global 数据只同步一次,只读共享。
|
||||
2. project 数据按 scope 独立运行。
|
||||
3. 项目跑完即可卸载运行态。
|
||||
4. 大部分改动都限制在基础设施层。
|
||||
1. 全局阶段一份配置。
|
||||
2. 项目阶段一份默认配置。
|
||||
3. 每个项目还能再覆盖一份配置。
|
||||
|
||||
这对于少量项目还能接受,但项目数到了几百个就不可维护了。
|
||||
|
||||
如果每个项目都允许自带一份 pipeline 配置,系统就失去“批量同步”的基本前提:
|
||||
|
||||
1. 无法判断哪些差异是业务差异,哪些是配置漂移。
|
||||
2. 无法稳定复用并发调度逻辑。
|
||||
3. 无法保证结果可复现。
|
||||
|
||||
结论是:多项目批量同步不应允许“每项目一套 pipeline 定义”。
|
||||
|
||||
@@ -37,6 +37,10 @@
|
||||
| `ABNORMAL` | 数据损坏或核心绑定异常 | 否 |
|
||||
| `DEPENDENCY_ERROR` | 依赖不可满足 | 否 |
|
||||
|
||||
bootstrap 特殊语义:
|
||||
- 如果节点在持久化恢复后存在绑定,但 bootstrap datasource refresh 未返回最新数据,则节点进入 `S17` 语义并被 `bootstrap_blocked` 隔离。
|
||||
- 如果节点仅残留在 persistence 中、没有绑定,且 datasource refresh 也未返回,则节点按 stale zombie 清理,不再进入 bind。
|
||||
|
||||
## 3. create / update 语义
|
||||
|
||||
### create
|
||||
@@ -64,6 +68,11 @@
|
||||
- 状态落地:`Decision.to_state` 对应 YAML `states[Sxx]`。
|
||||
- 不变量:每次落地后执行 `runtime.check_invariants()`。
|
||||
|
||||
bootstrap 额外输入:
|
||||
- `is_create_zombie`
|
||||
- `bootstrap_source_present`
|
||||
- `has_binding_record`
|
||||
|
||||
## 6. 关键一致性约束
|
||||
|
||||
- `UNCHECKED` 不允许带动作(`INV-1_UNCHECKED_ACTION_NONE`)。
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
# ECM Sync Quality Cleanup Execution Plan
|
||||
|
||||
## Goal
|
||||
|
||||
This cleanup pass keeps behavior unchanged and prioritizes deleting, merging, and unifying duplicated code on the single-pipeline path.
|
||||
|
||||
Acceptance remains unchanged:
|
||||
|
||||
- `backend/local_tests/sync_system_tests/depm_to_ecm_bootstrap_pull_then_project_push.pipeline.yaml` must still run successfully.
|
||||
- Local Summary and Remote Summary must keep the same structure and stable output shape.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Do not modify `schemas/`.
|
||||
- Do not modify `filtered_datasource/`.
|
||||
- Do not bypass schema validation.
|
||||
- Do not add fallback logic that changes behavior.
|
||||
- Prefer helper extraction, deletion, and merge over new abstraction layers.
|
||||
|
||||
## Execution Order
|
||||
|
||||
### Batch 1: Entry + Factory
|
||||
|
||||
Status: completed
|
||||
|
||||
- Turn `run.py` into a thin wrapper around `sync_state_machine.cli.main`.
|
||||
- Merge duplicated datasource lifecycle and resolved-config logging in `sync_state_machine/pipeline/factory.py`.
|
||||
- Verify with focused unit tests and the real DEPM to ECM profile.
|
||||
|
||||
### Batch 2: Pipeline Flow + Backend Runner
|
||||
|
||||
Status: completed
|
||||
|
||||
- Reduce repeated phase-completion, strategy-loop, reload-skip, and close-handling code in `sync_state_machine/pipeline/full_sync_pipeline.py`.
|
||||
- Reduce repeated orchestration code in `backend/app/thirdparty/ecm_sync/scripts/depm_to_ecm/run_pipeline.py` without changing CLI behavior or fail-fast rules.
|
||||
- Keep log text stable where practical.
|
||||
|
||||
### Batch 3: Datasource + Handler Base Cleanup
|
||||
|
||||
Status: completed
|
||||
|
||||
- Reduce duplicated handler registration and context injection between datasource base classes and subclasses.
|
||||
- Reduce duplicated node metadata initialization across handler subclasses by pushing safe defaults into handler base classes.
|
||||
- Delete domain-level constructor boilerplate where subclasses only repeat node type, node class, schema, or no-op `_create_node` wrappers.
|
||||
|
||||
### Batch 4: Domain Repeat Cleanup + Docs
|
||||
|
||||
Status: in progress
|
||||
|
||||
- Inspect domain handlers for repeated load/update helper logic and merge only stable repeated fragments into shared helpers.
|
||||
- Update docs so they only describe the real maintained execution path.
|
||||
- Keep backend runner documentation aligned with current script behavior.
|
||||
- Remove stale references only after code paths are confirmed stable.
|
||||
|
||||
## Validation Plan
|
||||
|
||||
1. Run targeted `ecm_sync_system` unit tests for the touched pipeline code.
|
||||
2. Run the real backend profile with the backend interpreter.
|
||||
3. Confirm the output still ends with pipeline completion plus Local Summary and Remote Summary tables.
|
||||
|
||||
## Current Focus
|
||||
|
||||
Current implementation focus is Batch 4.
|
||||
|
||||
- repeated load/update helpers under `sync_state_machine/domain/`
|
||||
- maintained execution-path docs under `docs/`
|
||||
|
||||
The intent is code reduction only, not behavior redesign.
|
||||
+469
-104
@@ -1,143 +1,508 @@
|
||||
# 运行配置业务说明
|
||||
# 运行配置写法说明
|
||||
|
||||
本文档从业务视角说明配置字段含义、对阶段行为的影响,以及推荐的覆盖方式。
|
||||
这份文档只讲一件事:现在 `ecm_sync_system` 的配置文件应该怎么写。
|
||||
|
||||
## 1. 推荐使用方式(先 default,再覆盖)
|
||||
重点不是把所有默认值抄一遍,而是说明当前真实生效的写法、哪些字段是内置能力、哪些字段来自业务侧扩展,以及单项目和多项目文件各自长什么样。
|
||||
|
||||
1. 先使用基线配置运行:
|
||||
- `run_profiles/preset/jsonl_to_jsonl.default.yaml`
|
||||
2. 若需项目定制,再复制为:
|
||||
- `run_profiles/jsonl_to_jsonl.local.yaml`
|
||||
3. 仅改有业务差异的字段,避免一次改太多导致排障困难。
|
||||
## 1. 先分清两类配置文件
|
||||
|
||||
占位符:
|
||||
- `${PROJECT_ROOT}` 会在运行时替换为项目根目录绝对路径。
|
||||
当前有两类 YAML:
|
||||
|
||||
## 2. 顶层字段(你在改什么)
|
||||
1. 单 pipeline 配置
|
||||
- 对应 `PipelineRunConfig`
|
||||
- 用来描述一条完整同步链路
|
||||
- 顶层通常包含:`node_types`、`local_datasource`、`remote_datasource`、`persist`、`logging`、`strategies`
|
||||
2. 多项目编排配置
|
||||
- 对应 `MultiProjectRunConfig`
|
||||
- 只负责描述“先跑全局,再按项目并发跑哪些项目”
|
||||
- 顶层包含:`global_config`、`default_project_config`、`max_concurrency`、`project_bootstrap_node_types`、`projects`
|
||||
|
||||
判断规则很简单:
|
||||
|
||||
- 如果文件里有 `global_config`、`default_project_config`、`projects`,它就是多项目编排配置。
|
||||
- 否则按单 pipeline 配置解析。
|
||||
|
||||
## 2. 配置加载约定
|
||||
|
||||
当前约定如下:
|
||||
|
||||
- 跟踪在仓库里的模板和示例,只写必要覆盖项,不重复展开默认值。
|
||||
- pipeline 启动后会打印一份 `Resolved Full Config`,它才是最终生效的完整配置。
|
||||
- 排障优先看 `Resolved Full Config`,不要只盯着输入 YAML。
|
||||
|
||||
## 3. 占位符规则
|
||||
|
||||
`ecm_sync_system` 核心库内置只认识一个占位符:
|
||||
|
||||
- `${PROJECT_ROOT}`:运行时替换为 `ecm_sync_system` 仓库根目录绝对路径。
|
||||
|
||||
例如:
|
||||
|
||||
```yaml
|
||||
persist:
|
||||
backend: sqlite
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
|
||||
```
|
||||
|
||||
需要特别区分的是:
|
||||
|
||||
- `${BACKEND_ROOT}`、`${ECM_SYNC_LOG_DIR}` 这类占位符不是核心库通用语义。
|
||||
- 它们通常是 backend 集成脚本在读取 profile 时额外替换的调用方语义。
|
||||
- 如果你不是通过 backend 的联调脚本运行,而是直接在 `ecm_sync_system` 内跑 profile,就不要假设这些占位符一定可用。
|
||||
|
||||
## 4. 单 pipeline 配置的当前写法
|
||||
|
||||
### 4.1 最小骨架
|
||||
|
||||
```yaml
|
||||
node_types:
|
||||
- company
|
||||
- supplier
|
||||
- project
|
||||
|
||||
local_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
|
||||
|
||||
remote_datasource:
|
||||
type: api
|
||||
api_base_url: http://127.0.0.1:8000/api/third/v2
|
||||
api_uid: your_uid
|
||||
api_secret: your_secret
|
||||
|
||||
persist:
|
||||
backend: sqlite
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
|
||||
|
||||
logging:
|
||||
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
|
||||
file_name_pattern: demo_{timestamp}.log
|
||||
|
||||
strategies:
|
||||
project:
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- name
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
```
|
||||
|
||||
### 4.2 顶层字段含义
|
||||
|
||||
- `node_types`
|
||||
- 本轮参与同步的业务类型列表。
|
||||
- 顺序会影响依赖链执行时机(当前按 node_type 串行执行 bind/create/update)。
|
||||
- `target_project_ids`
|
||||
- 兼容字段。作为两侧 datasource 的项目白名单兜底值。
|
||||
- 若 datasource 自身配置了 `target_project_ids`,则优先使用 datasource 级配置。
|
||||
- 本轮参与同步的节点类型列表。
|
||||
- 顺序会影响执行顺序,依赖链通常要把上游类型放前面。
|
||||
- `local_datasource` / `remote_datasource`
|
||||
- 分别定义两侧数据来源与执行端。
|
||||
- `strategies`
|
||||
- 每个业务类型的行为策略(依赖、自动绑定、创建方向、更新方向、跳过等)。
|
||||
- 定义两侧数据源类型和对应参数。
|
||||
- `persist`
|
||||
- 持久化开关与数据库位置。
|
||||
- 定义持久化后端及目标位置。
|
||||
- `logging`
|
||||
- 运行日志输出位置与级别。
|
||||
- 定义日志初始化、落盘位置和级别。
|
||||
- `strategies`
|
||||
- 定义每个 `node_type` 的同步策略。
|
||||
|
||||
## 3. 数据源字段(DataSource)
|
||||
## 5. datasource 字段怎么写
|
||||
|
||||
### 3.1 JSONL 数据源
|
||||
### 5.1 内置 datasource type
|
||||
|
||||
- `type=jsonl`:使用本地 JSONL 文件作为数据源。
|
||||
- `jsonl_dir`:JSONL 目录路径。
|
||||
- `read_only=false`:允许该侧执行 create/update 回写;若为 true,通常用于只读加载。
|
||||
- `target_project_ids=[]`:该侧项目白名单;为空时不进行项目筛选。
|
||||
- `handler_configs`:预留给各业务 handler 的扩展参数。
|
||||
核心库内置注册了两种 datasource type:
|
||||
|
||||
### 3.2 API 数据源
|
||||
1. `jsonl`
|
||||
2. `api`
|
||||
|
||||
- `type=api`:对接 HTTP API。
|
||||
- `api_base_url` / `api_uid` / `api_secret`:接口访问参数。
|
||||
- `api_debug`:开启后输出更多调试信息。
|
||||
- `poll_max_retries` / `poll_interval`:异步任务轮询次数与间隔。
|
||||
- `target_project_ids`:**必填**,该侧项目白名单;至少需要指定一个项目ID,以避免加载过多远程数据。
|
||||
- `handler_configs`:业务 handler 扩展参数。
|
||||
#### `jsonl` 写法
|
||||
|
||||
## 4. 策略字段(Strategy)
|
||||
```yaml
|
||||
local_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
|
||||
read_only: false
|
||||
handler_configs:
|
||||
project:
|
||||
data_id_filter:
|
||||
- project-id-1
|
||||
```
|
||||
|
||||
每个 node_type 都建议显式写出以下字段,便于审阅与排障。
|
||||
字段说明:
|
||||
|
||||
### 4.1 `update_direction`
|
||||
- `jsonl_dir`:JSONL 数据目录。
|
||||
- `read_only`:是否只读。
|
||||
- `handler_configs`:按 `node_type` 下发给 handler 的扩展参数。
|
||||
|
||||
- `push`:本地变更推送到远端。
|
||||
- `pull`:远端变更拉取到本地。
|
||||
- `bidirectional`:双向更新(需要明确冲突策略,谨慎使用)。
|
||||
- `none`:关闭 update 阶段。
|
||||
#### `api` 写法
|
||||
|
||||
### 4.2 `local_orphan_action` / `remote_orphan_action`
|
||||
```yaml
|
||||
remote_datasource:
|
||||
type: api
|
||||
api_base_url: http://127.0.0.1:8000/api/third/v2
|
||||
api_uid: your_uid
|
||||
api_secret: your_secret
|
||||
api_debug: true
|
||||
poll_max_retries: 10
|
||||
poll_interval: 0.5
|
||||
api_rate_limit_max_requests: 30
|
||||
api_rate_limit_window_seconds: 10.0
|
||||
handler_configs:
|
||||
project:
|
||||
data_id_filter:
|
||||
- remote-project-id-1
|
||||
```
|
||||
|
||||
- `create_remote`:本地孤儿创建到远端。
|
||||
- `create_local`:远端孤儿创建到本地。
|
||||
- `none`:不自动创建。
|
||||
- `delete_local` / `delete_remote`:当前主流程未作为默认路径,谨慎使用。
|
||||
字段说明:
|
||||
|
||||
### 4.3 `auto_bind` / `auto_bind_fields` / `pre_bind_data_id`
|
||||
- `api_debug`:开启后输出更详细的请求日志。
|
||||
- `poll_max_retries` / `poll_interval`:异步任务轮询控制。
|
||||
- `api_rate_limit_max_requests` / `api_rate_limit_window_seconds`:全局限流窗口。
|
||||
- `handler_configs`:同样按 `node_type` 下发业务参数。
|
||||
|
||||
- `auto_bind=true`:在 `S02(MISSING)` 上尝试自动匹配。
|
||||
- `auto_bind=false`:不会做自动匹配,也不会走自动创建识别路径;通常会进入阻断态等待人工处理。
|
||||
- `auto_bind_fields`:自动匹配使用的业务键字段。
|
||||
- `pre_bind_data_id`:显式 data_id 预绑定清单(通用能力),格式为:
|
||||
- `- local_data_id: "..."`
|
||||
` remote_data_id: "..."`
|
||||
- 规则:若两侧数据源都存在这对 data_id 对应节点,且当前无绑定记录,则直接绑定。
|
||||
- 不依赖 `auto_bind/auto_bind_fields`,即使 `auto_bind=false` 也会执行。
|
||||
- 不做业务键一对一匹配检查,按你给定的映射对优先执行。
|
||||
### 5.2 业务侧注册的 datasource type
|
||||
|
||||
### 4.4 `depend_fields`
|
||||
核心库允许业务系统在启动阶段注册自己的 datasource type。
|
||||
|
||||
- 定义依赖字段到依赖 node_type 的映射,例如:`company_id -> company`。
|
||||
- 依赖节点状态非 `NORMAL` 时,会在 bind 的依赖检查阶段进入 `DEPENDENCY_ERROR`。
|
||||
因此,配置里不只可以写 `jsonl` 和 `api`,还可以写业务侧注册出来的类型,例如 backend 里的 `depm_db`。
|
||||
|
||||
### 4.5 `skip_sync`
|
||||
当前 backend 的 `depm_db` 真实写法是:
|
||||
|
||||
- 运行配置中已不建议使用 `skip_sync`。
|
||||
- 推荐使用显式组合:
|
||||
- `local_orphan_action: none`
|
||||
- `remote_orphan_action: none`
|
||||
- `update_direction: none`
|
||||
- 当三者同时为 `none/none/none` 时,pipeline 会跳过该 node_type 的 bind/create/update。
|
||||
```yaml
|
||||
local_datasource:
|
||||
type: depm_db
|
||||
tenant_id: "019976df-0628-7c46-a896-81d3f290a7df"
|
||||
app:
|
||||
config_file: ${BACKEND_ROOT}/config.yaml
|
||||
overrides:
|
||||
database:
|
||||
url: mysql+aiomysql://root:123456@127.0.0.1/depm_prod_local_auto_test?charset=utf8mb4
|
||||
handler_configs:
|
||||
project:
|
||||
data_id_filter:
|
||||
- 019a62a5-2323-7be2-99c8-82534f8befa4
|
||||
```
|
||||
|
||||
### 4.6 Handler 级参数(`handler_configs`)
|
||||
这个例子说明两点:
|
||||
|
||||
- 建议将 handler 特有参数放在 datasource 的 `handler_configs.<node_type>` 下。
|
||||
- 示例:
|
||||
- `remote_datasource.handler_configs.supplier.load_mode: persisted_only`
|
||||
- `persisted_only` 表示只使用已持久化数据,不主动从外部拉新数据(具体以对应 handler 实现为准)。
|
||||
1. datasource 的可写字段由对应注册模型决定,不是所有 type 都共用同一套字段。
|
||||
2. 如果 `type` 没有先被注册,配置校验会直接报错,不会做降级处理。
|
||||
|
||||
## 5. 阶段行为与配置关系
|
||||
`depm_db` 这类业务扩展字段需要以业务侧实现为准。对 backend 当前实现来说:
|
||||
|
||||
- bind 阶段
|
||||
- `depend_fields` 决定是否做依赖校验。
|
||||
- `pre_bind_data_id` 先按显式 local/remote data_id 映射做预绑定。
|
||||
- `auto_bind/auto_bind_fields` 决定是否继续进行业务键自动绑定。
|
||||
- create 阶段
|
||||
- `local_orphan_action` / `remote_orphan_action` 决定孤儿是否创建。
|
||||
- update 阶段
|
||||
- `update_direction` 决定是否和向哪侧更新。
|
||||
- 写入后 reload(Pipeline 内置)
|
||||
- 当某个 node_type 的 create 或 update 发生实际写入成功后,pipeline 会自动对该 node_type 重新执行两侧 `load`。
|
||||
- 目的:让后续 update 判断基于最新远端/本地数据,支持“create 后再补一次 update”的链路。
|
||||
- 同步后一致性检查(Pipeline 内置)
|
||||
- 每个 node_type 在本轮同步后会基于绑定对执行数据一致性校验(比较时忽略 `id/_id` 字段)。
|
||||
- 若发现差异,会在日志中打印差异样本,并在汇总表新增 `Consistent` 列展示 `Y` / `N(差异数)`。
|
||||
- 全局跳过
|
||||
- `local_orphan_action=none` + `remote_orphan_action=none` + `update_direction=none`
|
||||
可直接跳过该类型的 bind/create/update。
|
||||
- `tenant_id`:DEPM 本地 datasource 的租户过滤 ID。
|
||||
- `app.config_file`:backend 配置文件路径,默认可落回 `backend/config.yaml`。
|
||||
- `app.overrides`:直接传给 backend `Settings()` 的覆盖项。
|
||||
|
||||
## 6. 覆盖配置建议
|
||||
## 6. strategy 写法
|
||||
|
||||
- 先在 default 跑通一轮,再做 local 覆盖。
|
||||
- 每次只改一类问题相关字段(例如只改 `project` 的依赖与自动绑定)。
|
||||
- 覆盖后优先看 `sync_log` 是否出现预期状态迁移。
|
||||
### 6.1 当前推荐写法
|
||||
|
||||
## 7. 持久化与日志字段
|
||||
`strategies` 现在推荐写成“按 `node_type` 为键”的映射,而不是手工写一长串列表。
|
||||
|
||||
### `persist`
|
||||
```yaml
|
||||
strategies:
|
||||
supplier:
|
||||
skip_post_check: true
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- credit_code
|
||||
depend_fields: {}
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: create_local
|
||||
update_direction: none
|
||||
project:
|
||||
domain_option:
|
||||
pull_group_plan_production_date: true
|
||||
config:
|
||||
auto_bind: false
|
||||
auto_bind_fields: []
|
||||
pre_bind_data_id:
|
||||
- local_data_id: local-project-id
|
||||
remote_data_id: remote-project-id
|
||||
depend_fields:
|
||||
company_id: company
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
```
|
||||
|
||||
- `enable`:是否开启持久化。
|
||||
- `db_path`:SQLite 路径。
|
||||
- `wipe_on_start`:启动是否清理历史持久化。
|
||||
### 6.2 常用字段
|
||||
|
||||
### `logging`
|
||||
- `auto_bind`
|
||||
- 是否开启自动绑定。
|
||||
- `auto_bind_fields`
|
||||
- 自动绑定使用的业务键字段。
|
||||
- `pre_bind_data_id`
|
||||
- 显式 local/remote data_id 绑定对。
|
||||
- `depend_fields`
|
||||
- 依赖字段到依赖 `node_type` 的映射。
|
||||
- `local_orphan_action` / `remote_orphan_action`
|
||||
- 孤儿节点创建策略。
|
||||
- `update_direction`
|
||||
- `push` / `pull` / `none`。
|
||||
- `post_check_ignore_fields`
|
||||
- 只影响最终一致性检查,不影响 create/update 差异检测。
|
||||
- `compare_ignore_fields`
|
||||
- schema diff / validate 时忽略的顶层字段。
|
||||
- `skip_sync`
|
||||
- 只跳过 create/update,不跳过 load/bind。
|
||||
- `skip_post_check`
|
||||
- 跳过该类型的 post-check reload 与一致性校验。
|
||||
- `domain_option`
|
||||
- 领域策略自己的扩展参数,由对应 domain 自己解释。
|
||||
|
||||
- `file_dir`:日志目录。
|
||||
- `file_name_pattern`:文件命名模板。
|
||||
- `level`:日志级别。
|
||||
### 6.3 `preset` 仍然可用
|
||||
|
||||
排障建议:运行失败时优先看日志中的 `状态机流转` 与 `reason=`,再对照 `docs/state_machine.md` 与 `docs/node_state_definition.md` 理解含义。
|
||||
如果只是想套一组预设策略,可以这样写:
|
||||
|
||||
```yaml
|
||||
strategies:
|
||||
project:
|
||||
preset: pull_only
|
||||
```
|
||||
|
||||
当前内置预设包括:
|
||||
|
||||
- `first_sync`
|
||||
- `daily_sync`
|
||||
- `repair_sync`
|
||||
- `pull_only`
|
||||
- `push_only`
|
||||
|
||||
### 6.4 已废弃的旧字段不要再写
|
||||
|
||||
以下 legacy key 已明确拒绝:
|
||||
|
||||
- `force_sync`
|
||||
- `load_mode`
|
||||
|
||||
如果你在 strategy 里写这两个字段,配置会直接报错。
|
||||
|
||||
当前替代方式是:
|
||||
|
||||
- 是否跳过同步,用 `skip_sync`
|
||||
- handler 特有加载参数,写到 `datasource.handler_configs.<node_type>`
|
||||
|
||||
## 7. persist 的当前写法
|
||||
|
||||
### 7.1 校验规则
|
||||
|
||||
当前 `persist` 配置遵循下面的规则:
|
||||
|
||||
1. `db_path` 和 `url` 至少要有一个。
|
||||
2. 当 `backend != sqlite` 时,必须显式提供 `url`。
|
||||
3. `wipe_on_start=true` 只会尝试删除可映射到文件路径的持久化文件;对 MySQL 这类远程数据库不会帮你删库。
|
||||
|
||||
### 7.2 SQLite 文件持久化
|
||||
|
||||
```yaml
|
||||
persist:
|
||||
backend: sqlite
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/simple_pipeline_sm.db
|
||||
enable: true
|
||||
wipe_on_start: false
|
||||
```
|
||||
|
||||
### 7.3 SQLAlchemy 持久化到 SQLite URL
|
||||
|
||||
```yaml
|
||||
persist:
|
||||
backend: sqlalchemy
|
||||
url: sqlite+aiosqlite:////abs/path/to/pipeline.db
|
||||
enable: true
|
||||
wipe_on_start: false
|
||||
```
|
||||
|
||||
### 7.4 SQLAlchemy 持久化到 MySQL
|
||||
|
||||
```yaml
|
||||
persist:
|
||||
backend: sqlalchemy
|
||||
url: mysql+aiomysql://root:123456@127.0.0.1/ecm_sync_system?charset=utf8mb4
|
||||
enable: true
|
||||
wipe_on_start: false
|
||||
```
|
||||
|
||||
当前语义:
|
||||
|
||||
- `sqlalchemy` backend 会自动创建它自己的 `nodes` 和 `bindings` 表及索引。
|
||||
- 它不会替你创建 MySQL 数据库本身,所以 `ecm_sync_system` 这个库需要提前存在。
|
||||
- 现有表结构已经兼容 MySQL,不需要再额外给主键字段手工指定前缀长度。
|
||||
|
||||
### 7.5 `backend_options` 什么时候用
|
||||
|
||||
`backend_options` 仍然保留,但现在优先使用显式字段:
|
||||
|
||||
- `db_path`
|
||||
- `url`
|
||||
- `enable`
|
||||
- `wipe_on_start`
|
||||
|
||||
除非某个自定义 backend 明确要求,否则不要把主连接信息再塞进 `backend_options`。
|
||||
|
||||
## 8. logging 的当前写法
|
||||
|
||||
```yaml
|
||||
logging:
|
||||
initialize: true
|
||||
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
|
||||
file_name_pattern: simple_pipeline_sm_{timestamp}.log
|
||||
level: 20
|
||||
suppress_node_logs: true
|
||||
```
|
||||
|
||||
字段说明:
|
||||
|
||||
- `initialize`
|
||||
- 是否初始化应用日志。
|
||||
- `file_path`
|
||||
- 直接指定日志文件绝对路径。
|
||||
- `file_dir`
|
||||
- 只指定目录时,实际文件名由 `file_name_pattern` 生成。
|
||||
- `file_name_pattern`
|
||||
- 支持 `{timestamp}` 占位。
|
||||
- `level`
|
||||
- 默认 `20`,即 `INFO`。
|
||||
- `suppress_node_logs`
|
||||
- 是否压缩逐节点日志噪音。
|
||||
|
||||
## 9. 多项目编排文件怎么写
|
||||
|
||||
### 9.1 结构
|
||||
|
||||
多项目文件不是把所有字段都堆在一起,而是引用两份单 pipeline 配置:
|
||||
|
||||
```yaml
|
||||
global_config: run_profiles/jsonl_to_jsonl.multi_project.global.yaml
|
||||
default_project_config: run_profiles/jsonl_to_jsonl.multi_project.project.yaml
|
||||
|
||||
project_bootstrap_node_types:
|
||||
- company
|
||||
- supplier
|
||||
|
||||
max_concurrency: 8
|
||||
|
||||
projects:
|
||||
- local_project_id: local-project-1
|
||||
remote_project_id: remote-project-1
|
||||
- local_project_id: local-project-2
|
||||
remote_project_id: remote-project-2
|
||||
```
|
||||
|
||||
字段说明:
|
||||
|
||||
- `global_config`
|
||||
- 全局公共节点要跑的单 pipeline 配置。
|
||||
- `default_project_config`
|
||||
- 每个项目默认使用的单 pipeline 配置。
|
||||
- `project_bootstrap_node_types`
|
||||
- 需要从 global scope 继承到项目 scope 的公共节点类型,例如 `company`、`supplier`。
|
||||
- `max_concurrency`
|
||||
- 项目并发数上限,默认 `20`。
|
||||
- `projects`
|
||||
- 项目映射清单。
|
||||
|
||||
### 9.2 项目项支持单独切配置
|
||||
|
||||
如果某个项目需要特殊策略,可以在项目项里覆盖 `config_path`:
|
||||
|
||||
```yaml
|
||||
projects:
|
||||
- local_project_id: local-project-1
|
||||
remote_project_id: remote-project-1
|
||||
- local_project_id: local-project-2
|
||||
remote_project_id: remote-project-2
|
||||
config_path: run_profiles/special_project.pipeline.yaml
|
||||
```
|
||||
|
||||
此时该项目不会再用 `default_project_config`,而是改用自己的单 pipeline 配置文件。
|
||||
|
||||
### 9.3 当前多项目 profile 的实际风格
|
||||
|
||||
以当前 backend 两项目推送链路为例,常见组合是:
|
||||
|
||||
- `local_datasource.type: depm_db`
|
||||
- `remote_datasource.type: api`
|
||||
- `persist.backend: sqlalchemy`
|
||||
- `persist.url: mysql+aiomysql://...`
|
||||
- `strategies.project.config.pre_bind_data_id` 明确写每个项目的 local/remote 映射
|
||||
|
||||
这类 profile 本质上仍然是“多项目编排文件 + 两份单 pipeline 文件”的组合,不要把它理解成第三种配置格式。
|
||||
|
||||
## 10. 当前推荐的写法习惯
|
||||
|
||||
### 10.1 只写必要覆盖项
|
||||
|
||||
推荐:
|
||||
|
||||
- 只写与你当前环境、项目过滤、持久化目标、策略差异直接相关的字段。
|
||||
|
||||
不推荐:
|
||||
|
||||
- 把默认值全部展开复制一遍。
|
||||
|
||||
原因很直接:
|
||||
|
||||
- 默认值一旦调整,你的大段本地 YAML 会悄悄过时。
|
||||
- 排障时也更难看出你到底改了什么。
|
||||
|
||||
### 10.2 handler 特有参数放 datasource 里
|
||||
|
||||
比如:
|
||||
|
||||
```yaml
|
||||
remote_datasource:
|
||||
type: api
|
||||
handler_configs:
|
||||
contract_settlement_detail:
|
||||
load_method: aggregate
|
||||
```
|
||||
|
||||
这类参数应该写在 `datasource.handler_configs.<node_type>`,不要错写到 strategy 里。
|
||||
|
||||
### 10.3 项目过滤显式下沉到 handler
|
||||
|
||||
例如:
|
||||
|
||||
```yaml
|
||||
handler_configs:
|
||||
project:
|
||||
data_id_filter:
|
||||
- project-id-1
|
||||
- project-id-2
|
||||
```
|
||||
|
||||
不要在 pipeline 外层再做一层临时过滤逻辑。项目范围应尽量体现在配置里,并由对应 handler 解释。
|
||||
|
||||
### 10.4 看到报错先修配置,不要做降级
|
||||
|
||||
当前系统对配置保持 fail-fast:
|
||||
|
||||
- datasource type 未注册会报错
|
||||
- legacy strategy key 会报错
|
||||
- `persist` 缺失目标会报错
|
||||
- profile 结构不符合 schema 会报错
|
||||
|
||||
这属于预期行为。优先修正配置,而不是在文档或调用侧加 fallback。
|
||||
|
||||
## 11. 参考文件
|
||||
|
||||
如果你想直接对照真实样例,优先看这些文件:
|
||||
|
||||
- `run_profiles/jsonl_to_api.yaml`
|
||||
- `run_profiles/jsonl_to_jsonl.multi_project.yaml`
|
||||
- backend 中的 `local_tests/sync_system_tests/depm_to_ecm_bootstrap_pull_then_project_push.two_projects.sqlalchemy.pipeline.yaml`
|
||||
|
||||
如果你想看配置模型定义,直接看:
|
||||
|
||||
- `sync_state_machine/config/pipeline_config.py`
|
||||
- `sync_state_machine/config/datasource_config.py`
|
||||
- `sync_state_machine/config/persist_config.py`
|
||||
- `sync_state_machine/config/multi_project_config.py`
|
||||
- `sync_state_machine/config/strategy_config.py`
|
||||
|
||||
排障时优先顺序:
|
||||
|
||||
1. 看输入 YAML。
|
||||
2. 看启动日志里的 `Resolved Full Config`。
|
||||
3. 看对应 datasource type 是否真的已经注册。
|
||||
4. 再看运行日志和状态机结果。
|
||||
|
||||
+14
-11
@@ -21,21 +21,22 @@
|
||||
- `S12C/S12U/S12D`:按动作拆分的执行失败状态
|
||||
- `S13`:对侧创建等待态(`PEER_CREATING/NONE/PENDING`)
|
||||
- `S14`:UPDATE 运行时降级为 `NONE` 并跳过
|
||||
- `S17`:bootstrap 加载异常隔离(存在绑定,但 datasource refresh 未返回最新节点)
|
||||
|
||||
## 3. 事件到代码映射
|
||||
|
||||
### bootstrap
|
||||
- `E01`:`engine/events.py::e01_bootstrap()`
|
||||
- `E01`:`sync_state_machine/engine/events.py::e01_bootstrap()`
|
||||
|
||||
### bind
|
||||
- `E10`:`engine/events.py::e10_bind_core()`
|
||||
- `E15`:`engine/events.py::e15_bind_dependency()`
|
||||
- `E16`:`engine/events.py::e16_bind_auto()`(自动绑定判定;`ZERO+create_enabled=true` 时进入 `S13`,等待创建副作用提交)
|
||||
- `E10`:`sync_state_machine/engine/events.py::e10_bind_core()`
|
||||
- `E15`:`sync_state_machine/engine/events.py::e15_bind_dependency()`
|
||||
- `E16`:`sync_state_machine/engine/events.py::e16_bind_auto()`(自动绑定判定;`ZERO+create_enabled=true` 时进入 `S13`,等待创建副作用提交)
|
||||
|
||||
### create/update prepare
|
||||
- `E20`:`engine/events.py::e20_create_prepare()`(spawn 成功从 `S13` 提交到 `S01`;失败保持 `S13`)
|
||||
- `E21`:`engine/events.py::e21_create_prepare()`
|
||||
- `E30`:`engine/events.py::e30_update_prepare()`
|
||||
- `E20`:`sync_state_machine/engine/events.py::e20_create_prepare()`(spawn 成功从 `S13` 提交到 `S01`;失败保持 `S13`)
|
||||
- `E21`:`sync_state_machine/engine/events.py::e21_create_prepare()`
|
||||
- `E30`:`sync_state_machine/engine/events.py::e30_update_prepare()`
|
||||
|
||||
### sync execute
|
||||
- `E40C_START/E40C`:CREATE 执行入口与结果回写
|
||||
@@ -43,14 +44,15 @@
|
||||
- `E40D_START/E40D`:DELETE 执行入口与结果回写
|
||||
|
||||
## 4. strategy 调用链(当前)
|
||||
- bind 主入口:`sync_system/strategy.py::DefaultSyncStrategy.bind()` -> `strategy_ops/bind_ops.py::run_bind()`
|
||||
- create 主入口:`sync_system/strategy.py::DefaultSyncStrategy.create()` -> `strategy_ops/create_ops.py::run_create()`
|
||||
- update 主入口:`sync_system/strategy.py::DefaultSyncStrategy.update()` -> `strategy_ops/update_ops.py::run_update()`
|
||||
- reset 主入口:`sync_system/strategy.py::BaseSyncStrategy.run_reset()` -> `strategy_ops/reset_ops.py::run_phase2_cleanup()`
|
||||
- bind 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.bind()`
|
||||
- create 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.create()`
|
||||
- update 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.update()`
|
||||
- reset 主入口:`sync_state_machine/pipeline/full_sync_pipeline.py::_load_persistence_state()` -> `sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`
|
||||
|
||||
## 5. context 字段(以 YAML 为准)
|
||||
|
||||
### bind 相关
|
||||
- `bootstrap_source_present`
|
||||
- `has_binding_record`
|
||||
- `core_binding_result` / `peer_core_binding_result` (`NORMAL|WARNING|ABNORMAL`)
|
||||
- `data_present`
|
||||
@@ -75,6 +77,7 @@
|
||||
## 6. 语义边界
|
||||
- strategy 负责业务语义计算(如依赖是否满足、自动绑定结果、差异检测)。
|
||||
- 状态机只负责迁移判定与不变量校验,不直接访问数据源。
|
||||
- bootstrap 阶段的数据源刷新由 pipeline 完成,状态机只消费 `bootstrap_source_present / has_binding_record / is_create_zombie` 这些已计算上下文。
|
||||
|
||||
## 7. 不变量
|
||||
- `INV-1_UNCHECKED_ACTION_NONE`
|
||||
|
||||
+46
-21
@@ -2,49 +2,74 @@
|
||||
|
||||
## 1. 全流程主线
|
||||
1. pipeline 装配组件并加载持久化。
|
||||
2. pipeline 调用 `strategy.bind()`。
|
||||
3. pipeline 调用 `strategy.create()` / `strategy.update()` 生成动作节点。
|
||||
4. pipeline 调用 `datasource.sync_all()` 执行动作并回写执行状态。
|
||||
5. pipeline 输出统计并持久化。
|
||||
2. pipeline 为 datasource 绑定 collection,但不在 bootstrap 阶段全量拉取业务数据。
|
||||
3. pipeline 在每个 `node_type` 开始执行前,单独加载该 `node_type` 的最新数据。
|
||||
4. pipeline 调用 `strategy.bind()`。
|
||||
5. pipeline 调用 `strategy.create()` / `strategy.update()` 生成动作节点。
|
||||
6. pipeline 调用 `datasource.sync_all()` 执行动作并回写执行状态。
|
||||
7. pipeline 输出统计并持久化。
|
||||
|
||||
## 2. reset 时序
|
||||
|
||||
### 2.1 加载时 reset(phase1 语义)
|
||||
- 在 `common/collection.py::load_from_persistence()` 执行。
|
||||
- 语义:仅恢复持久化状态(不在此阶段做状态归并)。
|
||||
### 2.1 加载时恢复
|
||||
- 在 `sync_state_machine/common/collection.py::load_from_persistence()` 执行。
|
||||
- 语义:仅恢复持久化状态,并记录本轮 `sync_log` 的 `persistence` 恢复来源;不在此阶段做状态归并。
|
||||
- 恢复后保留 `data/origin_data`,等待 bootstrap 阶段的 datasource 刷新覆盖最新数据。
|
||||
|
||||
### 2.2 加载后 cleanup(phase2 语义)
|
||||
- pipeline 调用 `BaseSyncStrategy.run_reset()`。
|
||||
- 实际实现:`strategy_ops/reset_ops.py::run_phase2_cleanup()`。
|
||||
### 2.2 bootstrap datasource refresh
|
||||
- pipeline 在持久化恢复后、`E01` 之前,按已恢复到内存的 `node_type` 做一次 datasource 刷新。
|
||||
- 实际实现:`sync_state_machine/pipeline/full_sync_pipeline.py::_bootstrap_refresh_latest_data()`。
|
||||
- 语义:
|
||||
- 如果 datasource 命中节点,则用最新数据覆盖持久化快照,并清除 `_loaded_from_persistence` 标记。
|
||||
- 如果 datasource 未命中节点,则节点保留持久化快照,供 `E01` 判断它是 stale zombie 还是 bootstrap 异常隔离对象。
|
||||
|
||||
### 2.3 加载后 cleanup
|
||||
- pipeline 在 datasource refresh 完成后调用 `run_phase2_cleanup()`。
|
||||
- 实际实现:`sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`。
|
||||
- 核心事件:`E01`(`S16 -> S00/S15`)。
|
||||
- `S16` 表示“持久化加载输入态”:本轮从持久化恢复出的节点集合(可包含上轮遗留节点,也可包含上轮已稳定节点)。
|
||||
- `S16` 表示“bootstrap 输入态”:持久化恢复出的节点,叠加 datasource refresh 后仍待 `E01` 判定的节点集合。
|
||||
- 判定:
|
||||
- `is_create_zombie=true` -> `S15`,节点删除(并尝试解绑);
|
||||
- `is_create_zombie=false` -> `S00`,节点归并为 `UNCHECKED/NONE/PENDING` 并清空 `error`。
|
||||
- `bootstrap_source_present=false && has_binding_record=false` -> `S15`,说明该节点仅残留在 persistence 中且未绑定,按 stale zombie 清理;
|
||||
- `bootstrap_source_present=false && has_binding_record=true` -> `S17`,说明该节点存在绑定,但 datasource refresh 未返回,进入 bootstrap 加载异常隔离;
|
||||
- 其余情况 -> `S00`,节点归并为 `UNCHECKED/NONE/PENDING` 并清空 `error`。
|
||||
|
||||
### 2.2.1 `enable_persistence` 与 bootstrap 关系
|
||||
- `enable_persistence=true`:先 `load_from_persistence()`,再对加载出的节点执行 `E01`(`S16` 域内处理)。
|
||||
- `enable_persistence=true`:先 `load_from_persistence()`,再做 bootstrap datasource refresh,最后对输入节点执行 `E01`(`S16` 域内处理)。
|
||||
- `enable_persistence=false`:本轮没有持久化输入,`S16/E01` 不参与;随后 Stage1 新加载节点直接进入本轮 bind/create/update/sync 主流程。
|
||||
|
||||
### 2.3 加载异常隔离(S17)
|
||||
### 2.4 加载异常隔离(S17)
|
||||
- 持久化中若出现非法枚举值(`action/status/binding_status`),节点标记为 `bootstrap_blocked`(`S17` 语义)。
|
||||
- 持久化恢复后若节点存在绑定,但 bootstrap datasource refresh 未返回该节点,也会进入 `S17` 语义并隔离。
|
||||
- 这类节点保留 `error`,用于人工排查。
|
||||
- 这类节点不会进入 `E01`,也不会参与 bind/create/update/sync 自动流程。
|
||||
|
||||
## 3. bind 流程(strategy_ops)
|
||||
- 入口:`strategy_ops/bind_ops.py::run_bind()`
|
||||
- 入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.bind()`
|
||||
- 入口状态:`S00`
|
||||
|
||||
### bind 编排步骤
|
||||
- strategy 先通过 `bind_ops.collect_bind_entry_nodes()` 收集两侧 `S00` 节点,作为 bind 入口工作集。
|
||||
- `phase_core_state()` 对入口工作集触发 `E10`,把“已有绑定记录”和“无绑定记录”的节点分流到 `S01/S03/S04/S02`。
|
||||
- `phase_dependency_check()` 对仍需继续处理的缺绑节点触发 `E15`,把缺数据、依赖不满足、语义错误的节点拦到 `S03/S05/S04`。
|
||||
- strategy 再通过 `bind_ops.collect_auto_bind_ready_nodes()` 收窄工作集,只保留当前 `S02 + binding_status=MISSING` 的节点进入自动绑定判定。
|
||||
- strategy 在顶层显式构造 orphan create 控制参数,并传给 `bind_ops.plan_auto_bind_updates()` 生成自动绑定/自动创建决策。
|
||||
- `apply_auto_bind_updates()` 对这些决策逐个触发 `E16`,把节点迁移到 `S01/S04/S05/S13`。
|
||||
- `refresh_bind_data_id()` 最后按绑定关系回填两侧节点的 `context.bind_data_id`。
|
||||
|
||||
### 核心绑定判定
|
||||
- `phase_core_state()` 内触发 `E10`。
|
||||
- 有绑定记录:根据本端/对端有效性进入 `S01/S03/S04`。
|
||||
- 无绑定记录:进入 `S02`。
|
||||
|
||||
注意:如果节点在 bootstrap 阶段已被隔离为 `S17` 语义(`bootstrap_blocked=true`),默认不会进入 bind 工作集。
|
||||
|
||||
### 依赖检查
|
||||
- `phase_dependency_check()` -> `check_dependencies_for_nodes()` 内触发 `E15`。
|
||||
- `data_present=true` 且 `dep_satisfied=true` -> 不迁移,节点保持在 `S02`,继续进入自动绑定判定。
|
||||
- `data_present=false` -> `S03`。
|
||||
- `dep_satisfied=false` -> `S05`。
|
||||
- `depend_fields` 为空 -> `S04`(无法自动处理,阻断后续自动创建)。
|
||||
- `depend_fields` 为空本身不会触发阻断;若节点数据存在,则等价于“当前依赖满足”,继续留在 `S02`。
|
||||
|
||||
### 自动绑定
|
||||
- `phase_auto_binding()` 内触发 `E16`。
|
||||
@@ -55,7 +80,7 @@
|
||||
## 4. create/update 准备
|
||||
|
||||
### create
|
||||
- 入口:`strategy_ops/create_ops.py::run_create()`
|
||||
- 入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.create()`
|
||||
- 运行时事件:`E20`(`E21/E22` 为配置保留分支,当前 strategy 未接入自动执行)
|
||||
- 运行时入口状态:`S13`
|
||||
- 说明:`E16` 负责识别“需自动创建”并将源节点置为 `S13`;业务代码执行对侧 spawn+bind 后,再通过 `E20` 将 `S13` 提交到 `S01`(成功);若 spawn 失败,`E20` 不迁移并保留在 `S13` 等待人工/重试。
|
||||
@@ -64,19 +89,19 @@
|
||||
- 新创建目标节点在 `S06/S10C` 阶段允许 `data_id` 为空;`CREATE SUCCESS`(`S11C`)必须携带非空 `result_data_id`,否则事件直接报错。
|
||||
|
||||
### update
|
||||
- 入口:`strategy_ops/update_ops.py::run_update()`
|
||||
- 入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.update()`
|
||||
- 事件:`E30`
|
||||
- 入口状态(source 节点):`S01`
|
||||
- `target_has_data_id=false` 或 `update_id_resolve_ok=false` -> `S08`
|
||||
- `has_diff=true` -> `S07`
|
||||
|
||||
### post-create 归并
|
||||
- 入口:`pipeline/full_sync_pipeline.py::normalize_create_success_to_update_entry()`
|
||||
- 入口:`sync_state_machine/pipeline/full_sync_pipeline.py::normalize_create_success_to_update_entry()`
|
||||
- 事件:`E41`
|
||||
- 语义:`CREATE` 执行成功后,节点先落 `S11C`(SUCCESS),再归并 `S11C -> S01`,作为同轮 update 起点。
|
||||
|
||||
## 5. 执行回写(datasource)
|
||||
- 入口:`datasource/datasource.py`
|
||||
- 入口:`sync_state_machine/datasource/datasource.py`
|
||||
- 事件:`E40C_START/E40C`、`E40U_START/E40U`、`E40D_START/E40D`
|
||||
- 当前约束:delete stage 尚未接入 strategy 主流程;若运行时检测到 `S09/DELETE` 节点,DataSource 会显式抛错阻断(避免散落删除执行)。
|
||||
- 执行准备入口状态:
|
||||
@@ -103,5 +128,5 @@
|
||||
| 隔离(人工) | `S17` | 不参与自动阶段,仅人工处理 |
|
||||
|
||||
## 7. 不变量保障
|
||||
- 每次事件落地后,`engine/events.py::_apply_decision()` 会执行 `runtime.check_invariants()`。
|
||||
- 每次事件落地后,`sync_state_machine/engine/events.py::_apply_decision()` 会执行 `runtime.check_invariants()`。
|
||||
- 违反 invariant 直接抛错,终止当前路径。
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
# 测试指南
|
||||
|
||||
本文档说明三件事:
|
||||
- 单元测试怎么跑。
|
||||
- 集成测试怎么组织。
|
||||
- 测试辅助工具如何使用,尤其是远程 ECM 环境初始化工具。
|
||||
|
||||
## 1. 测试分层
|
||||
|
||||
### 1.1 单元测试
|
||||
|
||||
位置:`tests/unit/`
|
||||
|
||||
目标:
|
||||
- 校验状态机不变量、事件约束、策略辅助逻辑。
|
||||
- 校验 datasource/handler 工具链、配置严格性、导入面稳定性。
|
||||
- 对具体 domain 的 update/create 行为做快速回归。
|
||||
|
||||
单元测试取舍原则:
|
||||
- 优先测文档里承诺的行为语义,不优先测单纯转发、setter、空包装器。
|
||||
- 如果一个测试只是在断言“函数 A 调了函数 B”,但不验证最终行为,通常应该删掉或改写成行为测试。
|
||||
- 如果需要 test double,优先让它承载真实行为断言,不保留仅为凑接口而存在的空壳类。
|
||||
|
||||
常用命令:
|
||||
- 全量单元测试:`pytest tests/unit -q`
|
||||
- 指定文件:`pytest tests/unit/test_update_ops.py -q`
|
||||
|
||||
适用场景:
|
||||
- 修改 `sync_state_machine/` 公共逻辑。
|
||||
- 修改某个 domain 的 handler/strategy。
|
||||
- 校验 schema 相关逻辑没有引入行为回退。
|
||||
|
||||
### 1.2 集成测试
|
||||
|
||||
位置:`tests/integration/`
|
||||
|
||||
目标:
|
||||
- 覆盖 pipeline 级行为,包括 load/bind/create/update/poll/persist。
|
||||
- 校验多节点、多轮次、mock datasource 与 UI smoke 场景。
|
||||
- 验证运行配置、状态机配置和工具链连通性。
|
||||
|
||||
常用命令:
|
||||
- 全量集成测试:`pytest tests/integration -q`
|
||||
- UI 烟测:`pytest tests/integration/test_ui_api_smoke.py -q`
|
||||
- 工具链专项:`pytest tests/integration/test_toolchain_config_and_graph.py -q`
|
||||
|
||||
适用场景:
|
||||
- 修改 pipeline、collection、状态机执行路径。
|
||||
- 修改持久化或跨 domain 依赖行为。
|
||||
- 修改运行配置解析、注册表装配等入口逻辑。
|
||||
|
||||
### 1.3 脚本化大规模回归
|
||||
|
||||
位置:`scripts/auto_test_domains/`
|
||||
|
||||
目标:
|
||||
- 用“多轮数据集 + 报告”的方式覆盖复杂业务 domain。
|
||||
- 适合真实接口联调、回归复现、差异收敛分析。
|
||||
|
||||
特点:
|
||||
- 这类脚本不直接纳入 pytest。
|
||||
- 它们更偏向测试编排与问题定位,而不是轻量级快速回归。
|
||||
|
||||
相关文档:
|
||||
- `docs/auto_test_spec/agent_auto_test_method.md`
|
||||
|
||||
## 2. 远程环境初始化工具
|
||||
|
||||
位置:`tools/remote_env.py`
|
||||
|
||||
这个工具是给真实 ECM 环境联调用的最小控制器,提供 4 个命令:
|
||||
- `clear`:停止服务、恢复 `.env`、清空自动测试专用 CouchDB 数据库。
|
||||
- `init`:停止服务,按 JSONL 数据目录执行 `run.py restore-database` 恢复测试数据库。
|
||||
- `run`:启动远程 ECM 服务,不阻塞当前命令。
|
||||
- `stop`:关闭远程 ECM 服务,并恢复原始 `.env`。
|
||||
|
||||
它不是 pipeline 执行器,只负责测试环境的基础生命周期管理。
|
||||
|
||||
### 2.1 配置项
|
||||
|
||||
示例配置:`tools/remote_env.example.yaml`
|
||||
|
||||
核心配置:
|
||||
- `couchdb.base_url` / `couchdb.database` / `couchdb.username` / `couchdb.password`
|
||||
- `service.ecm_root`
|
||||
- `service.python_path`
|
||||
- `data.data_dir`
|
||||
- `env_file.path`
|
||||
- `env_file.overrides`
|
||||
|
||||
按命令的最低要求:
|
||||
- `run` / `stop`:只要求 `service.*` 可用。
|
||||
- `clear`:要求 `service.*` 和 `couchdb.*` 可用。
|
||||
- `init`:要求 `service.*`、`couchdb.*`、`data.*` 全部可用。
|
||||
|
||||
可选配置:
|
||||
- `service.start_command`
|
||||
- `service.restore_command`
|
||||
- `service.ready_url`
|
||||
- `service.log_file`
|
||||
- `service.env`
|
||||
- `env_file.restore_on_stop`
|
||||
|
||||
支持占位符:
|
||||
- `${PROJECT_ROOT}`:当前仓库根目录。
|
||||
- `${CONFIG_DIR}`:配置文件所在目录。
|
||||
- `${ECM_ROOT}`:`service.ecm_root` 的值。
|
||||
- `${PYTHON}`:`service.python_path` 的值。
|
||||
- `${DATA_DIR}`:`data.data_dir` 的值。
|
||||
- `${COUCHDB_DATABASE}`:`couchdb.database` 的值。
|
||||
- `${COUCHDB_USERNAME}`:`couchdb.username` 的值。
|
||||
- `${COUCHDB_PASSWORD}`:`couchdb.password` 的值。
|
||||
- `${COUCHDB_URL}`:`couchdb.url` 的值。
|
||||
- `${COUCHDB_PORT}`:`couchdb.port` 的值。
|
||||
|
||||
### 2.2 基本用法
|
||||
|
||||
1. 复制示例配置并填入真实路径。
|
||||
2. 先执行 `init` 恢复测试数据库。
|
||||
3. 再执行 `run` 启动远程服务。
|
||||
4. 测试结束后执行 `stop`。
|
||||
|
||||
命令示例:
|
||||
|
||||
```bash
|
||||
python tools/remote_env.py --config tools/remote_env.example.yaml init
|
||||
python tools/remote_env.py --config tools/remote_env.example.yaml run
|
||||
python tools/remote_env.py --config tools/remote_env.example.yaml stop
|
||||
python tools/remote_env.py --config tools/remote_env.example.yaml clear
|
||||
```
|
||||
|
||||
### 2.3 命令行覆盖
|
||||
|
||||
如果只想临时覆盖少数字段,可以直接传命令行参数:
|
||||
|
||||
```bash
|
||||
python tools/remote_env.py \
|
||||
--config tools/remote_env.example.yaml \
|
||||
--db-name test_push_auto_tmp \
|
||||
--ecm-root /data/ecm_app \
|
||||
--python-path /data/venv/bin/python \
|
||||
init
|
||||
```
|
||||
|
||||
支持覆盖的字段:
|
||||
- `--db-name`
|
||||
- `--db-user`
|
||||
- `--db-password`
|
||||
- `--couchdb-base-url`
|
||||
- `--couchdb-url`
|
||||
- `--couchdb-port`
|
||||
- `--ecm-root`
|
||||
- `--python-path`
|
||||
- `--data-dir`
|
||||
- `--ready-url`
|
||||
|
||||
### 2.4 行为约束
|
||||
|
||||
- `clear` 和 `init` 会先执行停服,避免服务仍占用 CouchDB 连接。
|
||||
- `init` 不会自动执行 `run`;数据库恢复和服务启动分开控制。
|
||||
- `run` 如果检测到远端仓库下已有 `run.py` 进程在运行,会直接返回 `already_running`。
|
||||
- `.env` 只覆盖 `env_file.overrides` 中声明的键,未声明的键会原样保留。
|
||||
- 第一次覆盖 `.env` 时会自动备份,`stop` 和 `clear` 默认会恢复原始 `.env`。
|
||||
- 进程状态写到 `runtime/remote_env/state.json`,不需要手工配置 pid 文件。
|
||||
|
||||
## 3. 推荐测试流程
|
||||
|
||||
### 3.1 本地开发回归
|
||||
|
||||
1. 先跑 `pytest tests/unit -q`。
|
||||
2. 如果改动影响 pipeline,再跑目标集成测试。
|
||||
3. 如果改动影响真实接口流程,再使用 `tools/remote_env.py` 管理远程环境。
|
||||
4. 如果修改了配置语义或 pipeline 控制流,优先补一条直接覆盖设计语义的 pytest,而不是只补装配/转发测试。
|
||||
|
||||
### 3.2 真实接口联调
|
||||
|
||||
1. 用 `init` 恢复 ECM 测试库。
|
||||
2. 用 `run` 拉起 ECM 服务。
|
||||
3. 执行同步脚本或联调脚本。
|
||||
4. 查看 `runtime/remote_env/` 下的状态文件与日志。
|
||||
5. 完成后执行 `stop`。
|
||||
|
||||
### 3.3 复杂 domain 回归
|
||||
|
||||
1. 用 `scripts/auto_test_domains/` 中的 case 脚本准备多轮数据。
|
||||
2. 先缩小到单个 domain 复现。
|
||||
3. 确认差异与日志后,再扩大到聚合回归。
|
||||
|
||||
## 4. 相关文件
|
||||
|
||||
- `tests/README.md`
|
||||
- `tools/remote_env.py`
|
||||
- `tools/remote_env.example.yaml`
|
||||
- `docs/operations.md`
|
||||
- `docs/auto_test_spec/agent_auto_test_method.md`
|
||||
@@ -1 +1 @@
|
||||
{"_id": "a1d69bf1-b95e-4832-9258-5bc35c629e9a", "_rev": "2-e265a6cb3674c71d6957883177b270cb", "type": "company", "name": "金上公司", "short_name": "金上公司", "code": "028B", "company_type": "region", "parent_id": "5da30aa7-b560-416e-9223-284290692351", "path": ["5da30aa7-b560-416e-9223-284290692351", "a1d69bf1-b95e-4832-9258-5bc35c629e9a"], "path_name": ["集团公司", "金上公司"], "province": "510000", "city": "510100", "status": "active", "sort": 27, "created_by": "system", "created_by_name": "system", "created_at": "2025-05-30 09:00:09", "updated_by": "system", "updated_by_name": "system", "updated_at": "2025-09-25 20:08:42", "company_nature": ["owner"], "id": "a1d69bf1-b95e-4832-9258-5bc35c629e9a", "credit_code": null, "city_name": "成都市", "province_name": "四川省", "parent_name": "集团公司"}
|
||||
{"_id": "a1d69bf1-b95e-4832-9258-5bc35c629e9a", "_rev": "2-e265a6cb3674c71d6957883177b270cb", "type": "company", "name": "金上公司", "short_name": "金上公司", "code": "028B", "company_type": "region", "parent_id": "5da30aa7-b560-416e-9223-284290692351", "path": ["5da30aa7-b560-416e-9223-284290692351", "a1d69bf1-b95e-4832-9258-5bc35c629e9a"], "path_name": ["集团公司", "金上公司"], "province": "510000", "city": "510100", "status": "active", "sort": 27, "created_by": "system", "created_by_name": "system", "created_at": "2025-05-30 09:00:09", "updated_by": "system", "updated_by_name": "system", "updated_at": "2025-09-25 20:08:42", "company_nature": ["owner"], "id": "a1d69bf1-b95e-4832-9258-5bc35c629e9a", "credit_code": null, "city_name": "成都市", "province_name": "四川省", "parent_name": ""}
|
||||
|
||||
@@ -1,28 +1,28 @@
|
||||
{"_id": "0798431d-115a-4b1c-b95a-78470160fa9e", "_rev": "3-18cce2b2f9854c79aeb4329c31e6028c", "type": "supplier", "name": "沈阳中变电气有限责任公司", "short_name": "沈阳中变电气有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "210000", "city": "210100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91210100564677033W", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "210000", "210100"]}
|
||||
{"_id": "1098875c-e949-47b1-9f40-237eb507f890", "_rev": "3-0ef325ac17b9232224f5c9f924e2cf6f", "type": "supplier", "name": "中国水利水电第九工程局有限公司", "short_name": "中国水利水电第九工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "520000", "city": "520100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91520000214428387H", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "520000", "520100"]}
|
||||
{"_id": "1115a7f3-b2c1-4bfb-8c89-605250c402dd", "_rev": "3-6e0a5e272f35cd473d6cd82a1ad6cfa3", "type": "supplier", "name": "中国安能集团第三工程局有限公司", "short_name": "中国安能集团第三工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915101006331302278", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
|
||||
{"_id": "1ff5b102-2f9e-496c-8591-1a8e32407c7f", "_rev": "3-2857c4610fd9e9a9b8313dee83454969", "type": "supplier", "name": "中国葛洲坝集团三峡建设工程有限公司", "short_name": "中国葛洲坝集团三峡建设工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "420000", "city": "420500", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91420000615573462P", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "420000", "420500"]}
|
||||
{"_id": "4be9e1da-9ebf-4a2a-b6bd-14a41ac31858", "_rev": "5-b54a288a89b9e55cee26d42b08bb69a0", "type": "supplier", "name": "中国水利水电第七工程局有限公司", "short_name": "中国水利水电第七工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510000201827469M", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
|
||||
{"_id": "357c7eaf-5a33-4999-be94-79c42bfbd955", "_rev": "3-35879fa40a8fbe379f03c7a4fa8696ad", "type": "supplier", "name": "广州擎天实业有限公司", "short_name": "广州擎天实业有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "440300", "city": "440100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91440101231240749H", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "440000", "440100"]}
|
||||
{"_id": "2ca09c22-68b4-4e43-b9fa-619dcda5f325", "_rev": "3-ac12e05cedcda85821240a9066aa91ef", "type": "supplier", "name": "江苏苏博特新材料股份有限公司", "short_name": "江苏苏博特新材料股份有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "320000", "city": "320100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91320000768299302G", "created_at": "2025-09-25 20:10:43", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "320000", "320100"]}
|
||||
{"_id": "31b6b6d7-989b-4abe-a7e1-6d518f3a4020", "_rev": "3-bedf089583a4645867c7036dba8e7a9e", "type": "supplier", "name": "中国水利水电第十工程局有限公司", "short_name": "中国水利水电第十工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9151018120276341XK", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
|
||||
{"_id": "34db820f-860c-4c6e-9326-5c7cb385bf50", "_rev": "3-e475374124da6df3a40f38043efb7237", "type": "supplier", "name": "特变电工中发上海高压开关有限公司", "short_name": "特变电工中发上海高压开关有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "310000", "city": "310000", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9131012078784664XA", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "310000", "310000"]}
|
||||
{"_id": "4b069ce4-3687-4263-a1f2-4735c22c3f8e", "_rev": "3-9cccdfa9f992ae4f0ec07068c479487c", "type": "supplier", "name": "四川二滩国际工程咨询有限责任公司", "short_name": "四川二滩国际工程咨询有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510115201873799G", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
|
||||
{"_id": "2e465bd8-b240-4046-bf00-187d22823f7b", "_rev": "3-2629f9e1646b7e398fd8bb41a7fc47a2", "type": "supplier", "name": "哈尔滨电机厂有限责任公司", "short_name": "哈尔滨电机厂有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "230000", "city": "230100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "912301991270479655", "created_at": "2025-09-26 17:00:05", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "230000", "230100"]}
|
||||
{"_id": "3ab47229-6da8-4efb-aa44-b96866a2ce13", "_rev": "3-998c447f59be1475e690d1416384fb64", "type": "supplier", "name": "四川蜀府贸易有限公司", "short_name": "四川蜀府贸易有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915100005632853965", "created_at": "2025-09-26 17:00:10", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
|
||||
{"_id": "48a5f1a7-5d5e-41cf-8676-042f1d5316f1", "_rev": "2-d43e58afff0c5317401a22d075866c56", "field_metadata": null, "created_by": "a775915a-0bed-4a17-94f0-47b29916be90", "created_by_name": "集团公司管理员", "created_at": "2025-11-27 11:32:41", "updated_by": "a775915a-0bed-4a17-94f0-47b29916be90", "updated_by_name": "集团公司管理员", "updated_at": "2025-12-25 08:59:53", "deleted_at": null, "deleted_by": null, "deleted_by_name": null, "type": "supplier", "code": "czyy", "company_type": "other", "name": "常州液压成套设备厂有限公司", "short_name": "常州液压", "parent_id": null, "path": [], "path_name": [], "province": "320000", "city": "320400", "sort": 0, "status": "active", "company_nature": ["supplier"], "credit_code": "91320412250851852D", "country": "100000", "region_path": ["100000", "320000", "320400"]}
|
||||
{"_id": "67fa779d-2ce9-468f-b94f-a3d29f1f22c9", "_rev": "3-2c6e66d316216cfc460662bf4a40e612", "type": "supplier", "name": "杭州华新机电工程有限公司", "short_name": "杭州华新机电工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "330000", "city": "330100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "913301087227885306", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "330000", "330100"]}
|
||||
{"_id": "594a09af-8944-476d-b098-7aebe96ad4e8", "_rev": "3-fc6edc3d3384c95def0be02ee734a5b2", "type": "supplier", "name": "云南水电十四局东华建筑工程有限公司", "short_name": "云南水电十四局东华建筑工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "530000", "city": "530100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91530000713408946K", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "530000", "530100"]}
|
||||
{"_id": "6db1ed5b-f52a-4142-867c-d8b2731ead71", "_rev": "3-f769878777781eb98071dddc7c7c4968", "type": "supplier", "name": "中国水利水电第十四工程局有限公司", "short_name": "中国水利水电第十四工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "530000", "city": "530100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91530100216579074C", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "530000", "530100"]}
|
||||
{"_id": "538e01cd-85cb-42b3-b313-16977f7a5224", "_rev": "3-d9acfbf91ba6d83dc33c94b84127f2e3", "type": "supplier", "name": "中国水利水电第五工程局有限公司", "short_name": "中国水利水电第五工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510000205804264E", "created_at": "2025-09-26 17:00:12", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
|
||||
{"_id": "97be34dc-0a17-4ce5-baea-0f3ceee61103", "_rev": "4-4fbf33c81fff4a4a93ba8da7f99b786d", "type": "supplier", "name": "国电南京自动化股份有限公司", "short_name": "国电南京自动化股份有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "320000", "city": "320100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "913201007162522468", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "320000", "320100"]}
|
||||
{"_id": "77316fb4-6167-432b-94e5-89c3d8a6b900", "_rev": "3-2112fbd151033d97a2c90123a02c70a3", "type": "supplier", "name": "中国水利水电夹江水工机械有限公司", "short_name": "中国水利水电夹江水工机械有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "511100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91511126207500117U", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "511100"]}
|
||||
{"_id": "936c6316-8d87-4681-a3b5-b90ed4804a06", "_rev": "3-a9a9b85f9cdd45dc3646e66c9a1a41aa", "type": "supplier", "name": "中国电建集团成都勘测设计研究院有限公司", "short_name": "中国电建集团成都勘测设计研究院有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915100004507513971", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
|
||||
{"_id": "98892419-6a71-4d7e-a0ac-ad5a0a15f1d6", "_rev": "3-f5bb2f9abe284037054df3b323f2482f", "type": "supplier", "name": "中国电建集团贵阳勘测设计研究院有限公司", "short_name": "中国电建集团贵阳勘测设计研究院有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "520000", "city": "520100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9152000070966703X2", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "520000", "520100"]}
|
||||
{"_id": "9c80aace-ba73-4550-8fef-3080b51027d9", "_rev": "3-0dddd7f05fccce78e17fdf4ae9975f0a", "type": "supplier", "name": "中国水利水电第四工程局有限公司", "short_name": "中国水利水电第四工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "630000", "city": "630100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9163000022658124XK", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "630000", "630100"]}
|
||||
{"_id": "930dfaf9-708e-4ec0-95f4-521b10bf1c00", "_rev": "2-5c464399a48505836497072601a60159", "field_metadata": null, "created_by": "a775915a-0bed-4a17-94f0-47b29916be90", "created_by_name": "集团公司管理员", "created_at": "2025-11-27 11:33:18", "updated_by": "a775915a-0bed-4a17-94f0-47b29916be90", "updated_by_name": "集团公司管理员", "updated_at": "2025-12-25 08:59:53", "deleted_at": null, "deleted_by": null, "deleted_by_name": null, "type": "supplier", "code": "hnwz", "company_type": "other", "name": "某公司海南物资有限公司", "short_name": "海南物资", "parent_id": null, "path": [], "path_name": [], "province": "460000", "city": "460100", "sort": 0, "status": "active", "company_nature": ["supplier"], "credit_code": "91469007MA5TW", "country": "100000", "region_path": ["100000", "460000", "460100"]}
|
||||
{"_id": "ab5ab605-f27a-461b-9a88-73d1eafde5ed", "_rev": "3-a463c974fbcb84a524d1ee6ed72e64ec", "type": "supplier", "name": "东方电气集团东方电机有限公司", "short_name": "东方电气集团东方电机有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510600", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510600671415848H", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "510000", "510600"]}
|
||||
{"_id": "b4656ed6-5329-4673-93c8-d11e15d6ece5", "_rev": "3-0a11b989849efb63378b8a8fddb989e4", "type": "supplier", "name": "中国水利水电建设工程咨询西北有限公司", "short_name": "中国水利水电建设工程咨询西北有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "610000", "city": "610100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91610000220530812K", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "610000", "610100"]}
|
||||
{"_id": "e0ceec90-0ad7-417f-a669-fd4f5db07eff", "_rev": "4-71df8660a58c5628bb55f2d39a9264fb", "type": "supplier", "name": "西安西电变压器有限责任公司", "short_name": "西安西电变压器有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "610000", "city": "610100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91610104220601553B", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "610000", "610100"]}
|
||||
{"_id": "cd9e9224-a08f-4bed-81a6-b289b74506e8", "_rev": "3-572570411c03424fb3370e74b2f7f496", "type": "supplier", "name": "乐山一拉得电网自动化有限公司", "short_name": "乐山一拉得电网自动化有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "511100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91511100MA6281NC0N", "created_at": "2025-09-25 20:10:43", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "510000", "511100"]}
|
||||
{"_id": "e88c8c18-bc49-4bab-acc5-6fb78f4c7f8f", "_rev": "3-269fcaf6630593cb44ee3e6a1ffd4ef9", "type": "supplier", "name": "中国水利水电第六工程局有限公司", "short_name": "中国水利水电第六工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "210000", "city": "210100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9121011211756300XA", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "210000", "210100"]}
|
||||
{"_id": "0798431d-115a-4b1c-b95a-78470160fa9e", "_rev": "3-18cce2b2f9854c79aeb4329c31e6028c", "type": "supplier", "name": "沈阳中变电气有限责任公司", "short_name": "沈阳中变电气有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "210000", "city": "210100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91210100564677033W", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "210000", "210100"], "id": "0798431d-115a-4b1c-b95a-78470160fa9e", "city_name": "沈阳市", "province_name": "辽宁省", "country_name": "中国"}
|
||||
{"_id": "1098875c-e949-47b1-9f40-237eb507f890", "_rev": "3-0ef325ac17b9232224f5c9f924e2cf6f", "type": "supplier", "name": "中国水利水电第九工程局有限公司", "short_name": "中国水利水电第九工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "520000", "city": "520100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91520000214428387H", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "520000", "520100"], "id": "1098875c-e949-47b1-9f40-237eb507f890", "city_name": "贵阳市", "province_name": "贵州省", "country_name": "中国"}
|
||||
{"_id": "1115a7f3-b2c1-4bfb-8c89-605250c402dd", "_rev": "3-6e0a5e272f35cd473d6cd82a1ad6cfa3", "type": "supplier", "name": "中国安能集团第三工程局有限公司", "short_name": "中国安能集团第三工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915101006331302278", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "1115a7f3-b2c1-4bfb-8c89-605250c402dd", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
|
||||
{"_id": "1ff5b102-2f9e-496c-8591-1a8e32407c7f", "_rev": "3-2857c4610fd9e9a9b8313dee83454969", "type": "supplier", "name": "中国葛洲坝集团三峡建设工程有限公司", "short_name": "中国葛洲坝集团三峡建设工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "420000", "city": "420500", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91420000615573462P", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "420000", "420500"], "id": "1ff5b102-2f9e-496c-8591-1a8e32407c7f", "city_name": "宜昌市", "province_name": "湖北省", "country_name": "中国"}
|
||||
{"_id": "4be9e1da-9ebf-4a2a-b6bd-14a41ac31858", "_rev": "5-b54a288a89b9e55cee26d42b08bb69a0", "type": "supplier", "name": "中国水利水电第七工程局有限公司", "short_name": "中国水利水电第七工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510000201827469M", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "4be9e1da-9ebf-4a2a-b6bd-14a41ac31858", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
|
||||
{"_id": "357c7eaf-5a33-4999-be94-79c42bfbd955", "_rev": "3-35879fa40a8fbe379f03c7a4fa8696ad", "type": "supplier", "name": "广州擎天实业有限公司", "short_name": "广州擎天实业有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "440300", "city": "440100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91440101231240749H", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "440000", "440100"], "id": "357c7eaf-5a33-4999-be94-79c42bfbd955", "city_name": "广州市", "province_name": "广东省", "country_name": "中国"}
|
||||
{"_id": "2ca09c22-68b4-4e43-b9fa-619dcda5f325", "_rev": "3-ac12e05cedcda85821240a9066aa91ef", "type": "supplier", "name": "江苏苏博特新材料股份有限公司", "short_name": "江苏苏博特新材料股份有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "320000", "city": "320100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91320000768299302G", "created_at": "2025-09-25 20:10:43", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "320000", "320100"], "id": "2ca09c22-68b4-4e43-b9fa-619dcda5f325", "city_name": "南京市", "province_name": "江苏省", "country_name": "中国"}
|
||||
{"_id": "31b6b6d7-989b-4abe-a7e1-6d518f3a4020", "_rev": "3-bedf089583a4645867c7036dba8e7a9e", "type": "supplier", "name": "中国水利水电第十工程局有限公司", "short_name": "中国水利水电第十工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9151018120276341XK", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "31b6b6d7-989b-4abe-a7e1-6d518f3a4020", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
|
||||
{"_id": "34db820f-860c-4c6e-9326-5c7cb385bf50", "_rev": "3-e475374124da6df3a40f38043efb7237", "type": "supplier", "name": "特变电工中发上海高压开关有限公司", "short_name": "特变电工中发上海高压开关有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "310000", "city": "310000", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9131012078784664XA", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "310000", "310000"], "id": "34db820f-860c-4c6e-9326-5c7cb385bf50", "city_name": "直辖", "province_name": "上海市", "country_name": "中国"}
|
||||
{"_id": "4b069ce4-3687-4263-a1f2-4735c22c3f8e", "_rev": "3-9cccdfa9f992ae4f0ec07068c479487c", "type": "supplier", "name": "四川二滩国际工程咨询有限责任公司", "short_name": "四川二滩国际工程咨询有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510115201873799G", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "4b069ce4-3687-4263-a1f2-4735c22c3f8e", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
|
||||
{"_id": "2e465bd8-b240-4046-bf00-187d22823f7b", "_rev": "3-2629f9e1646b7e398fd8bb41a7fc47a2", "type": "supplier", "name": "哈尔滨电机厂有限责任公司", "short_name": "哈尔滨电机厂有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "230000", "city": "230100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "912301991270479655", "created_at": "2025-09-26 17:00:05", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "230000", "230100"], "id": "2e465bd8-b240-4046-bf00-187d22823f7b", "city_name": "哈尔滨市", "province_name": "黑龙江省", "country_name": "中国"}
|
||||
{"_id": "3ab47229-6da8-4efb-aa44-b96866a2ce13", "_rev": "3-998c447f59be1475e690d1416384fb64", "type": "supplier", "name": "四川蜀府贸易有限公司", "short_name": "四川蜀府贸易有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915100005632853965", "created_at": "2025-09-26 17:00:10", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "3ab47229-6da8-4efb-aa44-b96866a2ce13", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
|
||||
{"_id": "48a5f1a7-5d5e-41cf-8676-042f1d5316f1", "_rev": "2-d43e58afff0c5317401a22d075866c56", "field_metadata": null, "created_by": "a775915a-0bed-4a17-94f0-47b29916be90", "created_by_name": "集团公司管理员", "created_at": "2025-11-27 11:32:41", "updated_by": "a775915a-0bed-4a17-94f0-47b29916be90", "updated_by_name": "集团公司管理员", "updated_at": "2025-12-25 08:59:53", "deleted_at": null, "deleted_by": null, "deleted_by_name": null, "type": "supplier", "code": "czyy", "company_type": "other", "name": "常州液压成套设备厂有限公司", "short_name": "常州液压", "parent_id": null, "path": [], "path_name": [], "province": "320000", "city": "320400", "sort": 0, "status": "active", "company_nature": ["supplier"], "credit_code": "91320412250851852D", "country": "100000", "region_path": ["100000", "320000", "320400"], "id": "48a5f1a7-5d5e-41cf-8676-042f1d5316f1", "city_name": "常州市", "province_name": "江苏省", "country_name": "中国"}
|
||||
{"_id": "67fa779d-2ce9-468f-b94f-a3d29f1f22c9", "_rev": "3-2c6e66d316216cfc460662bf4a40e612", "type": "supplier", "name": "杭州华新机电工程有限公司", "short_name": "杭州华新机电工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "330000", "city": "330100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "913301087227885306", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "330000", "330100"], "id": "67fa779d-2ce9-468f-b94f-a3d29f1f22c9", "city_name": "杭州市", "province_name": "浙江省", "country_name": "中国"}
|
||||
{"_id": "594a09af-8944-476d-b098-7aebe96ad4e8", "_rev": "3-fc6edc3d3384c95def0be02ee734a5b2", "type": "supplier", "name": "云南水电十四局东华建筑工程有限公司", "short_name": "云南水电十四局东华建筑工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "530000", "city": "530100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91530000713408946K", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "530000", "530100"], "id": "594a09af-8944-476d-b098-7aebe96ad4e8", "city_name": "昆明市", "province_name": "云南省", "country_name": "中国"}
|
||||
{"_id": "6db1ed5b-f52a-4142-867c-d8b2731ead71", "_rev": "3-f769878777781eb98071dddc7c7c4968", "type": "supplier", "name": "中国水利水电第十四工程局有限公司", "short_name": "中国水利水电第十四工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "530000", "city": "530100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91530100216579074C", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "530000", "530100"], "id": "6db1ed5b-f52a-4142-867c-d8b2731ead71", "city_name": "昆明市", "province_name": "云南省", "country_name": "中国"}
|
||||
{"_id": "538e01cd-85cb-42b3-b313-16977f7a5224", "_rev": "3-d9acfbf91ba6d83dc33c94b84127f2e3", "type": "supplier", "name": "中国水利水电第五工程局有限公司", "short_name": "中国水利水电第五工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510000205804264E", "created_at": "2025-09-26 17:00:12", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "538e01cd-85cb-42b3-b313-16977f7a5224", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
|
||||
{"_id": "97be34dc-0a17-4ce5-baea-0f3ceee61103", "_rev": "4-4fbf33c81fff4a4a93ba8da7f99b786d", "type": "supplier", "name": "国电南京自动化股份有限公司", "short_name": "国电南京自动化股份有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "320000", "city": "320100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "913201007162522468", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "320000", "320100"], "id": "97be34dc-0a17-4ce5-baea-0f3ceee61103", "city_name": "南京市", "province_name": "江苏省", "country_name": "中国"}
|
||||
{"_id": "77316fb4-6167-432b-94e5-89c3d8a6b900", "_rev": "3-2112fbd151033d97a2c90123a02c70a3", "type": "supplier", "name": "中国水利水电夹江水工机械有限公司", "short_name": "中国水利水电夹江水工机械有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "511100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91511126207500117U", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "511100"], "id": "77316fb4-6167-432b-94e5-89c3d8a6b900", "city_name": "乐山市", "province_name": "四川省", "country_name": "中国"}
|
||||
{"_id": "936c6316-8d87-4681-a3b5-b90ed4804a06", "_rev": "3-a9a9b85f9cdd45dc3646e66c9a1a41aa", "type": "supplier", "name": "中国电建集团成都勘测设计研究院有限公司", "short_name": "中国电建集团成都勘测设计研究院有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915100004507513971", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "936c6316-8d87-4681-a3b5-b90ed4804a06", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
|
||||
{"_id": "98892419-6a71-4d7e-a0ac-ad5a0a15f1d6", "_rev": "3-f5bb2f9abe284037054df3b323f2482f", "type": "supplier", "name": "中国电建集团贵阳勘测设计研究院有限公司", "short_name": "中国电建集团贵阳勘测设计研究院有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "520000", "city": "520100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9152000070966703X2", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "520000", "520100"], "id": "98892419-6a71-4d7e-a0ac-ad5a0a15f1d6", "city_name": "贵阳市", "province_name": "贵州省", "country_name": "中国"}
|
||||
{"_id": "9c80aace-ba73-4550-8fef-3080b51027d9", "_rev": "3-0dddd7f05fccce78e17fdf4ae9975f0a", "type": "supplier", "name": "中国水利水电第四工程局有限公司", "short_name": "中国水利水电第四工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "630000", "city": "630100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9163000022658124XK", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "630000", "630100"], "id": "9c80aace-ba73-4550-8fef-3080b51027d9", "city_name": "西宁市", "province_name": "青海省", "country_name": "中国"}
|
||||
{"_id": "930dfaf9-708e-4ec0-95f4-521b10bf1c00", "_rev": "2-5c464399a48505836497072601a60159", "field_metadata": null, "created_by": "a775915a-0bed-4a17-94f0-47b29916be90", "created_by_name": "集团公司管理员", "created_at": "2025-11-27 11:33:18", "updated_by": "a775915a-0bed-4a17-94f0-47b29916be90", "updated_by_name": "集团公司管理员", "updated_at": "2025-12-25 08:59:53", "deleted_at": null, "deleted_by": null, "deleted_by_name": null, "type": "supplier", "code": "hnwz", "company_type": "other", "name": "华电海南物资有限公司", "short_name": "海南物资", "parent_id": null, "path": [], "path_name": [], "province": "460000", "city": "460100", "sort": 0, "status": "active", "company_nature": ["supplier"], "credit_code": "91469007MA5TW", "country": "100000", "region_path": ["100000", "460000", "460100"], "id": "930dfaf9-708e-4ec0-95f4-521b10bf1c00", "city_name": "海口市", "province_name": "海南省", "country_name": "中国"}
|
||||
{"_id": "ab5ab605-f27a-461b-9a88-73d1eafde5ed", "_rev": "3-a463c974fbcb84a524d1ee6ed72e64ec", "type": "supplier", "name": "东方电气集团东方电机有限公司", "short_name": "东方电气集团东方电机有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510600", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510600671415848H", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "510000", "510600"], "id": "ab5ab605-f27a-461b-9a88-73d1eafde5ed", "city_name": "德阳市", "province_name": "四川省", "country_name": "中国"}
|
||||
{"_id": "b4656ed6-5329-4673-93c8-d11e15d6ece5", "_rev": "3-0a11b989849efb63378b8a8fddb989e4", "type": "supplier", "name": "中国水利水电建设工程咨询西北有限公司", "short_name": "中国水利水电建设工程咨询西北有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "610000", "city": "610100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91610000220530812K", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "610000", "610100"], "id": "b4656ed6-5329-4673-93c8-d11e15d6ece5", "city_name": "西安市", "province_name": "陕西省", "country_name": "中国"}
|
||||
{"_id": "e0ceec90-0ad7-417f-a669-fd4f5db07eff", "_rev": "4-71df8660a58c5628bb55f2d39a9264fb", "type": "supplier", "name": "西安西电变压器有限责任公司", "short_name": "西安西电变压器有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "610000", "city": "610100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91610104220601553B", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "610000", "610100"], "id": "e0ceec90-0ad7-417f-a669-fd4f5db07eff", "city_name": "西安市", "province_name": "陕西省", "country_name": "中国"}
|
||||
{"_id": "cd9e9224-a08f-4bed-81a6-b289b74506e8", "_rev": "3-572570411c03424fb3370e74b2f7f496", "type": "supplier", "name": "乐山一拉得电网自动化有限公司", "short_name": "乐山一拉得电网自动化有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "511100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91511100MA6281NC0N", "created_at": "2025-09-25 20:10:43", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "510000", "511100"], "id": "cd9e9224-a08f-4bed-81a6-b289b74506e8", "city_name": "乐山市", "province_name": "四川省", "country_name": "中国"}
|
||||
{"_id": "e88c8c18-bc49-4bab-acc5-6fb78f4c7f8f", "_rev": "3-269fcaf6630593cb44ee3e6a1ffd4ef9", "type": "supplier", "name": "中国水利水电第六工程局有限公司", "short_name": "中国水利水电第六工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "210000", "city": "210100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9121011211756300XA", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "210000", "210100"], "id": "e88c8c18-bc49-4bab-acc5-6fb78f4c7f8f", "city_name": "沈阳市", "province_name": "辽宁省", "country_name": "中国"}
|
||||
|
||||
@@ -10,6 +10,8 @@ dependencies = [
|
||||
"PyYAML>=6.0,<7",
|
||||
"httpx>=0.27,<1",
|
||||
"aiosqlite>=0.20,<1",
|
||||
"SQLAlchemy>=2.0,<3",
|
||||
"greenlet>=3,<4",
|
||||
"pytz>=2024.1,<2027",
|
||||
"fastapi>=0.115,<1",
|
||||
"uvicorn>=0.30,<1",
|
||||
|
||||
@@ -10,4 +10,3 @@ filterwarnings =
|
||||
ignore:'asyncio.set_event_loop_policy' is deprecated and slated for removal in Python 3.16:DeprecationWarning:pytest_asyncio\.plugin
|
||||
# 也可以设置异步测试的默认 loop scope
|
||||
asyncio_default_fixture_loop_scope = function
|
||||
addopts = --ignore=scripts/test_api_sync.py --ignore=scripts/test_api_sync_top10.py --ignore=scripts/test_api_write_real_sample.py
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
@@ -10,54 +7,7 @@ PROJECT_ROOT = Path(__file__).resolve().parent
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from sync_state_machine.config import load_overrides_from_file
|
||||
from sync_state_machine.pipeline import build_config, run_pipeline_from_config
|
||||
|
||||
|
||||
def _resolve_config_path(raw_path: str) -> Path:
|
||||
path = Path(raw_path)
|
||||
if not path.is_absolute():
|
||||
path = PROJECT_ROOT / path
|
||||
return path.resolve()
|
||||
|
||||
|
||||
async def _main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Run sync pipeline from config override file")
|
||||
parser.add_argument(
|
||||
"--config_path",
|
||||
required=True,
|
||||
help="Path to run profile file (JSON/YAML, supports relative path, e.g. run_profiles/jsonl_to_jsonl.local.json)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
cfg_path = _resolve_config_path(args.config_path)
|
||||
if not cfg_path.exists() or not cfg_path.is_file():
|
||||
raise FileNotFoundError(f"Config file not found: {cfg_path}")
|
||||
|
||||
overrides = load_overrides_from_file(cfg_path, PROJECT_ROOT)
|
||||
config = build_config(project_root=PROJECT_ROOT, overrides=overrides)
|
||||
|
||||
mode = f"{config.local_datasource.type}->{config.remote_datasource.type}"
|
||||
print(f"🧩 Loaded config file: {cfg_path}")
|
||||
await run_pipeline_from_config(
|
||||
config,
|
||||
title=f"sync_state_machine pipeline ({mode})",
|
||||
print_summary=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
return asyncio.run(_main())
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted by user")
|
||||
return 130
|
||||
except Exception as exc:
|
||||
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
|
||||
return 1
|
||||
finally:
|
||||
logging.shutdown()
|
||||
from sync_state_machine.cli import main
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+19
-3
@@ -3,19 +3,35 @@
|
||||
- 路径:项目根目录 `run_profiles/`
|
||||
- 命名:建议使用 `*.local.yaml`(该目录下 YAML 默认会被 git 忽略)
|
||||
|
||||
当前约定:
|
||||
- 仓库里跟踪的配置文件只写必要字段,不重复抄默认值。
|
||||
- pipeline 启动时会打印 **Resolved Full Config**,把省略字段补齐。
|
||||
- 如果你要看完整字段和字段含义,不看模板,直接看 `docs/runtime_config_guide.md`。
|
||||
|
||||
模板来源:
|
||||
- `run_profiles/preset/jsonl_to_jsonl.default.yaml`
|
||||
- `run_profiles/preset/jsonl_to_api.default.yaml`
|
||||
- `run_profiles/preset/jsonl_to_jsonl.multi_project.default.yaml`
|
||||
|
||||
推荐流程:
|
||||
1. 从模板复制到本目录,例如:
|
||||
- `run_profiles/jsonl_to_jsonl.local.yaml`
|
||||
- `run_profiles/jsonl_to_api.local.yaml`
|
||||
- `run_profiles/jsonl_to_jsonl.local.yaml`
|
||||
- `run_profiles/jsonl_to_api.local.yaml`
|
||||
2. 按需修改后运行:
|
||||
- `ecm-sync-run --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
|
||||
- `ecm-sync-run --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
|
||||
|
||||
兼容旧入口:
|
||||
- `python run.py --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
|
||||
|
||||
多项目示例:
|
||||
- `run_profiles/jsonl_to_jsonl.multi_project.yaml`
|
||||
- `run_profiles/jsonl_to_jsonl.multi_project.global.yaml`
|
||||
- `run_profiles/jsonl_to_jsonl.multi_project.project.yaml`
|
||||
|
||||
支持占位符:
|
||||
- `${PROJECT_ROOT}`:运行时替换为项目根目录绝对路径。
|
||||
|
||||
现成示例:
|
||||
- `run_profiles/jsonl_to_api.yaml`:单项目 JSONL -> API 联调示例
|
||||
- `run_profiles/jsonl_to_api.test.yaml`:auto-test 使用的 JSONL -> API 示例
|
||||
- `run_profiles/jsonl_to_jsonl.multi_project.yaml`:多项目 JSONL -> JSONL 编排入口
|
||||
|
||||
@@ -2,20 +2,35 @@
|
||||
|
||||
该目录存放版本管理的运行配置模板(default)。
|
||||
|
||||
模板约定:
|
||||
- 这里只保留必要覆盖项。
|
||||
- 省略的字段会在加载时从默认 datasource 配置和各 domain 的 `default_config` 自动补齐。
|
||||
- 真正生效的全量配置,以启动日志里的 **Resolved Full Config** 为准。
|
||||
|
||||
当前模板:
|
||||
- `jsonl_to_jsonl.default.yaml`
|
||||
- `jsonl_to_api.default.yaml`
|
||||
- `jsonl_to_jsonl.multi_project.default.yaml`
|
||||
- `jsonl_to_jsonl.multi_project.global.default.yaml`
|
||||
- `jsonl_to_jsonl.multi_project.project.default.yaml`
|
||||
|
||||
使用方式:
|
||||
1. 复制模板到项目根目录 `run_profiles/`,并重命名为你自己的文件(如 `*.local.yaml`)。
|
||||
2. 修改后通过入口运行:
|
||||
2. 只改和当前环境/业务差异相关的字段。
|
||||
3. 修改后通过入口运行:
|
||||
- `ecm-sync-run --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
|
||||
|
||||
兼容旧入口:
|
||||
- `python run.py --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
|
||||
|
||||
多项目配置说明:
|
||||
- `jsonl_to_jsonl.multi_project.default.yaml` 是多 pipeline 编排入口。
|
||||
- `jsonl_to_jsonl.multi_project.global.default.yaml` 定义全局公共 domain 的 pipeline。
|
||||
- `jsonl_to_jsonl.multi_project.project.default.yaml` 定义单项目 pipeline 默认配置。
|
||||
- `project_bootstrap_node_types` 用于显式声明项目级 pipeline 需要从 global scope 继承到项目 scope 的公共节点类型,例如 `company`、`supplier`。
|
||||
|
||||
说明:
|
||||
- `run_profiles/*.yaml` / `run_profiles/*.yml` 默认被 `.gitignore` 忽略,用于本地私有配置。
|
||||
- 模板内可使用 `${PROJECT_ROOT}` 占位符。
|
||||
- 模板已显式展开 datasource 与 strategies 关键字段,便于集中审查。
|
||||
- 完整字段定义与字段含义见 `docs/runtime_config_guide.md`。
|
||||
- API 数据源支持全局限流配置:`api_rate_limit_max_requests` 与 `api_rate_limit_window_seconds`。
|
||||
@@ -1,31 +1,33 @@
|
||||
node_types:
|
||||
- company
|
||||
- supplier
|
||||
- project
|
||||
- contract
|
||||
- contract_change
|
||||
- construction_task
|
||||
- construction_task_detail
|
||||
- material
|
||||
- material_detail
|
||||
- contract_settlement
|
||||
- contract_settlement_detail
|
||||
- personnel
|
||||
- personnel_detail
|
||||
- preparation
|
||||
- production
|
||||
- lar
|
||||
- lar_change
|
||||
- project_detail_data
|
||||
- units
|
||||
- company
|
||||
- supplier
|
||||
- project
|
||||
- contract
|
||||
- contract_change
|
||||
- construction_task
|
||||
- construction_task_detail
|
||||
- material
|
||||
- material_detail
|
||||
- contract_settlement
|
||||
- contract_settlement_detail
|
||||
- personnel
|
||||
- personnel_detail
|
||||
- preparation
|
||||
- production
|
||||
- lar
|
||||
- lar_change
|
||||
- project_detail_data
|
||||
- units
|
||||
|
||||
local_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
|
||||
read_only: false
|
||||
target_project_ids:
|
||||
- "0f9c3e22-f4bb-4803-825a-4f238eeb06d5"
|
||||
- "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
|
||||
handler_configs: {}
|
||||
handler_configs:
|
||||
project:
|
||||
data_id_filter:
|
||||
- "0f9c3e22-f4bb-4803-825a-4f238eeb06d5"
|
||||
- "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
|
||||
|
||||
remote_datasource:
|
||||
type: api
|
||||
api_base_url: http://localhost:8000/api/third/v2
|
||||
@@ -35,256 +37,34 @@ remote_datasource:
|
||||
api_rate_limit_max_requests: 30
|
||||
api_rate_limit_window_seconds: 10.0
|
||||
poll_max_retries: 1
|
||||
poll_interval: 0.5
|
||||
target_project_ids:
|
||||
# 请替换为实际的项目ID,至少需要一个
|
||||
- "0f9c3e22-f4bb-4803-825a-4f238eeb06d5"
|
||||
- "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
|
||||
handler_configs:
|
||||
supplier: {}
|
||||
project:
|
||||
data_id_filter:
|
||||
- "0f9c3e22-f4bb-4803-825a-4f238eeb06d5"
|
||||
- "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
|
||||
|
||||
persist:
|
||||
backend: sqlite
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/simple_pipeline_sm.db
|
||||
enable: true
|
||||
wipe_on_start: false
|
||||
|
||||
logging:
|
||||
initialize: true
|
||||
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
|
||||
file_name_pattern: simple_pipeline_sm_{timestamp}.log
|
||||
level: 20
|
||||
|
||||
strategies:
|
||||
company:
|
||||
skip_sync: false
|
||||
preset: pull_only
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- credit_code
|
||||
depend_fields: {}
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: create_local
|
||||
update_direction: pull
|
||||
- credit_code
|
||||
|
||||
supplier:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- credit_code
|
||||
depend_fields: {}
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: create_local
|
||||
update_direction: pull
|
||||
preset: pull_only
|
||||
|
||||
project:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: false
|
||||
auto_bind_fields: []
|
||||
pre_bind_data_id: []
|
||||
depend_fields:
|
||||
company_id: company
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
contract:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- company_id
|
||||
- code
|
||||
depend_fields:
|
||||
project_id: project
|
||||
company_id: supplier
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
contract_change:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- contract_id
|
||||
- change_time
|
||||
- title
|
||||
depend_fields:
|
||||
project_id: project
|
||||
contract_id: contract
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
construction_task:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- contract_id
|
||||
- task_type
|
||||
depend_fields:
|
||||
contract_id: contract
|
||||
project_id: project
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
construction_task_detail:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- parent_id
|
||||
- date
|
||||
depend_fields:
|
||||
parent_id: construction_task
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
material:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- contract_id
|
||||
- material_type
|
||||
depend_fields:
|
||||
contract_id: contract
|
||||
project_id: project
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
material_detail:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- parent_id
|
||||
- date
|
||||
depend_fields:
|
||||
parent_id: material
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
contract_settlement:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- contract_id
|
||||
depend_fields:
|
||||
contract_id: contract
|
||||
project_id: project
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
contract_settlement_detail:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- parent_id
|
||||
- date
|
||||
depend_fields:
|
||||
parent_id: contract_settlement
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
personnel:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- code
|
||||
depend_fields:
|
||||
project_id: project
|
||||
contract_id: contract
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
personnel_detail:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- parent_id
|
||||
- date
|
||||
depend_fields:
|
||||
parent_id: personnel
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
preparation:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- code
|
||||
depend_fields:
|
||||
project_id: project
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
|
||||
production:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- code
|
||||
depend_fields:
|
||||
project_id: project
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
lar:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
depend_fields:
|
||||
project_id: project
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
lar_change:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- change_time
|
||||
- title
|
||||
depend_fields:
|
||||
project_id: project
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
project_detail_data:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- key
|
||||
depend_fields:
|
||||
project_id: project
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push # 未在 field_direction_overrides 中列出的 key 默认 push
|
||||
field_direction_key: key # 以节点 data.key 字段值作为方向查表键
|
||||
field_direction_overrides:
|
||||
ensure_production: pull # 远程 → 本地
|
||||
climb_production: pull # 远程 → 本地
|
||||
challenge_production: pull # 远程 → 本地
|
||||
units:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- serial_number
|
||||
depend_fields:
|
||||
project_id: project
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
post_check_ignore_fields:
|
||||
- is_exist
|
||||
|
||||
@@ -22,263 +22,46 @@ node_types:
|
||||
local_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
|
||||
read_only: false
|
||||
target_project_ids: []
|
||||
handler_configs: {}
|
||||
remote_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: ${PROJECT_ROOT}/remote_datasource/datasource
|
||||
read_only: false
|
||||
target_project_ids: []
|
||||
handler_configs: {}
|
||||
persist:
|
||||
backend: sqlite
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/full_sync_pipeline_jsonl_sm.db
|
||||
enable: true
|
||||
wipe_on_start: false
|
||||
logging:
|
||||
initialize: true
|
||||
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
|
||||
file_name_pattern: jsonl_pipeline_sm_{timestamp}.log
|
||||
level: 20
|
||||
strategies:
|
||||
company:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- credit_code
|
||||
depend_fields: {}
|
||||
- credit_code
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
user:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- username
|
||||
depend_fields: {}
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
supplier:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- credit_code
|
||||
depend_fields: {}
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
project:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- name
|
||||
depend_fields:
|
||||
company_id: company
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
contract:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- company_id
|
||||
- code
|
||||
depend_fields:
|
||||
project_id: project
|
||||
company_id: supplier
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
contract_change:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- contract_id
|
||||
- change_time
|
||||
- title
|
||||
depend_fields:
|
||||
project_id: project
|
||||
contract_id: contract
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
construction_task:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- contract_id
|
||||
- task_type
|
||||
depend_fields:
|
||||
contract_id: contract
|
||||
project_id: project
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
construction_task_detail:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- parent_id
|
||||
- date
|
||||
depend_fields:
|
||||
parent_id: construction_task
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
material:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- contract_id
|
||||
- material_type
|
||||
depend_fields:
|
||||
contract_id: contract
|
||||
project_id: project
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
material_detail:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- parent_id
|
||||
- date
|
||||
depend_fields:
|
||||
parent_id: material
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
contract_settlement:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- contract_id
|
||||
depend_fields:
|
||||
contract_id: contract
|
||||
project_id: project
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
contract_settlement_detail:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- parent_id
|
||||
- date
|
||||
depend_fields:
|
||||
parent_id: contract_settlement
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
personnel:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- code
|
||||
depend_fields:
|
||||
project_id: project
|
||||
contract_id: contract
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
personnel_detail:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- parent_id
|
||||
- date
|
||||
depend_fields:
|
||||
parent_id: personnel
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
preparation:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- code
|
||||
depend_fields:
|
||||
project_id: project
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
production:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- code
|
||||
depend_fields:
|
||||
project_id: project
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
- name
|
||||
lar:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
depend_fields:
|
||||
project_id: project
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
lar_change:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- change_time
|
||||
- title
|
||||
depend_fields:
|
||||
project_id: project
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
project_detail_data:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- key
|
||||
depend_fields:
|
||||
project_id: project
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
units:
|
||||
skip_sync: false
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- project_id
|
||||
- serial_number
|
||||
depend_fields:
|
||||
project_id: project
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
global_config: run_profiles/preset/jsonl_to_jsonl.multi_project.global.default.yaml
|
||||
default_project_config: run_profiles/preset/jsonl_to_jsonl.multi_project.project.default.yaml
|
||||
|
||||
project_bootstrap_node_types:
|
||||
- company
|
||||
- supplier
|
||||
|
||||
projects:
|
||||
- local_project_id: f3e02e84-e81c-4aa6-88d4-f5aa108c8227
|
||||
remote_project_id: f3e02e84-e81c-4aa6-88d4-f5aa108c8227
|
||||
- local_project_id: 0f9c3e22-f4bb-4803-825a-4f238eeb06d5
|
||||
remote_project_id: 0f9c3e22-f4bb-4803-825a-4f238eeb06d5
|
||||
@@ -0,0 +1,29 @@
|
||||
node_types:
|
||||
- company
|
||||
- supplier
|
||||
local_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: ${PROJECT_ROOT}/working_local_datasource/20260302-020003
|
||||
remote_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: ${PROJECT_ROOT}/runtime/tmp/multi_project_jsonl_remote
|
||||
persist:
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/multi_project_jsonl.db
|
||||
wipe_on_start: true
|
||||
logging:
|
||||
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
|
||||
file_name_pattern: multi_project_global_{timestamp}.log
|
||||
strategies:
|
||||
company:
|
||||
config:
|
||||
auto_bind_fields:
|
||||
- code
|
||||
- name
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
supplier:
|
||||
config:
|
||||
local_orphan_action: create_remote
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
@@ -0,0 +1,47 @@
|
||||
node_types:
|
||||
- project
|
||||
- contract
|
||||
- contract_change
|
||||
- construction_task
|
||||
- construction_task_detail
|
||||
- material
|
||||
- material_detail
|
||||
- contract_settlement
|
||||
- contract_settlement_detail
|
||||
- personnel
|
||||
- personnel_detail
|
||||
- preparation
|
||||
- production
|
||||
- lar
|
||||
- lar_change
|
||||
- project_detail_data
|
||||
- units
|
||||
local_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: ${PROJECT_ROOT}/working_local_datasource/20260302-020003
|
||||
remote_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: ${PROJECT_ROOT}/runtime/tmp/multi_project_jsonl_remote
|
||||
persist:
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/multi_project_jsonl.db
|
||||
logging:
|
||||
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
|
||||
file_name_pattern: multi_project_project_{timestamp}.log
|
||||
strategies:
|
||||
project:
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- name
|
||||
lar:
|
||||
config:
|
||||
local_orphan_action: create_remote
|
||||
lar_change:
|
||||
config:
|
||||
local_orphan_action: create_remote
|
||||
project_detail_data:
|
||||
config:
|
||||
local_orphan_action: create_remote
|
||||
units:
|
||||
config:
|
||||
local_orphan_action: create_remote
|
||||
@@ -59,6 +59,39 @@ class BaseResponseWithID(BaseModel):
|
||||
# 默认返回(已有 id)
|
||||
return data
|
||||
|
||||
|
||||
def _get_plan_node_date(node: Any) -> Any:
|
||||
if isinstance(node, dict):
|
||||
return node.get("date")
|
||||
return getattr(node, "date", None)
|
||||
|
||||
|
||||
def trim_plan_nodes_by_boundary_dates(
|
||||
plan_nodes: Optional[List[Any]],
|
||||
start_date: Optional[Any] = None,
|
||||
end_date: Optional[Any] = None,
|
||||
) -> List[Any]:
|
||||
if not plan_nodes:
|
||||
return []
|
||||
|
||||
cleaned_nodes = list(plan_nodes)
|
||||
|
||||
while (
|
||||
start_date is not None
|
||||
and cleaned_nodes
|
||||
and str(_get_plan_node_date(cleaned_nodes[0])) == str(start_date)
|
||||
):
|
||||
cleaned_nodes.pop(0)
|
||||
|
||||
while (
|
||||
end_date is not None
|
||||
and cleaned_nodes
|
||||
and str(_get_plan_node_date(cleaned_nodes[-1])) == str(end_date)
|
||||
):
|
||||
cleaned_nodes.pop()
|
||||
|
||||
return cleaned_nodes
|
||||
|
||||
class RequestApprovalMixin(BaseModel):
|
||||
"""
|
||||
请求审批相关的基础模型
|
||||
|
||||
@@ -24,7 +24,18 @@
|
||||
from pydantic import BaseModel, Field
|
||||
from typing import TypeVar, Generic, Optional, List, Dict, Any
|
||||
from ..project.enums import ProjectType
|
||||
from datetime import datetime
|
||||
# from backend.modules.admin.approval.models import ApprovalStep
|
||||
|
||||
|
||||
class ApprovalStep(BaseModel):
|
||||
"""审批流程中的一个步骤"""
|
||||
step_id: str = Field(description="步骤的唯一ID")
|
||||
operator_id: Optional[str] = Field("",description="操作人ID")
|
||||
operator_name: str = Field(description="操作人姓名")
|
||||
action: str = Field(description="执行的动作,如 'submit', 'approve', 'reject'")
|
||||
comment: Optional[str] = Field(None, description="审批意见")
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow, description="步骤的创建时间")
|
||||
# 定义泛型
|
||||
T = TypeVar('T', bound='BasePushData')
|
||||
|
||||
@@ -52,7 +63,7 @@ class SignatureMixin(BaseModel):
|
||||
class PushRequestMixin(BaseModel):
|
||||
"""推送ID字段混入"""
|
||||
push_id: str = Field(..., description="推送ID,客户端生成的唯一标识(UUIDv7,支持时间排序)")
|
||||
|
||||
steps : List[ApprovalStep] =Field(default_factory=list,description="当前数据的审批流程步骤") # 加入字段step用来记录审批流程
|
||||
|
||||
# ==================== 响应相关 ====================
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@ from pydantic import BaseModel, Field, field_validator, model_validator
|
||||
from fastapi import HTTPException
|
||||
from typing import List, Optional, Literal
|
||||
from ..common.validators import DateStr
|
||||
from ..common.base import BaseResponseWithID
|
||||
from ..common.base import BaseResponseWithID, trim_plan_nodes_by_boundary_dates
|
||||
|
||||
class PlanNode(BaseModel):
|
||||
date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"])
|
||||
@@ -40,6 +40,15 @@ class ConstructionResponse(BaseResponseWithID):
|
||||
# 2025-10-20新增字段,延迟时间
|
||||
delay_days: Optional[int] = Field(None, description="延迟时间(天)", examples=[0])
|
||||
|
||||
@model_validator(mode='after')
|
||||
def clean_response_plan_nodes(self):
|
||||
self.plan_node = trim_plan_nodes_by_boundary_dates(
|
||||
self.plan_node,
|
||||
start_date=self.plan_start_date,
|
||||
end_date=self.plan_complete_date,
|
||||
)
|
||||
return self
|
||||
|
||||
class ConstructionCreate(BaseModel):
|
||||
"""
|
||||
施工任务创建模型
|
||||
|
||||
@@ -2,7 +2,7 @@ from pydantic import Field, BaseModel,field_validator,model_validator
|
||||
from fastapi import HTTPException
|
||||
from typing import List, Optional, Literal
|
||||
from ..common.validators import DateStr
|
||||
from ..common.base import BaseResponseWithID
|
||||
from ..common.base import BaseResponseWithID, trim_plan_nodes_by_boundary_dates
|
||||
|
||||
class PlanNode(BaseModel):
|
||||
date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"])
|
||||
@@ -39,6 +39,15 @@ class ContractSettlementResponse(BaseResponseWithID):
|
||||
None, description="统计节点截止日期", examples=["2024-05-31"]
|
||||
)
|
||||
|
||||
@model_validator(mode='after')
|
||||
def clean_response_plan_nodes(self):
|
||||
self.plan_node = trim_plan_nodes_by_boundary_dates(
|
||||
self.plan_node,
|
||||
start_date=self.plan_start_date,
|
||||
end_date=self.plan_complete_date,
|
||||
)
|
||||
return self
|
||||
|
||||
class ContractSettlementCreate(BaseModel):
|
||||
"""
|
||||
合同结算创建模型
|
||||
|
||||
@@ -2,7 +2,7 @@ from pydantic import Field, BaseModel,field_validator,model_validator
|
||||
from fastapi import HTTPException
|
||||
from typing import List, Optional, Literal
|
||||
from ..common.validators import DateStr
|
||||
from ..common.base import BaseResponseWithID
|
||||
from ..common.base import BaseResponseWithID, trim_plan_nodes_by_boundary_dates
|
||||
|
||||
class PlanNode(BaseModel):
|
||||
date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"])
|
||||
@@ -52,6 +52,15 @@ class MaterialResponse(BaseResponseWithID):
|
||||
None, description="统计节点截止日期", examples=["2024-05-31"]
|
||||
)
|
||||
|
||||
@model_validator(mode='after')
|
||||
def clean_response_plan_nodes(self):
|
||||
self.plan_node = trim_plan_nodes_by_boundary_dates(
|
||||
self.plan_node,
|
||||
start_date=self.plan_start_date,
|
||||
end_date=self.plan_complete_date,
|
||||
)
|
||||
return self
|
||||
|
||||
class MaterialCreate(BaseModel):
|
||||
"""
|
||||
创建物资
|
||||
|
||||
+10
-1
@@ -2,7 +2,7 @@ from pydantic import Field, BaseModel,field_validator,model_validator
|
||||
from fastapi import HTTPException
|
||||
from typing import List, Optional, Literal
|
||||
from ..common.validators import DateStr
|
||||
from ..common.base import BaseResponseWithID
|
||||
from ..common.base import BaseResponseWithID, trim_plan_nodes_by_boundary_dates
|
||||
|
||||
class PlanNode(BaseModel):
|
||||
date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"])
|
||||
@@ -61,6 +61,15 @@ class PowerResponse(BaseResponseWithID):
|
||||
None, description="实际首批到场人数", examples=[4]
|
||||
)
|
||||
|
||||
@model_validator(mode='after')
|
||||
def clean_response_plan_nodes(self):
|
||||
self.plan_node = trim_plan_nodes_by_boundary_dates(
|
||||
self.plan_node,
|
||||
start_date=self.plan_start_date,
|
||||
end_date=self.plan_exit_date,
|
||||
)
|
||||
return self
|
||||
|
||||
class PowerCreate(BaseModel):
|
||||
"""
|
||||
创建力能
|
||||
|
||||
@@ -74,3 +74,12 @@ class ProjectInfoResponse(BaseModel):
|
||||
# 里程碑节点
|
||||
preparation_list: list[PreparationDetail] = Field(default_factory=list, description="里程碑节点列表")
|
||||
|
||||
|
||||
class ProjectListResponse(BaseModel):
|
||||
"""
|
||||
项目列表模型:分页
|
||||
"""
|
||||
page: int
|
||||
page_size: int
|
||||
total: int
|
||||
list: list[ProjectInfoResponse]
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
"""
|
||||
一个接口按 domain 返回项目聚合数据。
|
||||
"""
|
||||
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..construction.construction import ConstructionResponse
|
||||
from ..construction.construction_detail import ConstructionDetailResponse
|
||||
from ..contract.contract import ContractResponse
|
||||
from ..contract_settlement.contract_settlement import (
|
||||
ContractSettlementResponse,
|
||||
)
|
||||
from ..contract_settlement.contract_settlement_detail import (
|
||||
ContractSettlementDetailResponse,
|
||||
)
|
||||
from ..material.material import MaterialResponse
|
||||
from ..material.material_detail import MaterialDetailResponse
|
||||
from ..power.power import PowerResponse
|
||||
from ..power.power_detail import PowerDetail
|
||||
from ..project.project_base import ProjectResponseBase
|
||||
from ..project_extensions.contract_change import ContractChangeResponse
|
||||
from ..project_extensions.generator_unit import GeneratorUnitProject
|
||||
from ..project_extensions.lar import (
|
||||
LarChangeDetailSchema,
|
||||
LarDetailSchema,
|
||||
)
|
||||
from ..project_extensions.preparation import PreparationDetail
|
||||
from ..project_extensions.production import ProductionDetail
|
||||
from ..project_extensions.project_detail import ProjectDetailProject
|
||||
from ..project_multi_type import (
|
||||
GasTurbineExtension,
|
||||
HydropowerExtension,
|
||||
OffShoreWindExtension,
|
||||
OnShoreWindExtension,
|
||||
PhotovoltaicExtension,
|
||||
PumpedStorageExtension,
|
||||
ThermalPowerExtension,
|
||||
)
|
||||
|
||||
|
||||
ProjectAggregateDomain = Literal[
|
||||
"project",
|
||||
"project_detail",
|
||||
"generator_units",
|
||||
"lar",
|
||||
"lar_change",
|
||||
"contract_changes",
|
||||
"photovoltaic_extension",
|
||||
"onshore_wind_extension",
|
||||
"offshore_wind_extension",
|
||||
"hydropower_extension",
|
||||
"pumped_storage_extension",
|
||||
"gas_turbine_extension",
|
||||
"thermal_power_extension",
|
||||
"production_list",
|
||||
"preparation_list",
|
||||
"contracts",
|
||||
"materials",
|
||||
"material_details",
|
||||
"constructions",
|
||||
"construction_details",
|
||||
"contract_settlements",
|
||||
"contract_settlement_details",
|
||||
"power",
|
||||
"power_details",
|
||||
]
|
||||
|
||||
|
||||
ALL_PROJECT_AGGREGATE_DOMAINS: list[ProjectAggregateDomain] = [
|
||||
"project",
|
||||
"project_detail",
|
||||
"generator_units",
|
||||
"lar",
|
||||
"lar_change",
|
||||
"contract_changes",
|
||||
"photovoltaic_extension",
|
||||
"onshore_wind_extension",
|
||||
"offshore_wind_extension",
|
||||
"hydropower_extension",
|
||||
"pumped_storage_extension",
|
||||
"gas_turbine_extension",
|
||||
"thermal_power_extension",
|
||||
"production_list",
|
||||
"preparation_list",
|
||||
"contracts",
|
||||
"materials",
|
||||
"material_details",
|
||||
"constructions",
|
||||
"construction_details",
|
||||
"contract_settlements",
|
||||
"contract_settlement_details",
|
||||
"power",
|
||||
"power_details",
|
||||
]
|
||||
|
||||
|
||||
class ProjectAggregateData(BaseModel):
|
||||
"""按 domain 组织的项目聚合数据。"""
|
||||
|
||||
project: ProjectResponseBase | None = Field(None, description="项目基本信息")
|
||||
project_detail: list[ProjectDetailProject] = Field(default_factory=list, description="项目容量信息")
|
||||
generator_units: list[GeneratorUnitProject] = Field(default_factory=list, description="发电单元信息")
|
||||
lar: LarDetailSchema | None = Field(None, description="移民征地信息")
|
||||
lar_change: list[LarChangeDetailSchema] = Field(default_factory=list, description="移民变更信息")
|
||||
contract_changes: list[ContractChangeResponse] = Field(default_factory=list, description="合同变更信息")
|
||||
|
||||
photovoltaic_extension: PhotovoltaicExtension | None = Field(None, description="光伏项目扩展信息")
|
||||
onshore_wind_extension: OnShoreWindExtension | None = Field(None, description="陆风项目扩展信息")
|
||||
offshore_wind_extension: OffShoreWindExtension | None = Field(None, description="海风项目扩展信息")
|
||||
hydropower_extension: HydropowerExtension | None = Field(None, description="水电项目扩展信息")
|
||||
pumped_storage_extension: PumpedStorageExtension | None = Field(None, description="抽蓄项目扩展信息")
|
||||
gas_turbine_extension: GasTurbineExtension | None = Field(None, description="燃气轮机项目扩展信息")
|
||||
thermal_power_extension: ThermalPowerExtension | None = Field(None, description="火电项目扩展信息")
|
||||
|
||||
production_list: list[ProductionDetail] = Field(default_factory=list, description="投产节点列表")
|
||||
preparation_list: list[PreparationDetail] = Field(default_factory=list, description="里程碑节点列表")
|
||||
|
||||
contracts: list[ContractResponse] = Field(default_factory=list, description="合同列表")
|
||||
materials: list[MaterialResponse] = Field(default_factory=list, description="物资主表列表")
|
||||
material_details: list[MaterialDetailResponse] = Field(default_factory=list, description="物资明细列表")
|
||||
constructions: list[ConstructionResponse] = Field(default_factory=list, description="施工主表列表")
|
||||
construction_details: list[ConstructionDetailResponse] = Field(default_factory=list, description="施工明细列表")
|
||||
contract_settlements: list[ContractSettlementResponse] = Field(default_factory=list, description="合同结算主表列表")
|
||||
contract_settlement_details: list[ContractSettlementDetailResponse] = Field(default_factory=list, description="合同结算明细列表")
|
||||
power: list[PowerResponse] = Field(default_factory=list, description="力能主表列表")
|
||||
power_details: list[PowerDetail] = Field(default_factory=list, description="力能明细列表")
|
||||
|
||||
|
||||
class ProjectAggregateMeta(BaseModel):
|
||||
"""聚合请求回显和截断信息。"""
|
||||
|
||||
project_id: str = Field(..., description="项目ID")
|
||||
requested_domains: list[ProjectAggregateDomain] = Field(
|
||||
default_factory=lambda: list(ALL_PROJECT_AGGREGATE_DOMAINS),
|
||||
description="本次请求的 domain 列表",
|
||||
)
|
||||
domain_limit: int = Field(10000, description="每个 domain 的最大返回条数")
|
||||
truncated_domains: list[ProjectAggregateDomain] = Field(default_factory=list, description="被截断的 domain 列表")
|
||||
|
||||
|
||||
class ProjectAggregateResponse(BaseModel):
|
||||
"""项目聚合同步响应。"""
|
||||
|
||||
meta: ProjectAggregateMeta = Field(..., description="请求元信息")
|
||||
data: ProjectAggregateData = Field(
|
||||
default_factory=lambda: ProjectAggregateData.model_construct(),
|
||||
description="按 domain 分组的数据",
|
||||
)
|
||||
@@ -5,7 +5,7 @@
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import HTTPException
|
||||
from pydantic import Field, BaseModel, field_validator
|
||||
from pydantic import Field, BaseModel, field_validator, model_validator
|
||||
from ..common.push_system import BaseUpdateRequestSchema
|
||||
from ..common.base import BaseResponseWithID
|
||||
from ..common.datetime import get_current_date
|
||||
@@ -70,14 +70,15 @@ class ProjectResponseBase(BaseResponseWithID):
|
||||
complete_investment: float = Field(0.0, description="完成投资", examples=[80000.0])
|
||||
|
||||
# 其他信息
|
||||
progress_type: ProgressType = Field(..., description="推进分类", examples=[ProgressType.A])
|
||||
progress_type: ProgressType | None = Field(None, description="推进分类", examples=[ProgressType.A])
|
||||
key_constraints: KeyConstraints = Field(KeyConstraints.NONE, description="关键制约", examples=[KeyConstraints.NONE])
|
||||
key_constraints_remark: Optional[str] = Field("", description="关键制约备注", examples=["无"])
|
||||
status: Optional[str] = Field(None, description="项目状态")
|
||||
# status: Optional[str] = Field(None, description="项目状态")
|
||||
apply_file_ids: List[str] = Field(default_factory=list, description="申请表附件ID列表", examples=[[]])
|
||||
|
||||
# 拓展信息
|
||||
dbtc_score_list: Optional[list] = Field(default_factory=list, description='达标投产评分列表') # 这里项目用的是list而不是List
|
||||
# approval_status: Optional[str] = Field(None, description="审核状态", examples=["pending"])
|
||||
|
||||
|
||||
|
||||
@@ -265,11 +266,35 @@ class ProjectProductionCapacity(BaseModel):
|
||||
"""
|
||||
plan_production_date: DateStr | None = Field(None, description="集团公司计划投产日期", examples=["2024-08-01"])
|
||||
plan_full_production_date: DateStr | None = Field(None, description="集团公司计划全部投产日期", examples=["2024-09-01"])
|
||||
ic_plan_first_production_date: DateStr | None = Field(None, description="区域公司计划首批投产日期(只有新能源需要)",examples=["2024-08-01"])
|
||||
ic_plan_full_production_date: DateStr | None = Field(None, description="区域公司计划全投日期(只有新能源需要)",examples=["2024-09-01"])
|
||||
actual_production_date: DateStr | None = Field(None, description="实际投产日期(只有新能源需要)", examples=["2024-08-15"])
|
||||
actual_full_production_date: DateStr | None = Field(None, description="实际全投日期(只有新能源需要)", examples=["2024-09-10"])
|
||||
actual_production_list : list[_ActualProductionNode] | None = Field(None, description="实际投产节点列表(每次必须填写全部)(只有新能源需要)")
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def reject_ic_plan_dates(cls, data):
|
||||
if not isinstance(data, dict):
|
||||
return data
|
||||
|
||||
blocked_fields = [
|
||||
field for field in ("ic_plan_first_production_date", "ic_plan_full_production_date")
|
||||
if field in data
|
||||
]
|
||||
if blocked_fields:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"/production_capacity 接口不支持更新字段: {', '.join(blocked_fields)}"
|
||||
)
|
||||
return data
|
||||
|
||||
@property
|
||||
def ic_plan_first_production_date(self) -> str:
|
||||
return ""
|
||||
|
||||
@property
|
||||
def ic_plan_full_production_date(self) -> str:
|
||||
return ""
|
||||
|
||||
@field_validator("actual_production_date")
|
||||
def validate_actual_production_date(cls, v):
|
||||
if v is None:
|
||||
@@ -304,6 +329,7 @@ class ProjectSettlementInvestment(BaseModel):
|
||||
"""
|
||||
PROJECT_SETTLEMENT_INVESTMENT
|
||||
合同结算投资情况
|
||||
只有抽蓄-水电-煤电-气电项目类型可操作
|
||||
"""
|
||||
|
||||
implementation_estimate: float = Field(..., description="执行概算", examples=[80000.0])
|
||||
|
||||
@@ -21,8 +21,8 @@ class PostGeneratorUnits(BaseModel):
|
||||
|
||||
@field_validator('actual_production_date', mode="before")
|
||||
def validate_actual_date_not_future(cls, v: str) -> str:
|
||||
if v == "":
|
||||
return ""
|
||||
# if v == "":
|
||||
# return ""
|
||||
try:
|
||||
actual = datetime_date(v)
|
||||
except ValueError:
|
||||
|
||||
@@ -33,7 +33,7 @@ class PreparationDetail(BaseResponseWithID):
|
||||
code: Optional[str] = Field(None, description="里程碑节点编码")
|
||||
acquire_date: Optional[DateStr] = Field(None, description="取得时间")
|
||||
is_exist: Optional[bool] = Field(None, description="是否取得")
|
||||
approval_status: Optional[str] = Field(None, description="审批状态")
|
||||
# approval_status: Optional[str] = Field(None, description="审批状态")
|
||||
name: Optional[str] = Field(None, description="里程碑名称")
|
||||
group: Optional[str] = Field(None, description="分组条件", examples=['KGTJLS'])
|
||||
is_acquired: Optional[int] = Field(0, description="是否取得")
|
||||
# is_acquired: Optional[int] = Field(0, description="是否取得")
|
||||
|
||||
@@ -34,6 +34,6 @@ class ProductionDetail(BaseResponseWithID):
|
||||
code: Optional[str] = Field(None, description="投产节点编码")
|
||||
acquire_date: Optional[DateStr] = Field(None, description="取得时间")
|
||||
is_exist: Optional[bool] = Field(None, description="是否取得")
|
||||
approval_status: Optional[str] = Field(None, description="审批状态")
|
||||
# approval_status: Optional[str] = Field(None, description="审批状态")
|
||||
name: Optional[str] = Field(None, description="投产节点名称")
|
||||
is_acquired: Optional[int] = Field(0, description="是否取得")
|
||||
# is_acquired: Optional[int] = Field(0, description="是否取得")
|
||||
|
||||
@@ -1,8 +1,39 @@
|
||||
from pydantic import BaseModel, Field
|
||||
from fastapi import HTTPException
|
||||
from pydantic import BaseModel, Field, model_validator, field_validator
|
||||
from typing import List, Optional
|
||||
from enum import Enum
|
||||
from ..common.base import BaseResponseWithID
|
||||
|
||||
|
||||
class SupplierType(str, Enum):
|
||||
"""类型枚举"""
|
||||
PROJECT = "project" # 项目类型
|
||||
OTHER = "other" # 其他类型
|
||||
|
||||
@classmethod
|
||||
def supplier_list(cls):
|
||||
return [cls.PROJECT.value, cls.OTHER.value]
|
||||
|
||||
class SupplierNature(str, Enum):
|
||||
"""
|
||||
性质枚举
|
||||
"""
|
||||
CONSTRUCTION = "construction" # 施工单位
|
||||
DESIGN = "design" # 设计单位
|
||||
SUPERVISORY = "supervisory" # 监理单位
|
||||
SUPPLIER = "supplier" # 供应商
|
||||
SURVEY = "survey" # 2026.3.11 推送系统测试时发现与项目侧 org_role 不一致,已补充勘察单位
|
||||
GOVERMENT = "goverment" # 政府
|
||||
|
||||
@classmethod
|
||||
def supplier_type_list(cls):
|
||||
return [member.value for member in cls]
|
||||
|
||||
class SupplierStatus(str, Enum):
|
||||
"""状态枚举"""
|
||||
ACTIVE = "active" # 正常
|
||||
INACTIVE = "inactive" # 禁用
|
||||
|
||||
class SupplierDetailResponse(BaseResponseWithID):
|
||||
"""
|
||||
供应商详情模型
|
||||
@@ -17,15 +48,9 @@ class SupplierDetailResponse(BaseResponseWithID):
|
||||
status: str = Field(..., description="供应商状态", examples=["active"])
|
||||
sort: int = Field(..., description="排序", examples=[1])
|
||||
credit_code: Optional[str] = Field(None, description="信用代码", examples=["914403007109310121"])
|
||||
created_by: str = Field(..., description="创建者ID", examples=["user:001"])
|
||||
created_by_name: str = Field(..., description="创建者名称", examples=["张三"])
|
||||
created_at: str = Field(..., description="创建时间", examples=["2023-10-01T12:00:00Z"])
|
||||
updated_by: str = Field(..., description="更新者ID", examples=["user:001"])
|
||||
updated_by_name: str = Field(..., description="更新者名称", examples=["张三"])
|
||||
updated_at: str = Field(..., description="更新时间", examples=["2023-10-01T12:00:00Z"])
|
||||
province: Optional[str] = Field(None, description="省份编码", examples=[None])
|
||||
city: Optional[str] = Field(None, description="城市编码", examples=[None])
|
||||
company_nature: List[Optional[str]] = Field(None, description="供应商性质数组", examples=[['supplier']])
|
||||
company_nature: List[Optional[str]] = Field([], description="供应商性质数组", examples=[['supplier']])
|
||||
city_name: Optional[str] = Field(None, description="城市名称", examples=[None])
|
||||
province_name: Optional[str] = Field(None, description="省份名称", examples=[None])
|
||||
region_path: Optional[list] = Field(default_factory=list, description="供应商国家城市地址编码列表")
|
||||
@@ -41,3 +66,66 @@ class SupplierListResponse(BaseModel):
|
||||
page_size: int
|
||||
total: int
|
||||
list: List[SupplierDetailResponse] = []
|
||||
|
||||
|
||||
class SupplierCreateRequest(BaseModel):
|
||||
"""
|
||||
供应商推送创建请求模型
|
||||
"""
|
||||
name: str = Field(..., description="供应商名称", examples=["腾讯集团"])
|
||||
short_name: str = Field(..., description="供应商简称", examples=["腾讯"])
|
||||
code: str = Field(..., description="供应商编码", examples=["TX001"])
|
||||
company_type: SupplierType = Field(..., description="供应商类型", examples=["project"])
|
||||
parent_id: Optional[str] = Field(None, description="上级供应商ID", examples=[None])
|
||||
status: SupplierStatus = Field(..., description="供应商状态", examples=["active"])
|
||||
path: Optional[List[str]] = Field(default_factory=list, description="供应商路径 [..., 上级id, 本身id]")
|
||||
path_name: Optional[List[str]] = Field(default_factory=list, description="供应商路径名称 [..., 上级名称, 本身名称]")
|
||||
credit_code: str = Field(..., description="供应商信用代码", examples=["914403007109310121"])
|
||||
region_path: Optional[list] = Field(default_factory=list, description="供应商国家城市地址编码列表, 需对接集团侧地图组件, 没有对接可以不传递", examples=[[
|
||||
"100000",
|
||||
"420000",
|
||||
"420800"
|
||||
]])
|
||||
company_nature: List[SupplierNature] = Field(..., description="供应商性质数组 可多选",
|
||||
examples=[['construction',
|
||||
'supplier',
|
||||
'design',
|
||||
'survey',
|
||||
'supervisory',
|
||||
'goverment']])
|
||||
|
||||
@model_validator(mode='after')
|
||||
def validate_company_type_and_nature(self):
|
||||
data = self.model_dump()
|
||||
|
||||
company_nature = data.get("company_nature")
|
||||
if not company_nature:
|
||||
raise HTTPException(detail=f"供应商:company_nature为空", status_code=400)
|
||||
for cn in company_nature:
|
||||
if cn not in SupplierNature.supplier_type_list():
|
||||
raise HTTPException(detail=f"供应商:company_nature范围:{SupplierNature.supplier_type_list()}",
|
||||
status_code=400)
|
||||
|
||||
company_type = data.get("company_type")
|
||||
if not company_type:
|
||||
raise HTTPException(detail=f"供应商:company_type为空", status_code=400)
|
||||
if company_type not in SupplierType.supplier_list():
|
||||
raise HTTPException(
|
||||
detail=f"供应商:company_type范围({SupplierType.supplier_list()})", status_code=400)
|
||||
|
||||
for field in ["name", "short_name", "code", "credit_code", "status"]:
|
||||
field_value = data.get(field)
|
||||
if not field_value:
|
||||
raise HTTPException(detail=f"字段{field}为空", status_code=400)
|
||||
return self
|
||||
|
||||
@field_validator(
|
||||
"credit_code",
|
||||
mode="after"
|
||||
)
|
||||
def clean_credit_code(cls, field_value):
|
||||
if field_value is None:
|
||||
return None
|
||||
import re
|
||||
return re.sub(r'\s+', '', str(field_value))
|
||||
|
||||
|
||||
@@ -17,16 +17,17 @@ class UserListDetailItem(BaseResponseWithID):
|
||||
status: Optional[str] = Field(None, description="状态")
|
||||
email: Optional[EmailStr] = Field(None, description="邮箱地址")
|
||||
user_type: str = Field(..., description="用户类型")
|
||||
last_login_at: Optional[str] = Field(None, description="最后登录时间")
|
||||
updated_at: Optional[str] = Field(None, description="更新时间")
|
||||
# last_login_at: Optional[str] = Field(None, description="最后登录时间")
|
||||
# updated_at: Optional[str] = Field(None, description="更新时间")
|
||||
company_id: Optional[str] = Field(None, description="公司id")
|
||||
company_name: Optional[str] = Field(None, description="公司名称")
|
||||
created_at: Optional[str] = Field(None, description="创建时间")
|
||||
approval_status: Optional[str] = Field(None, description="审批状态")
|
||||
approval_time: Optional[str] = Field(None, description="审批通过时间")
|
||||
last_login_ip: Optional[str] = Field(None, description="最后登录ip")
|
||||
# created_at: Optional[str] = Field(None, description="创建时间")
|
||||
# approval_status: Optional[str] = Field(None, description="审批状态") # 不再区分审批状态和审批时间
|
||||
# approval_time: Optional[str] = Field(None, description="审批通过时间")
|
||||
# last_login_ip: Optional[str] = Field(None, description="最后登录ip")
|
||||
project_ids: Optional[List[str]] = Field([], description="权限内项目id")
|
||||
contract_ids: Optional[List[str]] = Field([], description="权限内合同id")
|
||||
phone_hash: Optional[str] = Field(None, description="手机号hash")
|
||||
apply_files: Optional[List[Dict]] = Field([],
|
||||
examples=[
|
||||
{"id": "file:001", "name": "文件1",
|
||||
|
||||
+32
-35
@@ -1,44 +1,41 @@
|
||||
# 脚本目录
|
||||
|
||||
## 文件说明
|
||||
本目录当前主要放两类脚本:
|
||||
- 自动化测试编排脚本
|
||||
- 本地数据集构建/分析脚本
|
||||
|
||||
### test_api_sync.py
|
||||
主要的API数据同步测试脚本。
|
||||
旧的 `scripts/test_api_sync.py` 与 `API_SYNC_GUIDE.md` 已不存在,不再作为当前入口。
|
||||
|
||||
**功能:**
|
||||
- 从远程API获取所有GET/LIST接口数据
|
||||
- 自动分页获取全量数据
|
||||
- 保存为JSONL格式到 `remote_datasource` 目录
|
||||
- 支持单项目模式或全量同步
|
||||
## 当前常用入口
|
||||
|
||||
### 自动化测试
|
||||
- `scripts/auto_test_init.py`
|
||||
- 初始化 auto-test 运行环境,负责清理状态、恢复测试数据、拉起后端。
|
||||
- `scripts/auto_test_clear.py`
|
||||
- 清理 auto-test 运行环境。
|
||||
- `scripts/run_auto_test.py`
|
||||
- 运行指定 domain 的多轮自动回归并输出报告。
|
||||
- `scripts/run_auto_test_all_domains.py`
|
||||
- 聚合运行多个 domain 的自动回归。
|
||||
- `scripts/run_large_scale_auto_test.py`
|
||||
- 大规模自动回归入口。
|
||||
|
||||
### 数据集与分析
|
||||
- `scripts/build_filtered_datasource_for_projects.py`
|
||||
- 从完整 datasource 中抽取指定项目,生成可复现的精简 JSONL 数据集。
|
||||
- `scripts/analyze_state_signatures.py`
|
||||
- 分析状态签名与差异。
|
||||
|
||||
## 推荐命令
|
||||
|
||||
这些命令目前可直接使用:
|
||||
|
||||
**运行方式:**
|
||||
```bash
|
||||
python scripts/test_api_sync.py
|
||||
python scripts/build_filtered_datasource_for_projects.py --help
|
||||
python scripts/run_auto_test.py --help
|
||||
```
|
||||
|
||||
**数据类型覆盖:**
|
||||
- 基础数据: company, user, supplier, project
|
||||
- 项目数据: contract, construction_task, material, contract_settlement, personnel
|
||||
- 项目扩展: preparation, production, lar, units (从all_info获取)
|
||||
- 明细数据: construction_task_detail, material_detail, contract_settlement_detail, personnel_detail
|
||||
## 进一步说明
|
||||
|
||||
**配置说明:**
|
||||
在文件中修改以下配置:
|
||||
```python
|
||||
BASE_URL = "http://localhost:8000/api/third/v2"
|
||||
UID = "your-uid"
|
||||
SECRET_KEY = "your-secret-key"
|
||||
TARGET_PROJECT_ID = "xxx-xxx-xxx" # 单项目模式,设为None则全量同步
|
||||
```
|
||||
|
||||
## 输出结果
|
||||
|
||||
数据保存在 `remote_datasource/datasource/` 目录下:
|
||||
- `company_1.jsonl` - 公司数据
|
||||
- `user_1.jsonl` - 用户数据
|
||||
- `project_1.jsonl` - 项目数据
|
||||
- ... 其他数据类型
|
||||
|
||||
## 更多文档
|
||||
|
||||
详细文档请查看项目根目录的 [API_SYNC_GUIDE.md](../API_SYNC_GUIDE.md)
|
||||
- 自动化测试总览:`docs/testing_guide.md`
|
||||
- 大规模自动回归方法:`docs/auto_test_spec/agent_auto_test_method.md`
|
||||
|
||||
@@ -292,8 +292,8 @@ def build_temp_run_profile_for_project_auto_bind(
|
||||
project_detail_push_keys: set[str] | None = None,
|
||||
) -> tuple[str, list[str]]:
|
||||
profile = load_yaml_config(config["pipeline"]["run_profile"])
|
||||
profile.setdefault("local_datasource", {})["target_project_ids"] = [local_project_id]
|
||||
profile.setdefault("remote_datasource", {})["target_project_ids"] = [remote_project_id]
|
||||
profile.setdefault("local_datasource", {}).setdefault("handler_configs", {}).setdefault("project", {})["data_id_filter"] = [local_project_id]
|
||||
profile.setdefault("remote_datasource", {}).setdefault("handler_configs", {}).setdefault("project", {})["data_id_filter"] = [remote_project_id]
|
||||
|
||||
project_strategy = profile.setdefault("strategies", {}).setdefault("project", {}).setdefault("config", {})
|
||||
project_strategy["auto_bind"] = False
|
||||
@@ -309,11 +309,11 @@ def build_temp_run_profile_for_project_auto_bind(
|
||||
project_strategy["update_direction"] = project_update_direction or "none"
|
||||
|
||||
if project_detail_push_keys:
|
||||
project_detail_config = profile.setdefault("strategies", {}).setdefault("project_detail_data", {}).setdefault("config", {})
|
||||
overrides = dict(project_detail_config.get("field_direction_overrides", {}))
|
||||
for key in project_detail_push_keys:
|
||||
overrides.pop(key, None)
|
||||
project_detail_config["field_direction_overrides"] = overrides
|
||||
fixed_pull_keys = {"ensure_production", "climb_production", "challenge_production"}
|
||||
if fixed_pull_keys.intersection(project_detail_push_keys):
|
||||
project_detail_runtime = profile.setdefault("strategies", {}).setdefault("project_detail_data", {})
|
||||
domain_option = project_detail_runtime.setdefault("domain_option", {})
|
||||
domain_option["pull_group_production_plans"] = False
|
||||
|
||||
runtime_dir = Path(report_root) / "runtime_configs"
|
||||
runtime_dir.mkdir(parents=True, exist_ok=True)
|
||||
@@ -365,7 +365,7 @@ def run_pipeline_round(
|
||||
merge_tree_contents(common_dataset_dir, paths["temp_data_dir"])
|
||||
enrich_supplier_display_fields(
|
||||
paths["temp_data_dir"],
|
||||
f"{config['project_root']}/filtered_datasource/datasource/filtered/region_1.jsonl",
|
||||
f"{config['project_root']}/working_local_datasource/filtered/region_1.jsonl",
|
||||
)
|
||||
normalize_contract_settlement_records(paths["temp_data_dir"])
|
||||
normalize_construction_task_plan_nodes(paths["temp_data_dir"])
|
||||
|
||||
+56
-11
@@ -15,6 +15,8 @@ from urllib import error, request
|
||||
|
||||
import yaml
|
||||
|
||||
from sync_state_machine.common.persistence import PersistenceBackend
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
@@ -312,26 +314,69 @@ def sqlite_rows(db_path: str | Path, sql: str, params: tuple[Any, ...] = ()) ->
|
||||
return [{key: row[key] for key in row.keys()} for row in rows]
|
||||
|
||||
|
||||
def persistence_summary(db_path: str | Path, node_type: str | None = None) -> dict[str, Any]:
|
||||
def persistence_rows(
|
||||
*,
|
||||
backend: str,
|
||||
sql: str,
|
||||
db_path: str | Path | None = None,
|
||||
url: str | None = None,
|
||||
limit: int = 200,
|
||||
) -> list[dict[str, Any]]:
|
||||
_, rows = PersistenceBackend.query_records(
|
||||
backend=backend,
|
||||
db_path=str(db_path) if db_path is not None else None,
|
||||
url=url,
|
||||
sql=sql,
|
||||
limit=limit,
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def persistence_summary(
|
||||
db_path: str | Path | None,
|
||||
node_type: str | None = None,
|
||||
*,
|
||||
backend: str = "sqlite",
|
||||
url: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
records = PersistenceBackend.records_summary(
|
||||
backend=backend,
|
||||
db_path=str(db_path) if db_path is not None else None,
|
||||
url=url,
|
||||
)
|
||||
summary: dict[str, Any] = {
|
||||
"backend": backend,
|
||||
"db_path": str(db_path),
|
||||
"db_exists": Path(db_path).exists(),
|
||||
"url": url,
|
||||
"db_exists": records.get("db_exists", False),
|
||||
"bindings": [],
|
||||
"node_status": [],
|
||||
}
|
||||
if not Path(db_path).exists():
|
||||
if not summary["db_exists"]:
|
||||
return summary
|
||||
|
||||
target_db_path = str(db_path) if db_path is not None else None
|
||||
if node_type:
|
||||
summary["bindings"] = sqlite_rows(
|
||||
db_path,
|
||||
"SELECT node_type, local_id, remote_id, last_updated FROM bindings WHERE node_type = ? ORDER BY last_updated DESC LIMIT 20",
|
||||
(node_type,),
|
||||
escaped_node_type = node_type.replace("'", "''")
|
||||
summary["bindings"] = persistence_rows(
|
||||
backend=backend,
|
||||
db_path=target_db_path,
|
||||
url=url,
|
||||
sql=(
|
||||
"SELECT node_type, local_id, remote_id, last_updated "
|
||||
f"FROM bindings WHERE node_type = '{escaped_node_type}' ORDER BY last_updated DESC LIMIT 20"
|
||||
),
|
||||
)
|
||||
summary["node_status"] = sqlite_rows(
|
||||
db_path,
|
||||
"SELECT collection_id, node_type, action, status, binding_status, COUNT(1) AS count FROM nodes WHERE node_type = ? GROUP BY collection_id, node_type, action, status, binding_status ORDER BY collection_id, action, status",
|
||||
(node_type,),
|
||||
summary["node_status"] = persistence_rows(
|
||||
backend=backend,
|
||||
db_path=target_db_path,
|
||||
url=url,
|
||||
sql=(
|
||||
"SELECT collection_id, node_type, action, status, binding_status, COUNT(1) AS count "
|
||||
f"FROM nodes WHERE node_type = '{escaped_node_type}' "
|
||||
"GROUP BY collection_id, node_type, action, status, binding_status "
|
||||
"ORDER BY collection_id, action, status"
|
||||
),
|
||||
)
|
||||
return summary
|
||||
|
||||
|
||||
@@ -1,208 +0,0 @@
|
||||
"""
|
||||
Run FullSyncPipeline: JSONL -> JSONL (filtered_datasource -> remote_datasource)
|
||||
|
||||
直接运行脚本,不使用 pytest。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
# 确保能导入项目模块
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
# 触发 DomainRegistry 注册
|
||||
import sync_system_new.domain # noqa: F401, E402
|
||||
|
||||
from sync_system_new.common.binding import BindingManager # noqa: E402
|
||||
from sync_system_new.common.collection import DataCollection # noqa: E402
|
||||
from sync_system_new.common.persistence import PersistenceBackend # noqa: E402
|
||||
from sync_system_new.common.registry import DomainRegistry # noqa: E402
|
||||
from sync_system_new.datasource.jsonl.datasource import JsonlDataSource # noqa: E402
|
||||
from sync_system_new.pipeline import FullSyncPipeline # noqa: E402
|
||||
from sync_system_new.sync_system.config import OrphanAction, UpdateDirection # noqa: E402
|
||||
|
||||
|
||||
LOCAL_DIR = PROJECT_ROOT / "filtered_datasource" / "datasource" / "ecm-2025-12-02"
|
||||
REMOTE_DIR = PROJECT_ROOT / "remote_datasource" / "datasource"
|
||||
PERSISTENCE_DB = PROJECT_ROOT / "test_results" / "full_sync_pipeline_jsonl_run.db"
|
||||
WIPE_DB_ON_START = False
|
||||
|
||||
NODE_TYPES: List[str] = [
|
||||
"company",
|
||||
# "supplier",
|
||||
"user",
|
||||
"project",
|
||||
"project_detail_data",
|
||||
"contract",
|
||||
"contract_change",
|
||||
"construction_task",
|
||||
"construction_task_detail",
|
||||
"material",
|
||||
"material_detail",
|
||||
"contract_settlement",
|
||||
"contract_settlement_detail",
|
||||
"personnel",
|
||||
"personnel_detail",
|
||||
"preparation",
|
||||
"production",
|
||||
"lar",
|
||||
"units",
|
||||
]
|
||||
|
||||
|
||||
class StrategyLogger:
|
||||
"""包装策略,打印执行完成信息。"""
|
||||
|
||||
def __init__(self, inner):
|
||||
self.inner = inner
|
||||
self.node_type = inner.node_type
|
||||
|
||||
async def bind(self):
|
||||
return await self.inner.bind()
|
||||
|
||||
async def create(self):
|
||||
created = await self.inner.create()
|
||||
print(f"✅ Strategy create: {self.node_type} (created={len(created)})")
|
||||
return created
|
||||
|
||||
async def update(self):
|
||||
updated = await self.inner.update()
|
||||
print(f"✅ Strategy update: {self.node_type} (updated={len(updated)})")
|
||||
return updated
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
PERSISTENCE_DB.parent.mkdir(parents=True, exist_ok=True)
|
||||
if WIPE_DB_ON_START and PERSISTENCE_DB.exists():
|
||||
PERSISTENCE_DB.unlink()
|
||||
|
||||
persistence = PersistenceBackend(str(PERSISTENCE_DB))
|
||||
await persistence.initialize()
|
||||
|
||||
local_ds = JsonlDataSource(LOCAL_DIR, read_only=True)
|
||||
remote_ds = JsonlDataSource(REMOTE_DIR, read_only=False)
|
||||
|
||||
for node_type in NODE_TYPES:
|
||||
handler_class = DomainRegistry.get_jsonl_handler(node_type)
|
||||
if handler_class is None:
|
||||
raise RuntimeError(f"Missing JSONL handler for node_type: {node_type}")
|
||||
local_ds.register_handler(handler_class(local_ds))
|
||||
remote_ds.register_handler(handler_class(remote_ds))
|
||||
|
||||
local_collection = DataCollection("local", persistence, auto_persist=False)
|
||||
remote_collection = DataCollection("remote", persistence, auto_persist=False)
|
||||
binding_manager = BindingManager(persistence, auto_persist=False)
|
||||
|
||||
strategies = []
|
||||
for node_type in NODE_TYPES:
|
||||
strategy_class = DomainRegistry.get_strategy_class(node_type)
|
||||
if strategy_class is None:
|
||||
raise RuntimeError(f"Missing strategy for node_type: {node_type}")
|
||||
strategy = strategy_class(node_type, local_collection, remote_collection, binding_manager)
|
||||
# 运行时强制只推不拉(不覆盖原有 depend_fields/auto_bind_fields)
|
||||
strategy.update_config(
|
||||
local_orphan_action=OrphanAction.CREATE_REMOTE,
|
||||
remote_orphan_action=OrphanAction.NONE,
|
||||
update_direction=UpdateDirection.PUSH,
|
||||
)
|
||||
if node_type == "contract":
|
||||
# contract 仅依赖 project_id,避免 company_id 为空导致依赖阻断
|
||||
strategy.update_config(depend_fields={"project_id": "project"})
|
||||
strategies.append(StrategyLogger(strategy))
|
||||
|
||||
pipeline = FullSyncPipeline(
|
||||
local_datasource=local_ds,
|
||||
remote_datasource=remote_ds,
|
||||
strategies=strategies,
|
||||
node_types=NODE_TYPES,
|
||||
load_order=NODE_TYPES,
|
||||
sync_order=NODE_TYPES,
|
||||
persistence=persistence,
|
||||
local_collection=local_collection,
|
||||
remote_collection=remote_collection,
|
||||
binding_manager=binding_manager,
|
||||
enable_persistence=True,
|
||||
fail_fast_on_precheck=False,
|
||||
)
|
||||
|
||||
stats = await pipeline.run()
|
||||
print(f"Pipeline finished. Stats: {stats}")
|
||||
|
||||
binding_counts = {}
|
||||
for node_type in NODE_TYPES:
|
||||
records = await binding_manager.get_all_records(node_type)
|
||||
binding_counts[node_type] = len(records)
|
||||
|
||||
def format_action(summary, key: str) -> str:
|
||||
action = summary.get(key, {})
|
||||
return f"{action.get('total', 0)}({action.get('success', 0)}/{action.get('failed', 0)})"
|
||||
|
||||
def format_total(summary) -> str:
|
||||
total = summary.get("total", {})
|
||||
return f"{total.get('total', 0)}({total.get('success', 0)}/{total.get('failed', 0)})"
|
||||
|
||||
def print_summary(title: str, action_summary: dict) -> None:
|
||||
print(f"\n=== {title} summary ===")
|
||||
print("doc_type: bindings | create(s/f) | update(s/f) | delete(s/f) | none | total(s/f)")
|
||||
|
||||
total_bindings = 0
|
||||
total_create = {"total": 0, "success": 0, "failed": 0}
|
||||
total_update = {"total": 0, "success": 0, "failed": 0}
|
||||
total_delete = {"total": 0, "success": 0, "failed": 0}
|
||||
total_none = 0
|
||||
total_all = {"total": 0, "success": 0, "failed": 0}
|
||||
|
||||
for node_type in NODE_TYPES:
|
||||
summary = action_summary.get(node_type, {})
|
||||
bindings = binding_counts.get(node_type, 0)
|
||||
create_str = format_action(summary, "create")
|
||||
update_str = format_action(summary, "update")
|
||||
delete_str = format_action(summary, "delete")
|
||||
none_count = summary.get("none", {}).get("total", 0)
|
||||
total_str = format_total(summary)
|
||||
|
||||
print(
|
||||
f"{node_type}: bindings={bindings} "
|
||||
f"create={create_str} update={update_str} delete={delete_str} "
|
||||
f"none={none_count} total={total_str}"
|
||||
)
|
||||
|
||||
total_bindings += bindings
|
||||
for bucket, key in (
|
||||
(total_create, "create"),
|
||||
(total_update, "update"),
|
||||
(total_delete, "delete"),
|
||||
):
|
||||
action = summary.get(key, {})
|
||||
bucket["total"] += action.get("total", 0)
|
||||
bucket["success"] += action.get("success", 0)
|
||||
bucket["failed"] += action.get("failed", 0)
|
||||
total_none += none_count
|
||||
total_all["total"] += summary.get("total", {}).get("total", 0)
|
||||
total_all["success"] += summary.get("total", {}).get("success", 0)
|
||||
total_all["failed"] += summary.get("total", {}).get("failed", 0)
|
||||
|
||||
print(
|
||||
"TOTAL: "
|
||||
f"bindings={total_bindings} "
|
||||
f"create={total_create['total']}({total_create['success']}/{total_create['failed']}) "
|
||||
f"update={total_update['total']}({total_update['success']}/{total_update['failed']}) "
|
||||
f"delete={total_delete['total']}({total_delete['success']}/{total_delete['failed']}) "
|
||||
f"none={total_none} "
|
||||
f"total={total_all['total']}({total_all['success']}/{total_all['failed']})"
|
||||
)
|
||||
|
||||
print_summary("Local", local_ds.get_action_summary())
|
||||
print_summary("Remote", remote_ds.get_action_summary())
|
||||
|
||||
await persistence.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -1,153 +0,0 @@
|
||||
"""
|
||||
run_full_sync_pipeline_jsonl_sm.py
|
||||
|
||||
独立执行:基于 sync_state_machine 的 JSONL -> JSONL 全量同步脚本。
|
||||
用途:纯本地可复现,不依赖远端 API;用于验证重构后 pipeline 主流程可跑通。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Dict, List
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
import sync_state_machine.domain # noqa: F401, E402
|
||||
from sync_state_machine.pipeline import create_jsonl_to_jsonl_pipeline # noqa: E402
|
||||
|
||||
|
||||
class JsonlSmConfig(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", validate_assignment=True)
|
||||
|
||||
local_jsonl_dir: str = str(PROJECT_ROOT / "filtered_datasource" / "datasource" / "ecm-2025-12-02")
|
||||
remote_jsonl_dir: str = str(PROJECT_ROOT / "remote_datasource" / "datasource")
|
||||
persistence_db: str = str(PROJECT_ROOT / "test_results" / "sync_state_machine" / "full_sync_pipeline_jsonl_sm.db")
|
||||
wipe_persistence_on_start: bool = False
|
||||
|
||||
target_project_ids: List[str] = Field(default_factory=lambda: ["f3e02e84-e81c-4aa6-88d4-f5aa108c8227"])
|
||||
project_auto_bind_fields: List[str] = Field(default_factory=lambda: ["name"])
|
||||
project_id_bindings: Dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
node_types: List[str] = Field(default_factory=lambda: [
|
||||
"company",
|
||||
"user",
|
||||
"project",
|
||||
"contract",
|
||||
"contract_change",
|
||||
"construction_task",
|
||||
"construction_task_detail",
|
||||
"material",
|
||||
"material_detail",
|
||||
"contract_settlement",
|
||||
"contract_settlement_detail",
|
||||
"personnel",
|
||||
"personnel_detail",
|
||||
"preparation",
|
||||
"production",
|
||||
"lar",
|
||||
"units",
|
||||
])
|
||||
|
||||
|
||||
def _load_dotenv() -> Dict[str, str]:
|
||||
dotenv_path = PROJECT_ROOT / ".env"
|
||||
if not dotenv_path.exists():
|
||||
return {}
|
||||
|
||||
values: Dict[str, str] = {}
|
||||
for raw_line in dotenv_path.read_text(encoding="utf-8").splitlines():
|
||||
line = raw_line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, value = line.split("=", 1)
|
||||
values[key.strip()] = value.strip().strip('"').strip("'")
|
||||
return values
|
||||
|
||||
|
||||
def build_config() -> JsonlSmConfig:
|
||||
cfg = JsonlSmConfig()
|
||||
env = _load_dotenv()
|
||||
cfg.wipe_persistence_on_start = (
|
||||
os.getenv("SPS_JSONL_WIPE_DB", env.get("SPS_JSONL_WIPE_DB", str(cfg.wipe_persistence_on_start))).lower()
|
||||
in {"1", "true", "yes"}
|
||||
)
|
||||
return cfg
|
||||
|
||||
|
||||
def _print_config_snapshot(config: JsonlSmConfig) -> None:
|
||||
snapshot = {
|
||||
"local_jsonl_dir": config.local_jsonl_dir,
|
||||
"remote_jsonl_dir": config.remote_jsonl_dir,
|
||||
"persistence_db": config.persistence_db,
|
||||
"wipe_persistence_on_start": config.wipe_persistence_on_start,
|
||||
"target_project_ids": config.target_project_ids,
|
||||
"project_auto_bind_fields": config.project_auto_bind_fields,
|
||||
"project_id_bindings": config.project_id_bindings,
|
||||
"node_types": config.node_types,
|
||||
}
|
||||
print(f"🔧 Resolved Config: {snapshot}")
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
pipeline = None
|
||||
log_path = None
|
||||
config = build_config()
|
||||
|
||||
try:
|
||||
log_dir = PROJECT_ROOT / "test_results" / "sync_state_machine"
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_path = log_dir / f"jsonl_pipeline_sm_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
|
||||
|
||||
print("\n" + "=" * 80)
|
||||
print("🚀 Creating sync_state_machine JSONL -> JSONL pipeline")
|
||||
print(f"📝 Log file: {log_path}")
|
||||
_print_config_snapshot(config)
|
||||
print("=" * 80)
|
||||
|
||||
Path(config.persistence_db).parent.mkdir(parents=True, exist_ok=True)
|
||||
pipeline = await create_jsonl_to_jsonl_pipeline(
|
||||
local_jsonl_dir=config.local_jsonl_dir,
|
||||
remote_jsonl_dir=config.remote_jsonl_dir,
|
||||
persistence_db=config.persistence_db,
|
||||
node_types=config.node_types,
|
||||
load_order=config.node_types,
|
||||
sync_order=config.node_types,
|
||||
enable_persistence=True,
|
||||
wipe_persistence_on_start=config.wipe_persistence_on_start,
|
||||
target_project_ids=config.target_project_ids,
|
||||
project_id_bindings=config.project_id_bindings,
|
||||
project_auto_bind_fields=config.project_auto_bind_fields,
|
||||
enable_project_bind_override=True,
|
||||
enable_project_filter_override=True,
|
||||
initialize_logger=True,
|
||||
logger_file_path=str(log_path),
|
||||
)
|
||||
|
||||
stats = await pipeline.run()
|
||||
print(f"\n✅ Pipeline completed: stats={stats}")
|
||||
print(f"🗃️ Persistence DB: {config.persistence_db}")
|
||||
return 0
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted by user")
|
||||
return 130
|
||||
except Exception as exc:
|
||||
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return 1
|
||||
finally:
|
||||
if pipeline is not None:
|
||||
await pipeline.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(asyncio.run(main()))
|
||||
@@ -1,179 +0,0 @@
|
||||
"""
|
||||
run_full_sync_pipeline_simple.py
|
||||
|
||||
极简测试脚本:演示使用工厂函数创建 Pipeline
|
||||
- 只需配置参数
|
||||
- 调用工厂函数
|
||||
- 执行 pipeline.run()
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# 项目路径
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
# 触发 Domain 注册
|
||||
import sync_system_new.domain # noqa: F401, E402
|
||||
|
||||
from sync_system_new.pipeline import create_jsonl_to_api_pipeline # noqa: E402
|
||||
|
||||
|
||||
# ========== 配置参数 ==========
|
||||
CONFIG = {
|
||||
# 路径
|
||||
"local_jsonl_dir": str(PROJECT_ROOT / "filtered_datasource" / "datasource" / "ecm-2025-12-02"),
|
||||
"persistence_db": str(PROJECT_ROOT / "_runtime" / "test_simple.db"),
|
||||
|
||||
# API
|
||||
"api_base_url": "http://localhost:8000/api/third/v2",
|
||||
"api_uid": "6b18c681-92d5-4c7f-802c-df413b6504c8",
|
||||
"api_secret": "1a28Az5dBD1Gc3URpmHwiR0XfzW7UtScQnqJQzgpzdg",
|
||||
|
||||
# 节点类型
|
||||
"node_types": [
|
||||
"company",
|
||||
"supplier",
|
||||
# "user",
|
||||
"project",
|
||||
"project_detail_data",
|
||||
"preparation",
|
||||
"production",
|
||||
"lar",
|
||||
"lar_change",
|
||||
"units",
|
||||
"contract",
|
||||
"contract_change",
|
||||
"construction_task",
|
||||
"construction_task_detail",
|
||||
"material",
|
||||
"material_detail",
|
||||
"contract_settlement",
|
||||
"contract_settlement_detail",
|
||||
"personnel",
|
||||
"personnel_detail",
|
||||
],
|
||||
|
||||
# 项目过滤和预绑定
|
||||
"target_project_ids": ["f3e02e84-e81c-4aa6-88d4-f5aa108c8227"],
|
||||
"project_id_bindings": {
|
||||
"f3e02e84-e81c-4aa6-88d4-f5aa108c8227": "f3e02e84-e81c-4aa6-88d4-f5aa108c8227",
|
||||
},
|
||||
|
||||
# Handler 自定义配置
|
||||
"handler_configs": {
|
||||
# supplier: 从持久化加载,不调用 API(后续运行时启用)
|
||||
"supplier": {"load_mode": "persisted_only"},
|
||||
},
|
||||
|
||||
# Skip/Force Sync 控制
|
||||
# skip_sync_types: 跳过同步(只执行 bind,不执行 create/update)
|
||||
# 后续运行时可配置: "skip_sync_types": ["supplier"]
|
||||
"skip_sync_types": ["supplier"],
|
||||
# 行为
|
||||
"wipe_persistence_on_start": False, # 改为 False,否则持久化数据会被清空
|
||||
"enable_persistence": True,
|
||||
|
||||
# API 调试
|
||||
"api_debug": True,
|
||||
"poll_max_retries": 1,
|
||||
"poll_interval": 2.0,
|
||||
}
|
||||
|
||||
|
||||
async def main():
|
||||
"""主函数 - 返回 True 表示成功,False 表示被中断"""
|
||||
from datetime import datetime
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
pipeline = None
|
||||
interrupted = False
|
||||
log_file = None
|
||||
original_stdout = None
|
||||
original_stderr = None
|
||||
|
||||
# 日志设置
|
||||
log_dir = Path(PROJECT_ROOT) / "test_results"
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_path = log_dir / f"simple_pipeline_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
|
||||
|
||||
try:
|
||||
# 重定向输出到日志文件
|
||||
log_file = open(log_path, "w", encoding="utf-8")
|
||||
|
||||
class Tee:
|
||||
def __init__(self, *streams):
|
||||
self._streams = streams
|
||||
def write(self, data):
|
||||
for stream in self._streams:
|
||||
stream.write(data)
|
||||
stream.flush()
|
||||
return len(data)
|
||||
def flush(self):
|
||||
for stream in self._streams:
|
||||
stream.flush()
|
||||
|
||||
original_stdout = sys.stdout
|
||||
original_stderr = sys.stderr
|
||||
sys.stdout = Tee(original_stdout, log_file)
|
||||
sys.stderr = Tee(original_stderr, log_file)
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("🚀 Creating Pipeline...")
|
||||
print(f"📝 Log file: {log_path}")
|
||||
print("=" * 60)
|
||||
|
||||
# 一行创建完整 Pipeline
|
||||
pipeline = await create_jsonl_to_api_pipeline(**CONFIG)
|
||||
|
||||
print(f"✅ Pipeline created!")
|
||||
print(f" - Node types: {len(pipeline.node_types)}")
|
||||
print(f" - Strategies: {len(pipeline.strategies)}")
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("🏃 Running Pipeline...")
|
||||
print("=" * 60)
|
||||
|
||||
# 执行同步(print_summary 已整合到 stage4_post_check 中)
|
||||
stats = await pipeline.run()
|
||||
|
||||
print("\n✅ Done!")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n\n⚠️ KeyboardInterrupt received, shutting down gracefully...")
|
||||
interrupted = True
|
||||
except Exception as e:
|
||||
print(f"\n\n❌ Error: {type(e).__name__}: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
if pipeline is not None:
|
||||
print("🧹 Cleaning up resources...")
|
||||
await pipeline.close()
|
||||
print("✅ Resources cleaned up")
|
||||
|
||||
# 恢复原始输出
|
||||
if original_stdout is not None:
|
||||
sys.stdout = original_stdout
|
||||
if original_stderr is not None:
|
||||
sys.stderr = original_stderr
|
||||
if log_file is not None:
|
||||
log_file.close()
|
||||
if not interrupted:
|
||||
print(f"\n📝 Log saved to: {log_path}")
|
||||
|
||||
return not interrupted # 返回 False 表示被中断
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
success = asyncio.run(main())
|
||||
sys.exit(0 if success else 130) # 130 是标准的 Ctrl+C 退出码
|
||||
except KeyboardInterrupt:
|
||||
# asyncio.run() 可能会重新抛出 KeyboardInterrupt
|
||||
print("\n⚠️ Program interrupted")
|
||||
sys.exit(130)
|
||||
@@ -1,143 +0,0 @@
|
||||
"""
|
||||
run_full_sync_pipeline_simple_sm.py
|
||||
|
||||
独立执行:基于 sync_state_machine 的 JSONL -> API Pipeline。
|
||||
用途:保留线上接口参数,直接执行重构后的推送链路(非 pytest)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Dict, List
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
import sync_state_machine.domain # noqa: F401, E402
|
||||
from sync_state_machine.pipeline import create_jsonl_to_api_pipeline # noqa: E402
|
||||
|
||||
|
||||
class SimpleSmConfig(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", validate_assignment=True)
|
||||
|
||||
local_jsonl_dir: str = str(PROJECT_ROOT / "filtered_datasource" / "datasource" / "ecm-2025-12-02")
|
||||
persistence_db: str = str(PROJECT_ROOT / "test_results" / "sync_state_machine" / "simple_pipeline_sm.db")
|
||||
api_base_url: str = "http://localhost:8000/api/third/v2"
|
||||
api_uid: str = "6b18c681-92d5-4c7f-802c-df413b6504c8"
|
||||
api_secret: str = "1a28Az5dBD1Gc3URpmHwiR0XfzW7UtScQnqJQzgpzdg"
|
||||
|
||||
node_types: List[str] = Field(default_factory=lambda: [
|
||||
"project",
|
||||
"contract",
|
||||
"contract_settlement",
|
||||
])
|
||||
target_project_ids: List[str] = Field(default_factory=lambda: ["f3e02e84-e81c-4aa6-88d4-f5aa108c8227"])
|
||||
project_id_bindings: Dict[str, str] = Field(
|
||||
default_factory=lambda: {
|
||||
"f3e02e84-e81c-4aa6-88d4-f5aa108c8227": "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
|
||||
}
|
||||
)
|
||||
|
||||
skip_sync_types: List[str] = Field(default_factory=lambda: ["supplier"])
|
||||
wipe_persistence_on_start: bool = False
|
||||
enable_persistence: bool = True
|
||||
api_debug: bool = True
|
||||
poll_max_retries: int = 1
|
||||
poll_interval: float = 2.0
|
||||
|
||||
|
||||
def _load_dotenv() -> Dict[str, str]:
|
||||
dotenv_path = PROJECT_ROOT / ".env"
|
||||
if not dotenv_path.exists():
|
||||
return {}
|
||||
|
||||
values: Dict[str, str] = {}
|
||||
for raw_line in dotenv_path.read_text(encoding="utf-8").splitlines():
|
||||
line = raw_line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, value = line.split("=", 1)
|
||||
values[key.strip()] = value.strip().strip('"').strip("'")
|
||||
return values
|
||||
|
||||
|
||||
def build_config() -> SimpleSmConfig:
|
||||
cfg = SimpleSmConfig()
|
||||
env = _load_dotenv()
|
||||
|
||||
cfg.api_base_url = os.getenv("SPS_API_BASE_URL", env.get("SPS_API_BASE_URL", cfg.api_base_url))
|
||||
cfg.api_uid = os.getenv("SPS_API_UID", env.get("SPS_API_UID", cfg.api_uid))
|
||||
cfg.api_secret = os.getenv("SPS_API_SECRET", env.get("SPS_API_SECRET", cfg.api_secret))
|
||||
return cfg
|
||||
|
||||
|
||||
def _print_config_snapshot(config: SimpleSmConfig) -> None:
|
||||
masked_secret = "***" if config.api_secret else ""
|
||||
snapshot = {
|
||||
"local_jsonl_dir": config.local_jsonl_dir,
|
||||
"persistence_db": config.persistence_db,
|
||||
"api_base_url": config.api_base_url,
|
||||
"api_uid": config.api_uid,
|
||||
"api_secret": masked_secret,
|
||||
"node_types": config.node_types,
|
||||
"target_project_ids": config.target_project_ids,
|
||||
"project_id_bindings": config.project_id_bindings,
|
||||
"skip_sync_types": config.skip_sync_types,
|
||||
"wipe_persistence_on_start": config.wipe_persistence_on_start,
|
||||
"enable_persistence": config.enable_persistence,
|
||||
"api_debug": config.api_debug,
|
||||
"poll_max_retries": config.poll_max_retries,
|
||||
"poll_interval": config.poll_interval,
|
||||
}
|
||||
print(f"🔧 Resolved Config: {snapshot}")
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
pipeline = None
|
||||
log_path = None
|
||||
config = build_config()
|
||||
|
||||
try:
|
||||
log_dir = PROJECT_ROOT / "test_results" / "sync_state_machine"
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_path = log_dir / f"simple_pipeline_sm_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
|
||||
|
||||
print("\n" + "=" * 80)
|
||||
print("🚀 Creating sync_state_machine JSONL -> API pipeline")
|
||||
print(f"📝 Log file: {log_path}")
|
||||
_print_config_snapshot(config)
|
||||
print("=" * 80)
|
||||
|
||||
pipeline = await create_jsonl_to_api_pipeline(
|
||||
**config.model_dump(),
|
||||
initialize_logger=True,
|
||||
logger_file_path=str(log_path),
|
||||
)
|
||||
|
||||
stats = await pipeline.run()
|
||||
print(f"\n✅ Pipeline completed: stats={stats}")
|
||||
print(f"🗃️ Persistence DB: {config.persistence_db}")
|
||||
return 0
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted by user")
|
||||
return 130
|
||||
except Exception as exc:
|
||||
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return 1
|
||||
finally:
|
||||
if pipeline is not None:
|
||||
await pipeline.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(asyncio.run(main()))
|
||||
@@ -15,6 +15,7 @@ if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
import sync_state_machine.domain # noqa: F401, E402
|
||||
from sync_state_machine.logging import configure_app_logging, get_logger # noqa: E402
|
||||
from sync_state_machine.pipeline import ( # noqa: E402
|
||||
build_config,
|
||||
run_pipeline_from_config,
|
||||
@@ -23,15 +24,17 @@ from sync_state_machine.config import load_named_preset_overrides # noqa: E402
|
||||
|
||||
|
||||
PRESET_NAME = "jsonl_sm"
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
try:
|
||||
overrides, preset_file, used_default = load_named_preset_overrides(PROJECT_ROOT, PRESET_NAME)
|
||||
source_label = "default" if used_default else "custom"
|
||||
print(f"🧩 Loaded preset ({source_label}): {preset_file}")
|
||||
|
||||
config = build_config(project_root=PROJECT_ROOT, overrides=overrides)
|
||||
configure_app_logging(config.logging, replace_handlers=True)
|
||||
logger.info("🧩 Loaded preset (%s): %s", source_label, preset_file)
|
||||
await run_pipeline_from_config(
|
||||
config,
|
||||
title="sync_state_machine JSONL -> JSONL pipeline",
|
||||
@@ -40,12 +43,10 @@ async def main() -> int:
|
||||
return 0
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted by user")
|
||||
logger.warning("⚠️ Interrupted by user")
|
||||
return 130
|
||||
except Exception as exc:
|
||||
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
logger.exception("❌ Pipeline failed: %s: %s", type(exc).__name__, exc)
|
||||
return 1
|
||||
|
||||
|
||||
|
||||
@@ -15,6 +15,7 @@ if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
import sync_state_machine.domain # noqa: F401, E402
|
||||
from sync_state_machine.logging import configure_app_logging, get_logger # noqa: E402
|
||||
from sync_state_machine.pipeline import ( # noqa: E402
|
||||
build_config,
|
||||
run_pipeline_from_config,
|
||||
@@ -23,15 +24,17 @@ from sync_state_machine.config import load_named_preset_overrides # noqa: E402
|
||||
|
||||
|
||||
PRESET_NAME = "simple_sm"
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
try:
|
||||
overrides, preset_file, used_default = load_named_preset_overrides(PROJECT_ROOT, PRESET_NAME)
|
||||
source_label = "default" if used_default else "custom"
|
||||
print(f"🧩 Loaded preset ({source_label}): {preset_file}")
|
||||
|
||||
config = build_config(project_root=PROJECT_ROOT, overrides=overrides)
|
||||
configure_app_logging(config.logging, replace_handlers=True)
|
||||
logger.info("🧩 Loaded preset (%s): %s", source_label, preset_file)
|
||||
await run_pipeline_from_config(
|
||||
config,
|
||||
title="sync_state_machine JSONL -> API pipeline",
|
||||
@@ -40,12 +43,10 @@ async def main() -> int:
|
||||
return 0
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted by user")
|
||||
logger.warning("⚠️ Interrupted by user")
|
||||
return 130
|
||||
except Exception as exc:
|
||||
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
logger.exception("❌ Pipeline failed: %s: %s", type(exc).__name__, exc)
|
||||
return 1
|
||||
|
||||
|
||||
|
||||
@@ -15,22 +15,25 @@ from .common.registry import DomainRegistry
|
||||
from .config import (
|
||||
ApiDataSourceConfig,
|
||||
BaseDataSourceConfig,
|
||||
DbDataSourceConfig,
|
||||
JsonlDataSourceConfig,
|
||||
PipelineRunConfig,
|
||||
build_config,
|
||||
build_config_from_file,
|
||||
build_default_config,
|
||||
get_datasource_config_model,
|
||||
get_datasource_factory,
|
||||
list_registered_datasource_types,
|
||||
load_named_preset_overrides,
|
||||
load_overrides_from_file,
|
||||
MultiProjectRunConfig,
|
||||
build_multi_project_config_from_file,
|
||||
is_multi_project_payload,
|
||||
register_datasource_config_model,
|
||||
register_datasource_factory,
|
||||
register_datasource_type,
|
||||
)
|
||||
from .datasource import ApiClient, ApiDataSource, BaseApiHandler, BaseDbHandler, BaseJsonlHandler, DbDataSource, JsonlDataSource
|
||||
from .pipeline import FullSyncPipeline, create_pipeline_from_config, run_pipeline_from_config
|
||||
from .datasource import ApiClient, ApiDataSource, BaseApiHandler, BaseJsonlHandler, JsonlDataSource
|
||||
from .pipeline import FullSyncPipeline, MultiProjectPipeline, create_pipeline_from_config, run_pipeline_from_config, run_profile_from_file
|
||||
|
||||
try:
|
||||
__version__ = version("ecm-sync-system")
|
||||
@@ -40,7 +43,7 @@ except PackageNotFoundError:
|
||||
|
||||
def ensure_domain_registry_loaded() -> None:
|
||||
"""Ensure built-in domain registrations are imported."""
|
||||
_domain # keep module import for side effects
|
||||
_ = _domain
|
||||
|
||||
|
||||
__all__ = [
|
||||
@@ -50,26 +53,29 @@ __all__ = [
|
||||
"PipelineRunConfig",
|
||||
"ApiDataSourceConfig",
|
||||
"BaseDataSourceConfig",
|
||||
"DbDataSourceConfig",
|
||||
"JsonlDataSourceConfig",
|
||||
"build_config",
|
||||
"build_config_from_file",
|
||||
"build_default_config",
|
||||
"get_datasource_config_model",
|
||||
"get_datasource_factory",
|
||||
"list_registered_datasource_types",
|
||||
"load_overrides_from_file",
|
||||
"load_named_preset_overrides",
|
||||
"MultiProjectRunConfig",
|
||||
"build_multi_project_config_from_file",
|
||||
"is_multi_project_payload",
|
||||
"register_datasource_config_model",
|
||||
"register_datasource_factory",
|
||||
"register_datasource_type",
|
||||
"ApiClient",
|
||||
"ApiDataSource",
|
||||
"DbDataSource",
|
||||
"JsonlDataSource",
|
||||
"BaseApiHandler",
|
||||
"BaseDbHandler",
|
||||
"BaseJsonlHandler",
|
||||
"FullSyncPipeline",
|
||||
"MultiProjectPipeline",
|
||||
"create_pipeline_from_config",
|
||||
"run_pipeline_from_config",
|
||||
"run_profile_from_file",
|
||||
]
|
||||
|
||||
+12
-12
@@ -5,8 +5,11 @@ import asyncio
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from .config import build_config, load_overrides_from_file
|
||||
from .pipeline import run_pipeline_from_config
|
||||
from .logging import get_logger, init_app_logger
|
||||
from .pipeline import run_profile_from_file
|
||||
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def package_project_root() -> Path:
|
||||
@@ -21,6 +24,7 @@ def resolve_config_path(raw_path: str, *, project_root: Path) -> Path:
|
||||
|
||||
|
||||
async def _main() -> int:
|
||||
init_app_logger(replace_handlers=True)
|
||||
project_root = package_project_root()
|
||||
|
||||
parser = argparse.ArgumentParser(description="Run sync pipeline from config override file")
|
||||
@@ -35,14 +39,10 @@ async def _main() -> int:
|
||||
if not cfg_path.exists() or not cfg_path.is_file():
|
||||
raise FileNotFoundError(f"Config file not found: {cfg_path}")
|
||||
|
||||
overrides = load_overrides_from_file(cfg_path, project_root)
|
||||
config = build_config(project_root=project_root, overrides=overrides)
|
||||
|
||||
mode = f"{config.local_datasource.type}->{config.remote_datasource.type}"
|
||||
print(f"🧩 Loaded config file: {cfg_path}")
|
||||
await run_pipeline_from_config(
|
||||
config,
|
||||
title=f"sync_state_machine pipeline ({mode})",
|
||||
logger.info("🧩 Loaded config file: %s", cfg_path)
|
||||
await run_profile_from_file(
|
||||
project_root=project_root,
|
||||
config_path=cfg_path,
|
||||
print_summary=True,
|
||||
)
|
||||
return 0
|
||||
@@ -52,10 +52,10 @@ def main() -> int:
|
||||
try:
|
||||
return asyncio.run(_main())
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted by user")
|
||||
logger.warning("⚠️ Interrupted by user")
|
||||
return 130
|
||||
except Exception as exc:
|
||||
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
|
||||
logger.exception("❌ Pipeline failed: %s: %s", type(exc).__name__, exc)
|
||||
return 1
|
||||
finally:
|
||||
logging.shutdown()
|
||||
|
||||
@@ -1,6 +1,12 @@
|
||||
from .types import SyncAction, SyncStatus, BindingStatus
|
||||
from .sync_node import SyncNode
|
||||
from .persistence import PersistenceBackend
|
||||
from .persistence import (
|
||||
PersistenceBackend,
|
||||
create_persistence_backend,
|
||||
get_registered_persistence_backends,
|
||||
register_persistence_backend,
|
||||
)
|
||||
from .persistence_backends import SQLAlchemyPersistenceBackend, SQLitePersistenceBackend
|
||||
from .collection import DataCollection
|
||||
from .binding import BindingManager
|
||||
|
||||
@@ -10,6 +16,11 @@ __all__ = [
|
||||
"BindingStatus",
|
||||
"SyncNode",
|
||||
"PersistenceBackend",
|
||||
"SQLitePersistenceBackend",
|
||||
"SQLAlchemyPersistenceBackend",
|
||||
"create_persistence_backend",
|
||||
"register_persistence_backend",
|
||||
"get_registered_persistence_backends",
|
||||
"DataCollection",
|
||||
"BindingManager",
|
||||
]
|
||||
|
||||
@@ -1,10 +1,13 @@
|
||||
import json
|
||||
import logging
|
||||
from typing import Optional, Dict, Any, List
|
||||
|
||||
from ..logging import get_logger, get_scope_display_name
|
||||
from .persistence import PersistenceBackend
|
||||
from .persistence_service import ScopedPersistenceView
|
||||
from .types import BindingStatus, BindingRecord
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class BindingManager:
|
||||
@@ -12,13 +15,53 @@ class BindingManager:
|
||||
BindingManager 负责维护不同 DataCollection 间节点的绑定关系。
|
||||
所有查询均在内存中进行,持久化仅作为备份。
|
||||
"""
|
||||
def __init__(self, persistence: PersistenceBackend, auto_persist: bool = True):
|
||||
def __init__(
|
||||
self,
|
||||
persistence: PersistenceBackend | ScopedPersistenceView,
|
||||
auto_persist: bool = False,
|
||||
scope: str = "global",
|
||||
global_fallback_manager: Optional["BindingManager"] = None,
|
||||
):
|
||||
self.persistence = persistence
|
||||
self.auto_persist = auto_persist
|
||||
self.scope = scope
|
||||
self._global_fallback_manager = global_fallback_manager
|
||||
# { node_type: { local_id: remote_id } }
|
||||
self._bindings: Dict[str, Dict[str, Optional[str]]] = {}
|
||||
self._deleted: Dict[str, set[str]] = {}
|
||||
|
||||
def set_global_fallback_manager(self, manager: Optional["BindingManager"]) -> None:
|
||||
self._global_fallback_manager = manager
|
||||
|
||||
async def _persistence_save_binding(self, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.save_binding(node_type, local_id, payload_dict)
|
||||
return
|
||||
await self.persistence.save_binding(self.scope, node_type, local_id, payload_dict)
|
||||
|
||||
async def _persistence_delete_binding(self, node_type: str, local_id: str) -> None:
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.delete_binding(node_type, local_id)
|
||||
return
|
||||
await self.persistence.delete_binding(self.scope, node_type, local_id)
|
||||
|
||||
async def _persistence_delete_bindings_bulk(self, node_type: str, local_ids: List[str]) -> None:
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.delete_bindings_bulk(node_type, local_ids)
|
||||
return
|
||||
await self.persistence.delete_bindings_bulk(self.scope, node_type, local_ids)
|
||||
|
||||
async def _persistence_save_bindings_bulk(self, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.save_bindings_bulk(node_type, bindings)
|
||||
return
|
||||
await self.persistence.save_bindings_bulk(self.scope, node_type, bindings)
|
||||
|
||||
async def _persistence_load_bindings(self, node_type: str) -> List[Dict[str, Any]]:
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
return await self.persistence.load_bindings(node_type)
|
||||
return await self.persistence.load_bindings(self.scope, node_type)
|
||||
|
||||
def _get_type_bindings(self, node_type: str) -> Dict[str, Optional[str]]:
|
||||
if node_type not in self._bindings:
|
||||
self._bindings[node_type] = {}
|
||||
@@ -55,7 +98,7 @@ class BindingManager:
|
||||
logger.debug(log_line)
|
||||
|
||||
if self.auto_persist:
|
||||
await self.persistence.save_binding(node_type, local_id, {"remote_id": remote_id})
|
||||
await self._persistence_save_binding(node_type, local_id, {"remote_id": remote_id})
|
||||
|
||||
async def unbind(self, node_type: str, local_id: str, manual: bool = False):
|
||||
"""
|
||||
@@ -80,14 +123,19 @@ class BindingManager:
|
||||
)
|
||||
|
||||
if self.auto_persist:
|
||||
await self.persistence.delete_binding(node_type, local_id)
|
||||
await self._persistence_delete_binding(node_type, local_id)
|
||||
else:
|
||||
self._get_deleted(node_type).add(local_id)
|
||||
|
||||
async def get_remote_id(self, node_type: str, local_id: str) -> Optional[str]:
|
||||
"""查询本地节点对应的远程 ID"""
|
||||
bindings = self._get_type_bindings(node_type)
|
||||
return bindings.get(local_id)
|
||||
remote_id = bindings.get(local_id)
|
||||
if remote_id is not None or local_id in bindings:
|
||||
return remote_id
|
||||
if self._global_fallback_manager is None:
|
||||
return None
|
||||
return await self._global_fallback_manager.get_remote_id(node_type, local_id)
|
||||
|
||||
async def get_local_id(self, node_type: str, remote_id: str) -> Optional[str]:
|
||||
"""查询远程节点对应的本地 ID (内存遍历)"""
|
||||
@@ -95,11 +143,13 @@ class BindingManager:
|
||||
for lid, rid in bindings.items():
|
||||
if rid == remote_id:
|
||||
return lid
|
||||
return None
|
||||
if self._global_fallback_manager is None:
|
||||
return None
|
||||
return await self._global_fallback_manager.get_local_id(node_type, remote_id)
|
||||
|
||||
async def get_record(self, node_type: str, local_id: str) -> Optional[BindingRecord]:
|
||||
"""获取结构化绑定记录"""
|
||||
remote_id = self._get_type_bindings(node_type).get(local_id)
|
||||
remote_id = await self.get_remote_id(node_type, local_id)
|
||||
if remote_id is None:
|
||||
return None
|
||||
return BindingRecord(
|
||||
@@ -119,25 +169,71 @@ class BindingManager:
|
||||
|
||||
async def load_from_persistence(self, node_type: str):
|
||||
"""从持久化层加载特定类型的绑定关系到内存"""
|
||||
raw_bindings = await self.persistence.load_bindings(node_type)
|
||||
raw_bindings = await self._persistence_load_bindings(node_type)
|
||||
bindings = self._get_type_bindings(node_type)
|
||||
bindings.clear()
|
||||
self._get_deleted(node_type).clear()
|
||||
for rb in raw_bindings:
|
||||
payload = rb.get("payload") or {}
|
||||
bindings[rb["local_id"]] = payload.get("remote_id")
|
||||
|
||||
async def persist(self) -> None:
|
||||
def import_bindings(self, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
|
||||
"""导入既有绑定到当前 manager,仅写内存,不触发持久化。"""
|
||||
target = self._get_type_bindings(node_type)
|
||||
target.clear()
|
||||
target.update(dict(bindings))
|
||||
self._get_deleted(node_type).clear()
|
||||
|
||||
async def prune_missing_nodes(
|
||||
self,
|
||||
*,
|
||||
local_collection,
|
||||
remote_collection,
|
||||
node_types: Optional[List[str]] = None,
|
||||
) -> int:
|
||||
target_types = list(node_types) if node_types is not None else list(self._bindings.keys())
|
||||
deleted_count = 0
|
||||
|
||||
for node_type in target_types:
|
||||
bindings = self._get_type_bindings(node_type)
|
||||
if not bindings:
|
||||
continue
|
||||
|
||||
valid_local_ids = {node.node_id for node in local_collection.filter(node_type=node_type)}
|
||||
valid_remote_ids = {node.node_id for node in remote_collection.filter(node_type=node_type)}
|
||||
|
||||
stale_local_ids = [
|
||||
local_id
|
||||
for local_id, remote_id in bindings.items()
|
||||
if local_id not in valid_local_ids or (remote_id is not None and remote_id not in valid_remote_ids)
|
||||
]
|
||||
|
||||
for local_id in stale_local_ids:
|
||||
del bindings[local_id]
|
||||
deleted_count += 1
|
||||
if self.auto_persist:
|
||||
await self._persistence_delete_binding(node_type, local_id)
|
||||
else:
|
||||
self._get_deleted(node_type).add(local_id)
|
||||
|
||||
return deleted_count
|
||||
|
||||
async def persist(self, *, exclude_node_types: Optional[List[str]] = None) -> None:
|
||||
"""将所有绑定关系写回持久化层"""
|
||||
if not self._bindings and not self._deleted:
|
||||
logger.info("🔍 [BindingManager.persist] 跳过:无绑定变更")
|
||||
return
|
||||
|
||||
excluded = set(node_type for node_type in (exclude_node_types or []) if node_type)
|
||||
|
||||
type_count = 0
|
||||
saved_total = 0
|
||||
deleted_total = 0
|
||||
touched_types: List[str] = []
|
||||
|
||||
for node_type, bindings in self._bindings.items():
|
||||
if node_type in excluded:
|
||||
continue
|
||||
deleted = list(self._get_deleted(node_type))
|
||||
binding_count = len(bindings)
|
||||
deleted_count = len(deleted)
|
||||
@@ -149,13 +245,19 @@ class BindingManager:
|
||||
touched_types.append(node_type)
|
||||
|
||||
if deleted:
|
||||
await self.persistence.delete_bindings_bulk(node_type, deleted)
|
||||
await self._persistence_delete_bindings_bulk(node_type, deleted)
|
||||
self._get_deleted(node_type).clear()
|
||||
|
||||
if bindings:
|
||||
await self.persistence.save_bindings_bulk(node_type, bindings)
|
||||
await self._persistence_save_bindings_bulk(node_type, bindings)
|
||||
|
||||
touched_text = ",".join(touched_types) if touched_types else "none"
|
||||
logger.info(
|
||||
f"🔍 [BindingManager.persist] 完成: types={type_count}, saved={saved_total}, deleted={deleted_total}, touched=[{touched_text}]"
|
||||
f"🔍 [BindingManager.persist] 完成: scope={get_scope_display_name(self.scope)}, types={type_count}, saved={saved_total}, deleted={deleted_total}, touched=[{touched_text}]"
|
||||
)
|
||||
|
||||
async def unload(self, *, node_types: Optional[List[str]] = None) -> None:
|
||||
target_types = list(node_types) if node_types is not None else list(self._bindings.keys())
|
||||
for node_type in target_types:
|
||||
self._bindings.pop(node_type, None)
|
||||
self._deleted.pop(node_type, None)
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
import uuid
|
||||
import copy
|
||||
from enum import Enum
|
||||
from typing import Dict, List, Optional, Any, Set, Callable, Literal
|
||||
from .sync_node import SyncNode
|
||||
from .types import SyncStatus
|
||||
from .persistence import PersistenceBackend
|
||||
from .persistence_service import ScopedPersistenceView
|
||||
|
||||
|
||||
class DataCollection:
|
||||
@@ -22,17 +24,23 @@ class DataCollection:
|
||||
def __init__(
|
||||
self,
|
||||
collection_id: str,
|
||||
persistence: Optional[PersistenceBackend] = None,
|
||||
scope: str = "global",
|
||||
persistence: Optional[PersistenceBackend | ScopedPersistenceView] = None,
|
||||
auto_persist: bool = False,
|
||||
sm_runtime: Optional[Any] = None,
|
||||
global_fallback_collection: Optional["DataCollection"] = None,
|
||||
):
|
||||
self.collection_id = collection_id
|
||||
self.scope = scope
|
||||
self.persistence = persistence
|
||||
self.auto_persist = auto_persist
|
||||
self._sm_runtime = sm_runtime
|
||||
self._global_fallback_collection = global_fallback_collection
|
||||
self._nodes: Dict[str, SyncNode] = {} # 内存缓存
|
||||
# data_id 全局唯一(按 collection 维度),用于 O(1) 反查 node_id
|
||||
self._data_id_to_node_id: Dict[str, str] = {}
|
||||
# node_id -> data_id 反向索引,用于 O(1) 维护 data_id 更新
|
||||
self._node_id_to_data_id: Dict[str, str] = {}
|
||||
# auto_persist=False 时,delete() 不会立即落库;这里记录删除集合,persist() 时统一写回
|
||||
self._deleted_node_ids: set[str] = set()
|
||||
|
||||
@@ -48,6 +56,83 @@ class DataCollection:
|
||||
"""设置 Collection 默认状态机 runtime。"""
|
||||
self._sm_runtime = runtime
|
||||
|
||||
def set_global_fallback_collection(self, collection: Optional["DataCollection"]) -> None:
|
||||
self._global_fallback_collection = collection
|
||||
|
||||
def _get_global_fallback(self) -> Optional["DataCollection"]:
|
||||
return self._global_fallback_collection
|
||||
|
||||
async def _persistence_save_node(self, node: SyncNode) -> None:
|
||||
if not self.persistence:
|
||||
return
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.save_node(self.collection_id, node)
|
||||
return
|
||||
await self.persistence.save_node(self.collection_id, self.scope, node)
|
||||
|
||||
async def _persistence_delete_node(self, node_id: str) -> None:
|
||||
if not self.persistence:
|
||||
return
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.delete_node(self.collection_id, node_id)
|
||||
return
|
||||
await self.persistence.delete_node(self.collection_id, self.scope, node_id)
|
||||
|
||||
async def _persistence_delete_nodes_bulk(self, node_ids: List[str]) -> None:
|
||||
if not self.persistence:
|
||||
return
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.delete_nodes_bulk(self.collection_id, node_ids)
|
||||
return
|
||||
await self.persistence.delete_nodes_bulk(self.collection_id, self.scope, node_ids)
|
||||
|
||||
async def _persistence_save_nodes_bulk(self, nodes: List[SyncNode]) -> None:
|
||||
if not self.persistence:
|
||||
return
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.save_nodes_bulk(self.collection_id, nodes)
|
||||
return
|
||||
await self.persistence.save_nodes_bulk(self.collection_id, self.scope, nodes)
|
||||
|
||||
async def _persistence_load_nodes(
|
||||
self,
|
||||
*,
|
||||
exclude_node_types: Optional[List[str]] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
if not self.persistence:
|
||||
return []
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
return await self.persistence.load_nodes(
|
||||
self.collection_id,
|
||||
exclude_node_types=exclude_node_types,
|
||||
)
|
||||
return await self.persistence.load_nodes(
|
||||
self.collection_id,
|
||||
self.scope,
|
||||
exclude_node_types=exclude_node_types,
|
||||
)
|
||||
|
||||
async def _persistence_load_bind_data_id_overrides(self, node_type: str) -> Dict[str, Optional[str]]:
|
||||
if not self.persistence:
|
||||
return {}
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
return await self.persistence.load_bind_data_id_overrides(node_type)
|
||||
return await self.persistence.load_bind_data_id_overrides(node_type, self.scope)
|
||||
|
||||
def _remove_data_id_index(self, node_id: str, data_id: Optional[str] = None) -> None:
|
||||
target_data_id = data_id
|
||||
if target_data_id is None:
|
||||
target_data_id = self._node_id_to_data_id.pop(node_id, None)
|
||||
else:
|
||||
self._node_id_to_data_id.pop(node_id, None)
|
||||
|
||||
if target_data_id and self._data_id_to_node_id.get(target_data_id) == node_id:
|
||||
self._data_id_to_node_id.pop(target_data_id, None)
|
||||
|
||||
def _set_data_id_index(self, node_id: str, data_id: str) -> None:
|
||||
self._data_id_to_node_id[data_id] = node_id
|
||||
self._node_id_to_data_id[node_id] = data_id
|
||||
|
||||
async def add(self, node: SyncNode):
|
||||
# 检查 node_id 唯一性
|
||||
if node.node_id in self._nodes:
|
||||
@@ -62,18 +147,17 @@ class DataCollection:
|
||||
raise ValueError(
|
||||
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
|
||||
)
|
||||
self._data_id_to_node_id[node.data_id] = node.node_id
|
||||
self._set_data_id_index(node.node_id, node.data_id)
|
||||
|
||||
self._nodes[node.node_id] = node
|
||||
self._deleted_node_ids.discard(node.node_id)
|
||||
if self.persistence and self.auto_persist:
|
||||
await self.persistence.save_node(self.collection_id, node)
|
||||
await self._persistence_save_node(node)
|
||||
|
||||
async def update(self, node: SyncNode):
|
||||
old_node = self._nodes.get(node.node_id)
|
||||
if old_node and old_node.data_id and old_node.data_id != "" and old_node.data_id != node.data_id:
|
||||
# node_id 不变但 data_id 变更,清理旧索引
|
||||
self._data_id_to_node_id.pop(old_node.data_id, None)
|
||||
old_data_id = self._node_id_to_data_id.get(node.node_id)
|
||||
if old_data_id and old_data_id != node.data_id:
|
||||
self._remove_data_id_index(node.node_id, old_data_id)
|
||||
|
||||
if node.data_id and node.data_id != "":
|
||||
existing_node_id = self._data_id_to_node_id.get(node.data_id)
|
||||
@@ -81,27 +165,33 @@ class DataCollection:
|
||||
raise ValueError(
|
||||
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
|
||||
)
|
||||
self._data_id_to_node_id[node.data_id] = node.node_id
|
||||
self._set_data_id_index(node.node_id, node.data_id)
|
||||
else:
|
||||
self._node_id_to_data_id.pop(node.node_id, None)
|
||||
self._nodes[node.node_id] = node
|
||||
self._deleted_node_ids.discard(node.node_id)
|
||||
if self.persistence and self.auto_persist:
|
||||
await self.persistence.save_node(self.collection_id, node)
|
||||
await self._persistence_save_node(node)
|
||||
|
||||
async def delete(self, node_id: str):
|
||||
old_node = self._nodes.get(node_id)
|
||||
if old_node and old_node.data_id and old_node.data_id != "":
|
||||
self._data_id_to_node_id.pop(old_node.data_id, None)
|
||||
self._remove_data_id_index(node_id)
|
||||
|
||||
if node_id in self._nodes:
|
||||
del self._nodes[node_id]
|
||||
if self.persistence:
|
||||
if self.auto_persist:
|
||||
await self.persistence.delete_node(self.collection_id, node_id)
|
||||
await self._persistence_delete_node(node_id)
|
||||
else:
|
||||
self._deleted_node_ids.add(node_id)
|
||||
|
||||
def get(self, node_id: str) -> Optional[SyncNode]:
|
||||
return self._nodes.get(node_id)
|
||||
node = self._nodes.get(node_id)
|
||||
if node is not None:
|
||||
return node
|
||||
fallback = self._get_global_fallback()
|
||||
if fallback is None:
|
||||
return None
|
||||
return fallback.get(node_id)
|
||||
|
||||
def get_by_node_id(self, node_id: str) -> Optional[SyncNode]:
|
||||
"""别名方法,与 get() 功能相同"""
|
||||
@@ -113,14 +203,23 @@ class DataCollection:
|
||||
|
||||
def get_node_id_by_data_id(self, data_id: str) -> Optional[str]:
|
||||
"""按 data_id 反查 node_id(O(1))。"""
|
||||
return self._data_id_to_node_id.get(data_id)
|
||||
node_id = self._data_id_to_node_id.get(data_id)
|
||||
if node_id is not None:
|
||||
return node_id
|
||||
fallback = self._get_global_fallback()
|
||||
if fallback is None:
|
||||
return None
|
||||
return fallback.get_node_id_by_data_id(data_id)
|
||||
|
||||
def get_by_data_id_any(self, data_id: str) -> Optional[SyncNode]:
|
||||
"""按 data_id 查找节点(不要求 node_type)(O(1))。"""
|
||||
node_id = self._data_id_to_node_id.get(data_id)
|
||||
if not node_id:
|
||||
if node_id:
|
||||
return self._nodes.get(node_id)
|
||||
fallback = self._get_global_fallback()
|
||||
if fallback is None:
|
||||
return None
|
||||
return self._nodes.get(node_id)
|
||||
return fallback.get_by_data_id_any(data_id)
|
||||
|
||||
def get_by_data_id(self, node_type: str, data_id: str) -> Optional[SyncNode]:
|
||||
"""按 node_type 和 data_id 查找节点"""
|
||||
@@ -359,7 +458,7 @@ class DataCollection:
|
||||
- data/origin_data 保留(由 DataSource 覆盖)
|
||||
|
||||
**初始化归并(E01)**:
|
||||
- 由 Pipeline 在加载后调用 run_reset() 触发 E01
|
||||
- 由 Pipeline 在加载后调用 `run_phase2_cleanup()` 触发 E01
|
||||
- CREATE 僵尸节点进入 S15 并删除
|
||||
- 其余节点进入 S00(UNCHECKED/NONE/PENDING)
|
||||
"""
|
||||
@@ -368,9 +467,10 @@ class DataCollection:
|
||||
|
||||
self._nodes = {}
|
||||
self._data_id_to_node_id = {}
|
||||
self._node_id_to_data_id = {}
|
||||
self._deleted_node_ids.clear()
|
||||
|
||||
node_dicts = await self.persistence.load_nodes(self.collection_id)
|
||||
node_dicts = await self._persistence_load_nodes()
|
||||
|
||||
from .types import SyncAction, SyncStatus, BindingStatus
|
||||
from .registry import DomainRegistry
|
||||
@@ -469,6 +569,7 @@ class DataCollection:
|
||||
context=context_value,
|
||||
)
|
||||
node.context["_loaded_from_persistence"] = True
|
||||
node.append_log("加载来源: persistence 恢复节点")
|
||||
except Exception:
|
||||
import logging
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -487,11 +588,11 @@ class DataCollection:
|
||||
raise ValueError(
|
||||
f"Duplicate data_id detected while loading: {node.data_id} already bound to node_id={existing_node_id}"
|
||||
)
|
||||
self._data_id_to_node_id[node.data_id] = node.node_id
|
||||
self._set_data_id_index(node.node_id, node.data_id)
|
||||
|
||||
node_types = {node.node_type for node in self._nodes.values()}
|
||||
for node_type in node_types:
|
||||
override_map = await self.persistence.load_bind_data_id_overrides(node_type)
|
||||
override_map = await self._persistence_load_bind_data_id_overrides(node_type)
|
||||
for node in self.filter(node_type=node_type):
|
||||
bind_data_id = override_map.get(node.node_id)
|
||||
if bind_data_id:
|
||||
@@ -499,24 +600,177 @@ class DataCollection:
|
||||
else:
|
||||
node.context.pop("bind_data_id", None)
|
||||
|
||||
async def persist(self) -> None:
|
||||
async def load_from_persistence_excluding(self, node_types: Optional[List[str]] = None):
|
||||
"""从持久化恢复数据,但跳过指定 node_type。"""
|
||||
if not self.persistence:
|
||||
return
|
||||
|
||||
self._nodes = {}
|
||||
self._data_id_to_node_id = {}
|
||||
self._node_id_to_data_id = {}
|
||||
self._deleted_node_ids.clear()
|
||||
|
||||
node_dicts = await self._persistence_load_nodes(exclude_node_types=node_types)
|
||||
|
||||
from .types import SyncAction, SyncStatus, BindingStatus
|
||||
from .registry import DomainRegistry
|
||||
|
||||
for node_dict in node_dicts:
|
||||
node_type = node_dict["node_type"]
|
||||
node_class = DomainRegistry.get_node_class(node_type)
|
||||
assert issubclass(node_class, SyncNode)
|
||||
|
||||
parse_errors: list[str] = []
|
||||
|
||||
raw_action = node_dict.get("action", SyncAction.NONE)
|
||||
if isinstance(raw_action, str):
|
||||
try:
|
||||
action_value = SyncAction(raw_action)
|
||||
except ValueError:
|
||||
parse_errors.append(f"invalid action={raw_action}")
|
||||
action_value = SyncAction.NONE
|
||||
elif isinstance(raw_action, SyncAction):
|
||||
action_value = raw_action
|
||||
else:
|
||||
if raw_action is None:
|
||||
action_value = SyncAction.NONE
|
||||
else:
|
||||
parse_errors.append(f"invalid action type={type(raw_action).__name__}")
|
||||
action_value = SyncAction.NONE
|
||||
|
||||
raw_status = node_dict.get("status", SyncStatus.PENDING)
|
||||
if isinstance(raw_status, str):
|
||||
try:
|
||||
status_value = SyncStatus(raw_status)
|
||||
except ValueError:
|
||||
parse_errors.append(f"invalid status={raw_status}")
|
||||
status_value = SyncStatus.FAILED
|
||||
elif isinstance(raw_status, SyncStatus):
|
||||
status_value = raw_status
|
||||
else:
|
||||
if raw_status is None:
|
||||
status_value = SyncStatus.PENDING
|
||||
else:
|
||||
parse_errors.append(f"invalid status type={type(raw_status).__name__}")
|
||||
status_value = SyncStatus.FAILED
|
||||
|
||||
raw_binding_status = node_dict.get("binding_status", BindingStatus.UNCHECKED)
|
||||
if isinstance(raw_binding_status, str):
|
||||
try:
|
||||
binding_status_value = BindingStatus(raw_binding_status)
|
||||
except ValueError:
|
||||
parse_errors.append(f"invalid binding_status={raw_binding_status}")
|
||||
binding_status_value = BindingStatus.ABNORMAL
|
||||
elif isinstance(raw_binding_status, BindingStatus):
|
||||
binding_status_value = raw_binding_status
|
||||
else:
|
||||
if raw_binding_status is None:
|
||||
binding_status_value = BindingStatus.UNCHECKED
|
||||
else:
|
||||
parse_errors.append(
|
||||
f"invalid binding_status type={type(raw_binding_status).__name__}"
|
||||
)
|
||||
binding_status_value = BindingStatus.ABNORMAL
|
||||
|
||||
raw_context = node_dict.get("context", {})
|
||||
context_value = raw_context if isinstance(raw_context, dict) else {}
|
||||
error_value = node_dict.get("error")
|
||||
|
||||
if parse_errors:
|
||||
context_value = dict(context_value)
|
||||
context_value["bootstrap_blocked"] = True
|
||||
context_value["bootstrap_block_reason"] = "invalid_persisted_enum"
|
||||
context_value["bootstrap_block_errors"] = parse_errors
|
||||
binding_status_value = BindingStatus.ABNORMAL
|
||||
action_value = SyncAction.NONE
|
||||
status_value = SyncStatus.FAILED
|
||||
details = "; ".join(parse_errors)
|
||||
error_value = f"LOAD_INVALID_ENUM: {details}"
|
||||
|
||||
node = node_class(
|
||||
node_id=node_dict["node_id"],
|
||||
data_id=node_dict.get("data_id", ""),
|
||||
depend_ids=node_dict.get("depend_ids", []),
|
||||
data=node_dict.get("data"),
|
||||
origin_data=node_dict.get("origin_data"),
|
||||
action=action_value,
|
||||
status=status_value,
|
||||
binding_status=binding_status_value,
|
||||
error=error_value,
|
||||
sync_log=None,
|
||||
context=context_value,
|
||||
)
|
||||
node.context["_loaded_from_persistence"] = True
|
||||
node.append_log("加载来源: persistence 恢复节点")
|
||||
self._nodes[node.node_id] = node
|
||||
|
||||
if node.data_id and node.data_id != "":
|
||||
existing_node_id = self._data_id_to_node_id.get(node.data_id)
|
||||
if existing_node_id is not None and existing_node_id != node.node_id:
|
||||
raise ValueError(
|
||||
f"Duplicate data_id detected while loading: {node.data_id} already bound to node_id={existing_node_id}"
|
||||
)
|
||||
self._set_data_id_index(node.node_id, node.data_id)
|
||||
|
||||
loaded_node_types = {node.node_type for node in self._nodes.values()}
|
||||
for node_type in loaded_node_types:
|
||||
override_map = await self._persistence_load_bind_data_id_overrides(node_type)
|
||||
for node in self.filter(node_type=node_type):
|
||||
bind_data_id = override_map.get(node.node_id)
|
||||
if bind_data_id:
|
||||
node.context["bind_data_id"] = bind_data_id
|
||||
else:
|
||||
node.context.pop("bind_data_id", None)
|
||||
|
||||
async def import_nodes(self, nodes: List[SyncNode]) -> None:
|
||||
"""导入现有节点到当前 collection,仅写内存,不触发持久化。"""
|
||||
for source_node in nodes:
|
||||
node = copy.deepcopy(source_node)
|
||||
if node.node_id in self._nodes:
|
||||
continue
|
||||
|
||||
if node.data_id and node.data_id != "":
|
||||
existing_node_id = self._data_id_to_node_id.get(node.data_id)
|
||||
if existing_node_id is not None and existing_node_id != node.node_id:
|
||||
raise ValueError(
|
||||
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
|
||||
)
|
||||
self._set_data_id_index(node.node_id, node.data_id)
|
||||
|
||||
self._nodes[node.node_id] = node
|
||||
self._deleted_node_ids.discard(node.node_id)
|
||||
|
||||
async def persist(self, *, exclude_node_types: Optional[List[str]] = None) -> None:
|
||||
"""将当前 collection 全量持久化到后端"""
|
||||
if not self.persistence:
|
||||
return
|
||||
|
||||
# 先落库删除(用于 auto_persist=False 的场景)
|
||||
if self._deleted_node_ids:
|
||||
await self.persistence.delete_nodes_bulk(self.collection_id, list(self._deleted_node_ids))
|
||||
await self._persistence_delete_nodes_bulk(list(self._deleted_node_ids))
|
||||
self._deleted_node_ids.clear()
|
||||
|
||||
if not self._nodes:
|
||||
return
|
||||
|
||||
await self.persistence.save_nodes_bulk(self.collection_id, list(self._nodes.values()))
|
||||
excluded = set(node_type for node_type in (exclude_node_types or []) if node_type)
|
||||
nodes_to_persist = [
|
||||
node for node in self._nodes.values() if node.node_type not in excluded
|
||||
]
|
||||
if not nodes_to_persist:
|
||||
return
|
||||
|
||||
async def filter_by_project_ids(self, target_project_ids: List[str]) -> int:
|
||||
"""按目标项目集合过滤当前 collection,返回删除数量。"""
|
||||
target_set = set(target_project_ids or [])
|
||||
await self._persistence_save_nodes_bulk(nodes_to_persist)
|
||||
|
||||
async def unload(self) -> None:
|
||||
self._nodes.clear()
|
||||
self._data_id_to_node_id.clear()
|
||||
self._node_id_to_data_id.clear()
|
||||
self._deleted_node_ids.clear()
|
||||
|
||||
async def filter_by_project_ids(self, data_id_filter: List[str]) -> int:
|
||||
"""按目标项目 data_id 集合过滤当前 collection,返回删除数量。"""
|
||||
target_set = set(data_id_filter or [])
|
||||
if not target_set:
|
||||
return 0
|
||||
|
||||
|
||||
@@ -1,21 +1,23 @@
|
||||
import aiosqlite
|
||||
import json
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sqlite3
|
||||
from typing import Optional, Dict, List, Any, TYPE_CHECKING
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any, Callable, ClassVar, Dict, List, Optional, TYPE_CHECKING
|
||||
|
||||
from sqlalchemy import create_engine, inspect, text
|
||||
from sqlalchemy.engine import URL, make_url
|
||||
|
||||
from ..config.persist_config import PersistConfig
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .sync_node import SyncNode
|
||||
|
||||
class PersistenceBackend:
|
||||
"""
|
||||
Persistence 负责同步系统运行时数据的持久化存储(aiosqlite 实现)。
|
||||
主要作为 DataCollection 和 BindingManager 的后端存储。
|
||||
"""
|
||||
def __init__(self, db_path: str = ":memory:", backend: str = "sqlite"):
|
||||
self.db_path = db_path
|
||||
self.backend = backend
|
||||
self.conn: Optional[aiosqlite.Connection] = None
|
||||
|
||||
class PersistenceBackend(ABC):
|
||||
"""同步系统持久化契约。"""
|
||||
|
||||
backend_name: ClassVar[str] = "abstract"
|
||||
|
||||
@staticmethod
|
||||
def _is_read_only_sql(sql: str) -> bool:
|
||||
@@ -27,385 +29,284 @@ class PersistenceBackend:
|
||||
cls,
|
||||
*,
|
||||
backend: str,
|
||||
db_path: str,
|
||||
db_path: str | None = None,
|
||||
url: str | None = None,
|
||||
sql: str,
|
||||
limit: int = 200,
|
||||
) -> tuple[List[str], List[Dict[str, Any]]]:
|
||||
if backend != "sqlite":
|
||||
raise NotImplementedError(f"records query backend not supported yet: {backend}")
|
||||
if not cls._is_read_only_sql(sql):
|
||||
raise ValueError("Only read-only SQL is allowed")
|
||||
|
||||
path = db_path
|
||||
if not os.path.exists(path):
|
||||
raise FileNotFoundError(f"DB not found: {path}")
|
||||
if backend == "sqlite":
|
||||
path = db_path
|
||||
if not path:
|
||||
raise ValueError("db_path is required for backend=sqlite")
|
||||
if not os.path.exists(path):
|
||||
raise FileNotFoundError(f"DB not found: {path}")
|
||||
|
||||
with sqlite3.connect(path) as conn:
|
||||
conn.row_factory = sqlite3.Row
|
||||
cursor = conn.execute(sql)
|
||||
rows = cursor.fetchmany(max(1, min(limit, 2000)))
|
||||
columns = [item[0] for item in (cursor.description or [])]
|
||||
data = [{k: row[k] for k in columns} for row in rows]
|
||||
return columns, data
|
||||
with sqlite3.connect(path) as conn:
|
||||
conn.row_factory = sqlite3.Row
|
||||
cursor = conn.execute(sql)
|
||||
rows = cursor.fetchmany(max(1, min(limit, 2000)))
|
||||
columns = [item[0] for item in (cursor.description or [])]
|
||||
data = [{k: row[k] for k in columns} for row in rows]
|
||||
return columns, data
|
||||
|
||||
if backend == "sqlalchemy":
|
||||
if not url:
|
||||
raise ValueError("url is required for backend=sqlalchemy")
|
||||
engine = create_engine(cls._to_sync_sqlalchemy_url(url))
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
result = conn.execute(text(sql))
|
||||
rows = result.fetchmany(max(1, min(limit, 2000)))
|
||||
columns = list(result.keys())
|
||||
data = [{column: row[index] for index, column in enumerate(columns)} for row in rows]
|
||||
return columns, data
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
raise NotImplementedError(f"records query backend not supported yet: {backend}")
|
||||
|
||||
@classmethod
|
||||
def records_summary(
|
||||
cls,
|
||||
*,
|
||||
backend: str,
|
||||
db_path: str,
|
||||
db_path: str | None = None,
|
||||
url: str | None = None,
|
||||
) -> Dict[str, Any]:
|
||||
db_exists = cls._target_exists(backend=backend, db_path=db_path, url=url)
|
||||
summary: Dict[str, Any] = {
|
||||
"backend": backend,
|
||||
"db_path": db_path,
|
||||
"db_exists": os.path.exists(db_path),
|
||||
"url": url,
|
||||
"db_exists": db_exists,
|
||||
"tables": [],
|
||||
"sample_bindings": [],
|
||||
}
|
||||
if not os.path.exists(db_path):
|
||||
return summary
|
||||
if backend != "sqlite":
|
||||
summary["error"] = f"records summary backend not supported yet: {backend}"
|
||||
if not db_exists:
|
||||
return summary
|
||||
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.row_factory = sqlite3.Row
|
||||
tables = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name"
|
||||
).fetchall()
|
||||
table_names = [str(r["name"]) for r in tables]
|
||||
|
||||
for name in table_names:
|
||||
count = conn.execute(f"SELECT COUNT(1) AS c FROM {name}").fetchone()["c"]
|
||||
summary["tables"].append({"name": name, "count": int(count)})
|
||||
|
||||
if "bindings" in table_names:
|
||||
sample = conn.execute(
|
||||
"SELECT node_type, local_id, remote_id FROM bindings ORDER BY last_updated DESC LIMIT 50"
|
||||
if backend == "sqlite":
|
||||
assert db_path is not None
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.row_factory = sqlite3.Row
|
||||
tables = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name"
|
||||
).fetchall()
|
||||
summary["sample_bindings"] = [
|
||||
{
|
||||
"node_type": row["node_type"],
|
||||
"local_id": row["local_id"],
|
||||
"remote_id": row["remote_id"],
|
||||
}
|
||||
for row in sample
|
||||
]
|
||||
table_names = [str(r["name"]) for r in tables]
|
||||
|
||||
for name in table_names:
|
||||
count = conn.execute(f"SELECT COUNT(1) AS c FROM {name}").fetchone()["c"]
|
||||
summary["tables"].append({"name": name, "count": int(count)})
|
||||
|
||||
if "bindings" in table_names:
|
||||
sample = conn.execute(
|
||||
"SELECT node_type, local_id, remote_id FROM bindings ORDER BY last_updated DESC LIMIT 50"
|
||||
).fetchall()
|
||||
summary["sample_bindings"] = [
|
||||
{
|
||||
"node_type": row["node_type"],
|
||||
"local_id": row["local_id"],
|
||||
"remote_id": row["remote_id"],
|
||||
}
|
||||
for row in sample
|
||||
]
|
||||
return summary
|
||||
|
||||
if backend == "sqlalchemy":
|
||||
if not url:
|
||||
summary["error"] = "url is required for backend=sqlalchemy"
|
||||
return summary
|
||||
engine = create_engine(cls._to_sync_sqlalchemy_url(url))
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
inspector = inspect(conn)
|
||||
table_names = sorted(inspector.get_table_names())
|
||||
|
||||
for name in table_names:
|
||||
count = conn.execute(text(f"SELECT COUNT(1) AS c FROM {name}")).scalar_one()
|
||||
summary["tables"].append({"name": name, "count": int(count)})
|
||||
|
||||
if "bindings" in table_names:
|
||||
sample = conn.execute(
|
||||
text("SELECT node_type, local_id, remote_id FROM bindings ORDER BY last_updated DESC LIMIT 50")
|
||||
).fetchall()
|
||||
summary["sample_bindings"] = [
|
||||
{
|
||||
"node_type": row[0],
|
||||
"local_id": row[1],
|
||||
"remote_id": row[2],
|
||||
}
|
||||
for row in sample
|
||||
]
|
||||
finally:
|
||||
engine.dispose()
|
||||
return summary
|
||||
|
||||
summary["error"] = f"records summary backend not supported yet: {backend}"
|
||||
return summary
|
||||
|
||||
async def initialize(self):
|
||||
"""异步初始化数据库连接和表结构"""
|
||||
if self.backend != "sqlite":
|
||||
raise NotImplementedError(f"Persistence backend not supported yet: {self.backend}")
|
||||
@staticmethod
|
||||
def _target_exists(*, backend: str, db_path: str | None, url: str | None) -> bool:
|
||||
if backend == "sqlite":
|
||||
return bool(db_path) and os.path.exists(db_path)
|
||||
if backend == "sqlalchemy":
|
||||
if not url:
|
||||
return False
|
||||
parsed = make_url(url)
|
||||
if parsed.get_backend_name() == "sqlite":
|
||||
database = parsed.database
|
||||
if not database or database == ":memory:":
|
||||
return True
|
||||
return os.path.exists(database)
|
||||
return True
|
||||
return False
|
||||
|
||||
# 确保目录存在
|
||||
if self.db_path != ":memory:":
|
||||
dir_path = os.path.dirname(self.db_path)
|
||||
if dir_path:
|
||||
os.makedirs(dir_path, exist_ok=True)
|
||||
@staticmethod
|
||||
def _to_sync_sqlalchemy_url(url: str) -> str:
|
||||
parsed = make_url(url)
|
||||
async_to_sync = {
|
||||
"sqlite+aiosqlite": "sqlite+pysqlite",
|
||||
"postgresql+asyncpg": "postgresql+psycopg",
|
||||
"mysql+aiomysql": "mysql+pymysql",
|
||||
}
|
||||
drivername = async_to_sync.get(parsed.drivername, parsed.drivername)
|
||||
sync_url: URL = parsed.set(drivername=drivername)
|
||||
return str(sync_url)
|
||||
|
||||
self.conn = await aiosqlite.connect(self.db_path)
|
||||
self.conn.row_factory = aiosqlite.Row
|
||||
await self._init_db()
|
||||
@abstractmethod
|
||||
async def initialize(self) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
async def _init_db(self):
|
||||
# 1. 节点表: 仅用于持久化存储 DataCollection 中的 SyncNode
|
||||
# 注意: depend_ids不持久化,每次从data实时计算
|
||||
await self.conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS nodes (
|
||||
collection_id TEXT NOT NULL,
|
||||
node_id TEXT NOT NULL,
|
||||
node_type TEXT NOT NULL,
|
||||
data_id TEXT,
|
||||
bind_data_id TEXT,
|
||||
data TEXT,
|
||||
origin_data TEXT,
|
||||
action TEXT,
|
||||
status TEXT,
|
||||
binding_status TEXT,
|
||||
error TEXT,
|
||||
sync_log TEXT,
|
||||
context TEXT,
|
||||
PRIMARY KEY (collection_id, node_id)
|
||||
)
|
||||
""")
|
||||
await self._ensure_nodes_schema()
|
||||
@abstractmethod
|
||||
async def save_node(self, collection_id: str, scope: str, node: "SyncNode") -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
# 2. 绑定表: 存储本地到远程的映射关系
|
||||
# 只保留 local_id / remote_id 两字段(remote_id 可空)
|
||||
await self._ensure_bindings_schema()
|
||||
await self.conn.commit()
|
||||
@abstractmethod
|
||||
async def save_nodes_bulk(self, collection_id: str, scope: str, nodes: List["SyncNode"]) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
async def _ensure_nodes_schema(self) -> None:
|
||||
cursor = await self.conn.execute("PRAGMA table_info(nodes)")
|
||||
rows = await cursor.fetchall()
|
||||
columns = [r[1] for r in rows] if rows else []
|
||||
if "bind_data_id" not in columns:
|
||||
await self.conn.execute("ALTER TABLE nodes ADD COLUMN bind_data_id TEXT")
|
||||
@abstractmethod
|
||||
async def delete_node(self, collection_id: str, scope: str, node_id: str) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
async def _ensure_bindings_schema(self) -> None:
|
||||
"""确保 bindings 表为 (node_type, local_id, remote_id) 结构。
|
||||
@abstractmethod
|
||||
async def delete_nodes_bulk(self, collection_id: str, scope: str, node_ids: List[str]) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
若检测到旧表含 payload 列,则进行迁移。
|
||||
"""
|
||||
cursor = await self.conn.execute("PRAGMA table_info(bindings)")
|
||||
rows = await cursor.fetchall()
|
||||
columns = [r[1] for r in rows] if rows else []
|
||||
@abstractmethod
|
||||
async def load_nodes(
|
||||
self,
|
||||
collection_id: str,
|
||||
scope: str,
|
||||
*,
|
||||
exclude_node_types: Optional[List[str]] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
raise NotImplementedError
|
||||
|
||||
if not columns:
|
||||
await self.conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS bindings (
|
||||
node_type TEXT NOT NULL,
|
||||
local_id TEXT NOT NULL,
|
||||
remote_id TEXT,
|
||||
last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (node_type, local_id)
|
||||
)
|
||||
""")
|
||||
return
|
||||
@abstractmethod
|
||||
async def save_binding(self, scope: str, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
if "payload" in columns and "remote_id" not in columns:
|
||||
# 迁移旧结构 payload -> remote_id
|
||||
await self.conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS bindings_new (
|
||||
node_type TEXT NOT NULL,
|
||||
local_id TEXT NOT NULL,
|
||||
remote_id TEXT,
|
||||
last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (node_type, local_id)
|
||||
)
|
||||
""")
|
||||
@abstractmethod
|
||||
async def save_bindings_bulk(self, scope: str, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
async with self.conn.execute("SELECT node_type, local_id, payload FROM bindings") as cur:
|
||||
old_rows = await cur.fetchall()
|
||||
for row in old_rows:
|
||||
try:
|
||||
payload_dict = json.loads(row[2]) if row[2] else {}
|
||||
except json.JSONDecodeError:
|
||||
payload_dict = {}
|
||||
remote_id = payload_dict.get("remote_id")
|
||||
await self.conn.execute(
|
||||
"INSERT OR REPLACE INTO bindings_new (node_type, local_id, remote_id) VALUES (?, ?, ?)",
|
||||
(row[0], row[1], remote_id),
|
||||
)
|
||||
@abstractmethod
|
||||
async def delete_binding(self, scope: str, node_type: str, local_id: str) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
await self.conn.execute("DROP TABLE bindings")
|
||||
await self.conn.execute("ALTER TABLE bindings_new RENAME TO bindings")
|
||||
@abstractmethod
|
||||
async def delete_bindings_bulk(self, scope: str, node_type: str, local_ids: List[str]) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
# --- DataCollection 支持 ---
|
||||
@abstractmethod
|
||||
async def load_bindings(self, scope: str, node_type: str) -> List[Dict[str, Any]]:
|
||||
raise NotImplementedError
|
||||
|
||||
async def save_node(self, collection_id: str, node: "SyncNode"):
|
||||
"""保存或更新单个节点(depend_ids不持久化,运行时从data实时计算)"""
|
||||
bind_data_id = None
|
||||
if isinstance(node.context, dict):
|
||||
value = node.context.get("bind_data_id")
|
||||
if value is not None and value != "":
|
||||
bind_data_id = str(value)
|
||||
await self.conn.execute("""
|
||||
INSERT OR REPLACE INTO nodes (
|
||||
collection_id, node_id, node_type, data_id, bind_data_id,
|
||||
data, origin_data, action, status, binding_status, error, sync_log, context
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""", (
|
||||
collection_id,
|
||||
node.node_id,
|
||||
node.node_type,
|
||||
node.data_id,
|
||||
bind_data_id,
|
||||
json.dumps(node.data) if node.data else None,
|
||||
json.dumps(node.origin_data) if node.origin_data else None,
|
||||
node.action.value,
|
||||
node.status.value,
|
||||
node.binding_status.value,
|
||||
node.error,
|
||||
node.sync_log,
|
||||
json.dumps(node.context) if node.context else None
|
||||
))
|
||||
await self.conn.commit()
|
||||
@abstractmethod
|
||||
async def load_bind_data_id_overrides(self, node_type: str, scope: str) -> Dict[str, Optional[str]]:
|
||||
raise NotImplementedError
|
||||
|
||||
async def save_nodes_bulk(self, collection_id: str, nodes: List["SyncNode"]):
|
||||
"""批量保存或更新节点 (单次提交,depend_ids不持久化)"""
|
||||
if not nodes:
|
||||
return
|
||||
rows = []
|
||||
for node in nodes:
|
||||
bind_data_id = None
|
||||
if isinstance(node.context, dict):
|
||||
value = node.context.get("bind_data_id")
|
||||
if value is not None and value != "":
|
||||
bind_data_id = str(value)
|
||||
rows.append((
|
||||
collection_id,
|
||||
node.node_id,
|
||||
node.node_type,
|
||||
node.data_id,
|
||||
bind_data_id,
|
||||
json.dumps(node.data) if node.data else None,
|
||||
json.dumps(node.origin_data) if node.origin_data else None,
|
||||
node.action.value,
|
||||
node.status.value,
|
||||
node.binding_status.value,
|
||||
node.error,
|
||||
node.sync_log,
|
||||
json.dumps(node.context) if node.context else None
|
||||
))
|
||||
@abstractmethod
|
||||
async def delete_scope_node_types(self, *, scope: str, node_types: List[str]) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
await self.conn.executemany("""
|
||||
INSERT OR REPLACE INTO nodes (
|
||||
collection_id, node_id, node_type, data_id, bind_data_id,
|
||||
data, origin_data, action, status, binding_status, error, sync_log, context
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""", rows)
|
||||
await self.conn.commit()
|
||||
@abstractmethod
|
||||
async def copy_nodes_between_scopes(
|
||||
self,
|
||||
*,
|
||||
collection_id: str,
|
||||
source_scope: str,
|
||||
target_scope: str,
|
||||
node_types: List[str],
|
||||
) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
async def delete_node(self, collection_id: str, node_id: str):
|
||||
"""删除单个节点"""
|
||||
await self.conn.execute("DELETE FROM nodes WHERE collection_id = ? AND node_id = ?", (collection_id, node_id))
|
||||
await self.conn.commit()
|
||||
@abstractmethod
|
||||
async def copy_bindings_between_scopes(
|
||||
self,
|
||||
*,
|
||||
source_scope: str,
|
||||
target_scope: str,
|
||||
node_types: List[str],
|
||||
) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
async def delete_nodes_bulk(self, collection_id: str, node_ids: List[str]):
|
||||
"""批量删除节点 (单次提交)"""
|
||||
if not node_ids:
|
||||
return
|
||||
rows = [(collection_id, node_id) for node_id in node_ids]
|
||||
await self.conn.executemany(
|
||||
"DELETE FROM nodes WHERE collection_id = ? AND node_id = ?",
|
||||
rows,
|
||||
@abstractmethod
|
||||
async def dump_to_runtime(self, filename: str) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def close(self) -> None:
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
PersistenceBackendFactory = Callable[[PersistConfig], PersistenceBackend]
|
||||
_PERSISTENCE_BACKEND_REGISTRY: dict[str, PersistenceBackendFactory] = {}
|
||||
|
||||
|
||||
def register_persistence_backend(name: str, factory: PersistenceBackendFactory, *, override: bool = False) -> None:
|
||||
backend_name = name.strip()
|
||||
if not backend_name:
|
||||
raise ValueError("backend name cannot be empty")
|
||||
if not override and backend_name in _PERSISTENCE_BACKEND_REGISTRY:
|
||||
raise ValueError(f"persistence backend already registered: {backend_name}")
|
||||
_PERSISTENCE_BACKEND_REGISTRY[backend_name] = factory
|
||||
|
||||
|
||||
def get_registered_persistence_backends() -> Dict[str, PersistenceBackendFactory]:
|
||||
return dict(_PERSISTENCE_BACKEND_REGISTRY)
|
||||
|
||||
|
||||
def create_persistence_backend(
|
||||
persist_config: PersistConfig,
|
||||
) -> PersistenceBackend:
|
||||
backend_name = (persist_config.backend or "").strip()
|
||||
if not backend_name:
|
||||
raise ValueError("persist.backend cannot be empty")
|
||||
|
||||
if not _PERSISTENCE_BACKEND_REGISTRY:
|
||||
register_persistence_backend(
|
||||
"sqlite",
|
||||
lambda config: SQLitePersistenceBackend(config),
|
||||
)
|
||||
await self.conn.commit()
|
||||
|
||||
async def load_nodes(self, collection_id: str) -> List[Dict[str, Any]]:
|
||||
"""从特定 collection 中加载所有节点(depend_ids初始化为空,稍后从data实时计算)"""
|
||||
async with self.conn.execute("SELECT * FROM nodes WHERE collection_id = ?", (collection_id,)) as cursor:
|
||||
rows = await cursor.fetchall()
|
||||
results = []
|
||||
for row in rows:
|
||||
results.append({
|
||||
"node_id": row["node_id"],
|
||||
"node_type": row["node_type"],
|
||||
"data_id": row["data_id"],
|
||||
"bind_data_id": row["bind_data_id"],
|
||||
"depend_ids": [], # 不从数据库加载,运行时从data实时计算
|
||||
"data": json.loads(row["data"]) if row["data"] else None,
|
||||
"origin_data": json.loads(row["origin_data"]) if row["origin_data"] else None,
|
||||
"action": row["action"],
|
||||
"status": row["status"],
|
||||
"binding_status": row["binding_status"],
|
||||
"error": row["error"],
|
||||
"sync_log": row["sync_log"],
|
||||
"context": json.loads(row["context"]) if row["context"] else {}
|
||||
})
|
||||
if results[-1]["bind_data_id"] and isinstance(results[-1]["context"], dict):
|
||||
results[-1]["context"]["bind_data_id"] = results[-1]["bind_data_id"]
|
||||
return results
|
||||
factory = _PERSISTENCE_BACKEND_REGISTRY.get(backend_name)
|
||||
if factory is None:
|
||||
available = ", ".join(sorted(_PERSISTENCE_BACKEND_REGISTRY)) or "none"
|
||||
raise NotImplementedError(f"Persistence backend not supported yet: {backend_name}. Available: {available}")
|
||||
return factory(persist_config)
|
||||
|
||||
# --- BindingManager 支持 ---
|
||||
|
||||
async def save_binding(self, node_type: str, local_id: str, payload_dict: Dict[str, Any]):
|
||||
"""保存或更新绑定关系及其元数据"""
|
||||
remote_id = payload_dict.get("remote_id") if payload_dict else None
|
||||
await self.conn.execute("""
|
||||
INSERT OR REPLACE INTO bindings (node_type, local_id, remote_id)
|
||||
VALUES (?, ?, ?)
|
||||
""", (node_type, local_id, remote_id))
|
||||
await self.conn.commit()
|
||||
def _register_builtin_persistence_backends() -> None:
|
||||
from .persistence_backends.sqlalchemy_backend import SQLAlchemyPersistenceBackend
|
||||
from .persistence_backends.sqlite_backend import SQLitePersistenceBackend
|
||||
|
||||
async def save_bindings_bulk(self, node_type: str, bindings: Dict[str, Optional[str]]):
|
||||
"""批量保存或更新绑定关系 (单次提交)"""
|
||||
if not bindings:
|
||||
return
|
||||
rows = []
|
||||
for local_id, remote_id in bindings.items():
|
||||
rows.append((node_type, local_id, remote_id))
|
||||
await self.conn.executemany(
|
||||
"INSERT OR REPLACE INTO bindings (node_type, local_id, remote_id) VALUES (?, ?, ?)",
|
||||
rows,
|
||||
)
|
||||
await self.conn.commit()
|
||||
register_persistence_backend("sqlite", lambda config: SQLitePersistenceBackend(config), override=True)
|
||||
register_persistence_backend("sqlalchemy", lambda config: SQLAlchemyPersistenceBackend(config), override=True)
|
||||
|
||||
async def delete_binding(self, node_type: str, local_id: str):
|
||||
"""删除绑定关系"""
|
||||
await self.conn.execute(
|
||||
"DELETE FROM bindings WHERE node_type = ? AND local_id = ?",
|
||||
(node_type, local_id)
|
||||
)
|
||||
await self.conn.commit()
|
||||
|
||||
async def delete_bindings_bulk(self, node_type: str, local_ids: List[str]):
|
||||
"""批量删除绑定关系 (单次提交)"""
|
||||
if not local_ids:
|
||||
return
|
||||
rows = [(node_type, local_id) for local_id in local_ids]
|
||||
await self.conn.executemany(
|
||||
"DELETE FROM bindings WHERE node_type = ? AND local_id = ?",
|
||||
rows,
|
||||
)
|
||||
await self.conn.commit()
|
||||
|
||||
async def load_bindings(self, node_type: str) -> List[Dict[str, Any]]:
|
||||
"""回写该类型下的所有绑定关系到内存"""
|
||||
async with self.conn.execute(
|
||||
"SELECT local_id, remote_id FROM bindings WHERE node_type = ?",
|
||||
(node_type,)
|
||||
) as cursor:
|
||||
rows = await cursor.fetchall()
|
||||
return [
|
||||
{"local_id": r["local_id"], "payload": {"remote_id": r["remote_id"]}}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
async def load_bind_data_id_overrides(self, node_type: str) -> Dict[str, Optional[str]]:
|
||||
"""基于 bindings 表计算 node_id -> bind_data_id 覆盖映射。
|
||||
|
||||
规则:
|
||||
- local 节点的 bind_data_id 取其绑定 remote 节点的 data_id
|
||||
- remote 节点的 bind_data_id 取其绑定 local 节点的 data_id
|
||||
"""
|
||||
sql = """
|
||||
SELECT b.local_id AS src_node_id, rn.data_id AS bind_data_id
|
||||
FROM bindings b
|
||||
LEFT JOIN nodes rn
|
||||
ON rn.node_id = b.remote_id
|
||||
AND rn.node_type = b.node_type
|
||||
WHERE b.node_type = ?
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT b.remote_id AS src_node_id, ln.data_id AS bind_data_id
|
||||
FROM bindings b
|
||||
LEFT JOIN nodes ln
|
||||
ON ln.node_id = b.local_id
|
||||
AND ln.node_type = b.node_type
|
||||
WHERE b.node_type = ?
|
||||
AND b.remote_id IS NOT NULL
|
||||
"""
|
||||
async with self.conn.execute(sql, (node_type, node_type)) as cursor:
|
||||
rows = await cursor.fetchall()
|
||||
result: Dict[str, Optional[str]] = {}
|
||||
for row in rows:
|
||||
src_node_id = row["src_node_id"]
|
||||
if not src_node_id:
|
||||
continue
|
||||
result[str(src_node_id)] = row["bind_data_id"]
|
||||
return result
|
||||
|
||||
async def dump_to_runtime(self, filename: str):
|
||||
"""将当前状态转储到 _runtime 文件夹 (使用 VACUUM INTO)"""
|
||||
runtime_path = os.path.join("_runtime", filename)
|
||||
if os.path.exists(runtime_path):
|
||||
os.remove(runtime_path)
|
||||
|
||||
# VACUUM INTO 是备份 SQLite 的推荐方式 (SQLite 3.27+)
|
||||
await self.conn.execute("VACUUM INTO ?", (runtime_path,))
|
||||
|
||||
async def close(self):
|
||||
"""关闭数据库连接(包含异常处理)"""
|
||||
if self.conn:
|
||||
try:
|
||||
await self.conn.close()
|
||||
except Exception as e:
|
||||
print(f"⚠️ Error closing persistence: {e}")
|
||||
finally:
|
||||
self.conn = None
|
||||
_register_builtin_persistence_backends()
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
from .sqlalchemy_backend import SQLAlchemyPersistenceBackend
|
||||
from .sqlite_backend import SQLitePersistenceBackend
|
||||
|
||||
__all__ = ["SQLitePersistenceBackend", "SQLAlchemyPersistenceBackend"]
|
||||
@@ -0,0 +1,428 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import Any, Dict, List, Optional, TYPE_CHECKING
|
||||
|
||||
from sqlalchemy import Column, Index, MetaData, PrimaryKeyConstraint, String, Table, Text, delete, insert, select
|
||||
from sqlalchemy.engine import make_url
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncConnection, create_async_engine
|
||||
|
||||
from ...config.persist_config import PersistConfig
|
||||
from ..persistence import PersistenceBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..sync_node import SyncNode
|
||||
|
||||
|
||||
class SQLAlchemyPersistenceBackend(PersistenceBackend):
|
||||
"""Generic SQLAlchemy persistence backend with explicit schema bootstrap."""
|
||||
|
||||
backend_name = "sqlalchemy"
|
||||
bulk_delete_batch_size = 1000
|
||||
collection_id_length = 255
|
||||
scope_length = 255
|
||||
node_id_length = 255
|
||||
node_type_length = 128
|
||||
data_id_length = 255
|
||||
bind_data_id_length = 255
|
||||
action_length = 32
|
||||
status_length = 32
|
||||
binding_status_length = 32
|
||||
remote_id_length = 255
|
||||
last_updated_length = 64
|
||||
|
||||
def __init__(self, persist_config: PersistConfig | str | None = None, **_: Any) -> None:
|
||||
if isinstance(persist_config, PersistConfig):
|
||||
self.persist_config = persist_config
|
||||
self.database_url = persist_config.resolved_url()
|
||||
self.db_path = persist_config.resolved_db_path()
|
||||
self.backend_options = dict(persist_config.backend_options)
|
||||
elif isinstance(persist_config, str):
|
||||
self.persist_config = None
|
||||
self.database_url = persist_config
|
||||
parsed = make_url(persist_config)
|
||||
self.db_path = str(parsed.database) if parsed.get_backend_name() == "sqlite" and parsed.database else None
|
||||
self.backend_options = {}
|
||||
else:
|
||||
raise ValueError("SQLAlchemyPersistenceBackend requires PersistConfig or database URL")
|
||||
|
||||
self.engine: Optional[AsyncEngine] = None
|
||||
self.metadata = MetaData()
|
||||
self.nodes = Table(
|
||||
"nodes",
|
||||
self.metadata,
|
||||
Column("collection_id", String(self.collection_id_length), nullable=False),
|
||||
Column("scope", String(self.scope_length), nullable=False, default="global"),
|
||||
Column("node_id", String(self.node_id_length), nullable=False),
|
||||
Column("node_type", String(self.node_type_length), nullable=False),
|
||||
Column("data_id", String(self.data_id_length), nullable=True),
|
||||
Column("bind_data_id", String(self.bind_data_id_length), nullable=True),
|
||||
Column("data", Text, nullable=True),
|
||||
Column("origin_data", Text, nullable=True),
|
||||
Column("action", String(self.action_length), nullable=True),
|
||||
Column("status", String(self.status_length), nullable=True),
|
||||
Column("binding_status", String(self.binding_status_length), nullable=True),
|
||||
Column("error", Text, nullable=True),
|
||||
Column("sync_log", Text, nullable=True),
|
||||
Column("context", Text, nullable=True),
|
||||
PrimaryKeyConstraint("collection_id", "scope", "node_id", name="pk_nodes"),
|
||||
Index("idx_nodes_collection_scope_type", "collection_id", "scope", "node_type"),
|
||||
Index("idx_nodes_collection_scope_data_id", "collection_id", "scope", "data_id"),
|
||||
Index("idx_nodes_scope_type_node_id", "scope", "node_type", "node_id"),
|
||||
)
|
||||
self.bindings = Table(
|
||||
"bindings",
|
||||
self.metadata,
|
||||
Column("scope", String(self.scope_length), nullable=False, default="global"),
|
||||
Column("node_type", String(self.node_type_length), nullable=False),
|
||||
Column("local_id", String(self.node_id_length), nullable=False),
|
||||
Column("remote_id", String(self.remote_id_length), nullable=True),
|
||||
Column("last_updated", String(self.last_updated_length), nullable=True),
|
||||
PrimaryKeyConstraint("scope", "node_type", "local_id", name="pk_bindings"),
|
||||
Index("idx_bindings_scope_type_remote", "scope", "node_type", "remote_id"),
|
||||
)
|
||||
|
||||
async def initialize(self) -> None:
|
||||
if self.db_path and self.db_path != ":memory:":
|
||||
dir_path = os.path.dirname(self.db_path)
|
||||
if dir_path:
|
||||
os.makedirs(dir_path, exist_ok=True)
|
||||
|
||||
engine_options = {"future": True}
|
||||
engine_options.update(self.backend_options)
|
||||
self.engine = create_async_engine(self.database_url, **engine_options)
|
||||
async with self.engine.begin() as conn:
|
||||
await conn.run_sync(self.metadata.create_all)
|
||||
|
||||
async def save_node(self, collection_id: str, scope: str, node: "SyncNode") -> None:
|
||||
row = self._node_row(collection_id=collection_id, scope=scope, node=node)
|
||||
async with self._begin() as conn:
|
||||
await self._replace_node_rows(conn, collection_id=collection_id, scope=scope, node_ids=[node.node_id])
|
||||
await conn.execute(insert(self.nodes), [row])
|
||||
|
||||
async def save_nodes_bulk(self, collection_id: str, scope: str, nodes: List["SyncNode"]) -> None:
|
||||
if not nodes:
|
||||
return
|
||||
rows = [self._node_row(collection_id=collection_id, scope=scope, node=node) for node in nodes]
|
||||
node_ids = [node.node_id for node in nodes]
|
||||
async with self._begin() as conn:
|
||||
await self._replace_node_rows(conn, collection_id=collection_id, scope=scope, node_ids=node_ids)
|
||||
await conn.execute(insert(self.nodes), rows)
|
||||
|
||||
async def delete_node(self, collection_id: str, scope: str, node_id: str) -> None:
|
||||
async with self._begin() as conn:
|
||||
await conn.execute(
|
||||
delete(self.nodes).where(
|
||||
self.nodes.c.collection_id == collection_id,
|
||||
self.nodes.c.scope == scope,
|
||||
self.nodes.c.node_id == node_id,
|
||||
)
|
||||
)
|
||||
|
||||
async def delete_nodes_bulk(self, collection_id: str, scope: str, node_ids: List[str]) -> None:
|
||||
if not node_ids:
|
||||
return
|
||||
async with self._begin() as conn:
|
||||
for batch in self._chunked(node_ids):
|
||||
await conn.execute(
|
||||
delete(self.nodes).where(
|
||||
self.nodes.c.collection_id == collection_id,
|
||||
self.nodes.c.scope == scope,
|
||||
self.nodes.c.node_id.in_(batch),
|
||||
)
|
||||
)
|
||||
|
||||
async def load_nodes(
|
||||
self,
|
||||
collection_id: str,
|
||||
scope: str,
|
||||
*,
|
||||
exclude_node_types: Optional[List[str]] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
stmt = select(self.nodes).where(
|
||||
self.nodes.c.collection_id == collection_id,
|
||||
self.nodes.c.scope == scope,
|
||||
)
|
||||
filtered_types = [node_type for node_type in (exclude_node_types or []) if node_type]
|
||||
if filtered_types:
|
||||
stmt = stmt.where(self.nodes.c.node_type.not_in(filtered_types))
|
||||
|
||||
async with self._connect() as conn:
|
||||
rows = (await conn.execute(stmt)).mappings().all()
|
||||
return [self._deserialize_node_row(row) for row in rows]
|
||||
|
||||
async def save_binding(self, scope: str, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
|
||||
row = self._binding_row(scope=scope, node_type=node_type, local_id=local_id, remote_id=payload_dict.get("remote_id") if payload_dict else None)
|
||||
async with self._begin() as conn:
|
||||
await self._replace_binding_rows(conn, scope=scope, node_type=node_type, local_ids=[local_id])
|
||||
await conn.execute(insert(self.bindings), [row])
|
||||
|
||||
async def save_bindings_bulk(self, scope: str, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
|
||||
if not bindings:
|
||||
return
|
||||
rows = [
|
||||
self._binding_row(scope=scope, node_type=node_type, local_id=local_id, remote_id=remote_id)
|
||||
for local_id, remote_id in bindings.items()
|
||||
]
|
||||
async with self._begin() as conn:
|
||||
await self._replace_binding_rows(conn, scope=scope, node_type=node_type, local_ids=list(bindings.keys()))
|
||||
await conn.execute(insert(self.bindings), rows)
|
||||
|
||||
async def delete_binding(self, scope: str, node_type: str, local_id: str) -> None:
|
||||
async with self._begin() as conn:
|
||||
await conn.execute(
|
||||
delete(self.bindings).where(
|
||||
self.bindings.c.scope == scope,
|
||||
self.bindings.c.node_type == node_type,
|
||||
self.bindings.c.local_id == local_id,
|
||||
)
|
||||
)
|
||||
|
||||
async def delete_bindings_bulk(self, scope: str, node_type: str, local_ids: List[str]) -> None:
|
||||
if not local_ids:
|
||||
return
|
||||
async with self._begin() as conn:
|
||||
for batch in self._chunked(local_ids):
|
||||
await conn.execute(
|
||||
delete(self.bindings).where(
|
||||
self.bindings.c.scope == scope,
|
||||
self.bindings.c.node_type == node_type,
|
||||
self.bindings.c.local_id.in_(batch),
|
||||
)
|
||||
)
|
||||
|
||||
async def load_bindings(self, scope: str, node_type: str) -> List[Dict[str, Any]]:
|
||||
stmt = select(self.bindings.c.local_id, self.bindings.c.remote_id).where(
|
||||
self.bindings.c.scope == scope,
|
||||
self.bindings.c.node_type == node_type,
|
||||
)
|
||||
async with self._connect() as conn:
|
||||
rows = (await conn.execute(stmt)).mappings().all()
|
||||
return [{"local_id": row["local_id"], "payload": {"remote_id": row["remote_id"]}} for row in rows]
|
||||
|
||||
async def load_bind_data_id_overrides(self, node_type: str, scope: str) -> Dict[str, Optional[str]]:
|
||||
local_join = (
|
||||
select(
|
||||
self.bindings.c.local_id.label("src_node_id"),
|
||||
self.nodes.c.data_id.label("bind_data_id"),
|
||||
)
|
||||
.select_from(
|
||||
self.bindings.outerjoin(
|
||||
self.nodes,
|
||||
(self.nodes.c.node_id == self.bindings.c.remote_id)
|
||||
& (self.nodes.c.node_type == self.bindings.c.node_type)
|
||||
& (self.nodes.c.scope == self.bindings.c.scope),
|
||||
)
|
||||
)
|
||||
.where(
|
||||
self.bindings.c.node_type == node_type,
|
||||
self.bindings.c.scope == scope,
|
||||
)
|
||||
)
|
||||
remote_join = (
|
||||
select(
|
||||
self.bindings.c.remote_id.label("src_node_id"),
|
||||
self.nodes.c.data_id.label("bind_data_id"),
|
||||
)
|
||||
.select_from(
|
||||
self.bindings.outerjoin(
|
||||
self.nodes,
|
||||
(self.nodes.c.node_id == self.bindings.c.local_id)
|
||||
& (self.nodes.c.node_type == self.bindings.c.node_type)
|
||||
& (self.nodes.c.scope == self.bindings.c.scope),
|
||||
)
|
||||
)
|
||||
.where(
|
||||
self.bindings.c.node_type == node_type,
|
||||
self.bindings.c.scope == scope,
|
||||
self.bindings.c.remote_id.is_not(None),
|
||||
)
|
||||
)
|
||||
async with self._connect() as conn:
|
||||
rows = (await conn.execute(local_join.union_all(remote_join))).mappings().all()
|
||||
result: Dict[str, Optional[str]] = {}
|
||||
for row in rows:
|
||||
src_node_id = row["src_node_id"]
|
||||
if src_node_id:
|
||||
result[str(src_node_id)] = row["bind_data_id"]
|
||||
return result
|
||||
|
||||
async def delete_scope_node_types(self, *, scope: str, node_types: List[str]) -> None:
|
||||
filtered_types = [node_type for node_type in node_types if node_type]
|
||||
if not filtered_types:
|
||||
return
|
||||
async with self._begin() as conn:
|
||||
await conn.execute(delete(self.nodes).where(self.nodes.c.scope == scope, self.nodes.c.node_type.in_(filtered_types)))
|
||||
await conn.execute(delete(self.bindings).where(self.bindings.c.scope == scope, self.bindings.c.node_type.in_(filtered_types)))
|
||||
|
||||
async def copy_nodes_between_scopes(
|
||||
self,
|
||||
*,
|
||||
collection_id: str,
|
||||
source_scope: str,
|
||||
target_scope: str,
|
||||
node_types: List[str],
|
||||
) -> None:
|
||||
if source_scope == target_scope or not node_types:
|
||||
return
|
||||
stmt = select(self.nodes).where(
|
||||
self.nodes.c.collection_id == collection_id,
|
||||
self.nodes.c.scope == source_scope,
|
||||
self.nodes.c.node_type.in_(node_types),
|
||||
)
|
||||
async with self._begin() as conn:
|
||||
rows = (await conn.execute(stmt)).mappings().all()
|
||||
target_rows = [dict(row, scope=target_scope) for row in rows]
|
||||
if not target_rows:
|
||||
return
|
||||
await self._replace_node_rows(
|
||||
conn,
|
||||
collection_id=collection_id,
|
||||
scope=target_scope,
|
||||
node_ids=[str(row["node_id"]) for row in target_rows],
|
||||
)
|
||||
await conn.execute(insert(self.nodes), target_rows)
|
||||
|
||||
async def copy_bindings_between_scopes(
|
||||
self,
|
||||
*,
|
||||
source_scope: str,
|
||||
target_scope: str,
|
||||
node_types: List[str],
|
||||
) -> None:
|
||||
if source_scope == target_scope or not node_types:
|
||||
return
|
||||
stmt = select(self.bindings).where(
|
||||
self.bindings.c.scope == source_scope,
|
||||
self.bindings.c.node_type.in_(node_types),
|
||||
)
|
||||
async with self._begin() as conn:
|
||||
rows = (await conn.execute(stmt)).mappings().all()
|
||||
target_rows = [dict(row, scope=target_scope) for row in rows]
|
||||
if not target_rows:
|
||||
return
|
||||
by_type: Dict[str, List[str]] = {}
|
||||
for row in target_rows:
|
||||
by_type.setdefault(str(row["node_type"]), []).append(str(row["local_id"]))
|
||||
for node_type, local_ids in by_type.items():
|
||||
await self._replace_binding_rows(conn, scope=target_scope, node_type=node_type, local_ids=local_ids)
|
||||
await conn.execute(insert(self.bindings), target_rows)
|
||||
|
||||
async def dump_to_runtime(self, filename: str) -> None:
|
||||
raise NotImplementedError("dump_to_runtime is only supported by the sqlite backend")
|
||||
|
||||
async def close(self) -> None:
|
||||
if self.engine is None:
|
||||
return
|
||||
await self.engine.dispose()
|
||||
self.engine = None
|
||||
|
||||
async def _replace_node_rows(
|
||||
self,
|
||||
conn: AsyncConnection,
|
||||
*,
|
||||
collection_id: str,
|
||||
scope: str,
|
||||
node_ids: List[str],
|
||||
) -> None:
|
||||
if not node_ids:
|
||||
return
|
||||
for batch in self._chunked(node_ids):
|
||||
await conn.execute(
|
||||
delete(self.nodes).where(
|
||||
self.nodes.c.collection_id == collection_id,
|
||||
self.nodes.c.scope == scope,
|
||||
self.nodes.c.node_id.in_(batch),
|
||||
)
|
||||
)
|
||||
|
||||
async def _replace_binding_rows(
|
||||
self,
|
||||
conn: AsyncConnection,
|
||||
*,
|
||||
scope: str,
|
||||
node_type: str,
|
||||
local_ids: List[str],
|
||||
) -> None:
|
||||
if not local_ids:
|
||||
return
|
||||
for batch in self._chunked(local_ids):
|
||||
await conn.execute(
|
||||
delete(self.bindings).where(
|
||||
self.bindings.c.scope == scope,
|
||||
self.bindings.c.node_type == node_type,
|
||||
self.bindings.c.local_id.in_(batch),
|
||||
)
|
||||
)
|
||||
|
||||
def _chunked(self, values: List[str]) -> List[List[str]]:
|
||||
if not values:
|
||||
return []
|
||||
batch_size = max(1, int(self.bulk_delete_batch_size))
|
||||
return [values[index : index + batch_size] for index in range(0, len(values), batch_size)]
|
||||
|
||||
def _node_row(self, *, collection_id: str, scope: str, node: "SyncNode") -> Dict[str, Any]:
|
||||
bind_data_id = None
|
||||
if isinstance(node.context, dict):
|
||||
value = node.context.get("bind_data_id")
|
||||
if value is not None and value != "":
|
||||
bind_data_id = str(value)
|
||||
return {
|
||||
"collection_id": collection_id,
|
||||
"scope": scope,
|
||||
"node_id": node.node_id,
|
||||
"node_type": node.node_type,
|
||||
"data_id": node.data_id,
|
||||
"bind_data_id": bind_data_id,
|
||||
"data": json.dumps(node.data) if node.data else None,
|
||||
"origin_data": json.dumps(node.origin_data) if node.origin_data else None,
|
||||
"action": node.action.value,
|
||||
"status": node.status.value,
|
||||
"binding_status": node.binding_status.value,
|
||||
"error": node.error,
|
||||
"sync_log": node.sync_log,
|
||||
"context": json.dumps(node.context) if node.context else None,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _binding_row(*, scope: str, node_type: str, local_id: str, remote_id: Optional[str]) -> Dict[str, Any]:
|
||||
return {
|
||||
"scope": scope,
|
||||
"node_type": node_type,
|
||||
"local_id": local_id,
|
||||
"remote_id": remote_id,
|
||||
"last_updated": None,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _deserialize_node_row(row: Dict[str, Any]) -> Dict[str, Any]:
|
||||
item = {
|
||||
"node_id": row["node_id"],
|
||||
"node_type": row["node_type"],
|
||||
"data_id": row["data_id"],
|
||||
"bind_data_id": row["bind_data_id"],
|
||||
"depend_ids": [],
|
||||
"data": json.loads(row["data"]) if row["data"] else None,
|
||||
"origin_data": json.loads(row["origin_data"]) if row["origin_data"] else None,
|
||||
"action": row["action"],
|
||||
"status": row["status"],
|
||||
"binding_status": row["binding_status"],
|
||||
"error": row["error"],
|
||||
"sync_log": row["sync_log"],
|
||||
"context": json.loads(row["context"]) if row["context"] else {},
|
||||
}
|
||||
if item["bind_data_id"] and isinstance(item["context"], dict):
|
||||
item["context"]["bind_data_id"] = item["bind_data_id"]
|
||||
return item
|
||||
|
||||
def _require_engine(self) -> AsyncEngine:
|
||||
if self.engine is None:
|
||||
raise RuntimeError("SQLAlchemyPersistenceBackend is not initialized")
|
||||
return self.engine
|
||||
|
||||
def _connect(self):
|
||||
return self._require_engine().connect()
|
||||
|
||||
def _begin(self):
|
||||
return self._require_engine().begin()
|
||||
@@ -0,0 +1,470 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import aiosqlite
|
||||
import json
|
||||
import os
|
||||
from typing import Any, Dict, List, Optional, TYPE_CHECKING
|
||||
|
||||
from ...config.persist_config import PersistConfig
|
||||
from ..persistence import PersistenceBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..sync_node import SyncNode
|
||||
|
||||
|
||||
class SQLitePersistenceBackend(PersistenceBackend):
|
||||
"""SQLite implementation of the persistence contract."""
|
||||
|
||||
backend_name = "sqlite"
|
||||
|
||||
def __init__(self, persist_config: PersistConfig | str | None = None, **_: Any) -> None:
|
||||
if isinstance(persist_config, PersistConfig):
|
||||
self.persist_config = persist_config
|
||||
resolved_db_path = persist_config.resolved_db_path()
|
||||
if not resolved_db_path:
|
||||
raise ValueError("sqlite persistence requires a resolvable db_path")
|
||||
self.db_path = resolved_db_path
|
||||
elif isinstance(persist_config, str):
|
||||
self.persist_config = None
|
||||
self.db_path = persist_config
|
||||
else:
|
||||
self.persist_config = None
|
||||
self.db_path = ":memory:"
|
||||
self.conn: Optional[aiosqlite.Connection] = None
|
||||
|
||||
async def initialize(self) -> None:
|
||||
if self.db_path != ":memory:":
|
||||
dir_path = os.path.dirname(self.db_path)
|
||||
if dir_path:
|
||||
os.makedirs(dir_path, exist_ok=True)
|
||||
|
||||
self.conn = await aiosqlite.connect(self.db_path)
|
||||
self.conn.row_factory = aiosqlite.Row
|
||||
await self._init_db()
|
||||
|
||||
async def _init_db(self) -> None:
|
||||
assert self.conn is not None
|
||||
await self.conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS nodes (
|
||||
collection_id TEXT NOT NULL,
|
||||
scope TEXT NOT NULL DEFAULT 'global',
|
||||
node_id TEXT NOT NULL,
|
||||
node_type TEXT NOT NULL,
|
||||
data_id TEXT,
|
||||
bind_data_id TEXT,
|
||||
data TEXT,
|
||||
origin_data TEXT,
|
||||
action TEXT,
|
||||
status TEXT,
|
||||
binding_status TEXT,
|
||||
error TEXT,
|
||||
sync_log TEXT,
|
||||
context TEXT,
|
||||
PRIMARY KEY (collection_id, scope, node_id)
|
||||
)
|
||||
""")
|
||||
await self._ensure_nodes_schema()
|
||||
await self._ensure_bindings_schema()
|
||||
await self.conn.commit()
|
||||
|
||||
async def _ensure_nodes_schema(self) -> None:
|
||||
assert self.conn is not None
|
||||
cursor = await self.conn.execute("PRAGMA table_info(nodes)")
|
||||
rows = await cursor.fetchall()
|
||||
columns = [r[1] for r in rows] if rows else []
|
||||
if "scope" not in columns:
|
||||
await self.conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS nodes_new (
|
||||
collection_id TEXT NOT NULL,
|
||||
scope TEXT NOT NULL DEFAULT 'global',
|
||||
node_id TEXT NOT NULL,
|
||||
node_type TEXT NOT NULL,
|
||||
data_id TEXT,
|
||||
bind_data_id TEXT,
|
||||
data TEXT,
|
||||
origin_data TEXT,
|
||||
action TEXT,
|
||||
status TEXT,
|
||||
binding_status TEXT,
|
||||
error TEXT,
|
||||
sync_log TEXT,
|
||||
context TEXT,
|
||||
PRIMARY KEY (collection_id, scope, node_id)
|
||||
)
|
||||
""")
|
||||
await self.conn.execute("""
|
||||
INSERT INTO nodes_new (
|
||||
collection_id, scope, node_id, node_type, data_id, bind_data_id,
|
||||
data, origin_data, action, status, binding_status, error, sync_log, context
|
||||
)
|
||||
SELECT
|
||||
collection_id, 'global', node_id, node_type, data_id, bind_data_id,
|
||||
data, origin_data, action, status, binding_status, error, sync_log, context
|
||||
FROM nodes
|
||||
""")
|
||||
await self.conn.execute("DROP TABLE nodes")
|
||||
await self.conn.execute("ALTER TABLE nodes_new RENAME TO nodes")
|
||||
cursor = await self.conn.execute("PRAGMA table_info(nodes)")
|
||||
rows = await cursor.fetchall()
|
||||
columns = [r[1] for r in rows] if rows else []
|
||||
if "bind_data_id" not in columns:
|
||||
await self.conn.execute("ALTER TABLE nodes ADD COLUMN bind_data_id TEXT")
|
||||
await self.conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_nodes_collection_scope_type ON nodes (collection_id, scope, node_type)"
|
||||
)
|
||||
await self.conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_nodes_collection_scope_data_id ON nodes (collection_id, scope, data_id)"
|
||||
)
|
||||
await self.conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_nodes_scope_type_node_id ON nodes (scope, node_type, node_id)"
|
||||
)
|
||||
|
||||
async def _ensure_bindings_schema(self) -> None:
|
||||
assert self.conn is not None
|
||||
cursor = await self.conn.execute("PRAGMA table_info(bindings)")
|
||||
rows = await cursor.fetchall()
|
||||
columns = [r[1] for r in rows] if rows else []
|
||||
|
||||
if not columns:
|
||||
await self.conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS bindings (
|
||||
scope TEXT NOT NULL DEFAULT 'global',
|
||||
node_type TEXT NOT NULL,
|
||||
local_id TEXT NOT NULL,
|
||||
remote_id TEXT,
|
||||
last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (scope, node_type, local_id)
|
||||
)
|
||||
""")
|
||||
return
|
||||
|
||||
if "scope" not in columns or ("payload" in columns and "remote_id" not in columns):
|
||||
await self.conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS bindings_new (
|
||||
scope TEXT NOT NULL DEFAULT 'global',
|
||||
node_type TEXT NOT NULL,
|
||||
local_id TEXT NOT NULL,
|
||||
remote_id TEXT,
|
||||
last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (scope, node_type, local_id)
|
||||
)
|
||||
""")
|
||||
|
||||
if "payload" in columns and "remote_id" not in columns:
|
||||
async with self.conn.execute("SELECT node_type, local_id, payload FROM bindings") as cur:
|
||||
old_rows = await cur.fetchall()
|
||||
for row in old_rows:
|
||||
try:
|
||||
payload_dict = json.loads(row[2]) if row[2] else {}
|
||||
except json.JSONDecodeError:
|
||||
payload_dict = {}
|
||||
remote_id = payload_dict.get("remote_id")
|
||||
await self.conn.execute(
|
||||
"INSERT OR REPLACE INTO bindings_new (scope, node_type, local_id, remote_id) VALUES (?, ?, ?, ?)",
|
||||
("global", row[0], row[1], remote_id),
|
||||
)
|
||||
else:
|
||||
async with self.conn.execute("SELECT node_type, local_id, remote_id FROM bindings") as cur:
|
||||
old_rows = await cur.fetchall()
|
||||
for row in old_rows:
|
||||
await self.conn.execute(
|
||||
"INSERT OR REPLACE INTO bindings_new (scope, node_type, local_id, remote_id) VALUES (?, ?, ?, ?)",
|
||||
("global", row[0], row[1], row[2]),
|
||||
)
|
||||
|
||||
await self.conn.execute("DROP TABLE bindings")
|
||||
await self.conn.execute("ALTER TABLE bindings_new RENAME TO bindings")
|
||||
await self.conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_bindings_scope_type_remote ON bindings (scope, node_type, remote_id)"
|
||||
)
|
||||
|
||||
async def save_node(self, collection_id: str, scope: str, node: "SyncNode") -> None:
|
||||
assert self.conn is not None
|
||||
bind_data_id = None
|
||||
if isinstance(node.context, dict):
|
||||
value = node.context.get("bind_data_id")
|
||||
if value is not None and value != "":
|
||||
bind_data_id = str(value)
|
||||
await self.conn.execute("""
|
||||
INSERT OR REPLACE INTO nodes (
|
||||
collection_id, scope, node_id, node_type, data_id, bind_data_id,
|
||||
data, origin_data, action, status, binding_status, error, sync_log, context
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""", (
|
||||
collection_id,
|
||||
scope,
|
||||
node.node_id,
|
||||
node.node_type,
|
||||
node.data_id,
|
||||
bind_data_id,
|
||||
json.dumps(node.data) if node.data else None,
|
||||
json.dumps(node.origin_data) if node.origin_data else None,
|
||||
node.action.value,
|
||||
node.status.value,
|
||||
node.binding_status.value,
|
||||
node.error,
|
||||
node.sync_log,
|
||||
json.dumps(node.context) if node.context else None,
|
||||
))
|
||||
await self.conn.commit()
|
||||
|
||||
async def save_nodes_bulk(self, collection_id: str, scope: str, nodes: List["SyncNode"]) -> None:
|
||||
assert self.conn is not None
|
||||
if not nodes:
|
||||
return
|
||||
rows = []
|
||||
for node in nodes:
|
||||
bind_data_id = None
|
||||
if isinstance(node.context, dict):
|
||||
value = node.context.get("bind_data_id")
|
||||
if value is not None and value != "":
|
||||
bind_data_id = str(value)
|
||||
rows.append((
|
||||
collection_id,
|
||||
scope,
|
||||
node.node_id,
|
||||
node.node_type,
|
||||
node.data_id,
|
||||
bind_data_id,
|
||||
json.dumps(node.data) if node.data else None,
|
||||
json.dumps(node.origin_data) if node.origin_data else None,
|
||||
node.action.value,
|
||||
node.status.value,
|
||||
node.binding_status.value,
|
||||
node.error,
|
||||
node.sync_log,
|
||||
json.dumps(node.context) if node.context else None,
|
||||
))
|
||||
|
||||
await self.conn.executemany("""
|
||||
INSERT OR REPLACE INTO nodes (
|
||||
collection_id, scope, node_id, node_type, data_id, bind_data_id,
|
||||
data, origin_data, action, status, binding_status, error, sync_log, context
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""", rows)
|
||||
await self.conn.commit()
|
||||
|
||||
async def delete_node(self, collection_id: str, scope: str, node_id: str) -> None:
|
||||
assert self.conn is not None
|
||||
await self.conn.execute(
|
||||
"DELETE FROM nodes WHERE collection_id = ? AND scope = ? AND node_id = ?",
|
||||
(collection_id, scope, node_id),
|
||||
)
|
||||
await self.conn.commit()
|
||||
|
||||
async def delete_nodes_bulk(self, collection_id: str, scope: str, node_ids: List[str]) -> None:
|
||||
assert self.conn is not None
|
||||
if not node_ids:
|
||||
return
|
||||
rows = [(collection_id, scope, node_id) for node_id in node_ids]
|
||||
await self.conn.executemany(
|
||||
"DELETE FROM nodes WHERE collection_id = ? AND scope = ? AND node_id = ?",
|
||||
rows,
|
||||
)
|
||||
await self.conn.commit()
|
||||
|
||||
async def load_nodes(
|
||||
self,
|
||||
collection_id: str,
|
||||
scope: str,
|
||||
*,
|
||||
exclude_node_types: Optional[List[str]] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
assert self.conn is not None
|
||||
sql = "SELECT * FROM nodes WHERE collection_id = ? AND scope = ?"
|
||||
params: List[Any] = [collection_id, scope]
|
||||
filtered_types = [node_type for node_type in (exclude_node_types or []) if node_type]
|
||||
if filtered_types:
|
||||
placeholders = ", ".join("?" for _ in filtered_types)
|
||||
sql += f" AND node_type NOT IN ({placeholders})"
|
||||
params.extend(filtered_types)
|
||||
|
||||
async with self.conn.execute(sql, tuple(params)) as cursor:
|
||||
rows = await cursor.fetchall()
|
||||
results = []
|
||||
for row in rows:
|
||||
item = {
|
||||
"node_id": row["node_id"],
|
||||
"node_type": row["node_type"],
|
||||
"data_id": row["data_id"],
|
||||
"bind_data_id": row["bind_data_id"],
|
||||
"depend_ids": [],
|
||||
"data": json.loads(row["data"]) if row["data"] else None,
|
||||
"origin_data": json.loads(row["origin_data"]) if row["origin_data"] else None,
|
||||
"action": row["action"],
|
||||
"status": row["status"],
|
||||
"binding_status": row["binding_status"],
|
||||
"error": row["error"],
|
||||
"sync_log": row["sync_log"],
|
||||
"context": json.loads(row["context"]) if row["context"] else {},
|
||||
}
|
||||
if item["bind_data_id"] and isinstance(item["context"], dict):
|
||||
item["context"]["bind_data_id"] = item["bind_data_id"]
|
||||
results.append(item)
|
||||
return results
|
||||
|
||||
async def save_binding(self, scope: str, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
|
||||
assert self.conn is not None
|
||||
remote_id = payload_dict.get("remote_id") if payload_dict else None
|
||||
await self.conn.execute("""
|
||||
INSERT OR REPLACE INTO bindings (scope, node_type, local_id, remote_id)
|
||||
VALUES (?, ?, ?, ?)
|
||||
""", (scope, node_type, local_id, remote_id))
|
||||
await self.conn.commit()
|
||||
|
||||
async def save_bindings_bulk(self, scope: str, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
|
||||
assert self.conn is not None
|
||||
if not bindings:
|
||||
return
|
||||
rows = [(scope, node_type, local_id, remote_id) for local_id, remote_id in bindings.items()]
|
||||
await self.conn.executemany(
|
||||
"INSERT OR REPLACE INTO bindings (scope, node_type, local_id, remote_id) VALUES (?, ?, ?, ?)",
|
||||
rows,
|
||||
)
|
||||
await self.conn.commit()
|
||||
|
||||
async def delete_binding(self, scope: str, node_type: str, local_id: str) -> None:
|
||||
assert self.conn is not None
|
||||
await self.conn.execute(
|
||||
"DELETE FROM bindings WHERE scope = ? AND node_type = ? AND local_id = ?",
|
||||
(scope, node_type, local_id),
|
||||
)
|
||||
await self.conn.commit()
|
||||
|
||||
async def delete_bindings_bulk(self, scope: str, node_type: str, local_ids: List[str]) -> None:
|
||||
assert self.conn is not None
|
||||
if not local_ids:
|
||||
return
|
||||
rows = [(scope, node_type, local_id) for local_id in local_ids]
|
||||
await self.conn.executemany(
|
||||
"DELETE FROM bindings WHERE scope = ? AND node_type = ? AND local_id = ?",
|
||||
rows,
|
||||
)
|
||||
await self.conn.commit()
|
||||
|
||||
async def load_bindings(self, scope: str, node_type: str) -> List[Dict[str, Any]]:
|
||||
assert self.conn is not None
|
||||
async with self.conn.execute(
|
||||
"SELECT local_id, remote_id FROM bindings WHERE scope = ? AND node_type = ?",
|
||||
(scope, node_type),
|
||||
) as cursor:
|
||||
rows = await cursor.fetchall()
|
||||
return [{"local_id": r["local_id"], "payload": {"remote_id": r["remote_id"]}} for r in rows]
|
||||
|
||||
async def load_bind_data_id_overrides(self, node_type: str, scope: str) -> Dict[str, Optional[str]]:
|
||||
assert self.conn is not None
|
||||
sql = """
|
||||
SELECT b.local_id AS src_node_id, rn.data_id AS bind_data_id
|
||||
FROM bindings b
|
||||
LEFT JOIN nodes rn
|
||||
ON rn.node_id = b.remote_id
|
||||
AND rn.node_type = b.node_type
|
||||
AND rn.scope = b.scope
|
||||
WHERE b.node_type = ?
|
||||
AND b.scope = ?
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT b.remote_id AS src_node_id, ln.data_id AS bind_data_id
|
||||
FROM bindings b
|
||||
LEFT JOIN nodes ln
|
||||
ON ln.node_id = b.local_id
|
||||
AND ln.node_type = b.node_type
|
||||
AND ln.scope = b.scope
|
||||
WHERE b.node_type = ?
|
||||
AND b.scope = ?
|
||||
AND b.remote_id IS NOT NULL
|
||||
"""
|
||||
async with self.conn.execute(sql, (node_type, scope, node_type, scope)) as cursor:
|
||||
rows = await cursor.fetchall()
|
||||
result: Dict[str, Optional[str]] = {}
|
||||
for row in rows:
|
||||
src_node_id = row["src_node_id"]
|
||||
if not src_node_id:
|
||||
continue
|
||||
result[str(src_node_id)] = row["bind_data_id"]
|
||||
return result
|
||||
|
||||
async def delete_scope_node_types(self, *, scope: str, node_types: List[str]) -> None:
|
||||
assert self.conn is not None
|
||||
filtered_types = [node_type for node_type in node_types if node_type]
|
||||
if not filtered_types:
|
||||
return
|
||||
|
||||
placeholders = ", ".join("?" for _ in filtered_types)
|
||||
await self.conn.execute(
|
||||
f"DELETE FROM nodes WHERE scope = ? AND node_type IN ({placeholders})",
|
||||
(scope, *filtered_types),
|
||||
)
|
||||
await self.conn.execute(
|
||||
f"DELETE FROM bindings WHERE scope = ? AND node_type IN ({placeholders})",
|
||||
(scope, *filtered_types),
|
||||
)
|
||||
await self.conn.commit()
|
||||
|
||||
async def copy_nodes_between_scopes(
|
||||
self,
|
||||
*,
|
||||
collection_id: str,
|
||||
source_scope: str,
|
||||
target_scope: str,
|
||||
node_types: List[str],
|
||||
) -> None:
|
||||
assert self.conn is not None
|
||||
if source_scope == target_scope or not node_types:
|
||||
return
|
||||
placeholders = ", ".join("?" for _ in node_types)
|
||||
sql = f"""
|
||||
INSERT OR REPLACE INTO nodes (
|
||||
collection_id, scope, node_id, node_type, data_id, bind_data_id,
|
||||
data, origin_data, action, status, binding_status, error, sync_log, context
|
||||
)
|
||||
SELECT
|
||||
collection_id, ?, node_id, node_type, data_id, bind_data_id,
|
||||
data, origin_data, action, status, binding_status, error, sync_log, context
|
||||
FROM nodes
|
||||
WHERE collection_id = ?
|
||||
AND scope = ?
|
||||
AND node_type IN ({placeholders})
|
||||
"""
|
||||
await self.conn.execute(sql, (target_scope, collection_id, source_scope, *node_types))
|
||||
await self.conn.commit()
|
||||
|
||||
async def copy_bindings_between_scopes(
|
||||
self,
|
||||
*,
|
||||
source_scope: str,
|
||||
target_scope: str,
|
||||
node_types: List[str],
|
||||
) -> None:
|
||||
assert self.conn is not None
|
||||
if source_scope == target_scope or not node_types:
|
||||
return
|
||||
placeholders = ", ".join("?" for _ in node_types)
|
||||
sql = f"""
|
||||
INSERT OR REPLACE INTO bindings (scope, node_type, local_id, remote_id)
|
||||
SELECT ?, node_type, local_id, remote_id
|
||||
FROM bindings
|
||||
WHERE scope = ?
|
||||
AND node_type IN ({placeholders})
|
||||
"""
|
||||
await self.conn.execute(sql, (target_scope, source_scope, *node_types))
|
||||
await self.conn.commit()
|
||||
|
||||
async def dump_to_runtime(self, filename: str) -> None:
|
||||
assert self.conn is not None
|
||||
runtime_path = os.path.join("_runtime", filename)
|
||||
if os.path.exists(runtime_path):
|
||||
os.remove(runtime_path)
|
||||
await self.conn.execute("VACUUM INTO ?", (runtime_path,))
|
||||
|
||||
async def close(self) -> None:
|
||||
if self.conn:
|
||||
try:
|
||||
await self.conn.close()
|
||||
except Exception as exc:
|
||||
print(f"⚠️ Error closing persistence: {exc}")
|
||||
finally:
|
||||
self.conn = None
|
||||
@@ -0,0 +1,85 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from .persistence import PersistenceBackend
|
||||
|
||||
|
||||
class PersistenceService:
|
||||
"""Shared persistence service that owns the backend lifecycle."""
|
||||
|
||||
def __init__(self, backend: PersistenceBackend):
|
||||
self._backend = backend
|
||||
|
||||
@property
|
||||
def backend(self) -> PersistenceBackend:
|
||||
return self._backend
|
||||
|
||||
async def initialize(self) -> None:
|
||||
await self._backend.initialize()
|
||||
|
||||
def view(self, scope: str) -> "ScopedPersistenceView":
|
||||
return ScopedPersistenceView(self._backend, scope=scope)
|
||||
|
||||
async def close(self) -> None:
|
||||
await self._backend.close()
|
||||
|
||||
|
||||
class ScopedPersistenceView:
|
||||
"""Thin scope-bound persistence view used by project runtimes."""
|
||||
|
||||
def __init__(self, backend: PersistenceBackend, *, scope: str):
|
||||
self._backend = backend
|
||||
self.scope = scope
|
||||
|
||||
@property
|
||||
def backend(self) -> PersistenceBackend:
|
||||
return self._backend
|
||||
|
||||
async def load_nodes(
|
||||
self,
|
||||
collection_id: str,
|
||||
*,
|
||||
exclude_node_types: Optional[List[str]] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
return await self._backend.load_nodes(
|
||||
collection_id,
|
||||
self.scope,
|
||||
exclude_node_types=exclude_node_types,
|
||||
)
|
||||
|
||||
async def save_node(self, collection_id: str, node) -> None:
|
||||
await self._backend.save_node(collection_id, self.scope, node)
|
||||
|
||||
async def save_nodes_bulk(self, collection_id: str, nodes) -> None:
|
||||
await self._backend.save_nodes_bulk(collection_id, self.scope, nodes)
|
||||
|
||||
async def delete_node(self, collection_id: str, node_id: str) -> None:
|
||||
await self._backend.delete_node(collection_id, self.scope, node_id)
|
||||
|
||||
async def delete_nodes_bulk(self, collection_id: str, node_ids: List[str]) -> None:
|
||||
await self._backend.delete_nodes_bulk(collection_id, self.scope, node_ids)
|
||||
|
||||
async def load_bindings(self, node_type: str) -> List[Dict[str, Any]]:
|
||||
return await self._backend.load_bindings(self.scope, node_type)
|
||||
|
||||
async def save_binding(self, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
|
||||
await self._backend.save_binding(self.scope, node_type, local_id, payload_dict)
|
||||
|
||||
async def save_bindings_bulk(self, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
|
||||
await self._backend.save_bindings_bulk(self.scope, node_type, bindings)
|
||||
|
||||
async def delete_binding(self, node_type: str, local_id: str) -> None:
|
||||
await self._backend.delete_binding(self.scope, node_type, local_id)
|
||||
|
||||
async def delete_bindings_bulk(self, node_type: str, local_ids: List[str]) -> None:
|
||||
await self._backend.delete_bindings_bulk(self.scope, node_type, local_ids)
|
||||
|
||||
async def load_bind_data_id_overrides(self, node_type: str) -> Dict[str, Optional[str]]:
|
||||
return await self._backend.load_bind_data_id_overrides(node_type, self.scope)
|
||||
|
||||
async def delete_scope_node_types(self, *, node_types: List[str]) -> None:
|
||||
await self._backend.delete_scope_node_types(scope=self.scope, node_types=node_types)
|
||||
|
||||
async def close(self) -> None:
|
||||
return None
|
||||
@@ -17,7 +17,7 @@ Domain 注册中心
|
||||
node_type="contract",
|
||||
schema=ContractResponse,
|
||||
node_class=ContractSyncNode,
|
||||
jsonl_handler_class=ContractJsonlHandler
|
||||
jsonl_handler_class=ContractJsonlHandler,
|
||||
)
|
||||
|
||||
# Collection 查询 SyncNode 类
|
||||
@@ -47,7 +47,6 @@ class DomainRegistration:
|
||||
strategy_class: Optional[Type[Any]] = None # 可选,部分 domain 可能只读
|
||||
jsonl_handler_class: Optional[Type[Any]] = None
|
||||
api_handler_class: Optional[Type[Any]] = None
|
||||
db_handler_class: Optional[Type[Any]] = None
|
||||
handler_classes: Dict[str, Type[Any]] = field(default_factory=dict)
|
||||
metadata: Dict[str, Any] = field(default_factory=dict) # 额外元数据
|
||||
|
||||
@@ -69,7 +68,6 @@ class DomainRegistry:
|
||||
_BUILTIN_HANDLER_ATTRS: Dict[str, str] = {
|
||||
"jsonl": "jsonl_handler_class",
|
||||
"api": "api_handler_class",
|
||||
"db": "db_handler_class",
|
||||
}
|
||||
|
||||
@classmethod
|
||||
@@ -81,7 +79,6 @@ class DomainRegistry:
|
||||
strategy_class: Optional[Type[Any]] = None,
|
||||
jsonl_handler_class: Optional[Type[Any]] = None,
|
||||
api_handler_class: Optional[Type[Any]] = None,
|
||||
db_handler_class: Optional[Type[Any]] = None,
|
||||
**metadata
|
||||
) -> None:
|
||||
"""
|
||||
@@ -94,7 +91,6 @@ class DomainRegistry:
|
||||
strategy_class: Strategy 子类(可选)
|
||||
jsonl_handler_class: JSONL Handler 类(可选)
|
||||
api_handler_class: API Handler 类(可选)
|
||||
db_handler_class: DB Handler 类(可选)
|
||||
**metadata: 额外的元数据
|
||||
|
||||
Raises:
|
||||
@@ -110,7 +106,6 @@ class DomainRegistry:
|
||||
strategy_class=strategy_class,
|
||||
jsonl_handler_class=jsonl_handler_class,
|
||||
api_handler_class=api_handler_class,
|
||||
db_handler_class=db_handler_class,
|
||||
metadata=metadata
|
||||
)
|
||||
for datasource_type, attr_name in cls._BUILTIN_HANDLER_ATTRS.items():
|
||||
@@ -187,10 +182,6 @@ class DomainRegistry:
|
||||
setattr(reg, attr_name, handler_class)
|
||||
|
||||
@classmethod
|
||||
def register_db_handler(cls, node_type: str, db_handler_class: Type[Any]) -> None:
|
||||
"""为已注册 domain 补充 DB Handler。"""
|
||||
cls.register_handler(node_type, "db", db_handler_class)
|
||||
|
||||
@classmethod
|
||||
def get_schema(cls, node_type: str) -> Type[BaseModel]:
|
||||
"""获取 Schema 类"""
|
||||
@@ -245,7 +236,6 @@ class DomainRegistry:
|
||||
print(f" Strategy Class: {reg.strategy_class.__name__ if reg.strategy_class else '❌ None'}")
|
||||
print(f" JSONL Handler: {reg.jsonl_handler_class.__name__ if reg.jsonl_handler_class else '❌ None'}")
|
||||
print(f" API Handler: {reg.api_handler_class.__name__ if reg.api_handler_class else '❌ None'}")
|
||||
print(f" DB Handler: {reg.db_handler_class.__name__ if reg.db_handler_class else '❌ None'}")
|
||||
extra_handler_types = sorted(
|
||||
handler_type for handler_type in reg.handler_classes if handler_type not in cls._BUILTIN_HANDLER_ATTRS
|
||||
)
|
||||
@@ -272,7 +262,7 @@ class DomainRegistry:
|
||||
"has_strategy": True,
|
||||
"has_jsonl_handler": True,
|
||||
"has_api_handler": True,
|
||||
"has_db_handler": False
|
||||
"handler_types": []
|
||||
},
|
||||
...
|
||||
}
|
||||
@@ -284,7 +274,6 @@ class DomainRegistry:
|
||||
"has_strategy": reg.strategy_class is not None,
|
||||
"has_jsonl_handler": reg.jsonl_handler_class is not None,
|
||||
"has_api_handler": reg.api_handler_class is not None,
|
||||
"has_db_handler": reg.db_handler_class is not None,
|
||||
"handler_types": sorted(reg.handler_classes.keys()),
|
||||
}
|
||||
return summary
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
from .persistence_backends.sqlalchemy_backend import SQLAlchemyPersistenceBackend
|
||||
|
||||
__all__ = ["SQLAlchemyPersistenceBackend"]
|
||||
@@ -0,0 +1,3 @@
|
||||
from .persistence_backends.sqlite_backend import SQLitePersistenceBackend
|
||||
|
||||
__all__ = ["SQLitePersistenceBackend"]
|
||||
@@ -1,4 +1,4 @@
|
||||
from .datasource_config import ApiDataSourceConfig, BaseDataSourceConfig, DbDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
|
||||
from .datasource_config import ApiDataSourceConfig, BaseDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
|
||||
from .persist_config import PersistConfig
|
||||
from .logging_config import LoggingConfig, resolve_log_file_path
|
||||
from ..datasource.type_registry import (
|
||||
@@ -24,18 +24,23 @@ from .strategy_config import (
|
||||
ConfigPresets,
|
||||
)
|
||||
from .pipeline_config import PipelineRunConfig, DEFAULT_NODE_TYPES
|
||||
from .multi_project_config import (
|
||||
MultiProjectItemConfig,
|
||||
MultiProjectRunConfig,
|
||||
build_multi_project_config_from_file,
|
||||
is_multi_project_payload,
|
||||
)
|
||||
from .builder import (
|
||||
build_default_config,
|
||||
build_config,
|
||||
build_config_from_file,
|
||||
apply_config_overrides,
|
||||
apply_env_overrides,
|
||||
config_snapshot,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"ApiDataSourceConfig",
|
||||
"BaseDataSourceConfig",
|
||||
"DbDataSourceConfig",
|
||||
"JsonlDataSourceConfig",
|
||||
"PipelineRunConfig",
|
||||
"DataSourceConfig",
|
||||
@@ -59,9 +64,13 @@ __all__ = [
|
||||
"StrategyRuntimeConfig",
|
||||
"ConfigPresets",
|
||||
"DEFAULT_NODE_TYPES",
|
||||
"MultiProjectItemConfig",
|
||||
"MultiProjectRunConfig",
|
||||
"build_multi_project_config_from_file",
|
||||
"is_multi_project_payload",
|
||||
"build_default_config",
|
||||
"build_config",
|
||||
"build_config_from_file",
|
||||
"apply_config_overrides",
|
||||
"apply_env_overrides",
|
||||
"config_snapshot",
|
||||
]
|
||||
|
||||
@@ -9,7 +9,7 @@ from ..common.registry import DomainRegistry
|
||||
from .datasource_config import DataSourceConfig
|
||||
from .pipeline_config import PipelineRunConfig
|
||||
from .run_presets import load_overrides_from_file
|
||||
from .strategy_config import ConfigPresets, StrategyRuntimeConfig
|
||||
from .strategy_config import ConfigPresets, StrategyRuntimeConfig, resolve_domain_option_config
|
||||
|
||||
|
||||
def _deep_merge(base: Mapping[str, Any], override: Mapping[str, Any]) -> Dict[str, Any]:
|
||||
@@ -28,19 +28,76 @@ def _deep_merge(base: Mapping[str, Any], override: Mapping[str, Any]) -> Dict[st
|
||||
return merged
|
||||
|
||||
|
||||
def _build_default_strategy_runtime_config(node_type: str) -> StrategyRuntimeConfig | None:
|
||||
strategy_class = DomainRegistry.get_strategy_class(node_type)
|
||||
if strategy_class is None:
|
||||
return None
|
||||
|
||||
config = strategy_class.default_config.model_copy(deep=True)
|
||||
config.domain_option = resolve_domain_option_config(
|
||||
getattr(strategy_class, "domain_option_model", None),
|
||||
config.domain_option,
|
||||
)
|
||||
|
||||
return StrategyRuntimeConfig(
|
||||
node_type=node_type,
|
||||
config=config,
|
||||
)
|
||||
|
||||
|
||||
def _default_strategy_map(node_types: list[str]) -> Dict[str, StrategyRuntimeConfig]:
|
||||
strategy_map: Dict[str, StrategyRuntimeConfig] = {}
|
||||
for node_type in node_types:
|
||||
strategy_class = DomainRegistry.get_strategy_class(node_type)
|
||||
if strategy_class is None:
|
||||
runtime_cfg = _build_default_strategy_runtime_config(node_type)
|
||||
if runtime_cfg is None:
|
||||
continue
|
||||
strategy_map[node_type] = StrategyRuntimeConfig(
|
||||
node_type=node_type,
|
||||
config=strategy_class.default_config.model_copy(deep=True),
|
||||
)
|
||||
strategy_map[node_type] = runtime_cfg
|
||||
return strategy_map
|
||||
|
||||
|
||||
def _merge_strategy_runtime_config(
|
||||
runtime_cfg: StrategyRuntimeConfig,
|
||||
raw_override: Any,
|
||||
) -> StrategyRuntimeConfig:
|
||||
override = dict(raw_override or {})
|
||||
override.pop("node_type", None)
|
||||
|
||||
deprecated_keys = [key for key in ("force_sync", "load_mode") if key in override]
|
||||
if deprecated_keys:
|
||||
joined = ", ".join(deprecated_keys)
|
||||
raise ValueError(
|
||||
f"Strategy override for {runtime_cfg.node_type!r} contains unsupported legacy keys: {joined}. "
|
||||
"Use skip_sync under strategies.<node_type> or datasource.handler_configs instead."
|
||||
)
|
||||
|
||||
preset = override.pop("preset", None)
|
||||
if preset:
|
||||
runtime_cfg.config = runtime_cfg.config.model_validate(
|
||||
_deep_merge(
|
||||
runtime_cfg.config.model_dump(),
|
||||
ConfigPresets.get_preset(str(preset)).model_dump(exclude_defaults=True),
|
||||
)
|
||||
)
|
||||
|
||||
if "config" in override and isinstance(override["config"], dict):
|
||||
runtime_cfg.config = runtime_cfg.config.model_validate(
|
||||
_deep_merge(runtime_cfg.config.model_dump(), override.pop("config"))
|
||||
)
|
||||
|
||||
if override:
|
||||
runtime_cfg.config = runtime_cfg.config.model_validate(
|
||||
_deep_merge(runtime_cfg.config.model_dump(), override)
|
||||
)
|
||||
|
||||
strategy_class = DomainRegistry.get_strategy_class(runtime_cfg.node_type)
|
||||
runtime_cfg.config.domain_option = resolve_domain_option_config(
|
||||
getattr(strategy_class, "domain_option_model", None) if strategy_class is not None else None,
|
||||
runtime_cfg.config.domain_option,
|
||||
)
|
||||
|
||||
return runtime_cfg
|
||||
|
||||
|
||||
def _apply_strategy_overrides(
|
||||
node_types: list[str],
|
||||
strategy_overrides: Any,
|
||||
@@ -49,8 +106,16 @@ def _apply_strategy_overrides(
|
||||
|
||||
if isinstance(strategy_overrides, list):
|
||||
for item in strategy_overrides:
|
||||
runtime_cfg = StrategyRuntimeConfig.model_validate(item)
|
||||
strategy_map[runtime_cfg.node_type] = runtime_cfg
|
||||
raw_item = dict(item or {})
|
||||
node_type = str(raw_item.get("node_type") or "")
|
||||
if not node_type:
|
||||
raise ValueError("Strategy override list item missing required node_type")
|
||||
|
||||
runtime_cfg = strategy_map.get(node_type)
|
||||
if runtime_cfg is None:
|
||||
runtime_cfg = _build_default_strategy_runtime_config(node_type) or StrategyRuntimeConfig(node_type=node_type)
|
||||
|
||||
strategy_map[node_type] = _merge_strategy_runtime_config(runtime_cfg, raw_item)
|
||||
return [strategy_map[n] for n in node_types if n in strategy_map]
|
||||
|
||||
if not isinstance(strategy_overrides, dict):
|
||||
@@ -58,28 +123,10 @@ def _apply_strategy_overrides(
|
||||
|
||||
for node_type, raw_override in strategy_overrides.items():
|
||||
if node_type not in strategy_map:
|
||||
strategy_map[node_type] = StrategyRuntimeConfig(node_type=node_type)
|
||||
strategy_map[node_type] = _build_default_strategy_runtime_config(node_type) or StrategyRuntimeConfig(node_type=node_type)
|
||||
|
||||
runtime_cfg = strategy_map[node_type]
|
||||
override = dict(raw_override or {})
|
||||
|
||||
preset = override.pop("preset", None)
|
||||
if preset:
|
||||
runtime_cfg.config = ConfigPresets.get_preset(str(preset))
|
||||
|
||||
override.pop("skip_sync", None)
|
||||
override.pop("force_sync", None)
|
||||
override.pop("load_mode", None)
|
||||
|
||||
if "config" in override and isinstance(override["config"], dict):
|
||||
runtime_cfg.config = runtime_cfg.config.model_validate(
|
||||
_deep_merge(runtime_cfg.config.model_dump(), override.pop("config"))
|
||||
)
|
||||
|
||||
if override:
|
||||
runtime_cfg.config = runtime_cfg.config.model_validate(
|
||||
_deep_merge(runtime_cfg.config.model_dump(), override)
|
||||
)
|
||||
strategy_map[node_type] = _merge_strategy_runtime_config(runtime_cfg, raw_override)
|
||||
|
||||
return [strategy_map[n] for n in node_types if n in strategy_map]
|
||||
|
||||
@@ -107,7 +154,6 @@ def build_default_config(project_root: Path) -> PipelineRunConfig:
|
||||
"read_only": False,
|
||||
"handler_configs": {},
|
||||
},
|
||||
"target_project_ids": [],
|
||||
"persist": {
|
||||
"db_path": str(project_root / "test_results" / "sync_state_machine" / "pipeline_default.db"),
|
||||
"enable": True,
|
||||
@@ -129,29 +175,17 @@ def build_default_config(project_root: Path) -> PipelineRunConfig:
|
||||
def apply_config_overrides(config: PipelineRunConfig, overrides: Dict[str, Any]) -> PipelineRunConfig:
|
||||
merged = _deep_merge(config.model_dump(), overrides)
|
||||
raw_strategies = merged.pop("strategies", None)
|
||||
legacy_strategy_overrides = merged.pop("strategy_overrides", None)
|
||||
|
||||
resolved = PipelineRunConfig.model_validate(merged)
|
||||
|
||||
if raw_strategies is not None:
|
||||
resolved.strategies = _apply_strategy_overrides(resolved.node_types, raw_strategies)
|
||||
else:
|
||||
resolved.strategies = _apply_strategy_overrides(resolved.node_types, legacy_strategy_overrides or {})
|
||||
resolved.strategies = _apply_strategy_overrides(resolved.node_types, {})
|
||||
|
||||
return resolved
|
||||
|
||||
|
||||
def apply_env_overrides(
|
||||
config: PipelineRunConfig,
|
||||
project_root: Path,
|
||||
*,
|
||||
dotenv_file: Optional[Union[str, Path]] = None,
|
||||
) -> PipelineRunConfig:
|
||||
raise NotImplementedError(
|
||||
"Environment-based overrides are removed. Use config files via overrides_file instead."
|
||||
)
|
||||
|
||||
|
||||
def build_config(
|
||||
*,
|
||||
project_root: Path,
|
||||
@@ -187,3 +221,11 @@ def build_config(
|
||||
config = apply_config_overrides(config, overrides)
|
||||
|
||||
return config
|
||||
|
||||
|
||||
def build_config_from_file(
|
||||
*,
|
||||
project_root: Path,
|
||||
file_path: Union[str, Path],
|
||||
) -> PipelineRunConfig:
|
||||
return build_config(project_root=project_root, overrides_file=Path(file_path))
|
||||
|
||||
@@ -11,7 +11,6 @@ class BaseDataSourceConfig(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", validate_assignment=True)
|
||||
|
||||
type: str
|
||||
target_project_ids: List[str] = Field(default_factory=list)
|
||||
handler_configs: Dict[str, Dict[str, Any]] = Field(default_factory=dict)
|
||||
|
||||
|
||||
@@ -22,11 +21,6 @@ class JsonlDataSourceConfig(BaseDataSourceConfig):
|
||||
read_only: bool = False
|
||||
|
||||
|
||||
class DbDataSourceConfig(BaseDataSourceConfig):
|
||||
|
||||
type: Literal["db"] = "db"
|
||||
|
||||
|
||||
class ApiDataSourceConfig(BaseDataSourceConfig):
|
||||
|
||||
type: Literal["api"] = "api"
|
||||
@@ -38,7 +32,6 @@ class ApiDataSourceConfig(BaseDataSourceConfig):
|
||||
poll_interval: float = 0.5
|
||||
api_rate_limit_max_requests: int = 10
|
||||
api_rate_limit_window_seconds: float = 10.0
|
||||
target_project_ids: List[str] = Field(min_length=1)
|
||||
|
||||
|
||||
DataSourceConfig = Annotated[
|
||||
@@ -52,5 +45,4 @@ DataSourceConfig = Annotated[
|
||||
|
||||
|
||||
register_datasource_config_model("jsonl", JsonlDataSourceConfig)
|
||||
register_datasource_config_model("db", DbDataSourceConfig)
|
||||
register_datasource_config_model("api", ApiDataSourceConfig)
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
|
||||
from .run_presets import load_overrides_from_file
|
||||
|
||||
|
||||
class MultiProjectItemConfig(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", validate_assignment=True)
|
||||
|
||||
local_project_id: str
|
||||
remote_project_id: str
|
||||
config_path: Optional[str] = None
|
||||
|
||||
@property
|
||||
def scope(self) -> str:
|
||||
return f"project_id:{self.local_project_id}"
|
||||
|
||||
|
||||
class MultiProjectRunConfig(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", validate_assignment=True)
|
||||
|
||||
global_config: str
|
||||
default_project_config: str
|
||||
max_concurrency: int = 20
|
||||
project_bootstrap_node_types: List[str] = []
|
||||
projects: List[MultiProjectItemConfig]
|
||||
|
||||
|
||||
def is_multi_project_payload(payload: Dict[str, Any]) -> bool:
|
||||
return all(key in payload for key in ("global_config", "default_project_config", "projects"))
|
||||
|
||||
|
||||
def build_multi_project_config_from_file(*, project_root: Path, file_path: str | Path) -> MultiProjectRunConfig:
|
||||
path = Path(file_path)
|
||||
overrides = load_overrides_from_file(path, project_root)
|
||||
return MultiProjectRunConfig.model_validate(overrides)
|
||||
@@ -17,6 +17,7 @@ meta:
|
||||
# represented as separate invocations with different inputs.
|
||||
context:
|
||||
is_create_zombie: { type: bool, desc: "初始化时判定为 CREATE 僵尸:action=CREATE 且 (status=FAILED 或 data_id 为空)" }
|
||||
bootstrap_source_present: { type: bool, desc: "bootstrap 阶段 datasource 刷新后,当前节点是否在最新数据源中命中" }
|
||||
|
||||
has_binding_record: { type: bool, desc: "binding_manager 里存在绑定记录 (remote_id/local_id 非空)" }
|
||||
core_binding_result: { type: enum, values: [NORMAL, WARNING, ABNORMAL], desc: "SyncDecisionEngine.determine_core_status() 给本节点的结果(仅在 has_binding_record=true 时有效)" }
|
||||
@@ -304,19 +305,30 @@ states:
|
||||
status: SKIPPED
|
||||
data_id_exist: true
|
||||
|
||||
S17:
|
||||
binding_status: ABNORMAL
|
||||
action: NONE
|
||||
status: FAILED
|
||||
data_id_exist: [false, true]
|
||||
desc: |
|
||||
bootstrap 加载异常隔离态
|
||||
持久化恢复后存在绑定,但 datasource 刷新未命中最新数据
|
||||
binding_status: ABNORMAL
|
||||
action: NONE
|
||||
status: FAILED
|
||||
data_id_exist: false | true
|
||||
|
||||
pseudo_states:
|
||||
S15:
|
||||
desc: "节点被清理删除(zombie cleanup)"
|
||||
S16:
|
||||
desc: "持久化加载输入态:本轮 E01 的输入节点集合(可包含上轮遗留与已稳定节点)"
|
||||
S17:
|
||||
desc: "加载异常隔离态:持久化枚举非法,保留错误供人工处理,不进入自动流程"
|
||||
|
||||
stages:
|
||||
bootstrap:
|
||||
desc: "启动/初始化(仅针对持久化加载出的节点):僵尸清理与归并到 S00"
|
||||
desc: "启动/初始化:持久化恢复后先做 datasource 刷新,再按 E01 清理僵尸或归并/隔离"
|
||||
entry_states: [S16]
|
||||
exit_states: [S00, S15]
|
||||
exit_states: [S00, S15, S17]
|
||||
|
||||
bind:
|
||||
desc: "绑定判定:core matrix +(可选)依赖检查 +(可选)自动绑定;stage 仅约束边界,不要求内部顺序"
|
||||
@@ -346,13 +358,23 @@ stages:
|
||||
transitions:
|
||||
E01:
|
||||
stage: bootstrap
|
||||
desc: "初始化:删CREATE僵尸;其余重置后进入起点"
|
||||
desc: "初始化:先依据 datasource 刷新结果判定 source 是否存在,再做僵尸清理/隔离/归并"
|
||||
from: [S16]
|
||||
outcomes:
|
||||
- when: { is_create_zombie: true }
|
||||
to: S15
|
||||
- when: { is_create_zombie: false }
|
||||
- when: { is_create_zombie: false, bootstrap_source_present: false, has_binding_record: false }
|
||||
to: S15
|
||||
emit:
|
||||
note: "持久化孤儿节点未命中最新数据源,按僵尸清理"
|
||||
- when: { is_create_zombie: false, bootstrap_source_present: false, has_binding_record: true }
|
||||
to: S17
|
||||
emit:
|
||||
note: "存在绑定但最新数据源未返回节点,进入加载异常隔离"
|
||||
- when: { is_create_zombie: false, bootstrap_source_present: true }
|
||||
to: S00
|
||||
emit:
|
||||
note: "bootstrap 已刷新到最新数据,归并到起点"
|
||||
|
||||
# --- Bind phase1 core ---
|
||||
E10:
|
||||
|
||||
@@ -1,13 +1,70 @@
|
||||
from typing import Any, Dict, Literal
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
||||
|
||||
|
||||
class PersistConfig(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", validate_assignment=True)
|
||||
|
||||
backend: Literal["sqlite", "sqlalchemy", "custom"] = "sqlite"
|
||||
db_path: str
|
||||
backend: str = "sqlite"
|
||||
db_path: Optional[str] = None
|
||||
url: Optional[str] = None
|
||||
backend_options: Dict[str, Any] = Field(default_factory=dict)
|
||||
enable: bool = True
|
||||
wipe_on_start: bool = False
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def _validate_target(cls, data: Any) -> Any:
|
||||
if not isinstance(data, dict):
|
||||
return data
|
||||
|
||||
normalized = dict(data)
|
||||
normalized["backend"] = (str(normalized.get("backend") or "sqlite").strip().lower() or "sqlite")
|
||||
normalized["db_path"] = cls._normalize_optional_text(normalized.get("db_path"))
|
||||
normalized["url"] = cls._normalize_optional_text(normalized.get("url"))
|
||||
|
||||
if not normalized["db_path"] and not normalized["url"]:
|
||||
raise ValueError("persist requires either db_path or url")
|
||||
if normalized["backend"] != "sqlite" and not normalized["url"]:
|
||||
raise ValueError(f"persist.url is required for backend={normalized['backend']}")
|
||||
return normalized
|
||||
|
||||
@staticmethod
|
||||
def _normalize_optional_text(value: Optional[str]) -> Optional[str]:
|
||||
if value is None:
|
||||
return None
|
||||
normalized = str(value).strip()
|
||||
return normalized or None
|
||||
|
||||
def resolved_url(self) -> str:
|
||||
if self.url:
|
||||
return self.url
|
||||
if self.backend == "sqlite" and self.db_path:
|
||||
return f"sqlite+aiosqlite:///{self.db_path}"
|
||||
raise ValueError(f"persist.url is required for backend={self.backend}")
|
||||
|
||||
def resolved_db_path(self) -> Optional[str]:
|
||||
if self.db_path:
|
||||
return self.db_path
|
||||
if not self.url:
|
||||
return None
|
||||
|
||||
from sqlalchemy.engine import make_url
|
||||
|
||||
parsed = make_url(self.url)
|
||||
if parsed.get_backend_name() != "sqlite":
|
||||
return None
|
||||
database = parsed.database
|
||||
if not database:
|
||||
return None
|
||||
return str(database)
|
||||
|
||||
def wipe_file_path(self) -> Optional[str]:
|
||||
db_path = self.resolved_db_path()
|
||||
if not db_path or db_path == ":memory:":
|
||||
return None
|
||||
return db_path
|
||||
|
||||
def display_target(self) -> str:
|
||||
return self.url or self.db_path or "<unset>"
|
||||
|
||||
@@ -38,7 +38,6 @@ class PipelineRunConfig(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", validate_assignment=True)
|
||||
|
||||
node_types: List[str] = Field(default_factory=lambda: list(DEFAULT_NODE_TYPES))
|
||||
target_project_ids: List[str] = Field(default_factory=list)
|
||||
|
||||
local_datasource: DataSourceConfig
|
||||
remote_datasource: DataSourceConfig
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import importlib
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Tuple
|
||||
|
||||
@@ -29,43 +28,6 @@ def _replace_placeholders(value: Any, project_root: Path) -> Any:
|
||||
return value
|
||||
|
||||
|
||||
def _import_from_path(import_path: str) -> Any:
|
||||
module_path, sep, attr_path = import_path.partition(":")
|
||||
if not module_path:
|
||||
raise ValueError(f"Invalid import path: {import_path}")
|
||||
|
||||
module = importlib.import_module(module_path)
|
||||
if not sep:
|
||||
return module
|
||||
|
||||
obj: Any = module
|
||||
for attr_name in attr_path.split("."):
|
||||
obj = getattr(obj, attr_name)
|
||||
return obj
|
||||
|
||||
|
||||
def _apply_datasource_bootstraps(payload: Dict[str, Any]) -> Dict[str, Any]:
|
||||
extensions = payload.pop("extensions", None)
|
||||
if extensions is None:
|
||||
return payload
|
||||
if not isinstance(extensions, dict):
|
||||
raise ValueError("extensions must be an object mapping")
|
||||
|
||||
bootstrap_paths = extensions.get("datasource_bootstraps") or []
|
||||
if not isinstance(bootstrap_paths, list):
|
||||
raise ValueError("extensions.datasource_bootstraps must be a list")
|
||||
|
||||
for item in bootstrap_paths:
|
||||
if not isinstance(item, str) or not item.strip():
|
||||
raise ValueError("extensions.datasource_bootstraps entries must be non-empty strings")
|
||||
loaded = _import_from_path(item)
|
||||
if not callable(loaded):
|
||||
raise TypeError(f"datasource bootstrap must resolve to a callable: {item}")
|
||||
loaded()
|
||||
|
||||
return payload
|
||||
|
||||
|
||||
def list_preset_names() -> List[str]:
|
||||
return sorted(PRESET_FILES.keys())
|
||||
|
||||
@@ -115,7 +77,8 @@ def load_overrides_from_file(file_path: Path, project_root: Path) -> Dict[str, A
|
||||
resolved = _replace_placeholders(payload, project_root)
|
||||
if not isinstance(resolved, dict):
|
||||
raise ValueError(f"Preset file must contain object mapping: {file_path}")
|
||||
return _apply_datasource_bootstraps(resolved)
|
||||
resolved.pop("extensions", None)
|
||||
return resolved
|
||||
|
||||
|
||||
def load_named_preset_overrides(project_root: Path, preset_name: str) -> Tuple[Dict[str, Any], Path, bool]:
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
Prefer importing from `sync_state_machine.config` directly.
|
||||
"""
|
||||
|
||||
from .datasource_config import ApiDataSourceConfig, DbDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
|
||||
from .datasource_config import ApiDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
|
||||
from .persist_config import PersistConfig
|
||||
from .logging_config import LoggingConfig
|
||||
from .strategy_config import (
|
||||
@@ -14,17 +14,22 @@ from .strategy_config import (
|
||||
ConfigPresets,
|
||||
)
|
||||
from .pipeline_config import PipelineRunConfig, DEFAULT_NODE_TYPES
|
||||
from .multi_project_config import (
|
||||
MultiProjectItemConfig,
|
||||
MultiProjectRunConfig,
|
||||
build_multi_project_config_from_file,
|
||||
is_multi_project_payload,
|
||||
)
|
||||
from .builder import (
|
||||
build_default_config,
|
||||
build_config,
|
||||
build_config_from_file,
|
||||
apply_config_overrides,
|
||||
apply_env_overrides,
|
||||
config_snapshot,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"ApiDataSourceConfig",
|
||||
"DbDataSourceConfig",
|
||||
"JsonlDataSourceConfig",
|
||||
"DataSourceConfig",
|
||||
"PersistConfig",
|
||||
@@ -36,9 +41,13 @@ __all__ = [
|
||||
"ConfigPresets",
|
||||
"PipelineRunConfig",
|
||||
"DEFAULT_NODE_TYPES",
|
||||
"MultiProjectItemConfig",
|
||||
"MultiProjectRunConfig",
|
||||
"build_multi_project_config_from_file",
|
||||
"is_multi_project_payload",
|
||||
"build_default_config",
|
||||
"build_config",
|
||||
"build_config_from_file",
|
||||
"apply_config_overrides",
|
||||
"apply_env_overrides",
|
||||
"config_snapshot",
|
||||
]
|
||||
|
||||
@@ -1,11 +1,21 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from enum import Enum
|
||||
from typing import Dict, List, Optional
|
||||
from typing import Any, Dict, List, Mapping, Type
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
|
||||
def resolve_domain_option_config(
|
||||
domain_option_model: Type[BaseModel] | None,
|
||||
domain_option: Mapping[str, Any] | None,
|
||||
) -> Dict[str, Any]:
|
||||
if domain_option_model is None:
|
||||
return dict(domain_option or {})
|
||||
return domain_option_model.model_validate(domain_option or {}).model_dump()
|
||||
|
||||
|
||||
class OrphanAction(str, Enum):
|
||||
CREATE_REMOTE = "create_remote"
|
||||
CREATE_LOCAL = "create_local"
|
||||
@@ -17,7 +27,6 @@ class OrphanAction(str, Enum):
|
||||
class UpdateDirection(str, Enum):
|
||||
PUSH = "push"
|
||||
PULL = "pull"
|
||||
BIDIRECTIONAL = "bidirectional"
|
||||
NONE = "none"
|
||||
|
||||
|
||||
@@ -40,18 +49,76 @@ class StrategyConfig(BaseModel):
|
||||
remote_orphan_action: OrphanAction = OrphanAction.NONE
|
||||
update_direction: UpdateDirection = UpdateDirection.PUSH
|
||||
|
||||
# 字段级同步方向覆盖。
|
||||
# field_direction_key: 节点 data 中用作查表键的字段名,如 "key"。
|
||||
# field_direction_overrides: 字段值 → UpdateDirection 的映射。
|
||||
# 未命中映射的节点沿用 update_direction 作为默认方向。
|
||||
field_direction_key: Optional[str] = None
|
||||
field_direction_overrides: Dict[str, UpdateDirection] = Field(default_factory=dict)
|
||||
# post-check 一致性比较时忽略的顶层字段。
|
||||
# 只影响最终一致性评估,不影响 create/update 的差异检测。
|
||||
post_check_ignore_fields: List[str] = Field(default_factory=list)
|
||||
|
||||
# post-check 一致性比较时,忽略列表型字段内各元素的特定子键。
|
||||
# 格式: {字段名: [子键, ...]},如 {"plan_node": ["is_audit"]}。
|
||||
# 用于排除后端硬写死、永远与本地不一致的字段,避免误报。
|
||||
post_check_ignore_list_item_fields: Dict[str, List[str]] = Field(default_factory=dict)
|
||||
|
||||
# schema diff / validate 时忽略的顶层字段。
|
||||
compare_ignore_fields: List[str] = Field(default_factory=list)
|
||||
|
||||
# 当 auto-bind 命中“多对一”候选时,允许稳定选择一条进行绑定,剩余候选按 orphan/create 流程继续处理。
|
||||
# 默认关闭,避免在一般场景下引入隐式匹配。
|
||||
allow_many_to_one_auto_bind: bool = False
|
||||
|
||||
# 是否跳过该类型的 create/update 阶段。
|
||||
skip_sync: bool = False
|
||||
|
||||
# 是否跳过该类型的 post-check reload 与一致性校验。
|
||||
skip_post_check: bool = False
|
||||
|
||||
# 领域策略运行参数,由各 strategy 的 domain_option_model 负责解释与校验。
|
||||
domain_option: Dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
def log_logic_warnings(self, *, node_type: str, logger: logging.Logger) -> None:
|
||||
direction = self.update_direction
|
||||
local_action = self.local_orphan_action
|
||||
remote_action = self.remote_orphan_action
|
||||
|
||||
if direction == UpdateDirection.PUSH:
|
||||
if local_action == OrphanAction.CREATE_LOCAL:
|
||||
logger.warning(
|
||||
f"[{node_type}] Config inconsistency: "
|
||||
f"update_direction=PUSH but local_orphan_action=CREATE_LOCAL (本地创建)。"
|
||||
f"应使用 CREATE_REMOTE (推送到远程)"
|
||||
)
|
||||
if remote_action == OrphanAction.CREATE_LOCAL:
|
||||
logger.warning(
|
||||
f"[{node_type}] Config inconsistency: "
|
||||
f"update_direction=PUSH but remote_orphan_action=CREATE_LOCAL (本地创建)。"
|
||||
f"PUSH 模式应设为 NONE 或 DELETE_REMOTE"
|
||||
)
|
||||
elif direction == UpdateDirection.PULL:
|
||||
if local_action == OrphanAction.CREATE_REMOTE:
|
||||
logger.warning(
|
||||
f"[{node_type}] Config inconsistency: "
|
||||
f"update_direction=PULL but local_orphan_action=CREATE_REMOTE (推送到远程)。"
|
||||
f"PULL 模式应设为 NONE 或 DELETE_LOCAL"
|
||||
)
|
||||
if remote_action in {OrphanAction.CREATE_REMOTE, OrphanAction.DELETE_LOCAL}:
|
||||
logger.warning(
|
||||
f"[{node_type}] Config inconsistency: "
|
||||
f"update_direction=PULL but remote_orphan_action={remote_action}。"
|
||||
f"PULL 模式建议使用 CREATE_LOCAL 或 NONE"
|
||||
)
|
||||
elif direction == UpdateDirection.NONE:
|
||||
if local_action != OrphanAction.NONE:
|
||||
logger.warning(
|
||||
f"[{node_type}] Config inconsistency: "
|
||||
f"update_direction=NONE but local_orphan_action={local_action}。"
|
||||
f"应设为 NONE"
|
||||
)
|
||||
if remote_action != OrphanAction.NONE:
|
||||
logger.warning(
|
||||
f"[{node_type}] Config inconsistency: "
|
||||
f"update_direction=NONE but remote_orphan_action={remote_action}。"
|
||||
f"应设为 NONE"
|
||||
)
|
||||
|
||||
|
||||
class StrategyRuntimeConfig(BaseModel):
|
||||
"""Pipeline 层策略运行配置(按 node_type)。"""
|
||||
|
||||
@@ -23,13 +23,22 @@ from .handler import (
|
||||
from .task_result import TaskResult, TaskStatus
|
||||
|
||||
# API DataSource
|
||||
from .api import ApiClient, ApiDataSource, BaseApiHandler
|
||||
|
||||
# DB DataSource
|
||||
from .db import DbDataSource, BaseDbHandler
|
||||
from .api import ApiClient, ApiDataSource, BaseApiHandler, register as register_api_datasource
|
||||
|
||||
# JSONL DataSource
|
||||
from .jsonl import JsonlDataSource, BaseJsonlHandler
|
||||
from .jsonl import JsonlDataSource, BaseJsonlHandler, register as register_jsonl_datasource
|
||||
|
||||
|
||||
_BUILTIN_DATASOURCE_TYPES_REGISTERED = False
|
||||
|
||||
|
||||
def ensure_builtin_datasource_types_registered(*, replace: bool = False) -> None:
|
||||
global _BUILTIN_DATASOURCE_TYPES_REGISTERED
|
||||
if _BUILTIN_DATASOURCE_TYPES_REGISTERED and not replace:
|
||||
return
|
||||
register_jsonl_datasource(replace=replace)
|
||||
register_api_datasource(replace=replace)
|
||||
_BUILTIN_DATASOURCE_TYPES_REGISTERED = True
|
||||
|
||||
__all__ = [
|
||||
# 基础类
|
||||
@@ -46,16 +55,13 @@ __all__ = [
|
||||
"TaskResult",
|
||||
"TaskStatus",
|
||||
"ValidationResult",
|
||||
"ensure_builtin_datasource_types_registered",
|
||||
|
||||
# API DataSource
|
||||
"ApiClient",
|
||||
"ApiDataSource",
|
||||
"BaseApiHandler",
|
||||
|
||||
# DB DataSource
|
||||
"DbDataSource",
|
||||
"BaseDbHandler",
|
||||
|
||||
# JSONL DataSource
|
||||
"JsonlDataSource",
|
||||
"BaseJsonlHandler",
|
||||
|
||||
@@ -1,15 +1,50 @@
|
||||
from __future__ import annotations
|
||||
|
||||
"""
|
||||
API DataSource 模块
|
||||
|
||||
提供基于 HTTP API 的数据源实现。
|
||||
"""
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from ..type_registry import register_datasource_type
|
||||
|
||||
from .client import ApiClient
|
||||
from .datasource import ApiDataSource
|
||||
from .handler import BaseApiHandler
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ...config.datasource_config import ApiDataSourceConfig
|
||||
|
||||
|
||||
def build_api_datasource(config: ApiDataSourceConfig, endpoint_name: str, **_: object):
|
||||
del endpoint_name
|
||||
return ApiDataSource(
|
||||
base_url=config.api_base_url,
|
||||
uid=config.api_uid,
|
||||
secret=config.api_secret,
|
||||
debug=config.api_debug,
|
||||
poll_max_retries=config.poll_max_retries,
|
||||
poll_interval=config.poll_interval,
|
||||
rate_limit_max_requests=config.api_rate_limit_max_requests,
|
||||
rate_limit_window_seconds=config.api_rate_limit_window_seconds,
|
||||
)
|
||||
|
||||
|
||||
def register(*, replace: bool = False) -> None:
|
||||
from ...config.datasource_config import ApiDataSourceConfig
|
||||
|
||||
register_datasource_type(
|
||||
"api",
|
||||
config_model=ApiDataSourceConfig,
|
||||
factory=build_api_datasource,
|
||||
replace=replace,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"ApiClient",
|
||||
"ApiDataSource",
|
||||
"BaseApiHandler",
|
||||
"build_api_datasource",
|
||||
"register",
|
||||
]
|
||||
|
||||
@@ -23,6 +23,7 @@ from datetime import datetime
|
||||
import httpx
|
||||
|
||||
from schemas.common.push_system import PushIdsSchema
|
||||
from ...logging import get_logger, get_scope_display_name
|
||||
|
||||
|
||||
def _build_default_push_steps() -> List[Dict[str, str]]:
|
||||
@@ -33,6 +34,7 @@ def _build_default_push_steps() -> List[Dict[str, str]]:
|
||||
class ApiClient:
|
||||
"""API HTTP 客户端"""
|
||||
_TRACE_TEXT_MAX_LEN = 1000
|
||||
_REQUEST_STATS_LOG_PREFIX = "📊 ApiClient request stats"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -41,6 +43,7 @@ class ApiClient:
|
||||
secret: str,
|
||||
debug: bool = False,
|
||||
log_file: Optional[Path] = None,
|
||||
session_label: Optional[str] = None,
|
||||
rate_limit_max_requests: int = 30,
|
||||
rate_limit_window_seconds: float = 10.0,
|
||||
):
|
||||
@@ -62,16 +65,58 @@ class ApiClient:
|
||||
self._client: Optional[httpx.AsyncClient] = None
|
||||
self._debug = debug
|
||||
self._log_file = log_file
|
||||
self._scope = ""
|
||||
self._session_label = (session_label or "").strip()
|
||||
self._trace_seq = 0
|
||||
self._trace_by_push_id: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
|
||||
self._trace_by_biz_id: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
|
||||
self._trace_by_op: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
|
||||
self._load_trace_by_item: Dict[tuple[str, str], deque[Dict[str, Any]]] = defaultdict(deque)
|
||||
self._logger = logging.getLogger(__name__)
|
||||
self._logger = get_logger(__name__)
|
||||
self._rate_limit_max_requests = int(rate_limit_max_requests)
|
||||
self._rate_limit_window_seconds = float(rate_limit_window_seconds)
|
||||
self._rate_limit_lock = asyncio.Lock()
|
||||
self._rate_limit_timestamps: deque[float] = deque()
|
||||
self._request_total = 0
|
||||
self._request_success = 0
|
||||
self._request_failure = 0
|
||||
|
||||
def set_scope(self, scope: Optional[str]) -> None:
|
||||
self._scope = str(scope or "").strip()
|
||||
|
||||
def set_session_label(self, label: Optional[str]) -> None:
|
||||
self._session_label = str(label or "").strip()
|
||||
|
||||
def get_log_session_label(self) -> str:
|
||||
if self._session_label:
|
||||
return self._session_label
|
||||
scope_label = get_scope_display_name(self._scope)
|
||||
return scope_label or self._scope
|
||||
|
||||
def _project_name_from_request(self, payload: Optional[Dict[str, Any]]) -> str:
|
||||
if not isinstance(payload, dict):
|
||||
return ""
|
||||
project_id = str(payload.get("project_id") or "").strip()
|
||||
if not project_id:
|
||||
return ""
|
||||
display = get_scope_display_name(f"project_id:{project_id}")
|
||||
if not display or display == f"project_id:{project_id}":
|
||||
return ""
|
||||
return display
|
||||
|
||||
def _resolve_request_session_label(
|
||||
self,
|
||||
*,
|
||||
request_params: Optional[Dict[str, Any]] = None,
|
||||
request_payload: Optional[Dict[str, Any]] = None,
|
||||
) -> str:
|
||||
payload_label = self._project_name_from_request(request_payload)
|
||||
if payload_label:
|
||||
return payload_label
|
||||
params_label = self._project_name_from_request(request_params)
|
||||
if params_label:
|
||||
return params_label
|
||||
return self.get_log_session_label()
|
||||
|
||||
def _is_rate_limit_enabled(self) -> bool:
|
||||
return self._rate_limit_max_requests > 0 and self._rate_limit_window_seconds > 0
|
||||
@@ -105,10 +150,55 @@ class ApiClient:
|
||||
|
||||
async def close(self):
|
||||
"""关闭 HTTP 客户端"""
|
||||
self.log_request_stats(level="INFO")
|
||||
if self._client:
|
||||
await self._client.aclose()
|
||||
self._client = None
|
||||
|
||||
def get_request_stats(self) -> Dict[str, int]:
|
||||
"""返回当前客户端生命周期内的请求统计。"""
|
||||
return {
|
||||
"total": self._request_total,
|
||||
"success": self._request_success,
|
||||
"failure": self._request_failure,
|
||||
}
|
||||
|
||||
def format_request_stats(self) -> str:
|
||||
"""格式化当前客户端生命周期内的请求统计。"""
|
||||
stats = self.get_request_stats()
|
||||
session_text = self.get_log_session_label()
|
||||
session_prefix = f"session={session_text} " if session_text else ""
|
||||
return (
|
||||
f"{self._REQUEST_STATS_LOG_PREFIX}: "
|
||||
f"{session_prefix}"
|
||||
f"total={stats['total']}, success={stats['success']}, failure={stats['failure']}"
|
||||
)
|
||||
|
||||
def log_request_stats(self, level: str = "DEBUG") -> None:
|
||||
"""输出当前客户端生命周期内的请求统计。"""
|
||||
self._log(self.format_request_stats(), level=level)
|
||||
|
||||
def _log_request_summary(
|
||||
self,
|
||||
*,
|
||||
method: str,
|
||||
endpoint: str,
|
||||
elapsed_seconds: float,
|
||||
request_params: Optional[Dict[str, Any]] = None,
|
||||
request_payload: Optional[Dict[str, Any]] = None,
|
||||
) -> None:
|
||||
if not self._debug:
|
||||
return
|
||||
session_text = self._resolve_request_session_label(
|
||||
request_params=request_params,
|
||||
request_payload=request_payload,
|
||||
)
|
||||
session_prefix = f"session={session_text} " if session_text else ""
|
||||
self._log(
|
||||
f"🔎 ApiClient request: {session_prefix}method={method.upper()} endpoint={endpoint} elapsed={elapsed_seconds:.3f}s",
|
||||
level="INFO",
|
||||
)
|
||||
|
||||
def _log(self, message: str, level: str = "DEBUG"):
|
||||
"""
|
||||
统一日志输出
|
||||
@@ -382,6 +472,7 @@ class ApiClient:
|
||||
client = self._get_client()
|
||||
url = f"{self._base_url}{endpoint}"
|
||||
await self._acquire_rate_limit_slot()
|
||||
self._request_total += 1
|
||||
|
||||
# 记录请求信息
|
||||
self._log(f"\n{'='*60}", level="DEBUG")
|
||||
@@ -433,6 +524,13 @@ class ApiClient:
|
||||
|
||||
# 记录响应时间
|
||||
elapsed = time.time() - start_time
|
||||
self._log_request_summary(
|
||||
method=method,
|
||||
endpoint=endpoint,
|
||||
elapsed_seconds=elapsed,
|
||||
request_params=params,
|
||||
request_payload=data if method.upper() in ['POST', 'PUT', 'DELETE'] else None,
|
||||
)
|
||||
self._log(f"Response Time: {elapsed:.3f}s", level="DEBUG")
|
||||
|
||||
# 检查 HTTP 状态
|
||||
@@ -453,14 +551,23 @@ class ApiClient:
|
||||
response=result,
|
||||
elapsed_ms=int(elapsed * 1000),
|
||||
)
|
||||
self._request_success += 1
|
||||
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self._request_failure += 1
|
||||
|
||||
# 记录错误
|
||||
elapsed = time.time() - start_time
|
||||
self._log_request_summary(
|
||||
method=method,
|
||||
endpoint=endpoint,
|
||||
elapsed_seconds=elapsed,
|
||||
request_params=params,
|
||||
request_payload=data if method.upper() in ['POST', 'PUT', 'DELETE'] else None,
|
||||
)
|
||||
error_msg = "\n========== [ApiClient HTTP Error] ==========\n"
|
||||
error_msg += f"Request: {method} {url}\n"
|
||||
error_msg += f"Params: {params}\n"
|
||||
|
||||
@@ -12,9 +12,10 @@ import json
|
||||
from typing import Optional
|
||||
from pathlib import Path
|
||||
from ..datasource import BaseDataSource
|
||||
from ..handler import NodeHandler, ensure_handler_compat
|
||||
from ..handler import NodeHandler
|
||||
from ..task_result import TaskResult
|
||||
from .client import ApiClient
|
||||
from ...logging import set_scope_display_name
|
||||
|
||||
|
||||
class ApiDataSource(BaseDataSource):
|
||||
@@ -57,6 +58,29 @@ class ApiDataSource(BaseDataSource):
|
||||
)
|
||||
self.poll_mode = poll_mode
|
||||
|
||||
def set_collection(self, collection) -> None:
|
||||
super().set_collection(collection)
|
||||
self.client.set_scope(getattr(collection, "scope", ""))
|
||||
|
||||
def _update_client_session_label_from_project_node(self, node) -> None:
|
||||
if self._collection is None or getattr(self._collection, "scope", "") == "global":
|
||||
return
|
||||
if getattr(node, "node_type", None) != "project":
|
||||
return
|
||||
if not hasattr(node, "get_data"):
|
||||
return
|
||||
|
||||
data = node.get_data()
|
||||
if not isinstance(data, dict):
|
||||
return
|
||||
|
||||
project_name = str(data.get("name") or "").strip()
|
||||
if not project_name:
|
||||
return
|
||||
|
||||
set_scope_display_name(self._collection.scope, f"project:{project_name}")
|
||||
self.client.set_session_label(f"project:{project_name}")
|
||||
|
||||
async def initialize(self):
|
||||
"""初始化数据源(初始化 HTTP 客户端)"""
|
||||
await self.client.initialize()
|
||||
@@ -65,23 +89,10 @@ class ApiDataSource(BaseDataSource):
|
||||
"""关闭数据源(关闭 HTTP 客户端)"""
|
||||
await self.client.close()
|
||||
|
||||
def register_handler(self, handler: NodeHandler) -> None:
|
||||
"""
|
||||
注册业务 Handler
|
||||
|
||||
Args:
|
||||
handler: API Handler 实例
|
||||
"""
|
||||
compatible_handler = ensure_handler_compat(handler)
|
||||
compatible_handler.set_api_client(self.client)
|
||||
compatible_handler.set_poll_mode(self.poll_mode)
|
||||
|
||||
# 调用基类注册方法
|
||||
super().register_handler(compatible_handler)
|
||||
|
||||
# 如果 collection 已设置,确保新 handler 拿到引用
|
||||
if self._collection is not None:
|
||||
compatible_handler.set_collection(self._collection) # type: ignore[arg-type]
|
||||
def _prepare_handler_for_registration(self, handler: NodeHandler) -> None:
|
||||
handler.set_api_client(self.client)
|
||||
handler.set_poll_mode(self.poll_mode)
|
||||
super()._prepare_handler_for_registration(handler)
|
||||
|
||||
@staticmethod
|
||||
def _fmt_trace_value(value, max_len: int = 600) -> str:
|
||||
@@ -188,6 +199,7 @@ class ApiDataSource(BaseDataSource):
|
||||
return None
|
||||
|
||||
def _on_node_loaded(self, handler: NodeHandler, node) -> None:
|
||||
self._update_client_session_label_from_project_node(node)
|
||||
data_id = node.data_id or ""
|
||||
if not data_id:
|
||||
return
|
||||
|
||||
@@ -15,11 +15,13 @@ BaseApiHandler - API Handler 基类
|
||||
import uuid
|
||||
import asyncio
|
||||
from abc import abstractmethod
|
||||
from copy import deepcopy
|
||||
from typing import ClassVar, Dict, List, Any, Optional, TYPE_CHECKING, TypeVar
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, ValidationError
|
||||
|
||||
from ...common.type_safety import get_pydantic_model_fields
|
||||
from ..handler import NodeHandler
|
||||
from ...logging import get_logger
|
||||
from ..handler import BaseNodeHandler
|
||||
from ..task_result import TaskResult
|
||||
from ..handler import ValidationResult # Import ValidationResult
|
||||
|
||||
@@ -29,9 +31,10 @@ if TYPE_CHECKING:
|
||||
from .client import ApiClient
|
||||
|
||||
T = TypeVar("T", bound=BaseModel)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class BaseApiHandler(NodeHandler[T]):
|
||||
class BaseApiHandler(BaseNodeHandler[T]):
|
||||
"""API Handler 基类"""
|
||||
|
||||
# 子类需要设置这些属性
|
||||
@@ -39,12 +42,22 @@ class BaseApiHandler(NodeHandler[T]):
|
||||
_node_class: type['SyncNode']
|
||||
_schema: Any
|
||||
|
||||
def __init__(self):
|
||||
# 子类应该在调用 super().__init__() 之前或之后设置 _node_type, _node_class, _schema
|
||||
def __init__(self, **handler_config: Any):
|
||||
# 子类可直接传入 node_class;少数特殊场景可显式覆盖 schema/node_type。
|
||||
node_class = handler_config.pop("node_class", None)
|
||||
node_type = handler_config.pop("node_type", None)
|
||||
schema = handler_config.pop("schema", None)
|
||||
|
||||
if node_class is not None and node_type is None:
|
||||
node_type = getattr(node_class, "node_type", None)
|
||||
if node_class is not None and schema is None:
|
||||
schema = getattr(node_class, "schema", None)
|
||||
|
||||
super().__init__(node_type=node_type, node_class=node_class, schema=schema)
|
||||
self.api_client: 'ApiClient' = None # type: ignore # 由 DataSource 注入
|
||||
self.poll_mode: str = "async" # async | sync
|
||||
self._collection: 'DataCollection' = None # type: ignore # 由 DataSource 注入
|
||||
self.handler_config: Dict[str, Any] = {}
|
||||
self.handler_config = dict(handler_config)
|
||||
# 子类在 __init__ 中设置此列表以声明 create/update 中涉及的 Pydantic schema 类;
|
||||
# post-check 将只比较这些 schema 覆盖的字段,过滤后端只读的派生字段。
|
||||
self.update_schemas: List[type] = []
|
||||
@@ -63,14 +76,17 @@ class BaseApiHandler(NodeHandler[T]):
|
||||
|
||||
def set_collection(self, collection: 'DataCollection') -> None:
|
||||
"""设置 Collection(由 DataSource 调用)"""
|
||||
self._collection = collection
|
||||
super().set_collection(collection)
|
||||
|
||||
def set_handler_config(self, config: Dict[str, Any]) -> None:
|
||||
self.handler_config = dict(config)
|
||||
def ensure_api_client(self) -> 'ApiClient':
|
||||
if self.api_client is None:
|
||||
raise RuntimeError("API client not set. Call set_api_client() before invoking API handler methods.")
|
||||
return self.api_client
|
||||
|
||||
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
|
||||
"""默认忽略项目过滤列表,具体子类按需覆盖。"""
|
||||
return
|
||||
def ensure_collection(self) -> 'DataCollection':
|
||||
if self._collection is None:
|
||||
raise RuntimeError("Collection not set. Call set_collection() before invoking handler methods.")
|
||||
return self._collection
|
||||
|
||||
@property
|
||||
def node_type(self) -> str:
|
||||
@@ -114,8 +130,7 @@ class BaseApiHandler(NodeHandler[T]):
|
||||
def _create_basic_node(self, data: Dict[str, Any], depend_ids: Optional[List[str]] = None) -> "SyncNode":
|
||||
"""创建或复用基础节点(默认使用 data['id'] 或 data['_id'])"""
|
||||
data_id = str(data.get("id") or data.get("_id") or "")
|
||||
if self._collection is None:
|
||||
raise RuntimeError("Collection not set. Call set_collection() before creating nodes.")
|
||||
self.ensure_collection()
|
||||
|
||||
if data_id:
|
||||
existing_node = self._collection.get_by_data_id(self.node_type, data_id)
|
||||
@@ -139,6 +154,44 @@ class BaseApiHandler(NodeHandler[T]):
|
||||
"""默认节点创建(子类可覆盖并补充 context)"""
|
||||
return self._create_basic_node(data)
|
||||
|
||||
def _get_committed_approval_snapshot(self, node: 'SyncNode') -> Dict[str, Any]:
|
||||
snapshot = node.context.get("approval_snapshot")
|
||||
return snapshot if isinstance(snapshot, dict) else {}
|
||||
|
||||
def _get_pending_approval_snapshot(self, node: 'SyncNode') -> Dict[str, Any]:
|
||||
snapshot = node.context.get("pending_approval_snapshot")
|
||||
return snapshot if isinstance(snapshot, dict) else {}
|
||||
|
||||
def _get_active_approval_snapshot(self, node: 'SyncNode') -> Dict[str, Any]:
|
||||
pending_snapshot = self._get_pending_approval_snapshot(node)
|
||||
if pending_snapshot:
|
||||
return pending_snapshot
|
||||
return self._get_committed_approval_snapshot(node)
|
||||
|
||||
def _get_request_steps(self, node: 'SyncNode') -> List[Dict[str, Any]]:
|
||||
snapshot = self._get_active_approval_snapshot(node)
|
||||
steps = snapshot.get("steps")
|
||||
if isinstance(steps, list):
|
||||
return deepcopy(steps)
|
||||
return []
|
||||
|
||||
def _steps_changed(self, node: 'SyncNode') -> bool:
|
||||
snapshot = self._get_active_approval_snapshot(node)
|
||||
return bool(snapshot.get("steps_changed"))
|
||||
|
||||
def _attach_request_steps(self, request_data: Dict[str, Any], node: Optional['SyncNode']) -> Dict[str, Any]:
|
||||
payload = dict(request_data)
|
||||
payload["steps"] = self._get_request_steps(node) if node is not None else []
|
||||
return payload
|
||||
|
||||
def _mark_approval_snapshot_synced(self, node: 'SyncNode') -> None:
|
||||
snapshot = self._get_active_approval_snapshot(node)
|
||||
if snapshot:
|
||||
committed_snapshot = deepcopy(snapshot)
|
||||
committed_snapshot.pop("steps_changed", None)
|
||||
node.context["approval_snapshot"] = committed_snapshot
|
||||
node.context.pop("pending_approval_snapshot", None)
|
||||
|
||||
def extract_id(self, data: Dict[str, Any]) -> str:
|
||||
"""从原始数据提取 ID(默认优先 id,其次 _id)"""
|
||||
return str(data.get("id") or data.get("_id") or "")
|
||||
@@ -216,15 +269,60 @@ class BaseApiHandler(NodeHandler[T]):
|
||||
except Exception as e:
|
||||
return str(e)
|
||||
|
||||
def _get_schema_diff(
|
||||
def _get_update_schema_payload(
|
||||
self,
|
||||
schema: Any,
|
||||
data: Dict[str, Any],
|
||||
*,
|
||||
require_valid: bool,
|
||||
) -> Dict[str, Any] | None:
|
||||
"""Update 内部实现:提取指定 schema 下的 payload。
|
||||
|
||||
这里明确只服务“update schema”,不负责 response model 一致性校验。
|
||||
这个方法只给 BaseApiHandler 的 update 入口方法复用,不作为 handler 入口。
|
||||
|
||||
处理顺序:
|
||||
1. 先尝试用 update schema 校验,成功则直接使用校验后的 model_dump。
|
||||
2. 如果校验失败且 require_valid=False,则退化为只抽取 schema 定义的字段。
|
||||
|
||||
之所以允许退化,是因为 update 比较时两边快照常常不完整:
|
||||
某些字段在旧快照缺失,但新数据已经有值。此时应该继续比较 schema
|
||||
相关字段,而不是把它误判成“无差异”。
|
||||
|
||||
空字符串和 None 会被统一过滤掉,因为当前 update handler 并不把它们
|
||||
当成“清空字段”的显式语义,而是当作“不参与本次更新”。
|
||||
"""
|
||||
try:
|
||||
validated = schema.model_validate(data)
|
||||
payload = validated.model_dump(exclude_unset=True)
|
||||
except ValidationError:
|
||||
if require_valid:
|
||||
return None
|
||||
|
||||
schema_fields = get_pydantic_model_fields(schema)
|
||||
if not schema_fields:
|
||||
return {}
|
||||
payload = {
|
||||
field_name: data.get(field_name)
|
||||
for field_name in schema_fields.keys()
|
||||
if field_name in data
|
||||
}
|
||||
|
||||
return {
|
||||
field_name: value
|
||||
for field_name, value in payload.items()
|
||||
if value not in (None, "")
|
||||
}
|
||||
|
||||
def get_update_schema_diff(
|
||||
self,
|
||||
schema,
|
||||
data: Dict[str, Any],
|
||||
origin_data: Dict[str, Any],
|
||||
node_id: Optional[str] = None
|
||||
node_id: Optional[str] = None,
|
||||
force_full_schema: bool = False,
|
||||
) -> Dict[str, Any] | None:
|
||||
"""
|
||||
获取数据在指定 schema 下的差异字段
|
||||
"""Update 入口:获取指定 update schema 下的差异字段。
|
||||
|
||||
Args:
|
||||
schema: Pydantic schema 类
|
||||
@@ -235,43 +333,37 @@ class BaseApiHandler(NodeHandler[T]):
|
||||
Returns:
|
||||
Dict[str, Any] | None: 有差异返回差异字段字典,无差异或字段不全返回 None
|
||||
"""
|
||||
from pydantic import ValidationError
|
||||
# 当前待发送的数据必须能构成合法 update payload;允许旧快照不完整。
|
||||
new_dict = self._get_update_schema_payload(schema, data, require_valid=True)
|
||||
origin_dict = self._get_update_schema_payload(schema, origin_data, require_valid=False)
|
||||
|
||||
try:
|
||||
new_model = schema.model_validate(data)
|
||||
origin_model = schema.model_validate(origin_data)
|
||||
|
||||
new_dict = new_model.model_dump(exclude_unset=True)
|
||||
origin_dict = origin_model.model_dump(exclude_unset=True)
|
||||
|
||||
if new_dict == origin_dict:
|
||||
return None
|
||||
|
||||
# 返回有差异的字段
|
||||
diff_fields = {k: v for k, v in new_dict.items() if origin_dict.get(k) != v}
|
||||
|
||||
if diff_fields:
|
||||
# 打印差异详情
|
||||
node_info = f"[{node_id}]" if node_id else ""
|
||||
print(f" [DIFF] {self.node_type}{node_info} schema={schema.__name__}")
|
||||
for k, v in diff_fields.items():
|
||||
orig_v = origin_dict.get(k)
|
||||
print(f" • {k}: {orig_v!r} → {v!r}")
|
||||
|
||||
return diff_fields if diff_fields else None
|
||||
|
||||
except ValidationError:
|
||||
# schema 字段不全,视为无差异
|
||||
if new_dict is None or origin_dict is None:
|
||||
return None
|
||||
|
||||
def _filter_update_data(
|
||||
if new_dict == origin_dict:
|
||||
if force_full_schema:
|
||||
return new_dict
|
||||
return None
|
||||
|
||||
diff_fields = {k: v for k, v in new_dict.items() if origin_dict.get(k) != v}
|
||||
|
||||
if diff_fields:
|
||||
node_info = f"[{node_id}]" if node_id else ""
|
||||
logger.debug(" [DIFF] %s%s schema=%s", self.node_type, node_info, schema.__name__)
|
||||
for k, v in diff_fields.items():
|
||||
orig_v = origin_dict.get(k)
|
||||
logger.debug(" • %s: %r → %r", k, orig_v, v)
|
||||
|
||||
return diff_fields if diff_fields else None
|
||||
|
||||
def build_update_request_data(
|
||||
self,
|
||||
new_data: Dict[str, Any],
|
||||
original_data: Dict[str, Any],
|
||||
schema: Optional[Any] = None
|
||||
schema: Optional[Any] = None,
|
||||
force_full_schema: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
过滤出需要更新的字段
|
||||
"""Update 入口:构造最终要发送给更新接口的请求数据。
|
||||
|
||||
规则:
|
||||
1. 提取 schema 定义的所有字段(包括必填)
|
||||
@@ -298,29 +390,12 @@ class BaseApiHandler(NodeHandler[T]):
|
||||
update_data[key] = new_value
|
||||
return update_data
|
||||
|
||||
# 1. 提取 schema 定义的所有字段
|
||||
schema_fields = set(schema_fields.keys())
|
||||
schema_data = {}
|
||||
has_change = False
|
||||
schema_data = self._get_update_schema_payload(schema, new_data, require_valid=True) or {}
|
||||
origin_schema_data = self._get_update_schema_payload(schema, original_data, require_valid=False) or {}
|
||||
|
||||
for key in schema_fields:
|
||||
new_value = new_data.get(key)
|
||||
|
||||
# 排除空值
|
||||
if new_value == "" or new_value is None:
|
||||
continue
|
||||
|
||||
schema_data[key] = new_value
|
||||
|
||||
# 检查是否变化
|
||||
original_value = original_data.get(key)
|
||||
if new_value != original_value:
|
||||
# 记录具体差异以便排查
|
||||
# print(f" [DEBUG] Field '{key}' changed: '{original_value}' -> '{new_value}'")
|
||||
has_change = True
|
||||
|
||||
# 2. 只有存在变化时才返回数据
|
||||
return schema_data if has_change else {}
|
||||
if force_full_schema or schema_data != origin_schema_data:
|
||||
return schema_data
|
||||
return {}
|
||||
|
||||
def _schema_changed_fields(
|
||||
self,
|
||||
@@ -328,16 +403,12 @@ class BaseApiHandler(NodeHandler[T]):
|
||||
new_data: Dict[str, Any],
|
||||
origin_data: Dict[str, Any],
|
||||
) -> List[str]:
|
||||
schema_fields = get_pydantic_model_fields(schema)
|
||||
if not schema_fields:
|
||||
return []
|
||||
changed: List[str] = []
|
||||
for field_name in schema_fields.keys():
|
||||
if new_data.get(field_name) != origin_data.get(field_name):
|
||||
changed.append(field_name)
|
||||
return changed
|
||||
new_payload = self._get_update_schema_payload(schema, new_data, require_valid=True) or {}
|
||||
origin_payload = self._get_update_schema_payload(schema, origin_data, require_valid=False) or {}
|
||||
field_names = sorted(set(new_payload) | set(origin_payload))
|
||||
return [field_name for field_name in field_names if new_payload.get(field_name) != origin_payload.get(field_name)]
|
||||
|
||||
def _build_update_schema_status(
|
||||
def build_update_schema_status(
|
||||
self,
|
||||
*,
|
||||
schema_name: str,
|
||||
@@ -347,6 +418,7 @@ class BaseApiHandler(NodeHandler[T]):
|
||||
will_update: bool,
|
||||
update_error: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Update 入口:生成 update schema 的状态说明文本。"""
|
||||
new_error = self._validate_schema(new_data, schema)
|
||||
origin_error = self._validate_schema(origin_data, schema)
|
||||
changed_fields = self._schema_changed_fields(schema, new_data, origin_data)
|
||||
|
||||
@@ -31,16 +31,16 @@ if TYPE_CHECKING:
|
||||
|
||||
from ..common.types import SyncStatus
|
||||
from ..common.type_safety import get_pydantic_model_fields
|
||||
from .handler import ensure_handler_compat
|
||||
from .task_result import TaskResult
|
||||
from ..engine import (
|
||||
StateMachineConfig,
|
||||
StateMachineRuntime,
|
||||
e40_sync_execute,
|
||||
)
|
||||
from ..logging import get_logger, get_scope_display_name
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class BaseDataSource(ABC):
|
||||
@@ -58,9 +58,9 @@ class BaseDataSource(ABC):
|
||||
# 1. 初始化 DataSource
|
||||
datasource = MyDataSource()
|
||||
|
||||
# 2. 注册 Handler
|
||||
datasource.register_handler(project_handler)
|
||||
datasource.register_handler(contract_handler)
|
||||
# 2. 添加 Handler
|
||||
datasource.add_handler(project_handler)
|
||||
datasource.add_handler(contract_handler)
|
||||
|
||||
# 3. 设置 Collection
|
||||
datasource.set_collection(collection)
|
||||
@@ -75,7 +75,8 @@ class BaseDataSource(ABC):
|
||||
def __init__(self, poll_max_retries: int = 1, poll_interval: float = 0.5):
|
||||
self._handlers: Dict[str, "NodeHandler"] = {}
|
||||
self._collection: Optional["DataCollection"] = None
|
||||
self._target_project_ids: List[str] = []
|
||||
self._endpoint_name: str = "unknown"
|
||||
self._datasource_type: str = self.__class__.__name__.lower()
|
||||
|
||||
# 按节点类型统计信息
|
||||
# {"company": {"loaded": 10, "synced": 10, "success": 10, "failed": 0}, ...}
|
||||
@@ -88,6 +89,17 @@ class BaseDataSource(ABC):
|
||||
self._poll_interval = max(0, poll_interval)
|
||||
self._sm_runtime: Optional[StateMachineRuntime] = None
|
||||
|
||||
def set_runtime_logging_context(self, *, endpoint_name: str, datasource_type: str) -> None:
|
||||
self._endpoint_name = endpoint_name
|
||||
self._datasource_type = datasource_type
|
||||
|
||||
def _datasource_log_prefix(self) -> str:
|
||||
scope = self._collection.scope if self._collection is not None else "unbound"
|
||||
return f"[{get_scope_display_name(scope)}][{self._endpoint_name}:{self._datasource_type}]"
|
||||
|
||||
def _node_log_prefix(self, node_type: str) -> str:
|
||||
return f"{self._datasource_log_prefix()}[{node_type}]"
|
||||
|
||||
def _ensure_sm_runtime(self) -> StateMachineRuntime:
|
||||
if self._sm_runtime is None:
|
||||
cfg_path = Path(__file__).resolve().parents[1] / "config" / "node_state_machine.yaml"
|
||||
@@ -164,6 +176,11 @@ class BaseDataSource(ABC):
|
||||
"""Datasource-specific hook after a FAILED result is applied to node."""
|
||||
return
|
||||
|
||||
def _prepare_handler_for_registration(self, handler: "NodeHandler") -> None:
|
||||
"""Allow subclasses to inject datasource-specific dependencies before registration."""
|
||||
if self._collection is not None:
|
||||
handler.set_collection(self._collection)
|
||||
|
||||
async def _handle_success_result(self, node: "SyncNode", result: "TaskResult") -> None:
|
||||
previous_data_id = node.data_id
|
||||
ok = self._apply_sync_execute_state(
|
||||
@@ -362,29 +379,18 @@ class BaseDataSource(ABC):
|
||||
|
||||
# ========== Handler 管理 ==========
|
||||
|
||||
def register_handler(self, handler: "NodeHandler") -> None:
|
||||
def add_handler(self, handler: "NodeHandler") -> None:
|
||||
"""
|
||||
注册 Handler
|
||||
添加 Handler 到当前 datasource
|
||||
|
||||
Args:
|
||||
handler: 节点处理器实例
|
||||
|
||||
示例:
|
||||
datasource.register_handler(ContractNodeHandler(api_client))
|
||||
datasource.add_handler(ContractNodeHandler(api_client))
|
||||
"""
|
||||
compatible_handler = ensure_handler_compat(handler)
|
||||
self._handlers[compatible_handler.node_type] = compatible_handler
|
||||
self._apply_target_project_ids_to_handler(compatible_handler)
|
||||
|
||||
def set_target_project_ids(self, target_project_ids: Optional[List[str]]) -> None:
|
||||
"""设置项目过滤列表,并在注册阶段下发给所有 handler。"""
|
||||
normalized = [str(pid) for pid in (target_project_ids or []) if str(pid)]
|
||||
self._target_project_ids = normalized
|
||||
for handler in self._handlers.values():
|
||||
self._apply_target_project_ids_to_handler(handler)
|
||||
|
||||
def _apply_target_project_ids_to_handler(self, handler: "NodeHandler") -> None:
|
||||
handler.set_target_project_ids(self._target_project_ids)
|
||||
self._prepare_handler_for_registration(handler)
|
||||
self._handlers[handler.node_type] = handler
|
||||
|
||||
def get_handler(self, node_type: str) -> "NodeHandler":
|
||||
"""获取 Handler"""
|
||||
@@ -432,7 +438,9 @@ class BaseDataSource(ABC):
|
||||
async def load_all(
|
||||
self,
|
||||
order: Optional[List[str]] = None,
|
||||
data_type: Optional[str] = None
|
||||
data_type: Optional[str] = None,
|
||||
*,
|
||||
raise_on_error: bool = False,
|
||||
) -> None:
|
||||
"""
|
||||
按依赖顺序加载所有数据
|
||||
@@ -474,7 +482,7 @@ class BaseDataSource(ABC):
|
||||
# Handler 加载数据并创建节点
|
||||
nodes = await handler.load()
|
||||
|
||||
print(f"Loaded nodes for {node_type}: {len(nodes)}")
|
||||
logger.debug("%s Loaded nodes for %s: %d", self._datasource_log_prefix(), node_type, len(nodes))
|
||||
|
||||
# 将节点写入 Collection:优先按 data_id 命中已有节点并更新
|
||||
await self._upsert_loaded_nodes(handler, node_type, nodes)
|
||||
@@ -482,7 +490,9 @@ class BaseDataSource(ABC):
|
||||
# 统计加载的节点数
|
||||
self._stats[node_type]["loaded"] += len(nodes)
|
||||
except Exception as exc:
|
||||
print(f"❌ [{node_type}] 加载失败: {exc}")
|
||||
logger.error("%s ❌ 加载失败: %s", self._node_log_prefix(node_type), exc)
|
||||
if raise_on_error:
|
||||
raise RuntimeError(f"[{node_type}] 加载失败: {exc}") from exc
|
||||
continue
|
||||
|
||||
# ========== 同步执行 ==========
|
||||
@@ -631,7 +641,7 @@ class BaseDataSource(ABC):
|
||||
|
||||
# 批量创建(传入节点列表)
|
||||
if create_nodes:
|
||||
logger.info(f"[{handler.node_type}] create提交开始: nodes={len(create_nodes)}")
|
||||
logger.debug(f"{self._node_log_prefix(handler.node_type)} create提交开始: nodes={len(create_nodes)}")
|
||||
try:
|
||||
create_results = await handler.create_all(create_nodes)
|
||||
create_success = 0
|
||||
@@ -648,12 +658,12 @@ class BaseDataSource(ABC):
|
||||
elif result.status == _TaskStatus.SKIPPED:
|
||||
create_skipped += 1
|
||||
await self._handle_task_result(handler, result, async_tasks)
|
||||
logger.info(
|
||||
f"[{handler.node_type}] create提交完成: success={create_success}, "
|
||||
logger.debug(
|
||||
f"{self._node_log_prefix(handler.node_type)} create提交完成: success={create_success}, "
|
||||
f"in_progress={create_in_progress}, failed={create_failed}, skipped={create_skipped}"
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"❌ [{handler.node_type}] create_all 异常: {e}")
|
||||
logger.error("❌ [%s] create_all 异常: %s", handler.node_type, e)
|
||||
for node in create_nodes:
|
||||
ok = self._apply_sync_execute_state(
|
||||
node,
|
||||
@@ -670,7 +680,7 @@ class BaseDataSource(ABC):
|
||||
for result in update_results:
|
||||
await self._handle_task_result(handler, result, async_tasks)
|
||||
except Exception as e:
|
||||
print(f"❌ [{handler.node_type}] update_all 异常: {e}")
|
||||
logger.error("❌ [%s] update_all 异常: %s", handler.node_type, e)
|
||||
for node in update_nodes:
|
||||
ok = self._apply_sync_execute_state(
|
||||
node,
|
||||
@@ -684,8 +694,8 @@ class BaseDataSource(ABC):
|
||||
|
||||
# 轮询异步任务
|
||||
if async_tasks and poll_async_tasks:
|
||||
logger.info(
|
||||
f"[{handler.node_type}] create已提交,push_id检查即将开始: pending={len(async_tasks)}, "
|
||||
logger.debug(
|
||||
f"{self._node_log_prefix(handler.node_type)} create已提交,push_id检查即将开始: pending={len(async_tasks)}, "
|
||||
f"first_wait={self._poll_interval:.3f}s"
|
||||
)
|
||||
await self._poll_async_tasks(handler, async_tasks)
|
||||
@@ -835,14 +845,14 @@ class BaseDataSource(ABC):
|
||||
return
|
||||
|
||||
if async_tasks and poll_interval > 0:
|
||||
logger.info(
|
||||
f"[{handler.node_type}] push_id首次检查前等待 {poll_interval:.3f}s, pending={len(async_tasks)}"
|
||||
logger.debug(
|
||||
f"{self._node_log_prefix(handler.node_type)} push_id首次检查前等待 {poll_interval:.3f}s, pending={len(async_tasks)}"
|
||||
)
|
||||
await asyncio.sleep(poll_interval)
|
||||
|
||||
while async_tasks and retry_count < max_retries:
|
||||
logger.info(
|
||||
f"[{handler.node_type}] push_id检查 {retry_count + 1}/{max_retries}, pending={len(async_tasks)}"
|
||||
logger.debug(
|
||||
f"{self._node_log_prefix(handler.node_type)} push_id检查 {retry_count + 1}/{max_retries}, pending={len(async_tasks)}"
|
||||
)
|
||||
# 批量查询
|
||||
task_ids = list(async_tasks.values())
|
||||
@@ -957,6 +967,10 @@ class BaseDataSource(ABC):
|
||||
|
||||
return summary
|
||||
|
||||
async def initialize(self) -> None:
|
||||
"""初始化数据源资源(子类可覆盖)"""
|
||||
return None
|
||||
|
||||
async def close(self) -> None:
|
||||
"""关闭数据源资源(子类可覆盖以实现特定的清理逻辑)"""
|
||||
pass # 基类默认空实现
|
||||
return None
|
||||
|
||||
@@ -1,6 +0,0 @@
|
||||
"""DB DataSource 模块。"""
|
||||
|
||||
from .datasource import DbDataSource
|
||||
from .handler import BaseDbHandler
|
||||
|
||||
__all__ = ["DbDataSource", "BaseDbHandler"]
|
||||
@@ -1,22 +0,0 @@
|
||||
"""DbDataSource - 数据库数据源协调层。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from ..datasource import BaseDataSource
|
||||
|
||||
|
||||
class DbDataSource(BaseDataSource):
|
||||
"""基于数据库 Handler 的数据源实现。"""
|
||||
|
||||
async def close(self) -> None:
|
||||
return None
|
||||
|
||||
def _on_node_loaded(self, handler, node) -> None:
|
||||
node.append_log(
|
||||
f"DB链路[load] node_type={node.node_type} data_id={node.data_id or '<empty>'}"
|
||||
)
|
||||
|
||||
def _on_in_progress_result(self, handler, node, result) -> None:
|
||||
node.append_log(
|
||||
f"DB链路[submit] action={node.action.value} data_id={node.data_id or '<empty>'}"
|
||||
)
|
||||
@@ -1,68 +0,0 @@
|
||||
"""BaseDbHandler - DB Handler 基类。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, List, Optional, Set, TypeVar, TYPE_CHECKING
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ...common.type_safety import get_pydantic_model_fields
|
||||
from ..handler import BaseNodeHandler
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ...common.sync_node import SyncNode
|
||||
from .datasource import DbDataSource
|
||||
|
||||
T = TypeVar("T", bound=BaseModel)
|
||||
|
||||
|
||||
class BaseDbHandler(BaseNodeHandler[T]):
|
||||
"""数据库 Handler 基类。"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
datasource: "DbDataSource",
|
||||
node_type: str,
|
||||
node_class: type["SyncNode[T]"],
|
||||
schema: type[T],
|
||||
):
|
||||
super().__init__(node_type, node_class, schema)
|
||||
self.datasource = datasource
|
||||
self.update_schemas: List[type] = []
|
||||
self._target_project_ids: Optional[Set[str]] = None
|
||||
|
||||
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
|
||||
if target_project_ids:
|
||||
self._target_project_ids = {str(item) for item in target_project_ids if str(item)}
|
||||
else:
|
||||
self._target_project_ids = None
|
||||
|
||||
def get_update_fields(self, *, exclude: frozenset = frozenset({"id"})) -> List[str]:
|
||||
fields: set[str] = set()
|
||||
for schema in self.update_schemas:
|
||||
for field_name in get_pydantic_model_fields(schema):
|
||||
if field_name not in exclude:
|
||||
fields.add(field_name)
|
||||
return sorted(fields)
|
||||
|
||||
def _create_basic_node(self, data: Dict[str, Any], depend_ids: Optional[List[str]] = None) -> "SyncNode[T]":
|
||||
data_id = self.extract_id(data)
|
||||
if self._collection is None:
|
||||
raise RuntimeError("Collection not set. Call set_collection() before creating nodes.")
|
||||
|
||||
if data_id:
|
||||
existing_node = self._collection.get_by_data_id(self.node_type, data_id)
|
||||
if existing_node is not None:
|
||||
existing_node.depend_ids = list(depend_ids or [])
|
||||
existing_node.set_data(data.copy())
|
||||
existing_node.set_origin_data(data.copy())
|
||||
return existing_node
|
||||
|
||||
node_id = self._collection.generate_node_id()
|
||||
return self.node_class(
|
||||
node_id=node_id,
|
||||
data_id=data_id,
|
||||
data=data.copy(),
|
||||
depend_ids=depend_ids or [],
|
||||
origin_data=data.copy(),
|
||||
)
|
||||
@@ -74,10 +74,6 @@ class NodeHandler(Protocol[T]):
|
||||
"""注入 handler 级别配置。默认可忽略。"""
|
||||
...
|
||||
|
||||
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
|
||||
"""注入项目过滤列表。默认可忽略。"""
|
||||
...
|
||||
|
||||
def set_api_client(self, client: "ApiClient") -> None:
|
||||
"""注入 API 客户端。默认可忽略。"""
|
||||
...
|
||||
@@ -341,11 +337,11 @@ class BaseNodeHandler(ABC, Generic[T]):
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
node_type: str,
|
||||
node_class: Type["SyncNode[T]"],
|
||||
schema: Type[T]
|
||||
node_type: str | None = None,
|
||||
node_class: Type["SyncNode[T]"] | None = None,
|
||||
schema: Type[T] | None = None,
|
||||
):
|
||||
self._node_type = node_type
|
||||
self._node_type = node_type or ""
|
||||
self._node_class = node_class
|
||||
self._schema = schema
|
||||
self._collection: Optional["DataCollection"] = None
|
||||
@@ -360,14 +356,34 @@ class BaseNodeHandler(ABC, Generic[T]):
|
||||
"""
|
||||
self._collection = collection
|
||||
|
||||
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
|
||||
"""默认忽略项目过滤能力。"""
|
||||
return
|
||||
|
||||
def set_handler_config(self, config: Dict[str, Any]) -> None:
|
||||
"""默认保存 handler 级别配置,供子类按需读取。"""
|
||||
self.handler_config = dict(config)
|
||||
|
||||
def get_data_id_filter(self) -> List[str]:
|
||||
if "data_id_filter" in self.handler_config:
|
||||
value = self.handler_config.get("data_id_filter", [])
|
||||
if isinstance(value, str):
|
||||
return [value] if value else []
|
||||
if isinstance(value, list):
|
||||
return [str(item) for item in value if str(item)]
|
||||
return []
|
||||
if self.node_type != "project" and self._collection is not None:
|
||||
return [node.data_id for node in self._collection.filter(node_type="project") if node.data_id]
|
||||
return []
|
||||
|
||||
def should_skip_by_data_id_filter(self, item: Dict[str, Any], item_id: str) -> bool:
|
||||
data_id_filter = set(self.get_data_id_filter())
|
||||
if not data_id_filter:
|
||||
return False
|
||||
if self.node_type == "project":
|
||||
return item_id not in data_id_filter
|
||||
|
||||
project_id = item.get("project_id")
|
||||
if project_id:
|
||||
return str(project_id) not in data_id_filter
|
||||
return False
|
||||
|
||||
def set_api_client(self, client: "ApiClient") -> None:
|
||||
"""默认忽略 API 客户端注入。"""
|
||||
return
|
||||
@@ -479,87 +495,6 @@ class BaseNodeHandler(ABC, Generic[T]):
|
||||
return self.create_node(data)
|
||||
|
||||
|
||||
class CompatibleNodeHandler(Generic[T]):
|
||||
"""兼容旧式 Handler,实现扩展后的可选接口并委托给原对象。"""
|
||||
|
||||
def __init__(self, handler: NodeHandler[T]):
|
||||
self._handler = handler
|
||||
|
||||
@property
|
||||
def node_type(self) -> str:
|
||||
return self._handler.node_type
|
||||
|
||||
@property
|
||||
def node_class(self) -> Type["SyncNode[T]"]:
|
||||
return self._handler.node_class
|
||||
|
||||
@property
|
||||
def schema(self) -> Type[T]:
|
||||
return self._handler.schema
|
||||
|
||||
def set_collection(self, collection: "DataCollection") -> None:
|
||||
self._handler.set_collection(collection)
|
||||
|
||||
def set_handler_config(self, config: Dict[str, Any]) -> None:
|
||||
method = getattr(self._handler, "set_handler_config", None)
|
||||
if callable(method):
|
||||
method(config)
|
||||
|
||||
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
|
||||
method = getattr(self._handler, "set_target_project_ids", None)
|
||||
if callable(method):
|
||||
method(target_project_ids)
|
||||
|
||||
def set_api_client(self, client: "ApiClient") -> None:
|
||||
method = getattr(self._handler, "set_api_client", None)
|
||||
if callable(method):
|
||||
method(client)
|
||||
|
||||
def set_poll_mode(self, mode: str) -> None:
|
||||
method = getattr(self._handler, "set_poll_mode", None)
|
||||
if callable(method):
|
||||
method(mode)
|
||||
|
||||
def get_update_fields(self, *, exclude: frozenset[str] = frozenset({"id"})) -> List[str]:
|
||||
method = getattr(self._handler, "get_update_fields", None)
|
||||
if callable(method):
|
||||
return method(exclude=exclude)
|
||||
return []
|
||||
|
||||
async def load(self) -> List["SyncNode"]:
|
||||
return await self._handler.load()
|
||||
|
||||
async def create_all(self, nodes: List["SyncNode"]) -> List[TaskResult]:
|
||||
return await self._handler.create_all(nodes)
|
||||
|
||||
async def update_all(self, nodes: List["SyncNode"]) -> List[TaskResult]:
|
||||
return await self._handler.update_all(nodes)
|
||||
|
||||
async def delete_all(self, nodes: List["SyncNode"]) -> List[TaskResult]:
|
||||
return await self._handler.delete_all(nodes)
|
||||
|
||||
async def poll_tasks(self, task_ids: List[str]) -> Dict[str, TaskResult]:
|
||||
return await self._handler.poll_tasks(task_ids)
|
||||
|
||||
async def validate(self, data: Dict[str, Any]) -> ValidationResult:
|
||||
return await self._handler.validate(data)
|
||||
|
||||
def extract_id(self, data: Dict[str, Any]) -> str:
|
||||
return self._handler.extract_id(data)
|
||||
|
||||
def _create_node(self, data: Dict[str, Any]) -> "SyncNode[T]":
|
||||
return self._handler._create_node(data)
|
||||
|
||||
def __getattr__(self, item: str) -> Any:
|
||||
return getattr(self._handler, item)
|
||||
|
||||
|
||||
def ensure_handler_compat(handler: NodeHandler[T]) -> NodeHandler[T]:
|
||||
if isinstance(handler, BaseNodeHandler) or isinstance(handler, CompatibleNodeHandler):
|
||||
return handler
|
||||
return CompatibleNodeHandler(handler)
|
||||
|
||||
|
||||
class NodeHandlerRegistry:
|
||||
"""节点处理器注册表"""
|
||||
|
||||
|
||||
@@ -1,13 +1,53 @@
|
||||
from __future__ import annotations
|
||||
|
||||
"""
|
||||
JSONL DataSource 模块
|
||||
|
||||
提供基于 JSONL 文件的数据源实现,用于测试和本地开发。
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from ..type_registry import register_datasource_type
|
||||
from .datasource import JsonlDataSource
|
||||
from .handler import BaseJsonlHandler
|
||||
from .path_utils import prepare_remote_jsonl_dir
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ...config.datasource_config import JsonlDataSourceConfig
|
||||
|
||||
|
||||
def build_jsonl_datasource(
|
||||
config: JsonlDataSourceConfig,
|
||||
endpoint_name: str,
|
||||
*,
|
||||
project_root: Path | None = None,
|
||||
):
|
||||
dir_path = Path(config.jsonl_dir)
|
||||
if endpoint_name == "remote":
|
||||
dir_path = prepare_remote_jsonl_dir(dir_path, project_root=project_root or Path.cwd())
|
||||
elif not dir_path.exists() or not dir_path.is_dir():
|
||||
raise FileNotFoundError(
|
||||
f"{endpoint_name}_datasource.jsonl_dir does not exist or is not a directory: {dir_path}"
|
||||
)
|
||||
return JsonlDataSource(dir_path, read_only=config.read_only)
|
||||
|
||||
|
||||
def register(*, replace: bool = False) -> None:
|
||||
from ...config.datasource_config import JsonlDataSourceConfig
|
||||
|
||||
register_datasource_type(
|
||||
"jsonl",
|
||||
config_model=JsonlDataSourceConfig,
|
||||
factory=build_jsonl_datasource,
|
||||
replace=replace,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"JsonlDataSource",
|
||||
"BaseJsonlHandler",
|
||||
"build_jsonl_datasource",
|
||||
"prepare_remote_jsonl_dir",
|
||||
"register",
|
||||
]
|
||||
|
||||
@@ -10,8 +10,10 @@ JsonlDataSource - JSONL 文件数据源
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, Set
|
||||
from typing import Dict, Any, Set, Callable, List, Optional
|
||||
|
||||
from ..datasource import BaseDataSource
|
||||
|
||||
@@ -30,7 +32,7 @@ class JsonlDataSource(BaseDataSource):
|
||||
|
||||
示例:
|
||||
datasource = JsonlDataSource(Path("filtered_datasource/datasource/ecm-2025-12-02"))
|
||||
datasource.register_handler(ContractJsonlHandler(datasource))
|
||||
datasource.add_handler(ContractJsonlHandler(datasource))
|
||||
|
||||
collection = DataCollection()
|
||||
datasource.set_collection(collection)
|
||||
@@ -46,6 +48,7 @@ class JsonlDataSource(BaseDataSource):
|
||||
# 用于 create/update/delete 的内存缓存
|
||||
self._data: Dict[str, Dict[str, Dict[str, Any]]] = {} # {node_type: {id: data}}
|
||||
self._dirty_types: Set[str] = set()
|
||||
self._node_items_cache: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
def _on_node_loaded(self, handler, node) -> None:
|
||||
node.append_log(
|
||||
@@ -62,6 +65,98 @@ class JsonlDataSource(BaseDataSource):
|
||||
f"JSONL链路[poll] task_id={task_id} status={result.status.value}"
|
||||
)
|
||||
|
||||
def invalidate_node_cache(self, node_type: str) -> None:
|
||||
self._node_items_cache.pop(node_type, None)
|
||||
|
||||
def _matching_files(self, node_type: str) -> List[Path]:
|
||||
if not self.dir_path.exists():
|
||||
return []
|
||||
pattern = re.compile(rf"^{re.escape(node_type)}_\d+\.jsonl$")
|
||||
return sorted(
|
||||
file_path
|
||||
for file_path in self.dir_path.iterdir()
|
||||
if pattern.match(file_path.name)
|
||||
)
|
||||
|
||||
def _build_cache_entry(
|
||||
self,
|
||||
*,
|
||||
node_type: str,
|
||||
extract_id: Callable[[Dict[str, Any]], str],
|
||||
) -> Dict[str, Any]:
|
||||
files = self._matching_files(node_type)
|
||||
signature = tuple(
|
||||
(file_path.name, file_path.stat().st_mtime_ns, file_path.stat().st_size)
|
||||
for file_path in files
|
||||
)
|
||||
cached = self._node_items_cache.get(node_type)
|
||||
if cached and cached.get("signature") == signature:
|
||||
return cached
|
||||
|
||||
items: List[Dict[str, Any]] = []
|
||||
data_id_index: Dict[str, List[Dict[str, Any]]] = {}
|
||||
project_index: Dict[str, List[Dict[str, Any]]] = {}
|
||||
no_project_items: List[Dict[str, Any]] = []
|
||||
|
||||
for file_path in files:
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
for line_num, line in enumerate(f, 1):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
|
||||
try:
|
||||
item = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
print(f"Warning: Invalid JSON in {file_path.name}:{line_num}")
|
||||
continue
|
||||
|
||||
items.append(item)
|
||||
|
||||
item_id = extract_id(item)
|
||||
if item_id:
|
||||
data_id_index.setdefault(str(item_id), []).append(item)
|
||||
|
||||
project_id = item.get("project_id")
|
||||
if project_id:
|
||||
project_index.setdefault(str(project_id), []).append(item)
|
||||
else:
|
||||
no_project_items.append(item)
|
||||
|
||||
entry = {
|
||||
"signature": signature,
|
||||
"items": items,
|
||||
"data_id_index": data_id_index,
|
||||
"project_index": project_index,
|
||||
"no_project_items": no_project_items,
|
||||
}
|
||||
self._node_items_cache[node_type] = entry
|
||||
return entry
|
||||
|
||||
def get_items_for_load(
|
||||
self,
|
||||
*,
|
||||
node_type: str,
|
||||
extract_id: Callable[[Dict[str, Any]], str],
|
||||
data_id_filter: Optional[List[str]] = None,
|
||||
filter_on_project_id: bool = False,
|
||||
) -> List[Dict[str, Any]]:
|
||||
entry = self._build_cache_entry(node_type=node_type, extract_id=extract_id)
|
||||
filters = [str(item) for item in (data_id_filter or []) if str(item)]
|
||||
if not filters:
|
||||
return list(entry["items"])
|
||||
|
||||
if not filter_on_project_id:
|
||||
matched: List[Dict[str, Any]] = []
|
||||
for data_id in filters:
|
||||
matched.extend(entry["data_id_index"].get(data_id, []))
|
||||
return matched
|
||||
|
||||
matched = list(entry["no_project_items"])
|
||||
for project_id in filters:
|
||||
matched.extend(entry["project_index"].get(project_id, []))
|
||||
return matched
|
||||
|
||||
# load_all 和 sync_all 继承自 BaseDataSource
|
||||
# 保存逻辑在 BaseJsonlHandler 中实现
|
||||
|
||||
|
||||
@@ -12,7 +12,6 @@ BaseJsonlHandler - JSONL Handler 基类
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from typing import Dict, List, Any, TYPE_CHECKING, Optional, Set
|
||||
from uuid import uuid4
|
||||
|
||||
@@ -44,31 +43,22 @@ class BaseJsonlHandler(BaseNodeHandler):
|
||||
def __init__(
|
||||
self,
|
||||
datasource: JsonlDataSource,
|
||||
node_type: str,
|
||||
node_class: type,
|
||||
schema: type
|
||||
node_class_or_node_type,
|
||||
node_class: type | None = None,
|
||||
schema: type | None = None,
|
||||
node_type: str | None = None,
|
||||
):
|
||||
super().__init__(node_type, node_class, schema)
|
||||
self.datasource = datasource
|
||||
self._target_project_ids: Optional[Set[str]] = None
|
||||
|
||||
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
|
||||
"""注入项目过滤列表,供 load 阶段预过滤使用。"""
|
||||
if target_project_ids:
|
||||
self._target_project_ids = {str(pid) for pid in target_project_ids if str(pid)}
|
||||
if isinstance(node_class_or_node_type, str):
|
||||
resolved_node_type = node_class_or_node_type
|
||||
resolved_node_class = node_class
|
||||
resolved_schema = schema
|
||||
else:
|
||||
self._target_project_ids = None
|
||||
resolved_node_class = node_class_or_node_type
|
||||
resolved_node_type = node_type or getattr(resolved_node_class, "node_type", None)
|
||||
resolved_schema = schema or getattr(resolved_node_class, "schema", None)
|
||||
|
||||
def _should_skip_item_by_project_filter(self, item: Dict[str, Any], item_id: str) -> bool:
|
||||
if not self._target_project_ids:
|
||||
return False
|
||||
if self.node_type == "project":
|
||||
return item_id not in self._target_project_ids
|
||||
|
||||
project_id = item.get("project_id")
|
||||
if project_id:
|
||||
return str(project_id) not in self._target_project_ids
|
||||
return False
|
||||
super().__init__(resolved_node_type, resolved_node_class, resolved_schema)
|
||||
self.datasource = datasource
|
||||
|
||||
async def load(self) -> List['SyncNode']:
|
||||
"""
|
||||
@@ -84,39 +74,25 @@ class BaseJsonlHandler(BaseNodeHandler):
|
||||
if not self.datasource.dir_path.exists():
|
||||
return nodes
|
||||
|
||||
# 查找所有匹配的文件:{node_type}_N.jsonl
|
||||
pattern = re.compile(rf"^{re.escape(self.node_type)}_\d+\.jsonl$")
|
||||
data_id_filter = self.get_data_id_filter()
|
||||
items = self.datasource.get_items_for_load(
|
||||
node_type=self.node_type,
|
||||
extract_id=self.extract_id,
|
||||
data_id_filter=data_id_filter,
|
||||
filter_on_project_id=self.node_type != "project",
|
||||
)
|
||||
|
||||
for file_path in self.datasource.dir_path.iterdir():
|
||||
if not pattern.match(file_path.name):
|
||||
for item in items:
|
||||
item_id = self.extract_id(item)
|
||||
if self.should_skip_by_data_id_filter(item, item_id):
|
||||
continue
|
||||
|
||||
# 读取文件
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
for line_num, line in enumerate(f, 1):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
node = self.create_node(item)
|
||||
nodes.append(node)
|
||||
|
||||
try:
|
||||
item = json.loads(line)
|
||||
item_id = self.extract_id(item)
|
||||
|
||||
if self._should_skip_item_by_project_filter(item, item_id):
|
||||
continue
|
||||
|
||||
# 创建节点
|
||||
node = self.create_node(item)
|
||||
nodes.append(node)
|
||||
|
||||
# 同时加载到缓存
|
||||
if item_id:
|
||||
bucket = self.datasource._data.setdefault(self.node_type, {})
|
||||
bucket[item_id] = item
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"Warning: Invalid JSON in {file_path.name}:{line_num}: {e}")
|
||||
# 继续处理下一行
|
||||
if item_id:
|
||||
bucket = self.datasource._data.setdefault(self.node_type, {})
|
||||
bucket[item_id] = item
|
||||
|
||||
return nodes
|
||||
|
||||
@@ -295,6 +271,7 @@ class BaseJsonlHandler(BaseNodeHandler):
|
||||
|
||||
# 清除该类型的 dirty 标记
|
||||
self.datasource._dirty_types.discard(self.node_type)
|
||||
self.datasource.invalidate_node_cache(self.node_type)
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error saving {self.node_type}: {str(e)}")
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from ...logging import get_logger
|
||||
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def prepare_remote_jsonl_dir(remote_dir: Path, *, project_root: Path) -> Path:
|
||||
project_root_path = project_root.resolve()
|
||||
resolved_remote_dir = remote_dir.resolve(strict=False)
|
||||
|
||||
if resolved_remote_dir.exists():
|
||||
if not resolved_remote_dir.is_dir():
|
||||
raise FileNotFoundError(
|
||||
f"remote_datasource.jsonl_dir exists but is not a directory: {resolved_remote_dir}"
|
||||
)
|
||||
return resolved_remote_dir
|
||||
|
||||
try:
|
||||
resolved_remote_dir.relative_to(project_root_path)
|
||||
except ValueError as exc:
|
||||
raise FileNotFoundError(
|
||||
"remote_datasource.jsonl_dir does not exist and auto-create is only allowed "
|
||||
f"under project root: {resolved_remote_dir} (project_root={project_root_path})"
|
||||
) from exc
|
||||
|
||||
resolved_remote_dir.mkdir(parents=True, exist_ok=True)
|
||||
logger.info("Created missing remote JSONL directory: %s", resolved_remote_dir)
|
||||
return resolved_remote_dir
|
||||
@@ -17,10 +17,7 @@ class AttachmentApiHandler(BaseApiHandler):
|
||||
"""附件 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "attachment"
|
||||
self._node_class = AttachmentSyncNode
|
||||
self._schema = AttachmentSchema
|
||||
super().__init__(node_class=AttachmentSyncNode, schema=AttachmentSchema)
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载附件列表(依赖 Project)"""
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user