Compare commits

..

42 Commits

Author SHA1 Message Date
strepsiades ba73b30a04 增加sqlalchemy后端,优化日志结构 2026-04-08 09:28:14 +08:00
strepsiades c377264610 实现了多project pipeline 2026-04-07 16:52:42 +08:00
strepsiades 4d6f499508 压制了无用日志 2026-04-07 16:14:00 +08:00
strepsiades 966f2e1270 初步实现多project pipeline 2026-04-07 14:45:19 +08:00
strepsiades e3eab5ade1 添加了重构文档 2026-04-07 11:25:19 +08:00
strepsiades aa31243bae 修复一个因为项目双向更新更新导致第二次更新无法进行的问题 2026-04-03 11:44:53 +08:00
strepsiades f5729d5a18 整理代码 2026-04-03 09:18:44 +08:00
strepsiades 543c3ff906 优化加载后删除节点的逻辑;和数据库查询性能 2026-04-02 19:31:04 +08:00
strepsiades 74df53889c 解决了部分detail load效率问题和load失败终止pipeline问题 2026-04-02 14:35:28 +08:00
strepsiades bb8ea931b5 优化大量推送问题 2026-04-02 11:14:44 +08:00
strepsiades f93a6c5c68 优化性能 2026-04-01 18:13:18 +08:00
strepsiades 9a3a34c58b 修改问题 2026-04-01 17:04:26 +08:00
strepsiades 7c49df94fc 修改多个问题,优化代码结构 2026-04-01 16:23:43 +08:00
strepsiades 8f4727a772 优化sync_system代码质量 2026-04-01 11:52:57 +08:00
strepsiades eb02fd32a5 修复了一个project_detail_data create失败报错的问题 2026-03-31 15:54:12 +08:00
strepsiades f0ce8dc37e 修改supplier model,删除了多余字段 2026-03-31 11:33:27 +08:00
strepsiades 14c20a6123 整理update逻辑 2026-03-31 08:48:34 +08:00
strepsiades bdc7bd5069 规范了update data的构建方式,统一使用build_update_request_data方法来处理更新数据的构建和过滤逻辑。这个方法会根据提供的原始数据、更新数据和对应的schema来生成最终的更新请求数据,同时会自动排除掉未变化的字段和空值字段,从而确保只发送必要的更新信息到后端接口。这种方式不仅简化了代码逻辑,还提高了代码的可维护性和一致性。 2026-03-30 14:51:01 +08:00
strepsiades f50dc1aed0 整体同步代码 2026-03-26 14:10:47 +08:00
strepsiades 4b81b83fd8 更新文档 2026-03-25 10:32:50 +08:00
strepsiades 67370657db 同步用户相关修改 2026-03-24 15:13:46 +08:00
strepsiades 8ee0131d47 修改schema 2026-03-24 10:27:14 +08:00
strepsiades 65bf2346a3 现在depend_fields可以处理列表类id 2026-03-24 10:18:13 +08:00
strepsiades 03a22b0c1c 将persist和后端分离,方便替换为其他数据库 2026-03-24 10:15:16 +08:00
strepsiades 7d92b3f1fe 更新了部分测试 2026-03-24 09:41:08 +08:00
strepsiades b51ed0175b 多pipeline同步支持 2026-03-24 08:40:40 +08:00
strepsiades 0812489f9d 增强validator和打印区别 2026-03-23 15:16:16 +08:00
strepsiades 4c380f1157 修正了preparation production的schema 2026-03-20 15:09:55 +08:00
strepsiades 2c09c61165 调整了pipeline执行顺序。现在按domain顺序读取数据源数据,防止前面domain的更新影响后面domain.
将target_project_ids替换为更通用的data_id_filter
2026-03-20 14:38:56 +08:00
strepsiades f9f175ee79 进一步清理代码和注册过程 2026-03-19 11:44:02 +08:00
strepsiades 32fa374e86 清理代码 2026-03-18 17:09:46 +08:00
strepsiades 6ae54e3edd Merge branch 'main' of https://git.getaix.tech/strepsiades/ecm_sync_system 2026-03-18 16:55:25 +08:00
strepsiades 841986740c 整理了初始化过程,现在更容易被继承了。 2026-03-18 16:48:53 +08:00
strepsiades 84b3b7652a 增加项目侧重构文档 2026-03-18 11:34:02 +08:00
strepsiades a677220f4d 增加项目侧重构文档 2026-03-12 16:03:26 +08:00
strepsiades 758b1c4f77 增加重构文档 2026-03-12 14:23:15 +08:00
strepsiades 0ae2db1808 增加字段config重构文档 2026-03-12 11:39:57 +08:00
strepsiades e786bd5bf0 增加一处注释,说明project_extension字段的来源和作用。 2026-03-11 16:16:37 +08:00
strepsiades a22c527daf 修改了investment的schema 2026-03-11 15:30:46 +08:00
strepsiades bd81c6dae6 现在progress_type允许为None 2026-03-11 14:49:35 +08:00
strepsiades b9fbf357a4 增加了一个测试工具用于初始化测试环境 2026-03-11 11:40:23 +08:00
strepsiades 1f68bd5ca8 增加了项目侧关于steps的处理。允许保存和提取steps信息。 2026-03-11 10:34:49 +08:00
225 changed files with 18325 additions and 4678 deletions
+1
View File
@@ -96,3 +96,4 @@ run_profiles/*.yml
_runtime/
test_results/
*.code-workspace
runtime/
+70 -6
View File
@@ -2,6 +2,42 @@
可作为命令行工具运行,也可作为 `pip install -e .` 的本地可编辑安装库接入其他项目。
## 0. 10 分钟上手
如果你是第一次进入这个仓库,先不要从所有文档开始读,先确认环境和最小工具链可用。
推荐顺序:
1. 准备 Python 3.9+ 环境(当前仓库已在 `pyproject.toml` 中声明 `requires-python >= 3.9`)。
2. 在仓库根目录创建并激活虚拟环境。
3. 以可编辑模式安装本项目。
4. 先跑配置校验,再跑一个轻量集成测试,确认本地环境可用。
5. 只在确认入口可运行后,再去看架构与状态机文档。
最小命令链:
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
python run.py --help
python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml
pytest tests/integration/test_toolchain_config_and_graph.py -q
```
如果你只是想直接运行仓库内现成配置,优先用仓库入口:
```bash
python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
```
如果你已经完成 `pip install -e .`,也可以使用安装后的命令行入口:
```bash
ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
```
面向第一次接触仓库的更完整说明见:`docs/getting_started.md`
## 1. 项目简介
本项目用于实现异构系统之间的数据推送:将项目侧管理平台的十余种业务数据,稳定推送到集团侧管理平台。
@@ -18,6 +54,7 @@
项目采用:`schema` 严格约束 + 状态机 + 节点模型 + 分层架构 + 注册式 `domain` + 数据源与同步系统解耦。
架构详见:
- `docs/getting_started.md`
- `docs/architecture.md`
- `docs/node_state_definition.md`
- `docs/state_machine.md`
@@ -58,15 +95,26 @@
## 5. 使用方法
先区分两种常见使用方式:
- 仓库内开发与调试:优先使用 `python run.py ...`,因为它不依赖 shell 中已经存在 `ecm-sync-run`
- 作为库或已安装命令使用:先执行 `pip install -e .`,再使用 `ecm-sync-run ...`
运行配置遵循一个约定:
- 跟踪在仓库里的 `run_profiles/*.yaml``run_profiles/preset/*.default.yaml` 只写必要的覆盖项。
- pipeline 启动时会打印并写入一份 **Resolved Full Config**,把省略的默认值全部补齐,便于排障和审阅。
- 字段含义与全量示例见:`docs/runtime_config_guide.md`
### JSONL 同步
- 基线配置(优先):`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
- 兼容旧入口:`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
- 使用自定义配置:先从 `run_profiles/preset/jsonl_to_jsonl.default.yaml` 复制到 `run_profiles/jsonl_to_jsonl.local.yaml`,按需修改后运行:`ecm-sync-run --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
- 仓库内直接运行:`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
- 已安装命令行入口:`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
- 多项目 JSONL 示例:`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.multi_project.default.yaml`
- 当前已经支持多 project 运行;标准写法是直接使用多项目编排配置文件,或在支持 implicit multi-project 的 profile 中由配置本身决定运行模式。
- 自定义配置:从 `run_profiles/preset/` 复制一份到 `run_profiles/*.local.yaml`,只改有差异的字段即可。
### API 同步
- 基线配置:`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_api.default.yaml`
- 兼容旧入口:`python run.py --config_path=run_profiles/preset/jsonl_to_api.default.yaml`
- 使用自定义配置:先从 `run_profiles/preset/jsonl_to_api.default.yaml` 复制到 `run_profiles/jsonl_to_api.local.yaml`,按需修改后运行:`ecm-sync-run --config_path=run_profiles/jsonl_to_api.local.yaml`
- 仓库内直接运行:`python run.py --config_path=run_profiles/preset/jsonl_to_api.default.yaml`
- 已安装命令行入口:`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_api.default.yaml`
- 如果是单项目联调,优先参考仓库内现成示例:`run_profiles/jsonl_to_api.yaml`
### 库模式使用
- 本地可编辑安装:`pip install -e .`
@@ -75,19 +123,34 @@
### 配置与校验
- 模板配置:`run_profiles/preset/*.default.yaml`
- 建议先跑 default,再在 `run_profiles/*.local.yaml` 做覆盖(该目录 YAML 已默认忽略)
- 跟踪中的模板与示例配置都采用“只写必要字段”的风格;启动日志里的 `Resolved Full Config` 才是最终生效配置。
- 状态机配置校验:`python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml`
- 状态机图生成:`python tools/render_graph.py --format mermaid --out docs/state_machine.mmd`
### 测试
- 全量:`pytest tests -q`
- 说明:`tests/README.md`
- 远程环境初始化工具:`python tools/remote_env.py --config tools/remote_env.example.yaml init|run|stop|clear`
- 测试总览:`docs/testing_guide.md`
### 其他工具
- UI 调试:`python -m ui.main --host 127.0.0.1 --port 8765`
- 运维与排障:`docs/operations.md`
- 远程测试环境:`tools/remote_env.py`
- 脚本化回归与数据集构建:`scripts/README.md`
## 文档导航
如果你是按任务查文档,建议按下面的顺序进入:
- 第一次接手仓库:`docs/getting_started.md`
- 想理解整体分层:`docs/architecture.md`
- 想理解状态与迁移:`docs/node_state_definition.md` + `docs/state_machine.md`
- 想调运行配置:`docs/runtime_config_guide.md` + `run_profiles/README.md`(包含多项目配置写法)
- 想接新数据源或改 handler`sync_state_machine/datasource/README.md` + `docs/library_integration_guide.md`
- 想排障:`docs/operations.md`
- 想补测试或跑联调:`docs/testing_guide.md` + `tests/README.md`
- 新人上手:`docs/getting_started.md`
- 总体架构:`docs/architecture.md`
- 状态定义:`docs/node_state_definition.md`
- 状态机规则:`docs/state_machine.md`
@@ -96,5 +159,6 @@
- 库化与接入:`docs/library_integration_guide.md`
- Collection 查询与升级:`docs/collection_query_and_upgrade_plan.md`
- 运维迭代:`docs/operations.md`
- 测试指南:`docs/testing_guide.md`
历史资料已归档到 `docs/archive/`
+14 -14
View File
@@ -3,11 +3,11 @@
## 1. 分层职责(pipeline + strategy + datasource + domain
### pipeline(编排层)
- 入口:`pipeline/full_sync_pipeline.py`
- 入口:`sync_state_machine/pipeline/full_sync_pipeline.py`
- 负责组装并驱动整条流程:加载持久化、执行 bind/create/update、调用 datasource 同步、持久化结果。
- reset 语义:
- 加载阶段仅恢复持久化状态(不做状态重置);
- 加载后通过 `BaseSyncStrategy.run_reset()` 触发 `E01`
- 加载后通过 `run_phase2_cleanup()` 触发 `E01`
- `create_zombie=true` -> `S15`(随后删除);
- `create_zombie=false` -> `S00`(归并为 `UNCHECKED/NONE/PENDING`,并清空 `error`)。
- 若持久化核心枚举非法(`action/status/binding_status`),节点进入加载异常隔离态 `S17`
@@ -24,29 +24,29 @@
- 非僵尸节点统一进入 `S00`,再进入本轮 bind/create/update 判定。
### strategy(决策组织层)
- 外部入口:`sync_system/strategy.py`
- 具体实现下沉:`sync_system/strategy_ops/`
- `bind_ops.py`: `run_bind()` + `phase_core_state()/phase_dependency_check()/phase_auto_binding()`
- `create_ops.py`: `run_create()`
- `update_ops.py`: `run_update()`
- 外部入口:`sync_state_machine/sync_system/strategy.py`
- 具体实现下沉:`sync_state_machine/sync_system/strategy_ops/`
- `bind_ops.py`: bind 阶段静态函数与纯工具
- `create_ops.py`: create 阶段静态函数与纯工具
- `update_ops.py`: update 阶段静态函数与纯工具
- `delete_ops.py`: `run_delete()`
- `reset_ops.py`: `get_phase1_reset_defaults()/run_phase2_cleanup()`
- strategy 负责收集上下文并调用状态机事件函数,不直接硬编码状态值。
- `reset_ops.py`: `run_phase2_cleanup()`
- strategy 负责 bind/create/update 的相位内部编排,并调用状态机事件函数,不直接硬编码状态值。
### datasource(执行与回写层)
- 入口:`datasource/datasource.py`
- 入口:`sync_state_machine/datasource/datasource.py`
- 执行 handlercreate/update/delete),并结合 `E40` 将执行结果落地到节点状态。
### domain(业务适配层)
- 目录:`domain/*`
- 目录:`sync_state_machine/domain/*`
- 负责业务字段映射、handler 细节与 schema 约束,不承担状态迁移判定。
## 2. 运行时契约(Runtime Contract
### 真源
- 状态机配置真源:`config/node_state_machine.yaml`
- 事件执行入口:`engine/events.py`
- 状态落地入口:`engine/state_apply.py`
- 状态机配置真源:`sync_state_machine/config/node_state_machine.yaml`
- 事件执行入口:`sync_state_machine/engine/events.py`
- 状态落地入口:`sync_state_machine/engine/state_apply.py`
### 执行规则
1. 业务层构造 `context`
+11 -9
View File
@@ -38,7 +38,7 @@ JSONL 数据源对应实现:
- 从目录中按 `{node_type}_N.jsonl` 规则发现并加载数据
- 批量 create/update/delete 时,先改内存缓存,再统一刷盘
- 支持 `read_only`,在只读模式下禁止写回
- 支持按项目过滤(`target_project_ids`)在 load 阶段预过滤
- 支持按项目过滤(`handler_configs.project.data_id_filter`)在 load 阶段预过滤
### 2.2 行为特点
@@ -53,7 +53,7 @@ JSONL 数据源对应实现:
- `type: jsonl`
- `jsonl_dir`: 数据目录
- `read_only`: 是否只读
- `target_project_ids`: 项目过滤列表
- `handler_configs.project.data_id_filter`: 项目过滤列表
- `handler_configs`: 按节点类型的 handler 参数
---
@@ -131,8 +131,8 @@ API 数据源中,校验分为三层:
- 例如 `ContractCreate.model_validate(...)``ContractUpdate.model_validate(...)`
- load 响应也会在解析后转为领域 schema
2. 通用更新过滤与差异识别
- `_filter_update_data`仅在 schema 相关字段有变化时提交更新
- `_get_schema_diff`:用于差异字段识别与调试输出
- `build_update_request_data`update 入口,仅在 schema 相关字段有变化时提交更新
- `get_update_schema_diff`update 入口,用于差异字段识别与调试输出
3. 轮询接口输入校验
- `PushIdsSchema` 校验 push_ids 格式后再请求 `/push/result`
@@ -159,8 +159,9 @@ local_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
read_only: false
target_project_ids: []
handler_configs: {}
handler_configs:
project:
data_id_filter: []
```
### 4.2 API 数据源配置示例(含限流)
@@ -181,9 +182,10 @@ remote_datasource:
api_rate_limit_max_requests: 10
api_rate_limit_window_seconds: 10.0
target_project_ids:
- "project-id-1"
handler_configs: {}
handler_configs:
project:
data_id_filter:
- "project-id-1"
```
### 4.3 配置模型位置
+111
View File
@@ -0,0 +1,111 @@
# ECM Sync Follow-up Plan
## 目标
本轮先落两件事:
1. 项目聚合读取接口,降低同步读取请求数。
2. 增强现有 validate,让 strategy 可以直接复用 schema diff validator,并输出聚合日志。
第三件事作为下一步:在 backend 里为 data mapping 增加静态样本测试,绕开 pipeline,直接验证本地样本和远程样本在 sync schema 层是否足够一致。
## 当前结论
### 1. 项目聚合读取接口
现有按业务拆散的读取方式请求数过高,不适合大规模同步。
本轮改为提供统一项目聚合接口,最小聚合范围:
- `project`
- `project_detail_data`
- `preparation`
- `production`
- `contract`
目标:
- 以项目为同步分片单位。
- 减少绝大多数项目级 fan-out 请求。
- 保持写入侧业务流程不变,只优化读取侧接口粒度。
接口返回的数据仍然基于现有 service 组装,不新增业务含义。
### 2. validate 增强方案
现有 validate 主要停留在 schema 校验,缺少可读的聚合 diff 输出。
本轮新增 schema diff validator,原则:
- 仍以现有 sync schema 为比较中心。
- 支持按 schema 聚合。
- 支持忽略字段配置。
- 支持输出字段级聚合统计。
- 可以被 strategy 直接引用,也可以脱离 pipeline 单独调用。
建议输出格式:
- `field_name`
- `mismatch_count / total_count`
- `schema_refs`
- `sample local/remote`
例如:
```text
ProjectDiff
code 12/30 [ProjectResponseBase, ProjectBaseInfoUpdate] sample[project-1]=""/"012A"
```
### 3. data mapping 静态测试
下一步在 depm backend 中做静态 mapping 测试,目标不是验证整条 pipeline,而是验证 mapping 本身。
测试输入:
- 本地数据静态文件
- 远程数据静态文件
要求:
- 本地样本通过本地 `data_mapping` 转成 sync schema。
- 远程样本直接按 sync schema 校验。
- 忽略掉少量必须动态生成或动态获取的字段。
- 通过统一 validator 输出字段级 diff 报告。
这类测试应当绕开推送系统,通过手工配对样本来验证 mapping,而不是混在 pipeline 里排查。
## 已落地项
### 项目聚合接口
backend 中新增项目同步聚合读取接口,按项目统一返回:
- `project`
- `project_detail_data`
- `preparation`
- `production`
- `contract`
### schema diff validator
同步引擎新增可独立调用的 schema diff validator
- 支持忽略顶层字段
- 支持忽略列表元素内特定子字段
- 支持输出字段级 mismatch 聚合统计
- strategy 在 update diff 阶段直接复用该 validator
## 下一步
### 先做
1.`project` 建第一批静态样本。
2. 用统一 validator 对本地样本和远程样本做 schema 层比较。
3. 只在 validator 配置里维护少量 ignore 字段,不额外引入 contract 文件。
### 暂不做
1. 不引入 schema 外的独立契约系统。
2. 不为所有 domain 一次性补齐静态样本。
3. 不把副作用字段强行纳入完全一致比较。
+258
View File
@@ -0,0 +1,258 @@
# Construction 领域重构指南
## 1. 范围
本文档应覆盖 construction 领域中两部分字段的并集:
- 集团侧 `enum_config` 中 construction 相关的全部字段
- depm 中所有已经使用代码 enum、代码中的初始化列表、硬编码 dict、类 enum、`config` 表、`dict` 表承载,并且会影响施工初始化、手动创建、手动编辑、接口展示或业务校验的相关字段
当前这份文档先给出 construction 领域的总范围和主线判断,再把本轮优先落地对象收敛到一条主线:
- `construction.section_type_code`
以下复杂问题本轮先不展开具体迁移方案,但仍属于 construction 领域后续必须补齐的梳理范围:
- `task_id`
- `task_type`
- 其它施工节点层级和映射规则特别复杂的问题
也就是说,施工文档不是永久只看 section,而是本轮先把 section 这一层讲清楚,同时把 task 相关复杂问题明确记为后续梳理项,而不是视为领域外问题。
## 2. 当前现状
construction 领域当前至少有两类配置来源:
### 2.1 `construction_section_config`
当前按项目类型配置 section 信息,例如光伏下的:
- `code`
- `name`
- `weight`
- `project_root_nodes`
这份配置同时承担:
- section 候选项来源
- 施工初始化入口信息来源
### 2.2 `construction_node_config`
当前按项目类型配置节点信息,例如:
- `task_code`
- `type_id`
- `type_name`
- `unit`
- `weight`
- `is_primary`
- `section`
这份配置描述的是节点补充信息,不是字段级配置。
## 3. 重构目标
construction 领域本轮的目标不是重写全部施工配置,而是:
- 把 section 候选项迁到新的字段级配置
- 保留 `construction_node_config` 作为自由业务配置
- 让施工初始化和手动校验可以逐步复用统一的字段级入口
## 4. `construction.section_type_code` 的新结构
`construction_section_config` 应改造成新的字段级 key
- `construction.section_type_code`
它的 JSON 中必须包含:
- `field_config`
并且可以额外包含业务自定义字段,例如:
- `description`
- `init_config`
推荐结构:
```json
{
"description": "施工区字段配置",
"field_config": [
{"value": "GF", "label": "光伏场区", "weight": 1.4},
{"value": "SY", "label": "升压站", "weight": 1.4},
{"value": "SC", "label": "送出(自建)", "weight": 1.2},
{"value": "SCDW", "label": "送出(电网建设)", "weight": 1.0}
],
"init_config": {
"root_nodes_by_value": {
"GF": [
{"type_id": "01", "type_code": "01", "type_name": "光伏区土建工程"},
{"type_id": "03", "type_code": "03", "type_name": "光伏区安装工程"}
],
"SY": [
{"type_id": "02", "type_code": "02", "type_name": "升压站土建工程"}
]
}
}
}
```
这里的原则是:
- `field_config` 负责 section 候选项
- `init_config` 负责 section 相关初始化附属数据
- 两者可以同处一个 JSON 中,不需要再拆第二份 section 配置
## 5. `construction_node_config` 的定位
`construction_node_config` 不迁到字段级配置。
它仍然是自由业务配置,负责描述:
- 可能节点有哪些种类
- 哪些节点是主节点
- 节点单位、权重等补充信息
建议只做一项统一性改造:
- 顶层补 `description`
## 6. 需要迁移到 `field_config` 的功能
construction 领域应重点梳理下面两类功能:
### 6.1 项目初始化链路
需要明确:
- 初始化 section 时,候选项是否改由 `get_field_options(project_type, "construction.section_type_code")` 提供
- 初始化 root node 时,是否从同一配置对象中的业务自定义字段读取附属数据
也就是说,初始化链路应逐步从“直接解释旧 `construction_section_config`”迁移到“先读取字段级配置,再读取其业务附属字段”。
### 6.2 手动创建 / 编辑链路
需要明确:
- 如果某接口允许手动传 `section_type_code`
- 是否应改为调用 `validate_field_value(project_type, "construction.section_type_code", value)`
这样可以保证手动写入与初始化使用同一套候选项来源。
## 7. 当前迁移示例
### 示例 1section 候选项读取
当前:
- 通过 construction 私有 helper 直接读取并解释 `construction_section_config`
目标:
- 通过 `ConfigService.get_field_options(project_type, "construction.section_type_code")` 获取 section 列表
### 示例 2section 合法值校验
当前:
- 可能由业务代码自行判断,或者根本没有统一校验入口
目标:
- 统一走 `ConfigService.validate_field_value(...)`
### 示例 3:初始化附属数据读取
当前:
- 直接从旧 `construction_section_config` 读取 `project_root_nodes`
目标:
-`construction.section_type_code` 的完整配置对象中读取业务自定义字段
## 8. 梳理清单
construction 领域后续梳理时,至少要回答下面这些问题:
1. 哪些字段属于集团侧 `enum_config` 已覆盖或可对应的字段。
2. 哪些字段在初始化时使用候选项,哪些字段在手动创建时也要校验。
3. 哪些逻辑仍在直接解释旧 JSON,而不是通过统一 helper 读取。
4. 哪些逻辑仍在使用硬编码白名单。
5. `construction_node_config` 中哪些数据只是节点补充信息,哪些其实已经在承担字段候选项职责。
如果发现某字段既参与初始化又参与手动创建,就应优先考虑迁到字段级配置。
## 9. 按本轮要求的改造落点
construction 领域这轮不是泛泛地“改配置”,而是明确围绕 3 条链路收口:
- 初始化
- create 校验
- 下拉列表
### 9.1 初始化
当前施工初始化本来就不是 dict 驱动,而是:
- `construction_section_config`
- `ConstructionTypeTree`
- `ConstructionTypeMapping`
因此本轮初始化改造的重点不是“把 dict 搬到 config”,而是:
-`section_type_code` 的候选项读取统一收口到新的 config 工具
- 保留类型树和 root node 等结构化数据,但不再让 section 候选项散落在私有 helper 里
### 9.2 Create 校验
当前 `ConstructionService.create()` / `create_for_scdw()` 已经在 service 开头做了较强校验:
- `section_type_code` 来自 `construction_section_config`
- `section_sub_type_code` 来自 section config 和类型树映射
- `type_id` 来自施工类型树或 SCDW 专用配置
本轮目标是:
- `section_type_code` 统一改成通过 `ConfigService.validate_field_value(project_type, "construction.section_type_code", value)` 校验
- `section_sub_type_code``type_id` 继续走配置/类型树校验,不做降级
### 9.3 下拉列表
当前施工下拉接口已经基本同源:
- `/construction/section/type/list`
- `/construction/section/sub/type/list`
- `/construction/task/type/list`
它们当前都来自 `construction_section_config + 类型树`
本轮目标不是重做接口,而是:
- 让 section 级选项由新的 config 工具统一返回
- 保证下拉、初始化、create 校验继续同源
### 9.4 清理项
construction 领域这轮优先清理的是“section 候选项读取分散”问题,不是一次性重写全部施工结构。
完成后应清理:
- 只负责 section 候选项翻译的旧 helper 直接解析逻辑
- 不再适合作为 section 真源的旧硬编码映射
但下面这些不属于本轮删除目标:
- `ConstructionTypeTree`
- `ConstructionTypeMapping`
- 与节点树生成强绑定的结构化配置
## 10. 测试关注点
construction 领域 service 层集成测试至少覆盖:
1. 初始化:项目初始化后生成的 section / root node / 主施工任务是否来自目标 config。
2. 下拉:section / sub type / task type 列表是否和初始化使用同源配置。
3. Create:合法 `section_type_code``section_sub_type_code``type_id` 可以创建,非法值明确报错。
+108
View File
@@ -0,0 +1,108 @@
# Construction Task 领域重构指南
## 1. 范围
本文档对应 backend 的 `construction` 主表,也就是施工任务本体。
当前优先关注:
- `construction.section_type_code`
- `construction.task_type`
- `construction.status`
## 2. 字段盘点
### 2.1 `section_type_code`
当前实现:
- 本地有硬编码 `section_type_id_dict`
- 也有 `construction_section_config` 作为配置来源
- 模型里是 `String`
- 创建 schema 里是 `str`
- 同步 schema 对应字段名是 `construction_section`,类型是 `str`
当前使用:
- 是施工初始化和施工任务分区的关键字段
- construction 服务和 config helper 都直接依赖它
项目侧取值范围:
- 本地明确值:`GF``SY``SC``SCDW`
集团侧差异:
- 当前没看到集团侧 `enum_config` 有独立的 Construction 顶层段
- 同步 schema 里有对应字段,但仍是普通字符串
结论:
- 应迁到 `field_config`
- 继续沿用 `construction.section_type_code`
### 2.2 `task_type`
当前实现:
- 模型里是 `String`
- 创建 schema 里是 `str`
- 本地只有一个层级 enum `ConstructionCategory`,值是 `01``02``0201``03``0301``04``05`
当前使用:
- 施工树和类型树逻辑会使用它
- 但业务任务实际取值往往来自施工树节点,而不只是 `ConstructionCategory`
项目侧取值范围:
- 如果按层级 category 看,明确值是:`01``02``0201``03``0301``04``05`
- 如果按真实任务类型看,当前无法从单一代码 enum 列出完整集合,因为还受施工树/`construction_node_config` 影响
集团侧差异:
- 同步 schema 里 `task_type` 仍是 `str`
- 当前没有看到集团侧 `enum_config` 对这个字段给出统一候选项表
结论:
- 不能简单做成单一代码 enum
- 更适合拆成“层级 category”与“真实任务类型”两层定义
### 2.3 `status`
当前实现:
- 模型里是 `SmallInteger`
- 同步 schema 里是 `Optional[int]`
- 本地有 `progress_status_dict`
项目侧取值范围:
- `-1`
- `0`
- `1`
集团侧差异:
- 当前没看到集团侧 `enum_config` 独立列出 Construction.status
- 同步 schema 仍是 `int`
结论:
- 保留数值状态字段
- 需要统一是走代码常量、dict 映射还是后续字段配置
## 3. 重点结论
- `section_type_code` 是典型的字段级配置候选。
- `task_type` 当前最复杂,不能直接当成一个静态 enum 处理。
- `status` 目前是简单数值状态,但也需要统一正式定义源。
## 4. 待修改项清单
1.`section_type_code` 正式迁到 `field_config`
2. 清理 `section_type_code` 的硬编码 dict 与 config helper 双来源。
3. 拆分 `task_type` 的“层级 category”与“真实任务类型”语义。
4. 明确哪些 `task_type` 值来自施工树,哪些来自配置表。
5.`status` 建立统一常量定义,避免散落数值字面量。
+192
View File
@@ -0,0 +1,192 @@
# Contract 领域重构指南
## 1. 范围
本文档只做 Contract 领域的项目侧梳理。
当前优先关注:
- `contract.contract_type`
- `contract.code`
- `contract.currency`
## 2. 字段盘点
### 2.1 `contract_type`
当前实现:
- `app/modules/contract/enums.py` 定义了 `ContractType`
- 取值:`product``service``project`
- `Contract` 模型里是 `String`
- `ContractCreate` 里是 `str | None`
- 同步 schema `schemas/contract/contract.py` 里也是 `str`
当前使用:
- 是合同主表核心分类字段
- 合同结算服务里会按 `contract_type == "project"` 分支处理工程类合同
项目侧取值范围:
- `product`
- `service`
- `project`
集团侧差异:
- 集团侧 `enum_config.Contract.contract_type` 当前只有 `product``service`
- 项目侧比集团侧多一个 `project`
- 同步 schema 当前没有把它收紧成 enum
结论:
- 保留为代码 enum
- 先统一“是否正式保留 `project`
### 2.2 `code`
当前实现:
- 模型里是普通 `String`
- 没有本地正式 enum
- 初始化侧实际依赖 `contract_config` / `contract_code_dict` / `supply_chain_group`
当前使用:
- 初始化合同时,`code` 承担业务子类型编码职责,例如 `EPC``PC``KY``JL`
- 不是单纯的“自由合同编号”
项目侧取值范围:
- 当前无法从单一代码 enum 列出权威全集
- 本地已明确出现的典型值包括:`EPC``PC``PF``CG``OSC``KY``JL``E``SCYW``ZJ``TY``NB``XB``ZB``GS``CN``QT`
集团侧差异:
- 集团侧 `enum_config.Contract.code` 已明确把这类值当成正式候选项,并且按项目类型变化
- 同步 schema 里的 `code` 仍是普通 `str`
- 项目侧现在没有把这组值沉成统一的字段配置或正式 enum
结论:
- 不建议继续当自由字符串处理
- 更适合迁到 `field_config`,或至少统一走配置表来源
### 2.3 `currency`
当前实现:
- 模型里是 `String`
- 默认值 `CNY`
- schema 里是普通字符串
当前使用:
- 当前主要是合同基础信息字段
- 没看到复杂分支逻辑
项目侧取值范围:
- 当前代码里唯一明确值是默认值 `CNY`
集团侧差异:
- 当前没看到集团侧 `enum_config` 对这个字段给出正式候选项
- 同步 schema 里也是 `str`
结论:
- 先保持字符串字段
- 如果未来真的出现多币种,再决定是否需要正式枚举
## 3. 重点结论
- `contract_type` 应保留为代码 enum,但要先明确是否保留 `project`
- `code` 当前实际上承担合同业务子类型职责,不应继续当自由字符串处理。
- `currency` 当前没有足够证据说明要做枚举化。
## 4. 待修改项清单
1. 统一 `ContractType` 的正式定义源。
2. 确认 `contract_type=project` 是否继续作为正式值保留。
3.`contract_type` 在 schema 层补 enum 校验。
4.`contract.code` 从“自由字符串”改成统一配置来源。
5. 评估 `contract.code` 是否直接迁到 `field_config`
6. 区分清楚 `contract.code` 与“合同编号/contract_sn”的职责,避免一个字段同时承担两层语义。
## 5. 按本轮要求的改造落点
contract 领域本轮要直接收口 3 条链路:
- 初始化
- create 校验
- 下拉列表
### 5.1 初始化
当前合同初始化已经走 `contract_config`
因此这轮初始化改造的核心不是重做合同初始化,而是:
-`contract.code` 明确成正式字段级配置来源
- 后续让合同初始化、物资初始化、前端下拉都围绕这套配置工作
`contract_type` 继续保留为代码 enum,不迁到 `field_config`
### 5.2 Create 校验
当前 `ContractService.create()` 的校验主要依赖:
- `contarct_type_dict`
- `contract_code_dict`
- service / product 的硬编码白名单
这和初始化使用的 `contract_config` 不是同一真源。
本轮目标是:
- `contract_type` 继续按代码 enum 校验
- `contract.code` 改成通过新的 config 工具校验
- 避免继续新增 `if contract_type == ... and code in [...]` 这种硬编码白名单
### 5.3 下拉列表
当前合同下拉接口:
- `/contract/type/list`
- `/contract/sub-type/list`
其中:
- `type/list` 直接读代码字典
- `sub-type/list` 先写死过滤列表,再读代码字典
这和初始化 `contract_config` 不一致。
本轮目标是:
- 合同类型列表继续由代码 enum 提供
- 合同子类型列表改为动态读取 config
- 保证前端看到的 code 列表和初始化、create 校验同源
### 5.4 清理项
本轮 contract 领域改造完成后,应清理:
- 只用于 `contract.code` 下拉和校验的旧硬编码 code 白名单
- 只承担旧合同子类型翻译的代码字典入口
但下面这些不应误删:
- `ContractType` enum
- 其它真正稳定的固定语义 enum
## 6. 测试关注点
contract 领域 service 层集成测试至少覆盖:
1. 初始化:`contract_config` 能正确初始化合同,并继续驱动下游结算/物资链路。
2. 下拉:合同子类型下拉是否来自目标 config,且与初始化 code 集一致。
3. Create:合法 `contract_type + code` 创建成功,非法 code 明确报错。
+149
View File
@@ -0,0 +1,149 @@
# Contract Settlement 领域重构指南
## 1. 范围
本文档对应 backend 的 `contract_settlement` 主表。
当前优先关注:
- `contract_settlement.contract_type`
- `contract_settlement.status`
- `contract_settlement.payment_method`
## 2. 字段盘点
### 2.1 `contract_type`
当前实现:
- 模型里是 `String`
- 没有单独 enum 类型绑定到模型
项目侧取值范围:
- 当前主要跟随关联合同的 `contract_type`
- 代码现状至少包含:`product``service``project`
集团侧差异:
- 集团侧 `enum_config.ContractSettlement.contract_type` 当前只有 `product``service`
- 项目侧如果继续沿用 `project`,会比集团侧多一个值
- 当前仓库里没有 standalone 的同步 schema 来收紧这个字段
结论:
- 先与合同主表 `contract_type` 保持一致
- 再确认结算领域是否允许 `project`
### 2.2 `status`
当前实现:
- 模型里是 `SmallInteger`
- 默认值 `0`
项目侧取值范围:
- 当前没有独立 enum 类
- 从集团侧和现有语义看,应至少覆盖 `-1``0``1`
集团侧差异:
- 集团侧 `enum_config.ContractSettlement.status` 明确给出 `-1``0``1`
- 项目侧模型没有沉成统一常量或 enum
结论:
- 应补统一状态定义源
### 2.3 `payment_method`
当前实现:
- 本地 `PaymentMethod` enum 有两个值:`node_payment``progress_payment`
- 模型里仍是 `String`
- `SettlePaymentMethodsSelect` 手动校验合法值
当前使用:
- 合同结算服务会按 `payment_method` 分支校验和处理
项目侧取值范围:
- `node_payment`
- `progress_payment`
集团侧差异:
- 当前没看到集团侧 `enum_config` 为该字段给出正式候选项
- 当前仓库里也没有 standalone 的同步 schema 覆盖该字段
结论:
- 保留为本地代码 enum
- 需要把“手动 validator”提升为统一 schema / service 约束
## 3. 重点结论
- `contract_settlement` 里最稳定的枚举字段是 `payment_method`,应继续保留为代码 enum。
- `contract_type` 不能单独拍板,必须跟合同主表一起决定是否保留 `project`
- `status` 当前只是模型整数,没有正式定义源,需要补齐。
## 4. 待修改项清单
1. 明确 `contract_settlement.contract_type` 是否允许 `project`
2.`status` 补正式常量或 enum 定义。
3.`payment_method` 的合法值校验统一沉到正式 schema/enum,而不是只在个别 schema 手动校验。
4. 明确结算主表与合同主表在 `contract_type` 上是否必须同源。
## 5. 按本轮要求的改造落点
contract_settlement 领域不属于项目初始化第一批重点,但它直接涉及“写死下拉改动态”这件事。
### 5.1 初始化
本领域当前没有独立的项目初始化候选项链路。
`contract_type` 主要跟随合同主表,`payment_method` 主要出现在结算业务选择和后续分支处理中。
### 5.2 Create / 业务校验
当前 `payment_method` 的合法值主要靠:
- `PaymentMethod` enum
- `SettlePaymentMethodsSelect` schema 里的手动 validator
而 service `payment_methods_select()` 本身没有再做额外校验。
本轮目标是:
- 保留 `payment_method` 的固定语义 enum 属性
- 但在业务层入口也要能通过统一 config 工具或统一 enum 工具做显式校验
- 不把校验只留在 schema 里
### 5.3 下拉列表
当前支付方式只有选择接口,没有正式的动态下拉接口。
因此这轮要补的不是“替换已有动态接口”,而是:
- 把现有写死支付方式候选项明确收口
- 如果前端需要单独下拉接口,应改成动态返回,而不是继续依赖前端或 schema 写死
同理,`payment_type``advance_type` 这类当前靠本地 enum 固定的候选项,也应按同样思路处理。
### 5.4 清理项
本轮 contract_settlement 领域完成后,应避免继续保留:
- 只存在于 schema validator、但没有统一出口的支付方式候选项定义
- 前端和后端各自写死一份支付方式列表
## 6. 测试关注点
contract_settlement 领域 service 层集成测试至少覆盖:
1. `payment_method` 合法值更新成功,非法值明确报错。
2. 如果补了下拉接口,下拉返回是否与 service 校验同源。
3. `contract_type` 跟随合同主表时,不出现与合同主表不一致的脏数据。
+157
View File
@@ -0,0 +1,157 @@
# 项目侧字段分流总表草案
## 1. 目的
这份总表只回答一件事:
- 哪些字段继续保留为代码 enum / 代码常量。
- 哪些字段应迁到 `field_config`
- 哪些字段暂时不能直接收口,必须先补领域边界或映射关系。
## 2. 分流原则
优先按三类处理:
- 保留代码 enum:值域固定、语义稳定、业务分支已直接依赖。
- 迁到 `field_config`:值域跟 `project_type`、初始化模板或业务配置强相关。
- 暂缓收口:当前 backend 与 sync schema 映射不完整,或领域边界本身还没定。
## 3. 总表
| 领域 | 字段 | 当前主要来源 | 建议归类 | 备注 |
| --- | --- | --- | --- | --- |
| project | `project_type` | 代码 enum + 初始化字典 | 保留代码 enum | 核心路由字段 |
| project | `project_subtype` | 自由字符串 | 迁到 `field_config` | 建议 key`project.project_subtype` |
| project | `syxs` | 自由字符串 | 迁到 `field_config` | 建议 key`project.syxs` |
| project | `status` | 代码 enum + 服务推导 | 保留代码 enum | 正式集合含 `early`,并清理历史 `NULL` |
| project | `progress_type` | 代码 enum + DB enum | 保留代码 enum | 已与 sync schema 对齐 |
| project | `key_constraints` | 代码 enum 分叉 + DB enum | 保留代码 enum | 项目侧保留 `OTHER`,备注集团侧使用 `QT` |
| contract | `contract_type` | 代码 enum | 保留代码 enum | 先确认是否保留 `project` |
| contract | `code` | 初始化字典 / config / 自由字符串 | 迁到 `field_config` | 建议 key`contract.code` |
| contract | `currency` | 自由字符串 | 暂不处理 | 当前仅见 `CNY` |
| construction_task | `section_type_code` | 硬编码 dict + config | 迁到 `field_config` | 建议 key`construction.section_type_code` |
| construction_task | `task_type` | 类型树 + 字符串字段 | 暂缓收口 | 先拆 category 与真实任务类型 |
| construction_task | `status` | 数值字段 + dict | 保留代码常量 | 先统一常量定义源 |
| material | `material_type` | 本地列表 + 自由字符串 | 迁到 `field_config` | 建议 key`material.material_type` |
| material | `status` | 数值字段 | 暂缓收口 | 先和 `supply_status` 划边界 |
| material | `supply_status` | 代码 enum | 保留代码常量 | 先确认是否为正式推送字段 |
| material | `approval_party` | 本地候选项 | 暂不处理 | 当前无集团侧对应 |
| contract_settlement | `contract_type` | 跟随合同主表 | 暂缓收口 | 先确认是否允许 `project` |
| contract_settlement | `status` | 数值字段 | 保留代码常量 | 需补统一定义源 |
| contract_settlement | `payment_method` | 代码 enum | 保留代码 enum | 不建议迁 `field_config` |
| personnel | `personnel_type` | 本地代码常量 | 保留代码 enum | 本地真实领域是人员资质 |
| personnel | `task_type` | 本地候选项常量 | 保留代码常量 | 先补 schema 白名单 |
| personnel | `is_duty` | 数值常量 | 保留代码常量 | 与集团侧 `status` 不是同一字段 |
| preparation | `code` | 初始化字典 + 自由字符串 | 迁到 `field_config` | 建议 key`preparation.code` |
| preparation | `approval_status` | 通用审批体系 | 接入通用审批定义 | 不单独建 `field_config` |
| preparation | `is_exist` | 布尔字段 | 暂不处理 | 不属于候选项字段 |
| production | `code` | 初始化字典 + 自由字符串 | 迁到 `field_config` | 建议 key`production.code` |
| production | `approval_status` | 通用审批体系 | 接入通用审批定义 | 不单独建 `field_config` |
| production | `is_exist` | 布尔字段 | 暂不处理 | 不属于候选项字段 |
## 4. 第一批建议落地项
优先进入 `field_config` 的字段:
- `project.project_subtype`
- `project.syxs`
- `contract.code`
- `construction.section_type_code`
- `material.material_type`
- `preparation.code`
- `production.code`
优先保留并收紧 schema 的代码 enum 字段:
- `project.project_type`
- `project.status`
- `project.progress_type`
- `project.key_constraints`
- `contract.contract_type`
- `contract_settlement.payment_method`
优先先做边界确认、暂不直接配置化的字段:
- `construction.task_type`
- `material.status`
- `contract_settlement.contract_type`
## 5. 使用方式
这份表可以直接作为后续代码改造顺序的输入:
1. 先处理“迁到 `field_config`”的一组字段。
2. 再处理“保留代码 enum,但 schema 还没收口”的一组字段。
3. 最后处理“暂缓收口”的结构性问题字段。
## 6. 按本轮目标拆出的第一批字段
这次不是单纯做字段分类,而是要直接服务于 3 条链路:
- 项目初始化
- create 业务校验
- 前端下拉列表
因此第一批字段不能只看“适不适合进 `field_config`”,还要看它能不能尽快把这 3 条链路收口到同一真源。
### 6.1 第一批优先收口字段
建议第一批优先处理:
- `contract.code`
- `material.material_type`
- `construction.section_type_code`
- `preparation.code`
- `production.code`
- `power.code`
原因:
- 都属于 code 类字段
- 都直接影响初始化、创建或下拉其中至少两条链路
- 当前都存在 config / dict / 代码常量混用问题,或者已具备明显收口价值
### 6.2 每个字段要同时回答的 3 个问题
对上面每个字段,后续文档和代码改造都应同时回答:
1. 初始化时从哪里拿候选项。
2. create service 开头用哪个 config key 做校验。
3. 前端下拉接口用哪个 config key 返回选项。
如果这 3 个问题的答案不是同一个 key,就说明还没有真正收口。
### 6.3 本轮不优先收口但要避免误迁的字段
下面这些字段虽然也和 enum 有关,但本轮不应抢先塞进字段级配置:
- `project.status`
- `project.progress_type`
- `project.key_constraints`
- `contract.contract_type`
- `contract_settlement.payment_method`
原因是:
- 这些字段语义更固定
- 业务分支已经直接依赖
- 本轮重点是先把“按项目类型变化的 code 类字段”收口,而不是把固定状态类枚举全部配置化
### 6.4 对物资字段的特别说明
`material.material_type` 虽然在表里只是一行,但它实际上牵扯 4 套来源:
- 初始化合同候选项:`contract_config`
- 物资初始化补充:`material_contract_*`
- 前端下拉:`material_types`
- create 内部默认展示/单位:`contract_product_type_list`
所以 `material.material_type` 不是简单迁一个 key,而是本轮最需要先统一真源的一项。
### 6.5 本轮完成后的字段判定标准
一个字段只有同时满足下面 3 条,才算本轮真正完成收口:
- 初始化使用它对应的 field config
- create 校验使用它对应的 field config
- 下拉列表返回也使用它对应的 field config
+757
View File
@@ -0,0 +1,757 @@
# init_project_data 现状分析
## 1. 范围
本文档只分析 backend 里的 `ProjectService.init_project_data)` 当前到底做了什么。
重点回答 4 个问题:
- 这个函数当前创建了哪些业务数据。
- 它直接读取了哪些 `config``dict`
- 哪些 `dict` 是真的在负责初始化,哪些只是历史残留。
- 如果把这里 `dict` 负责的部分迁到 `config`,复杂度到底高不高。
## 2. 触发时机
`init_project_data)` 在项目审批通过后执行。
调用链是:
- `ProjectService.audit_project()`
- `ProjectService._audit_approval()`
- `ProjectService.init_project_data)`
也就是说,这不是“创建项目主表时立即执行”的逻辑,而是“审批通过后批量灌初始化数据”的逻辑。
## 3. 函数当前做的 6 件事
按执行顺序,当前函数会做下面 6 件事:
1. 读取当前 `project_type` 整个 category 下的全部 config。
2. 读取 6 组 dict 数据。
3. 初始化 `preparation` 里程碑节点。
4. 初始化 `production` 投产节点。
5. 初始化 `contract` 合同,并在合同初始化内部继续初始化 `contract_settlement``material`
6. 初始化 `power` 力能配置、`construction` 施工任务、`archive` 电子档案。
注意:
- 这个函数表面上不长,但它本质上是一个“项目初始化编排器”。
- 真正复杂的地方不只在它自己,而在它调用出去的各个 `init_from_*`
## 4. 直接读取了什么
### 4.1 直接读取的 config
函数一开始会调用:
- `config_service.get_category_config(category=project.project_type)`
这一步不是只读某一个 key,而是把当前 `project_type` 这个 category 下的全部 config 都读出来,再做:
- `configs = {config_key: decoded_json}`
但在 `init_project_data)` 自己内部,真正直接用到的 config 只有 1 个:
- `contract_config`
也就是说:
- 直接读取:整类 config
- 直接消费:只有 `contract_config`
- 其余 config 在这个函数里只是被一并装进了 `configs`,并没有直接使用
### 4.2 直接读取的 dict
函数会一次性读取 6 个 dict key:
- `preparation_{project_type}`
- `production_{project_type}`
- `construction_contract_{project_type}`
- `service_contract_{project_type}`
- `material_contract_{project_type}`
- `power_{project_type}`
但这 6 个 key 不是都真正被当前函数消费了。
## 5. 哪些数据是怎么创建出来的
### 5.1 Preparation
创建来源:
- 直接使用 dict`preparation_{project_type}`
创建方式:
- 读取 dict 项的 `code``name`
- 先删项目下旧的 `Preparation`
- 再批量插入新的 `Preparation`
涉及配置数量:
- `config`: 0
- `dict`: 1 组
当前种子数据条目数:
- `preparation_photovoltaic`: 12 条
- `preparation_onshore_wind`: 13 条
- `preparation_offshore_wind`: 14 条
结论:
- 这是最典型的“dict 直接承担初始化模板”场景。
- 结构非常简单,迁到 `config` 不复杂。
### 5.2 Production
创建来源:
- 直接使用 dict`production_{project_type}`
创建方式:
- 读取 dict 项的 `code``name`
- 先删项目下旧的 `Production`
- 再批量插入新的 `Production`
涉及配置数量:
- `config`: 0
- `dict`: 1 组
当前种子数据条目数:
- `production_photovoltaic`: 9 条
- `production_onshore_wind`: 9 条
- `production_offshore_wind`: 9 条
结论:
-`preparation` 一样,是平铺型 code/name 列表。
- 迁到 `config` 也不复杂。
### 5.3 Contract
创建来源:
- 直接使用 config`contract_config`
创建方式:
- `init_project_data)``configs["contract_config"]` 取出列表
- `ContractService.init_from_project()` 根据每项的 `code``name``contract_type` 创建合同
- 先删项目下旧合同,再批量插入新合同
后续联动:
- 初始化 `contract_settlement`
- 初始化 `material`
涉及配置数量:
- `config`: 1 个 key
- `dict`: 0 组直接参与合同创建
当前 `contract_config` 条目数:
- `photovoltaic`: 21 条
- `onshore_wind`: 20 条
- `offshore_wind`: 20 条
结论:
- 合同初始化本体已经是 config 驱动,不是 dict 驱动。
- 所以这里真正要迁移的不是“合同本身”,而是合同初始化后还依赖的那些配套 dict。
### 5.4 Material
创建来源:
- 不是 `init_project_data)` 直接创建。
- 它是 `contract.init_from_project()` 内部继续调用 `material.init_from_contract()` 创建。
当前来源拆分:
- 合同列表来源:`contract_config`
- 物资单位来源:`material_contract_{project_type}` dict
也就是说,当前 `material` 初始化其实是两段拼起来的:
- 先由 `contract_config` 决定会创建哪些货物类合同
- 再由 `material_contract_*` dict 给这些物资补 `unit`
涉及配置数量:
- `config`: 1 个上游 key `contract_config`
- `dict`: 1 组 `material_contract_{project_type}`
当前种子数据条目数:
- `material_contract_photovoltaic`: 14 条
- `material_contract_onshore_wind`: 12 条
- `material_contract_offshore_wind`: 7 条
这里有一个关键现状:
- `material` 实际创建多少条,取决于 `contract_config` 里有多少 `product` 合同。
- `material_contract_*` dict 只负责补充单位和名称,不负责决定是否创建。
- 因此这两套来源当前不是同一个真源。
已看到的分叉:
- 光伏 `contract_config` 里的货物类候选项和 `material_contract_photovoltaic` 并不完全一致。
- 陆风也存在类似分叉。
- 海风更明显:当前能看到 `contract_config` 已有较完整货物类合同,但 `dict` 侧条目明显更少。
结论:
- `material` 不是简单的“把一个 dict 搬到 config”。
- 真正要做的是先合并“合同货物候选项”和“物资单位/展示信息”两套来源。
- 这块复杂度中等,不是最高,但已经不是平铺搬运。
### 5.5 Power
创建来源:
- 直接使用 dict`power_{project_type}`
创建方式:
- 读取 dict 项的 `code``name`
- 统一绑定到 `contract_data.get("EPC")`
- 先删项目下旧 `Power`
- 再批量插入新 `Power`
涉及配置数量:
- `config`: 0
- `dict`: 1 组
当前种子数据条目数:
- `power_photovoltaic`: 1 条
- `power_onshore_wind`: 2 条
- `power_offshore_wind`: 2 条
结论:
- 也是平铺型初始化模板,迁到 `config` 不复杂。
- 但它和合同存在一条强耦合:默认绑定 `EPC` 合同。
### 5.6 Construction
创建来源:
- `init_project_data)` 自己没有直接给施工传 dict 或 config 数据
- 只调用:`create_constructions_for_project(tenant_id, project_id)`
但施工初始化内部会继续读取:
- `construction_section_config`
并依赖:
- `BuildMethod`
- `ConstructionTypeTree`
- `ConstructionTypeMapping`
实际创建内容:
-`construction_section_config.sections[*].project_root_nodes` 创建项目根单位工程
- 基于这些根节点再创建主施工任务
- 对质量验收类施工继续创建施工树
- 对主施工继续自动补次级施工
涉及配置数量:
- `config`: 至少 1 个 key `construction_section_config`
- `dict`: 0
- 另外强依赖数据库里的施工类型树,不是简单 JSON 配置
结论:
- 施工初始化本来就不是 dict 驱动,已经是 config + 类型树驱动。
- 这里真正复杂的不是“dict 要不要迁 config”,而是施工本身就已经是结构化初始化。
### 5.7 Archive
创建来源:
- 不依赖 dict
- 也不依赖 `init_project_data)` 读出来的 config
当前方式:
-`template/{project_type}_archive.xlsx` 模板文件初始化电子档案树
涉及配置数量:
- `config`: 0(对 `_init_project_data` 而言)
- `dict`: 0
- 外部模板文件: 1 份 Excel
结论:
- 电子档案初始化不属于这次 dict/config 收敛的核心问题。
## 6. 这 6 组 dict 里,哪些真的在负责事情
### 6.1 直接被 `init_project_data)` 消费的 dict
- `preparation_{project_type}`
- `production_{project_type}`
- `power_{project_type}`
这 3 组 dict 是真的在当前函数里直接承担初始化模板职责。
### 6.2 被 `init_project_data)` 取了,但当前函数里没有直接消费的 dict
- `construction_contract_{project_type}`
- `service_contract_{project_type}`
- `material_contract_{project_type}`
其中:
- `construction_contract_*`:当前只看到被 `init_project_data)` 取出,但没有实际使用。
- `service_contract_*`:当前也只看到被 `init_project_data)` 取出,但没有实际使用。
- `material_contract_*`:当前函数本身没用,但会在 `MaterialService.init_from_contract()` 里被重新读取一次,用于补单位。
这说明当前 `init_project_data)` 至少存在两类历史包袱:
- 取了但没用的 dict key
- 同一套语义在上游合同 config 和下游 material dict 之间分叉
## 7. 把 dict 迁到 config,会不会太复杂
### 7.1 不复杂的部分
下面 3 组迁移到 config,复杂度不高:
- `preparation_{project_type}`
- `production_{project_type}`
- `power_{project_type}`
原因很简单:
- 它们当前就是平铺的 `code/name` 列表
- 服务接口已经接受 `list[dict]`
- 初始化逻辑只做“删旧 + 批量重建”
- 不依赖复杂树结构,也不依赖额外业务联动
如果迁移:
- `preparation``production` 很适合直接改成 `field_config`
- `power` 更像一个小型初始化模板,也可以直接放到 config,但不一定要走 `field_config` 这个名字
### 7.2 中等复杂的部分
`material_contract_{project_type}` 迁移到 `config`,复杂度中等。
原因不是数据量大,而是来源分叉:
- 哪些货物类记录会被创建,取决于 `contract_config`
- 每种物资的显示名、单位,又取决于 `material_contract_*` dict
所以这块不是“dict 搬家”,而是“先统一真源”。
更稳妥的做法是:
- 先把 `contract_config``material_contract_*` 合并成一套正式结构
- 再决定是否把单位、显示名、parts 一并并到 `material.field_config` 或其它 config key
### 7.3 高复杂的部分其实不在 dict
真正复杂的初始化不是 dict,而是施工:
- 它已经依赖 `construction_section_config`
- 又依赖 `ConstructionTypeTree`
- 又依赖 `ConstructionTypeMapping`
- 还会继续生成施工树和次级施工
所以如果目标是“降低 `init_project_data` 初始化复杂度”,优先清理 dict 只能解决一部分表层问题,不能解决施工初始化本身的结构复杂度。
## 8. 额外现状问题
### 8.1 `init_project_data)` 读了整类 config,但只直接用了 `contract_config`
这会导致:
- 函数看起来像“config 驱动”
- 实际上只有合同初始化是直接 config 驱动
### 8.2 有 3 个 dict key 在当前函数里不是实际输入
- `construction_contract_*`
- `service_contract_*`
- `material_contract_*`
其中前 2 个当前更像历史残留。
### 8.3 当前只明显覆盖了 3 类项目类型的初始化种子
从现有 `dict.csv` / `config.csv` 看,当前明确有初始化种子的主要是:
- `photovoltaic`
- `onshore_wind`
- `offshore_wind`
`project_type` 枚举里其实还有:
- `pumped_storage`
- `hydropower`
- `thermal_power`
- `gas_turbine`
也就是说,`project_type` 枚举范围和初始化种子覆盖范围并不一致。
### 8.4 海风配置目前看起来不完整
当前能直接看到:
- `offshore_wind``contract_config`
- 也有 `preparation_*``production_*``power_*``material_contract_*` dict
但当前 `config.csv` 里没看到海风的:
- `construction_section_config`
- `construction_node_config`
- `material_types`
- `supply_chain_group`
这意味着海风初始化链路当前很可能并不完整,至少配置层不齐。
## 9. 结论
可以把 `init_project_data)` 当前的配置来源,拆成 4 类理解:
1. 纯 dict 初始化:`preparation``production``power`
2. 纯 config 初始化:`contract`
3. config + dict 混合初始化:`material`
4. config + 类型树 + 模板文件初始化:`construction``archive`
所以“把这里 dict 负责的东西迁到 config 会不会太复杂”的答案不是一个统一结论:
- `preparation``production``power`:不复杂,可以迁。
- `material_contract`:中等复杂,先统一真源再迁。
- `construction`:复杂点不在 dict,本来就不是 dict 驱动。
## 10. 建议落地顺序
1. 先把 `init_project_data)` 里“取了但没直接用”的 `construction_contract_*``service_contract_*` 从函数里清出来,确认是否还能删掉。
2. 再把 `preparation_{project_type}``production_{project_type}``power_{project_type}` 迁到 config。
3. 再处理 `material_contract_*`,但前提是先统一 `contract_config` 和 material 初始化所需的单位/展示信息来源。
4. 施工初始化不要和这批 dict 迁移混做,单独处理。
## 10.1 按本轮改造目标,初始化链路具体要改什么
如果把这次目标收敛成“初始化统一走配置工具”,那么 `init_project_data)` 不应该再只是分析对象,而是明确要往下面的形态改。
### A. 保留并整理的 config
初始化链路应继续保留并整理这几类 config:
- `contract_config`:合同初始化候选项
- `construction_section_config`:施工区域 / 单位工程 / 任务入口配置
- `material_types`:物资类型、展示名、单位、parts 等字段配置
- 后续新增的字段级配置 key,例如:
- `preparation.code`
- `production.code`
- `power.code`
这里的关键不是 config 数量多少,而是把“初始化真的要用的候选项”收口到 config 工具统一读取。
### B. 应从 dict 迁出的初始化候选项
下面这几类当前属于平铺型初始化模板,适合迁出 dict:
- `preparation_{project_type}`
- `production_{project_type}`
- `power_{project_type}`
迁移后的目标形态是:
- `init_project_data)` 不再直接读这些 dict key
- 改由 `ConfigService.get_field_options()` 或等价工具读取字段候选项
- service 仍只负责“删旧 + 批量重建”,不再关心候选项来自 dict 还是 config
### C. 应去掉的不适当 dict 用法
本轮最典型的不适当 dict 用法是:
- `material_contract_{project_type}``remark` 存物资单位
这类做法的问题不是“它现在不能工作”,而是:
- 含义不显式
- 初始化、创建、下拉三条链路都没法自然共用
- 后续导出 CSV / 校验时也不容易发现结构错误
迁移方向应是:
-`unit``label``parts` 等字段并回正式 config 结构
- 不再让单位这类业务字段挂在 dict `remark`
### D. `init_project_data)` 迁移后的读取原则
迁移后,`init_project_data)` 本身应只做 2 件事:
- 找到当前项目类型下所需的候选项配置
- 把候选项列表传给各领域初始化 service
它不应该继续承担:
- 现场拼 JSON 结构
- 自己手工区分 dict / config / remark
- 同时取一堆历史 key 再由下游各自二次分叉读取
### E. 初始化改造完成后的判定标准
初始化侧改造完成后,至少应满足:
- `preparation``production``power` 初始化不再依赖 dict
- `material` 初始化不再依赖 dict `remark` 补单位
- 初始化候选项与前端下拉、create 校验使用同一套 config 真源
- `init_project_data)` 中不再保留只读取不用的 dict key
## 11. 从接口创建视角反看这些字段是不是已经“主要 config 化”
只看 `init_project_data)`,容易得出“系统已经比较 config 驱动”的印象。
但如果再抽样看接口创建路径,会发现当前真实情况是:
- 初始化层里,确实有一部分已经走 `config`
- 但创建接口里,很多 enum-like 字段仍然主要靠代码常量或 service 层校验
- schema 层大多还是普通 `str`,并没有把这些候选值正式收口到统一配置来源
### 11.1 Contract 创建
接口入参:
- `ContractCreate` 里的 `contract_type``code` 都只是普通字符串
创建时字段来源:
- `contract_type``code` 直接来自接口传参
- 合同名称不是前端传,而是 service 根据 `contract_code_dict` 动态拼出来
当前校验方式:
- schema 层:只有长度约束,没有枚举白名单
- service 层:
- `code` 必须在 `contract_code_dict`
- `contract_type` 必须在 `contarct_type_dict`
- `service` 合同再限制为 `JL``KY``E``SCYW`
- `product` 合同再限制为一组硬编码产品 code
结论:
- 合同初始化是 `contract_config` 驱动
- 但合同接口创建仍然主要依赖代码字典和硬编码白名单
- 这说明“初始化层已 config 化”并不等于“合同域已经完成 config 收口”
### 11.2 Material 创建
接口入参:
- `MaterialCreate.material_type``approval_party` 也都是普通字符串
创建时字段来源:
- `material_type` 直接来自接口传参
- `show_name``unit` 不是前端传,而是 `MaterialService.get_material_info_by_code()``contract_product_type_list` 里反查
- 创建物资时还会同步创建一个 `product` 合同,合同 code 直接复用 `material_type`
当前校验方式:
- schema 层:
- `material_type` 只是示例值来自 `contract_product_type_list`
- `approval_party` 只有示例,没有正式白名单
- service 层:
- 强校验的是 `contract_quantity > 0`、项目存在、tenant 存在
- `material_type` 本身没有显式抛错白名单
- 如果 code 不在 `contract_product_type_list``get_material_info_by_code()` 会退回默认值 `支架/组`
- 随后又会去调用合同创建,而合同创建对 `code` 有一层更严格的白名单校验
结论:
- material 创建不是 config 驱动,而是代码常量驱动
- 而且 `material_type` 的直接校验位置并不清晰,存在“本服务默认值兜底,合同服务再拦截”的现状
- 这正是还需要整理 config/真源的证据,不适合直接认为已经收口完成
### 11.3 Construction 创建
接口入参:
- `ConstructionCreate.section_type_code``section_sub_type_code``type_id` 都是普通字符串
创建时字段来源:
- `section_type_code``section_sub_type_code``type_id` 直接来自接口传参
- `section_type_name``section_sub_type_name``task_code``task_type` 则由 service 根据配置和类型树反查生成
当前校验方式:
- schema 层:只有字符串长度和数值范围,没有枚举白名单
- service 层:
- `section_type_code` 必须存在于 `construction_section_config`
- `section_sub_type_code` 必须能映射到统一类型节点
- `type_id` 必须能映射到施工类型树节点
- `SCDW` 分支还会单独校验专用任务编码列表
结论:
- 施工创建是这 3 个抽样域里最接近“配置/类型树驱动”的
- 但即便如此,接口 schema 本身仍未把候选值正式声明出来,真正校验仍落在 service 里
### 11.4 总判断
如果只问“初始化层面能不能看出 enum_config 类字段怎么来的”,答案是:
- 能看出一部分。
- `preparation``production``power` 明显还来自 dict。
- `contract` 初始化候选项明显来自 `contract_config`
- `material` 初始化候选项来自 `contract_config`,单位补充来自 `material_contract_*` dict。
- `construction` 初始化候选项来自 config + 类型树。
但如果进一步问“那是不是已经主要用 config 写了,所以没必要再整理 config”,答案是否定的。
原因是当前系统仍然存在 3 个没有收口的问题:
- 初始化来源和接口创建来源不是同一套真源。
- schema 层没有把这些字段正式建模成受控候选值。
- service 层仍保留大量代码字典、硬编码白名单和默认兜底。
所以现在做 config 整理仍然有必要,而且必要性不在“把所有东西都搬到 config”,而在:
- 明确每个字段的唯一真源
- 消除 dict/config/代码常量三头并存
- 让初始化和接口创建共用同一套候选值与校验逻辑
## 12. 前端下拉选项当前是通过什么路径返回的
除了创建接口本身,前端通常会先调一些 list 接口拿可选项。
这部分如果和初始化来源不一致,就会出现:
- 初始化能创建出来的值,不一定和前端可选项完全一致
- 前端下拉看到的值,也不一定就是创建 service 实际校验使用的那套真源
下面按合同、物资、施工 3 类抽样。
### 12.1 Contract 的前端下拉路径
当前相关接口主要有:
- `/contract/type/list`
- `/contract/sub-type/list`
返回路径:
- `router -> ContractService.get_contract_type_list()`
- `router -> ContractService.get_sub_type_list(contract_type)`
数据来源:
- `get_contract_type_list()` 直接读代码里的 `contarct_type_dict`
- `get_sub_type_list()` 先根据传入 `contract_type` 选一组硬编码 `filter_key_list`
- 再去代码里的 `contract_code_dict` 组装返回值
和初始化是否一致:
- 不一致。
- 初始化合同候选项来自 `contract_config`
- 但前端拿到的合同类型/子类型下拉,来自代码字典和硬编码列表。
这意味着:
- 前端创建合同时看到的 code/type 候选,不是直接从初始化同源配置拿出来的。
- 如果未来 `contract_config` 改了,但代码字典没改,就会分叉。
### 12.2 Material 的前端下拉路径
当前至少有两条相关路径:
- `/material/material-types`
- `/material-detail/contract-product-type/list`
第一条路径:`/material/material-types`
- `router` 先根据 `project_id` 查项目类型
- 再调用 `config_service.get_config_json(project.project_type, "material_types")`
- 返回 `material_types` 里的 `label/value`
它的数据来源是:
- config`material_types`
第二条路径:`/material-detail/contract-product-type/list`
- `router -> MaterialDetailService.get_material_part_options()`
- service 内部同样会先读项目的 `material_types` config
- 再按当前 material 的 `material_type` 找到对应 kit 和 parts
它的数据来源也是:
- config`material_types`
但同时,物资模块里还保留了老的代码常量路径:
- `MaterialService.get_material_info_by_code()` 直接读 `contract_product_type_list`
- `MaterialDetailService.get_kit_options()` 也直接读 `contract_product_type_list`
和初始化是否一致:
- 也不一致。
- 前端下拉主要已经走 `material_types` config。
- 但初始化物资时,候选项来源还是 `contract_config`,单位来源还是 `material_contract_*` dict。
- 而创建 service 内部补 `show_name/unit` 时,又还在读代码常量 `contract_product_type_list`
这是当前分叉最明显的一类:
- 前端选项:config
- 初始化来源:config + dict
- 创建补充逻辑:代码常量
### 12.3 Construction 的前端下拉路径
当前相关接口主要有:
- `/construction/section/type/list`
- `/construction/section/sub/type/list`
- `/construction/task/type/list`
返回路径:
- `router -> ConstructionService.get_section_type_list(project_type)`
- `router -> ConstructionService.get_section_sub_type_list(section_type_code, project_type)`
- `router -> ConstructionService.get_task_type_list(section_type_code, section_sub_type_code, project_type)`
数据来源:
- `section_type_list` 通过 `config_helper.section_type_id_dict()` 读取 `construction_section_config`
- `section_sub_type_list` 通过 `config_helper.section_type_project_root_node_mapping()` 读取 `construction_section_config.project_root_nodes`
- `task_type_list` 则继续基于 `construction_section_config``ConstructionTypeTree` 生成
和初始化是否一致:
- 基本一致。
- 施工初始化本来就是 `construction_section_config + 类型树` 驱动。
- 前端下拉接口也是同一条配置/类型树路径。
所以施工是目前 3 类里最接近“初始化与前端下拉同源”的。
### 12.4 总结
从前端下拉接口角度看,当前 3 类字段的真实状态可以概括成:
1. Contract:前端下拉走代码字典,和初始化 `contract_config` 不一致。
2. Material:前端下拉主要走 `material_types` config,但初始化和创建内部仍分别依赖 `contract_config``material_contract_*` dict、`contract_product_type_list` 代码常量,分叉最严重。
3. Construction:前端下拉和初始化都主要走 `construction_section_config + 类型树`,相对一致。
所以如果后续要做收口,优先级也比较明确:
- 合同:先统一 `contract_config` 与前端 `type/sub-type list`
- 物资:先统一 `material_types``contract_config``material_contract_*``contract_product_type_list` 四套来源。
- 施工:优先补 schema/创建校验与配置同源,配置来源本身相对已经收敛。
+201
View File
@@ -0,0 +1,201 @@
# Material 领域重构指南
## 1. 范围
本文档只做 Material 主表梳理。
当前优先关注:
- `material.material_type`
- `material.status`
- `material.supply_status`
- `material.approval_party`
## 2. 字段盘点
### 2.1 `material_type`
当前实现:
- 本地没有独立 enum 类
- 主要来源是 `contract_product_type_list`
- 模型里是 `String`
- 创建 schema 里是 `str`
- 同步 schema 里也是 `str`
当前使用:
- 是物资主类型字段
- schema 示例和业务逻辑都直接依赖 `contract_product_type_list`
项目侧取值范围:
- 当前本地明确值:`TY``ZJ``NB``XB``ZB``GS``CN``DL``XT``CBJDB``SX``ECXT``QT`
集团侧差异:
- 集团侧 `enum_config.Material.material_type` 是按项目类型变化的
- 光伏下有:`ZJ``TY``NB``XB``ZB``GS``CN``QT`
- 陆风/海风下有:`FJ``TT``XB``ZB``GS``CN``QT`
- 项目侧当前使用的本地列表明显偏光伏,且代码里还存在命名差异:例如本地用 `XT` / `CBJDB` / `SX` / `ECXT`,集团侧常见的是 `35KV` / `CYJD` / `SVG` / `ECPG`
- 同步 schema 里 `material_type` 仍是 `str`
结论:
- 应迁到 `field_config`
- 不能继续只靠本地 `contract_product_type_list`
### 2.2 `status`
当前实现:
- 模型里是 `SmallInteger`
- 同步 schema 里是 `Optional[int]`
项目侧取值范围:
- 当前未看到单独 enum 类
- 现有注释和使用都表明它是数值状态字段
集团侧差异:
- 集团侧 `enum_config.Material.status` 给的是 `-1``0``1`
- 项目侧没有在 material 模块里沉成统一常量
结论:
- 应统一正式状态定义源
### 2.3 `supply_status`
当前实现:
- 模型里是 `SmallInteger`
- 本地有 `SupplyStatus` enum`-1``0``1``None`
项目侧取值范围:
- `-1`
- `0`
- `1`
- `None`
集团侧差异:
- 集团侧 Material.status 也是按 `-1``0``1` 表达进度状态
- 项目侧存在 `status``supply_status` 两个相近字段,语义有重叠
结论:
- 需要明确 `status``supply_status` 哪个才是正式推送字段
### 2.4 `approval_party`
当前实现:
- 模型里是 `String`
- schema 里是 `str`
- 当前示例值是 `jiagong``yigong`
项目侧取值范围:
- `jiagong`
- `yigong`
集团侧差异:
- 当前没看到集团侧 `enum_config` 对这个字段给出正式候选项
- 同步 schema 里也没有这个字段
结论:
- 先保留本地代码级候选项
- 后续如需推送再决定是否外提
## 3. 重点结论
- `material_type` 是标准的 `field_config` 候选字段,不能再继续绑死在本地光伏型列表上。
- `status``supply_status` 当前语义重叠,必须先定正式推送字段。
- `approval_party` 暂时还是本地专有字段,不需要现在就硬并到集团侧。
## 4. 待修改项清单
1.`material_type` 正式迁到 `field_config`
2. 按项目类型重建 `material_type` 候选项,不再只用本地光伏型列表。
3. 统一 `35KV/XT``CYJD/CBJDB``SVG/SX``ECPG/ECXT` 这类命名差异。
4. 明确 `status``supply_status` 的职责边界。
5.`material_type` 在 schema 层增加统一校验入口。
## 5. 按本轮要求的改造落点
material 是本轮最需要针对性改造的领域之一,因为它当前同时存在:
- 初始化来源分叉
- create 校验分叉
- 下拉来源分叉
### 5.1 初始化
当前 material 初始化不是单源:
- 候选项来自 `contract_config` 里的 product 合同
- 单位和部分展示信息来自 `material_contract_{project_type}` dict
而且 `unit` 现在还挂在 dict `remark` 上。
本轮目标是:
- 明确 `material.material_type` 的正式配置来源
-`label``unit``parts` 等附属字段纳入正式 config 结构
- 去掉“从 dict remark 补单位”这种不适当用法
### 5.2 Create 校验
当前 create 路径里:
- `material_type` 入参本身没有正式 config 校验
- `show_name``unit``contract_product_type_list` 反查
- 随后又会调用合同创建,由合同 service 再做一层 code 白名单校验
这说明 material create 现在不是单一入口校验。
本轮目标是:
-`MaterialService.create()` 开头基于 config 校验 `material_type`
-`get_field_option_map()` 之类的方法直接拿 `label``unit``parts`
- 不再依赖 `contract_product_type_list` 和默认兜底值
### 5.3 下拉列表
当前物资下拉已经部分走 config:
- `/material/material-types``material_types`
- `/material-detail/contract-product-type/list` 也读 `material_types`
但模块里仍保留了旧的硬编码路径:
- `get_material_info_by_code()`
- `get_kit_options()`
- `contract_product_type_list`
本轮目标是:
- 让物资类型和部件列表统一从新的 config 工具返回
- 下拉、初始化、create 校验全部围绕同一套 `material_type` 配置
### 5.4 清理项
material 领域这轮改造完成后,应优先清理:
- `material_contract_*` 中用 `remark` 承载单位的旧逻辑
- `contract_product_type_list` 里只服务于 `material_type` 真源的旧定义
- `get_material_info_by_code()` 这类旧反查入口
## 6. 测试关注点
material 领域 service 层集成测试至少覆盖:
1. 初始化:项目初始化后 material 的 `material_type``show_name``unit` 是否来自目标 config。
2. 下拉:物资类型和部件列表是否来自同一 config,且与初始化同源。
3. Create:合法 `material_type` 能创建,非法 `material_type` 明确报错,不再依赖默认兜底。
+167
View File
@@ -0,0 +1,167 @@
# Personnel 领域重构指南
## 1. 范围
当前仓库里没有独立的 `personnel` 项目业务模块。
最接近的本地实现是:
- `app/modules/safety/models/safety_personnel_qualification.py`
- `app/modules/safety/schemas/personnel_qualification.py`
因此本文档先按“人员资质”这一真实落点梳理,而不是假设仓库里已经有独立 `personnel` 主表。
当前优先关注:
- `personnel_type`
- `task_type`
- `is_duty`
## 2. 字段盘点
### 2.1 `personnel_type`
当前实现:
- 模型里通过类常量定义两个值:`safe``special`
- 模型字段本身是 `String`
- schema 里仍是 `str`
项目侧取值范围:
- `safe`
- `special`
集团侧差异:
- 集团侧 `enum_config.Personnel` 里并没有 `personnel_type`
- 集团侧更像是在描述“人员/吊装设备”等资源配置编码,而不是人员资质类型
- 当前仓库里也没有 standalone 的 sync schema 对这个领域建模
结论:
- 这是本地代码级 enum 字段
- 可保留为代码 enum
### 2.2 `task_type`
当前实现:
- 模型里通过 `PERSONNEL_TASK_TYPE` 类常量给出候选项
- 字段本身是 `String`
- schema 里仍是 `str`
项目侧取值范围:
- `电工`
- `焊工`
- `高空作业人员`
- `无人机操作员`
- `其他`
集团侧差异:
- 集团侧 `enum_config.Personnel` 没有这个字段
- 当前仓库里也没有 sync schema 对应
结论:
- 这是本地代码级候选项字段
- 可以保留为代码常量,但要补 schema 校验
### 2.3 `is_duty`
当前实现:
- 模型里通过类常量定义:`1` 在场,`2` 退场
- 字段本身是 `SmallInteger`
- schema 里是 `int`
项目侧取值范围:
- `1`
- `2`
集团侧差异:
- 集团侧 `enum_config.Personnel.status` 给的是 `-1``0``1`
- 这更像资源进度状态,不是本地 `is_duty` 语义
- 两边不是同一字段,不应强行对齐
结论:
- 保留为本地数值常量字段
## 3. 重点结论
- 当前所谓 `personnel` 领域,在本地真实落点是“人员资质”,不是集团侧 `Personnel` 那套资源配置模型。
- 因此这里最重要的不是照抄集团侧,而是先承认两个系统的领域边界不一致。
## 4. 待修改项清单
1. 明确 `personnel` 在本地的正式领域名是否就是 `safety_personnel_qualification`
2.`personnel_type` 补正式 enum/schema 校验。
3.`task_type` 补统一白名单校验。
4. 不要把集团侧 `Personnel.code/status` 直接映射到本地 `personnel_type/is_duty`
## 5. 按本轮要求的改造落点
personnel 领域不是这轮“项目初始化 + 项目类型 field_config”第一批重点,但仍需要按统一口径写清楚 create、下拉和清理边界。
### 5.1 初始化
本地人员资质领域当前没有项目初始化批量灌数链路。
因此本轮不把它纳入 `init_project_data)` 第一批改造范围。
### 5.2 Create 校验
当前 `PersonnelQualificationService.create()` 已直接在 service 层校验:
- `personnel_type`
- `task_type`
校验来源是本地模型类常量:
- `PERSONNEL_TYPE_SAFE`
- `PERSONNEL_TYPE_SPECIAL`
- `PERSONNEL_TASK_TYPE`
这类字段当前更像固定语义本地枚举,不是按 `project_type` 变化的字段级配置候选项。
本轮目标是:
- 继续把它们视为本地固定候选项
- 统一 create / update / list 使用同一套定义源
- 不误迁到项目类型 field_config
### 5.3 下拉列表
当前已有:
- `/safety/personnel/task_type/list`
它直接返回 `PERSONNEL_TASK_TYPE`
本轮这里不要求强行改成 `project_type` 相关 config,而是要求:
- 明确它是本地固定候选项
- 前端下拉和 create 校验继续同源
### 5.4 清理项
personnel 领域本轮的清理重点不是删 enum,而是避免错误地把集团侧 `Personnel` 语义强行映射进来。
应避免:
- 把本地 `personnel_type``task_type` 误改造成和集团侧 `code/status` 一一对应
- 同一字段在 service 和下拉各自维护两套值
## 6. 测试关注点
personnel 领域 service 层集成测试至少覆盖:
1. `personnel_type` 合法值和非法值创建校验。
2. `special` 人员必须校验 `task_type`
3. `task_type/list` 返回值与 create 校验使用同源常量。
+697
View File
@@ -0,0 +1,697 @@
# 项目侧字段级配置重构计划
## 1. 目标
本次重构的目标,不是先统一项目侧全部 config / enum / dict 的历史实现,而是先补一套项目侧自己的“字段级配置总表”和配套工具,使下面这类问题有统一解法:
- 某个字段在某个 `project_type` 下有哪些可选项。
- 这些可选项的中文名是什么。
- 业务代码如何方便地做手动校验。
- 初始化流程如何从同一套字段级配置中读取候选项。
- 哪些字段应该继续在代码里用 enum 处理。
- 哪些字段应该迁到配置文件处理。
这套能力对齐集团侧 `enum_config.json` 的思路,但项目侧不照搬集团侧的存储形式,而是基于现有 `config` 表和 CSV 数据维护方式落地。
因此,本次重构除了建设字段级配置总表本身,还需要同步完成一件事:
- 梳理字段边界,明确哪些字段属于“固定语义、固定值集合、适合保留在代码 enum 中”,哪些字段属于“随 `project_type` 或初始化流程变化、应迁到字段级配置中”。
也就是说,这次不是单纯补一个配置表,而是要同时建立一套可执行的字段分流规则:
- enum 字段:继续留在代码中,由 enum 表达固定约束。
- field_config 字段:迁到配置中,由字段级配置表达候选项。
## 1.1 本轮实际要改的内容
这次文档和后续代码改造,不再停留在“抽象上应该统一”的层面,而是直接围绕下面 5 件事展开。
### 0. 通用能力
要补两类公共能力:
- 增强现有配置类,在 `ConfigService` 上新增字段级配置读取、映射、校验工具。
- 增强基于 JSON 导出 CSV 的工具,使字段级配置和现有 config 维护流程能直接联动。
### 1. 优化项目初始化过程
目标不是只分析 `init_project_data)`,而是把初始化链路改成统一从配置工具取候选项:
- 整理项目初始化当前实际使用到的 config。
- 把适合迁移的初始化候选项改为通过新的 config 工具获取后创建。
- 去掉不适当的 dict 用法,例如把单位塞在 dict `remark` 里。
### 2. 优化创建数据的验证
这次不改 schema 目录类型系统,但要改 service 层入口。
要求是:
- 在 create 类接口对应的 service 业务入口开头,基于 config 工具校验 code 类字段。
- 不再继续新增代码字典白名单、硬编码 if/else 白名单。
### 3. 统一下拉列表来源
这次不仅要梳理下拉列表,还要把它们收口到新的 config 工具:
- 梳理所有 code 类下拉列表当前返回路径。
- 能迁移的下拉统一改成通过 config 工具返回。
- 少量现有写死的下拉列表,例如合同结算类型列表,也要改成动态获取。
### 4. 删除废弃 enum / 硬编码字典
完成上述迁移后,要清掉已经不再适合作为真源的:
- 废弃 enum
- 废弃硬编码 dict
- 只用于旧下拉或旧创建校验的代码常量
### 5. 自动化测试脚本
这次不是只改文档和代码,还要补自动化测试,覆盖至少 3 类集成场景:
- 初始化脚本 / 初始化 service 集成测试
- 各业务下拉列表获取集成测试
- 各业务 create service 集成测试
测试脚本写在 `tests` 下,不写到项目根目录。
## 2. 重构范围
本次重构只覆盖一类字段:
- 对应集团侧 `enum_config.json` 相应字段的项目侧字段。
-`project_type` 相关。
- 和项目初始化过程相关。
也就是说,本次优先处理的是“初始化候选项字段”,而不是全部字符串字段。
优先关注的业务范围包括:
- project 初始化时需要的字段候选项
- contract 初始化相关字段
- construction 初始化相关字段
- material 初始化相关字段
- power / preparation / production 中与项目类型初始化直接相关的字段
从现状代码看,这些初始化来源当前混杂在:
- Python enum
- Python 硬编码 dict
- `dict_data`
- `config` 表中的自由 JSON
- 外部业务大表或类型树
本次要做的不是一次性清理所有来源,而是为这些字段补一个统一的字段级配置出口。
对本轮来说,这里的“统一出口”不是泛指,而是明确服务于 3 条业务链路:
- 初始化:通过统一配置工具取候选项并创建数据
- 创建:通过统一配置工具校验 code 类字段
- 下拉:通过统一配置工具返回选项列表
对本轮来说,这里的“统一出口”不是泛指,而是明确服务于 3 条业务链路:
- 初始化:通过统一配置工具取候选项并创建数据
- 创建:通过统一配置工具校验 code 类字段
- 下拉:通过统一配置工具返回选项列表
## 3. 不在本次范围内的内容
以下内容明确不在本次重构范围:
- 接口返回层的翻译统一
- `dict` 体系的整体重构
- 所有历史自由 config 的一次性收敛
- 所有外部树和业务大表的一次性字段化
- schema 目录下的类型系统改造
- 运行时直接读取 JSON 文件作为配置源
这里补充两条边界:
- 不允许为了兼容旧逻辑继续新增 dict `remark` 之类的隐式字段承载方式。
- 不允许为了减少改动,在新工具落地后继续保留“前端下拉走 config、初始化走 dict、create 校验走代码字典”的三头并存状态。
这里补充两条边界:
- 不允许为了兼容旧逻辑继续新增 dict `remark` 之类的隐式字段承载方式。
- 不允许为了减少改动,在新工具落地后继续保留“前端下拉走 config、初始化走 dict、create 校验走代码字典”的三头并存状态。
因此,本次允许字段级配置中的 `value / label` 与现有 dict 翻译短期重复。后续如有需要,可再通过脚本生成 dict 或做统一翻译收敛。
## 4. 核心设计
### 4.1 新增“字段级配置总表”概念
项目侧需要新增一层固定格式的字段级配置。
这层配置的职责只有一个:
- 对外统一回答“某字段在某项目类型下有哪些可选项”。
它不是为了取代所有业务 config,而是为了给业务代码、初始化流程、推送系统提供统一的读取接口。
### 4.2 存储方式
字段级配置继续存放在现有 `config` 表中,并通过现有 CSV 数据维护。
建议约束如下:
- `category = project_type`
- `config_type = json`
- `config_key` 不强制按规则自动拼接
- 代码里仍允许手动传入 key 查询
虽然推荐把 key 命名为 `model_name.field_name`,例如:
- `construction.section_type_code`
- `material.material_type`
- `contract.code`
但这只是推荐命名,不作为框架自动推导前提。
也就是说:
- 本次不建设“根据 model + field 自动拼 config key”的机制
- 只建设“拿到 key 后,按统一 schema 读取字段级配置”的机制
### 4.3 固定 JSON 结构
字段级配置的 `default_value` / `config_value` 必须是一个 JSON object。
对于字段级配置,只强约束一件事:
- 顶层必须有 `field_config`
其中:
- `field_config` 必须是列表
- 列表中每个元素至少有 `value``label`
推荐最小结构如下:
```json
{
"field_config": [
{"value": "GF", "label": "光伏场区", "weight": 1.4},
{"value": "SY", "label": "升压站", "weight": 1.4}
]
}
```
`field_config` 外,其它顶层字段不做统一规定,由各业务自己定义。
例如,某些字段配置可能会额外包含:
- `description`
- `init_config`
- 其他与该字段强相关的附属配置
这些字段允许存在,但不纳入统一字段级 schema 强约束。
### 4.4 非字段级 config 的统一要求
对于仍然保留的自由业务 config,不要求包含 `field_config`
但建议默认增加一个顶层字段:
- `description`
这样至少能保证所有 config 在文档和工具层面具备最低限度的语义说明。
## 5. `field_config` Schema 设计
### 5.1 字段候选项结构
字段级配置中的 `field_config` 列表项,最少包含:
- `value: str`
- `label: str`
允许附加字段,例如:
- `weight`
- `unit`
- `section`
- `legacy`
- `flag`
- `skip_init`
- `skip_create`
这些附加字段不要求在所有字段中统一,但它们必须和值本身强相关。
### 5.2 建议的 Pydantic 模型
建议新增专门的 Pydantic schema,用于字段级配置校验。
```python
from pydantic import BaseModel, ConfigDict, Field
class FieldOptionItem(BaseModel):
model_config = ConfigDict(extra="allow")
value: str = Field(..., description="候选值")
label: str = Field(..., description="显示名")
```
说明:
- 只需要识别 `field_config` 这个字段
- 只需要校验 `field_config``list[FieldOptionItem]`
- 其它顶层字段不在统一 schema 里写死
## 6. `ConfigService` 统一承载
字段级配置相关的辅助方法统一放到 `ConfigService`,不再额外拆新的独立 service。
也就是说,这次应增强的是:
- `app/modules/system/services/config.py`
而不是在别处再起一套并行 helper。
### 6.1 建议新增的方法
```python
async def get_field_config(project_type: str, key: str, tenant_id: str | None = None) -> dict[str, Any]:
...
async def get_field_options(project_type: str, key: str, tenant_id: str | None = None) -> list[dict[str, Any]]:
...
async def get_field_option_map(project_type: str, key: str, tenant_id: str | None = None) -> dict[str, dict[str, Any]]:
...
async def get_field_label(project_type: str, key: str, value: str, tenant_id: str | None = None) -> str | None:
...
async def validate_field_value(
project_type: str,
key: str,
value: str,
tenant_id: str | None = None,
*,
error_prefix: str | None = None,
) -> None:
...
```
如果后续需要覆盖“批量校验多个字段”或“直接返回下拉结构”的场景,可以继续补:
```python
async def get_field_options_for_frontend(project_type: str, key: str, tenant_id: str | None = None) -> list[dict[str, Any]]:
...
async def validate_field_values(project_type: str, items: list[tuple[str, str]], tenant_id: str | None = None) -> None:
...
```
如果后续需要覆盖“批量校验多个字段”或“直接返回下拉结构”的场景,可以继续补:
```python
async def get_field_options_for_frontend(project_type: str, key: str, tenant_id: str | None = None) -> list[dict[str, Any]]:
...
async def validate_field_values(project_type: str, items: list[tuple[str, str]], tenant_id: str | None = None) -> None:
...
```
### 6.2 各接口职责
`get_field_config()`
- 读取指定 `project_type + key` 的 JSON 配置
- 找出顶层 `field_config`
- 校验 `field_config` 是否满足 `list[FieldOptionItem]`
- 返回完整配置对象
`get_field_options()`
- 返回 `field_config` 列表
- 供初始化、下拉、候选项获取直接使用
`get_field_option_map()`
- 返回 `{value: option}` 的映射
- 供快速查找、补充属性读取使用
`get_field_label()`
- 根据 `value` 返回 `label`
- 本次主要服务于业务内部使用,不承担接口翻译统一职责
`validate_field_value()`
- 手动调用
- 校验给定值是否在当前项目类型该字段的候选项中
- 校验失败时抛出业务异常
另外,本轮会直接用到两种典型调用方式:
- 初始化时:`get_field_options()` / `get_field_option_map()`
- 创建和编辑时:`validate_field_value()`
- 下拉接口时:`get_field_options_for_frontend()``get_field_options()`
另外,本轮会直接用到两种典型调用方式:
- 初始化时:`get_field_options()` / `get_field_option_map()`
- 创建和编辑时:`validate_field_value()`
- 下拉接口时:`get_field_options_for_frontend()``get_field_options()`
这里特别强调:
- `validate` 是手动调用的
- 本次不做自动注入所有模型或 schema 的统一校验框架
- 目标是让业务层“方便调用”,不是一次性接管所有校验入口
### 6.3 `ConfigService` 需要补齐的基础能力
`ConfigService` 当前除了读取外,还需要补齐两类基础能力:
#### 1. JSON 序列化 / 反序列化统一承载
不应继续让调用方自己到处 `msgspec.json.decode(...)`
`ConfigService` 应统一负责:
- 读取 JSON 配置时反序列化
- 更新 JSON 配置时序列化
- 在字段级配置方法中复用这一套逻辑
也就是说,配置的 JSON 处理应沉到 `ConfigService` 本身,而不是散落在各业务模块里。
#### 2. 关键读取方法加缓存
`ConfigService` 中的关键读取方法应增加缓存。
优先包括:
- `get_config_json()`
- `get_field_config()`
- `get_field_options()`
- `get_field_option_map()`
- `get_field_label()`
要求:
- 读接口走缓存
- 更新配置后显式失效相关缓存
- 不允许缓存让旧配置长期滞留
`validate_field_value()` 自身不需要单独缓存,但应复用已经缓存过的字段选项读取结果。
### 6.4 实现要求
工具类内部应统一处理:
- 读取 `config` 表 JSON
- 用 Pydantic 校验 `field_config`
- 构建 options / map / label
- 输出清晰错误,便于发现配置格式问题
不允许的处理方式:
- 遇到 schema 错误时静默降级
- 遇到缺字段时自动猜测兼容成其他结构
如果配置不满足 `field_config` schema,应直接报错。
## 7. JSON / CSV 工具重整
本次不要求运行时直接读取 JSON 文件。
但为了方便调试和维护,应把现有 `scripts/tool/csv_data_tools/` 按新的字段级配置目标重新整理。
目标是:
- 一个 key 对应一个 JSON 文件
- JSON 文件仍然采用 config 行列表格式
- 工具负责 JSON 和 `config.csv`、数据库之间的转换
这块和本轮目标直接对应:
- 字段级配置调整后,可以直接导出为 `config.csv`
- 初始化改造前后,可以快速对比 config 差异
- 下拉列表和创建校验依赖的 config,可以单独导出检查,不再手工翻长 CSV
这块和本轮目标直接对应:
- 字段级配置调整后,可以直接导出为 `config.csv`
- 初始化改造前后,可以快速对比 config 差异
- 下拉列表和创建校验依赖的 config,可以单独导出检查,不再手工翻长 CSV
建议目录:
- `scripts/initdata/json_config/`
建议能力:
- `validate-json-config`: 校验 JSON 文件是否满足 config 行结构,以及 `field_config` 约束
- `json-to-csv`: 从 `json_config` 目录批量生成 `config.csv`
- `json-to-db`: 从 `json_config` 目录批量导入数据库
- `csv-to-json`: 从 `config.csv` 按 key 导出 JSON 文件
- `db-to-json`: 从数据库导出 JSON 文件用于调试
当前已有的 `scripts/tool/csv_data_tools/csv_data_tools.py` 可以复用思路,但需要围绕“字段级配置总表 + 一 key 一文件”的目标重新整理。
## 8. 领域重构指南
主计划只定义系统级方案。具体领域的梳理应拆到独立文件,例如:
- [construction.md](/Users/zyl/my_projects/ecm_sync_system/docs/enum_refactor/construction.md)
- [project.md](/Users/zyl/my_projects/ecm_sync_system/docs/enum_refactor/project.md)
后续 contract / material 等领域也应按同样方式分别整理。
每个领域文档都不是只看“集团侧已有字段”或者“准备迁到 `field_config` 的字段”。
每个领域都必须同时覆盖两部分字段:
- depm 中所有已经使用以下来源之一的字段:代码 enum、代码中的初始化列表、硬编码 dict、类 enum、`config` 表、`dict`
- 集团侧 `enum_config` 中该领域涉及的全部字段
两部分做并集,不做交集。
也就是说:
- `enum_config` 中出现的字段,必须全部纳入领域文档
- 即使某字段暂时没有出现在 `enum_config` 中,只要 depm 已经在用代码 enum / 类 enum / 初始化列表 / 硬编码 dict / `config` 表 / `dict` 表承载它,并且它影响初始化、手动创建、编辑、校验、展示,也必须纳入领域文档
这里要特别避免一种错误写法:
- 不能把领域范围写成“只讨论和集团侧 `enum_config` 直接对应的字段”
- 不能因为某字段暂时只有 depm 本地 enum / 类 enum,而集团侧还没有对应项,就把它排除在领域文档之外
每个领域文档都应至少覆盖以下内容:
### 8.1 先做字段总表
每个领域文档开头都应先列出字段总表,至少回答:
- 集团侧 `enum_config` 覆盖了哪些字段
- depm 侧当前还额外维护了哪些相关字段
- 每个字段当前使用的来源是什么:代码 enum、初始化列表、硬编码 dict、类 enum、`config` 表、`dict` 表,或它们的混用
- 每个字段分别影响哪些链路:项目初始化、手动创建、手动编辑、接口展示、历史数据兼容、业务校验
如果同一个字段在多个来源重复定义,也必须明确记出来,而不是只记录“最终准备保留哪个”。
### 8.2 整理不查表的 enum 类数据
不是只有查表型数据才需要整理。
如果某领域存在不依赖字典表、也不依赖 `config` 表的纯 enum / 纯硬编码候选项,同样要梳理:
- 当前定义位置在哪
- 是否重复定义
- 是否和集团侧 `enum_config` 对应字段有关
- 是否参与初始化、手动创建、手动编辑、展示翻译或历史校验
- 是否需要继续保留为代码 enum
- 是否应该迁到新的 `field_config`
也就是说,纯 enum 数据如果和初始化字段、手动创建字段有关,也在本轮视野内。
### 8.3 梳理哪些功能需要迁移到新的 `field_config`
每个领域都需要明确列出:
- 哪些初始化功能依赖字段候选项
- 哪些手动创建 / 编辑功能依赖字段候选项
- 哪些历史白名单校验需要迁移到 `validate_field_value()`
- 哪些直接遍历旧 JSON 的逻辑需要迁移到 `get_field_options()` / `get_field_option_map()`
- 哪些字段虽然目前由 `dict` 表或代码 enum 承载,但实际上已经在承担“按项目类型变化的候选项字段”职责
并且必须给出示例。
示例类型包括:
- 初始化时从旧 config 直接遍历候选项,改为从 `field_config` 读取
- 手动创建时从硬编码 dict 判断合法值,改为调用 `validate_field_value()`
- 展示名称时从字段选项里直接拿 `label`
对本轮来说,每个领域文档都至少要把下面 3 条链路写清楚:
- 初始化:这个字段初始化时从哪里取候选项,迁移后改成哪个 config key
- 创建:这个字段 create service 当前怎么校验,迁移后改成哪个 `validate_field_value()` 调用
- 下拉:这个字段前端当前通过哪个接口取值,迁移后改成哪个 config key 统一返回
对本轮来说,每个领域文档都至少要把下面 3 条链路写清楚:
- 初始化:这个字段初始化时从哪里取候选项,迁移后改成哪个 config key
- 创建:这个字段 create service 当前怎么校验,迁移后改成哪个 `validate_field_value()` 调用
- 下拉:这个字段前端当前通过哪个接口取值,迁移后改成哪个 config key 统一返回
### 8.4 必须完整对齐集团侧 `enum_config`
领域梳理时,不能只挑 depm 里已经处理过的字段,也不能只挑“看起来最容易改”的字段。
每个领域都必须把集团侧 `enum_config` 中该领域涉及的字段逐一对齐到 depm,明确说明它现在处于哪一种状态:
- depm 已有同名或同语义字段,并且来源清晰
- depm 有近似字段,但命名、语义、来源或取值范围不完全一致
- depm 还没有正式落点,只能先记录缺口
对齐 `enum_config` 时,每个字段都应继续检查两条链路:
- 项目初始化时,取值从哪里来,如何展开
- 手动创建 / 更新时,取值如何校验,是否和初始化使用同一来源
如果两条链路来源不一致,应明确记录并给出迁移方案。
## 9. 实施步骤
### 第一阶段:公共能力
### 第一阶段:公共能力
- 明确字段级配置固定 JSON 结构
- 新增 `FieldOptionItem`
-`ConfigService` 中新增字段级配置方法
- 给关键读取方法加缓存
- 补齐 JSON 序列化 / 反序列化统一处理
- 明确错误处理规则:配置结构不合法时直接报错
- 增强 JSON / CSV 工具,支持字段级配置导入导出
### 第二阶段:初始化改造
- 逐项整理 `init_project_data)` 当前真实依赖的 config / dict
-`preparation``production``power` 这类平铺型初始化模板迁到 config
- 整理 `material` 初始化,把单位等附属属性从不适当的 dict 备注迁出
- 清理 `init_project_data)` 中取了但不该继续作为真源的 dict 依赖
### 第三阶段:创建校验改造
- 为 contract / material / construction 等 create service 入口补基于 config 的 code 校验
- 把当前分散在 service 内的硬编码白名单逐步迁到 config 工具
- 保留真正应该继续存在的代码 enum,不把固定语义状态误迁到 config
### 第四阶段:下拉列表改造
### 第四阶段:下拉列表改造
- 梳理 code 类下拉接口
- 下拉统一改成通过 config 工具返回
- 少量仍写死的下拉改为动态获取
- 保证前端下拉、初始化候选项、create 校验三者同源
### 第五阶段:清理和测试
- 删除废弃 enum 和废弃代码字典
- 删除不再适合作为真源的 dict 依赖
- 补 service 层集成测试和初始化测试
- 梳理 code 类下拉接口
- 下拉统一改成通过 config 工具返回
- 少量仍写死的下拉改为动态获取
- 保证前端下拉、初始化候选项、create 校验三者同源
### 第五阶段:清理和测试
- 删除废弃 enum 和废弃代码字典
- 删除不再适合作为真源的 dict 依赖
- 补 service 层集成测试和初始化测试
## 10. 产出要求
本次重构最终至少要产出:
- 一套明确的字段级配置 schema
- 一套明确的 `ConfigService` 字段级配置接口
- 一套明确的缓存与缓存失效规则
- 一套 JSON / CSV / DB 三者之间的转换与维护规范
- 一组领域级重构文档
- 一组初始化 / 下拉 / create 的 service 层集成测试
- 一组初始化 / 下拉 / create 的 service 层集成测试
判断本次重构是否成功的标准不是“历史实现全部统一”,而是:
- 后续新增一个与项目类型初始化相关的字段时,知道应优先写入字段级配置总表
- 业务代码可以通过 `ConfigService` 统一获取 options / map / label / validate
- 初始化流程和手动创建校验可以逐步收敛到同一来源
- 前端下拉、初始化、create 校验不再各自读不同真源
## 11. 自动化测试要求
这次测试不是只补 unit test,而是以 service 层集成测试为主。
至少覆盖下面 3 组:
### 11.1 初始化测试
- 覆盖项目初始化入口
- 校验初始化创建出的 preparation / production / contract / material / power / construction 数据是否来自目标 config
- 校验错误配置时直接报错,不做静默降级
### 11.2 下拉列表测试
- 覆盖 contract / material / construction 等核心下拉接口或对应 service
- 校验返回项是否来自预期 config key
- 校验下拉返回和初始化候选项是否一致
### 11.3 Create 校验测试
- 直接走 service 层 create
- 对合法 code 校验通过
- 对非法 code 明确抛错
- 校验不再依赖旧硬编码字典或 dict `remark`
### 11.4 测试组织方式
- 测试代码写在 `tests/`
- 优先写 service 层集成测试,不在项目根目录放临时脚本
- 如果需要初始化测试数据,应复用现有测试体系,不改 `schemas/`、不改 `filtered_datasource/`
- 前端下拉、初始化、create 校验不再各自读不同真源
## 11. 自动化测试要求
这次测试不是只补 unit test,而是以 service 层集成测试为主。
至少覆盖下面 3 组:
### 11.1 初始化测试
- 覆盖项目初始化入口
- 校验初始化创建出的 preparation / production / contract / material / power / construction 数据是否来自目标 config
- 校验错误配置时直接报错,不做静默降级
### 11.2 下拉列表测试
- 覆盖 contract / material / construction 等核心下拉接口或对应 service
- 校验返回项是否来自预期 config key
- 校验下拉返回和初始化候选项是否一致
### 11.3 Create 校验测试
- 直接走 service 层 create
- 对合法 code 校验通过
- 对非法 code 明确抛错
- 校验不再依赖旧硬编码字典或 dict `remark`
### 11.4 测试组织方式
- 测试代码写在 `tests/`
- 优先写 service 层集成测试,不在项目根目录放临时脚本
- 如果需要初始化测试数据,应复用现有测试体系,不改 `schemas/`、不改 `filtered_datasource/`
+133
View File
@@ -0,0 +1,133 @@
# Preparation 领域重构指南
## 1. 范围
本文档对应 backend 的 `preparation` 里程碑节点表。
当前优先关注:
- `preparation.code`
- `preparation.approval_status`
- `preparation.is_exist`
## 2. 字段盘点
### 2.1 `code`
当前实现:
- 模型里是 `String`
- 没有本地 enum 类
- 项目初始化时,节点集合来自 `preparation_{project_type}` 字典数据
- 同步 schema `project_extensions/preparation.py` 里也是 `str`
项目侧取值范围:
- 当前无法从单一代码 enum 列出完整范围
- 已知典型值包括:`JSZB``HZBA``KYNS``TZJC``YDYS``LCPF``YDPF``JRPF``HPPF``SGXK``SBPF`
集团侧差异:
- 集团侧 `enum_config.Preparation.code` 已明确把该字段作为按项目类型变化的正式候选项
- 本地当前仍主要靠初始化字典/字符串承载
结论:
- 更适合迁到 `field_config`
- 不应继续只靠初始化字典和自由字符串
### 2.2 `approval_status`
当前实现:
- backend 依赖 `ApprovalMixin`
- 本地 `PreparationApproval` schema 仍使用 int
- 同步 schema `PreparationDetail` 里是 `Optional[str]`
项目侧取值范围:
- 当前本地未在 preparation 模块中沉成独立 enum
集团侧差异:
- 同步 schema 有字段,但没有正式 enum
结论:
- 应统一接入通用审批状态体系
### 2.3 `is_exist`
当前实现:
- 模型里是 `Boolean`
- backend 和 sync schema 均有该字段
项目侧取值范围:
- `true`
- `false`
集团侧差异:
- 当前没看到 enum_config 对这个字段给出候选项
结论:
- 保持布尔字段即可
## 3. 重点结论
- `preparation.code` 是标准的初始化候选项字段,最适合迁到 `field_config`
- `approval_status` 不需要在本领域单独发明枚举,应直接接入通用审批体系。
- `is_exist` 是布尔存在性字段,不在本轮字段级配置重点内。
## 4. 待修改项清单
1.`preparation.code` 正式迁到 `field_config`
2. 区分“节点编码范围”与“审批状态/存在性”这两类字段,不要混在一个配置源里。
3.`approval_status` 接入通用审批状态定义。
## 5. 按本轮要求的改造落点
preparation 属于本轮最适合先收口的一类:
- 初始化平铺
- code 字段明确
- 旧代码里已经出现多份硬编码映射
### 5.1 初始化
当前 preparation 初始化直接来自 `preparation_{project_type}` dict。
本轮目标是:
-`preparation.code` 迁到 field config
- 初始化改成通过 config 工具读取候选项再批量创建
- 不再依赖 dict 作为初始化真源
### 5.2 Create 校验
当前 `PreparationCreate` / `PreparationService` 没有统一的 code 白名单校验入口。
本轮目标是:
- 在 create / update 对应的 service 入口开头,基于 config 校验 `preparation.code`
- 不再新增本地 code 白名单或名称反查表来兜底
### 5.3 下拉列表
preparation 当前没有单独的“节点编码下拉接口”,前端更多拿的是项目实例列表 `/preparation/list`
这轮文档上的要求是:
- 如果后续需要独立 code 候选项下拉,应直接从 `preparation.code` field config 返回
## 6. 测试关注点
preparation 领域 service 层集成测试至少覆盖:
1. 初始化:项目初始化时生成的节点是否来自目标 field config。
2. Create / Update:合法 `code` 可以写入,非法 `code` 明确报错。
3. 列表:项目实例列表顺序是否来自配置顺序,而不是硬编码 `sort_code`
+128
View File
@@ -0,0 +1,128 @@
# Production 领域重构指南
## 1. 范围
本文档对应 backend 的 `production` 投产节点表。
当前优先关注:
- `production.code`
- `production.approval_status`
- `production.is_exist`
## 2. 字段盘点
### 2.1 `code`
当前实现:
- 模型里是 `String`
- 没有本地 enum 类
- 项目初始化时,节点集合来自 `production_{project_type}` 字典数据
- 同步 schema `project_extensions/production.py` 里也是 `str`
项目侧取值范围:
- 当前无法从单一 enum 文件列出完整范围
- 已知典型值包括:`SYZDSD``SPBW``QBBW``KHWC240``YJSC``DBTC``CQZS``WGJS``JGJS`
集团侧差异:
- 集团侧 `enum_config.Production.code` 已把该字段按项目类型列成正式候选项
- 本地仍主要靠初始化字典和自由字符串
结论:
- 更适合迁到 `field_config`
### 2.2 `approval_status`
当前实现:
- backend 依赖 `ApprovalMixin`
- 本地 `ProductionApproval` schema 仍使用 int
- 同步 schema `ProductionDetail` 里是 `Optional[str]`
项目侧取值范围:
- 当前本地未在 production 模块中沉成独立 enum
集团侧差异:
- 同步 schema 有字段,但没有正式 enum
结论:
- 应统一接入通用审批状态体系
### 2.3 `is_exist`
当前实现:
- 模型里是 `Boolean`
- backend 和 sync schema 都有
项目侧取值范围:
- `true`
- `false`
集团侧差异:
- 当前没看到 enum_config 对该字段给候选项
结论:
- 保持布尔字段即可
## 3. 重点结论
- `production.code``preparation.code` 一样,属于典型的初始化节点编码字段,应迁到 `field_config`
- `approval_status` 仍应走通用审批体系,而不是 production 自己维护一套值域。
- `is_exist` 不属于本轮需要抽成字段候选表的重点字段。
## 4. 待修改项清单
1.`production.code` 正式迁到 `field_config`
2. 将项目初始化里对 `production_{project_type}` 的依赖逐步收敛到统一字段配置入口。
3.`approval_status` 接入通用审批状态定义。
## 5. 按本轮要求的改造落点
production 和 preparation 一样,属于本轮适合优先收口的平铺型节点编码领域。
### 5.1 初始化
当前 production 初始化直接来自 `production_{project_type}` dict。
本轮目标是:
-`production.code` 迁到 field config
- 初始化改成通过 config 工具读取候选项后批量创建
- 不再依赖 dict 作为初始化真源
### 5.2 Create 校验
当前 production 领域也没有统一的 code 候选项校验入口。
本轮目标是:
- 在 create / update 的 service 入口开头,基于 config 校验 `production.code`
- 不再继续用本地硬编码列表承担合法值白名单职责
### 5.3 下拉列表
production 当前也没有单独的 code 候选项下拉接口,更多是返回项目实例列表 `/production/list`
本轮要求是:
- 如果需要单独 code 候选项下拉,应直接从 `production.code` field config 返回
## 6. 测试关注点
production 领域 service 层集成测试至少覆盖:
1. 初始化:项目初始化时生成的投产节点是否来自目标 field config。
2. Create / Update:合法 `code` 可以写入,非法 `code` 明确报错。
3. 列表:项目实例列表顺序是否来自配置顺序,而不是硬编码 `sort_code`
+399
View File
@@ -0,0 +1,399 @@
# Project 领域重构指南
## 1. 范围
本文档只做 Project 领域的项目侧梳理。
必须覆盖两类字段:
- 集团侧 `enum_config` 对应到 Project 的字段:`project_type``project_subtype``syxs``status`
- depm 本地已经做成 enum / 类 enum,或者已经承担初始化、创建校验、业务分支职责的字段:`progress_type``key_constraints``build_methods`
本文重点只回答四件事:
- 当前有几种实现
- 模型层怎么存
- 是否真的用到了
- 创建、校验、初始化时怎么取值,以及下一步怎么改
## 2. 字段盘点
### 2.1 `project_type`
当前实现:
- `app/common/enums.py` 有一套 `ProjectType`
- `app/modules/project/enums.py` 又有一套 `ProjectType`
- `Project` 模型里是 `String`
- `ProjectCreate` / `ProjectUpdate` 里是 `str`
当前使用:
- `ProjectService.init_project_data)` 用它作为初始化路由键,去取 `preparation_{project_type}``production_{project_type}``power_{project_type}` 等配置
- construction 模块的部分接口和工具也直接复用 `ProjectType`
当前问题:
- 定义重复
- 模型层不是 enum
- 创建 / 更新入口没有真正按 enum 校验
项目侧取值范围:
- 当前代码里的正式候选项是:`photovoltaic``onshore_wind``offshore_wind``pumped_storage``hydropower``thermal_power``gas_turbine`
- `app/common/enums.py``app/modules/project/enums.py`、初始化 `dict_data.csv` 当前三处是一致的
集团侧差异:
- 当前未看到和项目侧不同的取值项
- 这个字段目前可视为两边一致
- 但同步 schema 里的 `project_type` 字段当前仍写成 `str`,并没有直接收紧成 `ProjectType`
结论:
- 保留为代码 enum
- 它是核心路由字段,不迁到 `field_config`
### 2.2 `project_subtype`
当前实现:
- 模型里是 `String`
- `ProjectCreate` / `ProjectUpdate` 里是 `str`
- 没看到本地正式 enum
当前使用:
- 当前主要是项目主表字段
- 在 Project 模块内没看到稳定初始化使用点
当前问题:
- 没有统一候选项来源
- 没有统一校验
- 现在只是“字符串字段”
项目侧取值范围:
- 目前列不出权威范围
- 当前项目侧没有正式 enum、没有 `field_config`、没有 schema 白名单
- 从代码看,它现在理论上就是任意字符串
项目侧可举例范围:
- 目前只能借集团侧样例来猜测,例如光伏下可能是 `centralized``distributed`
集团侧差异:
- 集团侧至少给出了光伏下的两个候选项:`centralized``distributed`
- 项目侧目前一个正式值都没有落下来
- 其它项目类型在集团侧也基本还是空占位,所以当前只能确认“项目侧缺正式范围,集团侧只给了局部样例”
- 同步 schema 对应字段名是 `sub_type`,当前类型是 `Optional[str]`,也还没有 enum 化
困难:
- 项目侧没有正式定义源,无法从代码中反推出完整候选项集合
- 集团侧除光伏外也没有给出有效值,导致现在只能列出局部范围,不能写成全项目类型完整表
结论:
- 改为 `field_config`
- 建议 key`project.project_subtype`
### 2.3 `syxs`
当前实现:
- 模型里是 `String`
- 当前 `ProjectCreate` / `ProjectUpdate` 里没有这个字段
- 没看到本地正式 enum
当前使用:
- 在 Project 模块内暂时没看到明确创建、初始化或校验使用点
- 当前更像是“模型里已有字段,但项目侧链路尚未收敛”的状态
当前问题:
- 有字段,但没有统一入口
- 没有统一候选项来源
- 没有统一校验
项目侧取值范围:
- 目前列不出权威范围
- 项目模型里有这个字段,但 Project 创建 schema 里没有该字段
- 代码中也没找到项目侧正式 enum 或统一校验入口
项目侧可举例范围:
- 当前只能参考集团侧海风场景,例如 `hssy``lssy`
集团侧差异:
- 集团侧至少给出了海风下的两个候选项:`hssy``lssy`
- 项目侧目前还没有正式范围、正式入口和正式校验
- 其它项目类型在集团侧也基本还是空占位,所以当前差异主要是“集团侧有局部定义,项目侧还没落下来”
- 同步 schema 中 `syxs` 当前也是 `Optional[str]`,还没有 enum 化
困难:
- 项目侧没有正式输入入口,也没有权威候选项定义
- 集团侧只有海风场景给了有效值,因此暂时只能列局部范围
结论:
- 改为 `field_config`
- 建议 key`project.syxs`
### 2.4 `status`
当前实现:
- `app/modules/project/enums.py` 里有 `ProjectStatus`
- `app/common/enums.py` 里也有一套 `ProjectStatus`
- 模型里是 `String`
- 响应 schema 里是 `str`
当前使用:
- `ProjectService.change_project_status()` 会根据 `investment_decision_date``actual_construction_date``actual_production_date``actual_full_production_date` 计算并写回 `status`
当前问题:
- 定义重复
- 模型层不是 enum
- 当前状态值由服务逻辑推导,但接口层没有统一约束
- 当前正式状态集合必须补上 `early`
- 当前库里还存在 `NULL`,这说明状态机和历史数据都还没收口干净
项目侧取值范围:
- 目标正式值应为:`early``preparation``started``partially_connected``completed`
- 当前代码定义和服务推导还没有完整覆盖这 5 个值
- 当前实际数据里还存在 `NULL`,这应视为待清理异常值,而不是正式取值
集团侧差异:
- 项目侧最终也要纳入 `early`
- 因此这个字段的目标差异不是“是否要 early”,而是“项目侧何时补齐 early 并清掉 NULL”
- 同步 schema 里的 `status` 当前也是 `Optional[str]`,没有正式 enum 类型
结论:
- 保留为代码 enum
- 正式状态集合按 `early``preparation``started``partially_connected``completed` 收口
- `NULL` 不是正式值,必须通过数据修复和状态回填消除
### 2.5 `progress_type`
当前实现:
- `app/common/enums.py``app/modules/project/enums.py` 都定义了 `ProgressType`
- 模型里是 `SQLEnum(ProgressType)`
- `ProjectInfoResponse` 里使用 `ProgressType`
- `ProjectCreate` 里仍是 `str`
当前使用:
- 是 Project 主表正式字段
- `project_detail_data` 初始化时会写入 `progress_type`
- 列表筛选也会直接按它查询
当前问题:
- 定义重复
- 模型层已收紧,但创建入口还没收紧
项目侧取值范围:
- 当前正式值是:`A``B``C`
- `app/common/enums.py``app/modules/project/enums.py`、初始化 `dict_data.csv` 当前是一致的
集团侧差异:
- 集团侧同步 schema 中,`progress_type` 也是正式 enum
- 同步 schema 的取值项是:`A``B``C`
- 这一点和项目侧代码 enum 一致
- 当前真正的差异不在取值范围,而在项目侧创建入口仍是 `str`,还没有像同步 schema 一样收紧到 enum 类型
困难:
- 这个字段属于 depm 本地已枚举化字段,不是集团侧 `Project` 段直出字段
- 因此这里只能写“项目侧权威范围明确,集团侧暂无直连字段”
结论:
- 保留为代码 enum
- 不迁到 `field_config`
### 2.6 `key_constraints`
当前实现:
- `app/common/enums.py``app/modules/project/enums.py` 都定义了 `KeyConstraints`
- 两套定义已分叉:common 里有 `OTHER`project 模块里没有
- 模型里是 `SQLEnum(KeyConstraints)`
- `ProjectInfoResponse` 里使用 `KeyConstraints`
- `ProjectSquareUpdate` 里还是 `str`
当前使用:
- 是 Project 主表正式字段
- `project_detail_data` 初始化时会和 `progress_type` 一起写入
- `project_detail_data` 审批回写时也会更新 Project 主表
当前问题:
- 定义分叉
- 主表是 enum,但部分接口入参仍是字符串
- 项目侧正式口径应保留 `OTHER`
项目侧取值范围:
- 项目侧正式值应为:`NONE``LAND``TRANSMISSION``BOTH``OTHER`
- 当前 `app/common/enums.py` 和初始化 `dict_data.csv` 已包含这 5 个值
- 当前 `app/modules/project/enums.py` 还少 `OTHER`
集团侧差异:
- 集团侧同步 schema 中,`key_constraints` 也是正式 enum
- 同步 schema 的取值项是:`NONE``LAND``TRANSMISSION``BOTH``QT`
- 项目侧正式口径保留 `OTHER`
- 这里需要备注的差异只有一条:集团侧用 `QT`,项目侧用 `OTHER`
结论:
- 保留为代码 enum
- 项目侧按 `OTHER` 收口
- 在同步和推送链路上单独备注 `OTHER/QT` 差异
- 同时补齐 project 模块那套缺失的 `OTHER`
### 2.7 `build_methods`
当前实现:
- `app/modules/project/enums.py` 里有 `BuildMethod`
- 模型里是 `String`
- `ProjectCreate` 里是 `str`
- 响应 schema 通过 `enum_to_build_method_name` 做名称映射
当前使用:
- construction 服务里确实使用了 `project.build_methods`
- 它会影响施工分支判断,例如 `power``mixed``none``self`
当前问题:
- 模型层不是 enum
- 创建入口不是 enum
- 已经参与业务分支,但没有统一校验
项目侧取值范围:
- 当前代码 enum 给出的值是:`self``power``mixed``none`
- construction 分支判断当前也确实在使用这 4 个值
集团侧差异:
- 集团侧同步 schema 里没有 `build_methods` 这个名字,对应字段是 `scgc_type`
- `scgc_type` 当前在同步 schema 里是 `Optional[str]`
- 目前没有看到同步 schema 中为 `scgc_type` 定义正式 enum
也就是说:
- 语义上两边大概率是在说同一个字段
- 但集团侧当前还没有把它收紧成正式 enum
- 因此这个字段更像“项目侧本地先有 enum,集团侧 schema 还停留在字符串字段”
困难:
- 这是项目侧本地业务字段,不是集团侧 `Project` 段直出字段
- 项目侧虽然已有 enum,但模型和 schema 还没完全收紧
结论:
- 保留为代码 enum
- 不迁到 `field_config`
- 需要把模型层和 schema 层一起收紧
## 3. 重点结论
- `project_type` 是初始化路由键,必须保留为代码 enum。
- `project_subtype``syxs` 是典型的按项目类型变化字段,应迁到 `field_config`
- `progress_type``key_constraints``build_methods` 都已经在项目侧真实使用,不是可忽略字段;它们应保留为代码 enum,但当前实现没有收紧干净。
- `status` 也应保留为代码 enum,正式集合必须包含 `early`,并且要清掉历史 `NULL`
- `key_constraints` 项目侧正式口径保留 `OTHER`,只需要在文档和同步链路上备注集团侧使用 `QT`
## 4. 待修改项清单
1. 统一 `ProjectType` 定义,只保留一套正式 enum。
2. 统一 `ProjectStatus` 定义,并把 `early` 纳入正式状态集合。
3. 统一 `ProgressType` 定义,只保留一套正式 enum。
4. 统一 `KeyConstraints` 定义,项目侧正式保留 `OTHER`
5.`project.project_subtype` 改为字段级配置,放入 `field_config`
6.`project.syxs` 改为字段级配置,放入 `field_config`
7.`project_type` 在 schema 层增加正式 enum 校验,不再使用裸 `str`
8.`progress_type` 在创建入口增加正式 enum 校验。
9.`build_methods` 在 schema 层增加正式 enum 校验。
10.`key_constraints` 在仍使用字符串入参的接口上补 enum 校验。
11. 清理 `Project.status` 的历史 `NULL` 数据,并补状态回填逻辑,确保 `NULL` 不再出现。
12. 评估 `Project` 模型中的 `project_type``status``build_methods` 是否要收紧为数据库 enum;如果暂不收紧,也要先统一 schema 校验。
13.`project_subtype``syxs` 增加统一读取入口:`get_field_options()` / `validate_field_value()`
14. 明确 `status` 的唯一来源:是继续由 `change_project_status()` 推导,还是允许手动写入。
## 5. 推荐落地顺序
1. 先统一 `ProjectType``ProgressType``KeyConstraints``ProjectStatus` 的定义源。
2. 再补 schema 层校验,把 `project_type``progress_type``build_methods``key_constraints` 的裸字符串入口收紧。
3. 先补 `status=early` 和历史 `NULL` 清理,再把 `project_subtype``syxs` 迁到 `field_config`
## 6. 按本轮要求的改造落点
project 领域这轮要区分两类字段:
- 固定语义枚举:继续保留代码 enum
- 按项目类型变化的候选项:迁到 field config
### 6.1 初始化
Project 本身不直接批量初始化子表候选项,但 `project_type` 是整个初始化流程的路由键。
因此本轮在 project 领域的初始化要求是:
- `project_type` 继续保留代码 enum,作为 config 路由键
- `project_subtype``syxs` 如果进入初始化或默认值链路,应直接从 field config 读取
### 6.2 Create / Update 校验
这轮 project 领域要分开处理:
- `project_type``status``progress_type``key_constraints``build_methods`:继续按代码 enum 收紧
- `project_subtype``syxs`:通过新的 config 工具校验
也就是说,project 领域不是“全部配置化”,而是“固定 enum 和 field config 分流后分别收紧”。
### 6.3 下拉列表
当前 project 领域里:
- `project_type``progress_type``key_constraints``build_methods` 更适合继续由代码 enum 提供候选项
- `project_subtype``syxs` 后续如果提供前端下拉,应直接走 field config
本轮目标是明确边界,避免把本地固定枚举误迁到 config,也避免把按项目类型变化字段继续留成自由字符串。
### 6.4 清理项
project 领域本轮改造完成后,应清理:
- 重复的 enum 定义源
- 仍以裸字符串接收固定 enum 字段的入口
- `status` 历史 `NULL`
## 7. 测试关注点
project 领域 service 层集成测试至少覆盖:
1. `project_type``progress_type``key_constraints``build_methods` 的合法/非法值校验。
2. `project_subtype``syxs` 的 field config 读取与校验。
3. `status` 状态推导补齐 `early` 且不再产生 `NULL`
-1
View File
@@ -93,7 +93,6 @@
- endpoint 创建 handler
- endpoint 统一注入:
- `handler_config`
- `target_project_ids`
- datasource-specific runtime object(如 api client
最终让 `factory` 只负责:
+128
View File
@@ -0,0 +1,128 @@
# 快速上手
这份文档只服务第一次接手 `ecm_sync_system` 的开发者。目标不是完整讲架构,而是让你先跑起来、知道从哪里入手,再去看更细的说明。
## 1. 先做什么
第一次进入仓库时,建议按下面顺序执行:
1. 准备 Python 3.9+。
2. 在仓库根目录创建并激活虚拟环境。
3. 执行 `pip install -e .` 安装当前项目。
4. 跑状态机配置校验,确认基础工具链正常。
5. 跑一条轻量测试,确认测试与渲染工具链可用。
6. 再去看架构文档和状态机文档。
推荐命令:
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
python run.py --help
python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml
pytest tests/integration/test_toolchain_config_and_graph.py -q
```
## 2. 仓库里最重要的目录
- `sync_state_machine/`
- 真正的引擎实现目录。大多数“核心代码入口”都在这里。
- `run_profiles/`
- 运行配置与模板。
- `tests/`
- 单元测试与集成测试。
- `tools/`
- 状态机校验、图渲染、远程环境等工具。
- `scripts/`
- 自动化测试编排和数据集相关脚本。
- `docs/`
- 面向开发者的设计、运行和排障文档。
## 3. 先读哪几篇文档
按任务来看:
- 想知道系统整体怎么分层:`docs/architecture.md`
- 想知道状态机怎么工作:`docs/node_state_definition.md``docs/state_machine.md`
- 想知道一次同步是怎么跑的:`docs/sync_flow.md`
- 想调运行配置:`docs/runtime_config_guide.md`(包含单项目、多项目、`persist.url``sqlalchemy`、业务侧注册 datasource 的当前写法)
- 想接新 datasource 或 handler`sync_state_machine/datasource/README.md``docs/library_integration_guide.md`
- 想看测试体系:`docs/testing_guide.md``tests/README.md`
- 想做排障:`docs/operations.md`
## 4. 新人最容易踩的坑
### 4.1 先用仓库入口,不要默认 shell 里已经有命令
如果你还没执行过 `pip install -e .`,那么 `ecm-sync-run` 不一定可用。
首次调试建议优先用:
```bash
python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
```
安装完成后,再切换到:
```bash
ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
```
### 4.2 核心实现路径以 `sync_state_machine/` 为准
阅读历史文档时,如果看到类似下面的写法:
- `pipeline/full_sync_pipeline.py`
- `sync_system/strategy.py`
- `datasource/datasource.py`
- `engine/events.py`
请优先理解为它们在当前仓库中的真实位置分别是:
- `sync_state_machine/pipeline/full_sync_pipeline.py`
- `sync_state_machine/sync_system/strategy.py`
- `sync_state_machine/datasource/datasource.py`
- `sync_state_machine/engine/events.py`
## 5. 如果你要开始改代码
### 5.1 改 domain 行为
优先查看:
- `sync_state_machine/domain/<node_type>/`
- `sync_state_machine/sync_system/strategy_ops/`
- `tests/unit/` 中对应测试
### 5.2 改状态机语义
优先查看:
- `sync_state_machine/config/node_state_machine.yaml`
- `sync_state_machine/engine/events.py`
- `sync_state_machine/engine/state_apply.py`
改完建议至少执行:
```bash
python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml
pytest tests/integration/test_toolchain_config_and_graph.py -q
```
### 5.3 改运行配置或 datasource
优先查看:
- `docs/runtime_config_guide.md`
- `run_profiles/README.md`
- `sync_state_machine/datasource/README.md`
## 6. 出错先看哪里
推荐排查顺序:
1. 先看运行日志中的 `Resolved Full Config`
2. 再看节点 `sync_log / status / error`
3. 再对照 `docs/operations.md``docs/state_machine.md`
4. 如果是配置问题,先让校验报错,不要绕过 schema 或状态机校验。
+19 -7
View File
@@ -11,7 +11,7 @@
推荐接入方式:
- `sync_state_machine` 负责同步引擎、状态机、pipeline、datasource 抽象。
- 业务系统负责提供自己的数据源适配层,不要反向让 `sync_state_machine` 依赖业务系统。
- 业务系统负责提供自己的 datasource / handler 适配层,不要反向让 `sync_state_machine` 依赖业务系统。
---
@@ -46,13 +46,18 @@ ecm-sync-run --config_path=run_profiles/preset/jsonl_to_api.default.yaml
- `sync_state_machine.build_config`
- `sync_state_machine.build_default_config`
- `sync_state_machine.run_pipeline_from_config`
- `sync_state_machine.create_pipeline_from_config`
- `sync_state_machine.build_config_from_file`
- `sync_state_machine.PipelineRunConfig`
- `sync_state_machine.JsonlDataSourceConfig`
- `sync_state_machine.ApiDataSourceConfig`
- `sync_state_machine.DomainRegistry`
配置加载分两步:
1. 先构造或读取 `PipelineRunConfig`
2. 再交给 `create_pipeline_from_config()` 创建 pipeline
内置 domain 注册会在导入包时自动完成。
---
@@ -200,7 +205,7 @@ class DemoJsonlHandler(BaseJsonlHandler):
---
### 6.2 如果 backend 是数据库:建议写“数据库 handler / adapter,不要让引擎直接依赖 ORM
### 6.2 如果 backend 是数据库:建议写 handler / adapter,不要让引擎直接依赖 ORM
推荐结构:
@@ -234,6 +239,7 @@ backend/
- 不要在 `sync_state_machine` 包里 import backend 的 model/repository
- 不要让 `DomainRegistry` 注册逻辑散落在 backend 业务层之外
- 不要在 datasource 之外做额外的 project 过滤;项目范围应通过 `handler_configs.project.data_id_filter` 下沉到具体 handler 解释
---
@@ -268,6 +274,7 @@ backend/
- `strategy_class`
- `jsonl_handler_class`
- `api_handler_class`
- 以及任意通过 `register_handler()` 追加的 datasource handler
示例:
@@ -282,9 +289,11 @@ DomainRegistry.register(
jsonl_handler_class=DemoJsonlHandler,
api_handler_class=DemoApiHandler,
)
DomainRegistry.register_handler("demo", "custom_remote", DemoCustomHandler)
```
如果是 backend 接入,建议在 backend 的 integration 启动阶段集中注册,而不是分散在 controller / service 里。
如果是 backend 接入,建议在启动入口集中注册,而不是分散在 controller / service 里。
---
@@ -311,7 +320,9 @@ async def run_demo_sync() -> None:
"api_base_url": "https://example.com",
"api_uid": "xxx",
"api_secret": "xxx",
"target_project_ids": ["demo-project"],
"handler_configs": {
"project": {"data_id_filter": ["demo-project"]}
},
},
}
@@ -320,7 +331,7 @@ async def run_demo_sync() -> None:
await pipeline.run()
```
如果 backend 自己提供本地数据库数据,不要把 backend model 直接灌进 pipeline;先经过 handler / mapper 转成 schema 再进入引擎。
如果 backend 自己提供本地数据库数据,不要把 backend model 直接灌进 pipeline;先经过 handler / mapper 转成 schema 再进入引擎。
---
@@ -353,8 +364,9 @@ async def run_demo_sync() -> None:
## 11. 相关文件
- 包入口:[sync_state_machine/__init__.py](../sync_state_machine/__init__.py)
- CLI[sync_state_machine/cli.py](../sync_state_machine/cli.py)
- CLI / 运行入口[sync_state_machine/cli.py](../sync_state_machine/cli.py)
- 配置工厂:[sync_state_machine/config](../sync_state_machine/config)
- datasource 基础层:[sync_state_machine/datasource](../sync_state_machine/datasource)
- pipeline 工厂:[sync_state_machine/pipeline/factory.py](../sync_state_machine/pipeline/factory.py)
- domain 注册:[sync_state_machine/common/registry.py](../sync_state_machine/common/registry.py)
- backend 集成入口:[backend/app/thirdparty/ecm_sync/services/sync_runner.py](../backend/app/thirdparty/ecm_sync/services/sync_runner.py)
+627
View File
@@ -0,0 +1,627 @@
# 多项目同步重构方案
## 1. 背景和目标
### 背景
当前 `multi_project_pipeline` 的主要问题:
1. 顶层概念过重,在单项目同步之上又叠了一层特殊编排。
2. 通过共享快照传递 `collection` / `binding` / 运行态,边界不清晰。
3. 配置入口过多,存在 `global_config` / `default_project_config` / `config_path` 多层模型。
4. 项目完成后哪些对象可以释放,没有明确生命周期。
### 目标
新方案只解决四件事:
1. 顶层仍然是 pipeline,但直接写新的 `ProjectBatchPipeline`
2. `global` 数据只同步一次,不复制到每个项目,不在每个项目中重复校验。
3. 改动尽量限制在 `persistence / collection / binding_manager / datasource / pipeline` 这些基础层。
4. 项目级运行态支持 `load -> sync -> persist -> unload -> close`
### 一个明确结论
这里不建议先做复杂的 session 框架。
第一阶段最合适的方式是:
1.`sync_project(...)` 里创建一个很薄的 `ProjectScopeRuntime`
2. 它只是把“当前项目 scope 下的一组对象”收拢起来。
3. 如果你临时不想单独建类,也可以先写成函数内局部变量。
但从清理顺序和可读性考虑,推荐保留 `ProjectScopeRuntime` 这个薄对象。
它不是复杂 session 系统,本质上就是“单个项目的一次运行上下文”。
## 2. 新的 Batch Project Pipeline 和伪代码
### 结构结论
顶层对象建议是:`ProjectBatchPipeline`
它负责:
1. 跑一次 global phase。
2. 并发调度所有项目。
3. 管理 batch 级共享对象。
4. 管理项目级对象的创建和清理。
它不负责:
1. 业务字段映射。
2. handler 细节。
3. strategy 逻辑。
### 共享对象和项目级对象
#### 整个 batch 共享的对象
1. `BatchConfig`
2. `StateMachineRuntime`
3. `Logger`
4. `PersistenceService`
5. `global` 只读运行态
6. datasource 的共享静态资源,例如 API client、认证信息、限流器、handler registry
7. semaphore 和任务调度器
#### 每个项目内创建的对象
1. `ProjectScopeRuntime`
2. `local_collection`
3. `remote_collection`
4. `binding_manager`
5. 当前项目 scope 对应的 persistence view
6. 当前项目的 local datasource runtime
7. 当前项目的 remote datasource runtime
8. 当前项目统计对象
### Global 数据处理原则
这点直接定死:
1. `global` 数据不复制到每个 project scope。
2. `global` 数据不在每个项目里重复 bootstrap。
3. `global` 数据不在每个项目里重复 post-check。
项目运行时如果需要共享节点或绑定:
1. 先查当前 project scope。
2. 未命中再回退到 `global` 只读运行态。
### 伪代码
```python
class ProjectBatchPipeline:
def __init__(self, config: BatchConfig):
self.config = config
self.runtime = build_state_machine_runtime()
self.logger = build_logger(config.logging)
self.persistence = PersistenceService(config.persist)
self.global_runtime = None
self.shared_local_ds = build_shared_local_datasource_resources(config.local_datasource)
self.shared_remote_ds = build_shared_remote_datasource_resources(config.remote_datasource)
async def run(self) -> None:
self.global_runtime = await self.sync_global()
await self.dispatch_projects()
async def sync_global(self) -> "GlobalReadonlyRuntime":
...
async def dispatch_projects(self) -> None:
...
async def sync_project(self, local_project_id: str, remote_project_id: str) -> None:
scope_runtime = await self.create_project_scope_runtime(local_project_id, remote_project_id)
try:
await scope_runtime.load()
await run_project_sync(scope_runtime, self.config.project_node_types)
await scope_runtime.persist()
finally:
await scope_runtime.unload()
await scope_runtime.close()
```
```python
class ProjectScopeRuntime:
def __init__(self, scope: str, persistence_view, global_runtime, shared_local_ds, shared_remote_ds):
self.scope = scope
self.persistence_view = persistence_view
self.local_collection = DataCollection(...)
self.remote_collection = DataCollection(...)
self.binding_manager = BindingManager(...)
self.local_datasource = build_local_datasource_runtime(shared_local_ds, self.local_collection)
self.remote_datasource = build_remote_datasource_runtime(shared_remote_ds, self.remote_collection)
self.global_runtime = global_runtime
async def load(self) -> None: ...
async def persist(self) -> None: ...
async def unload(self) -> None: ...
async def close(self) -> None: ...
```
## 3. 组件说明
### 3.1 Persistence
#### 角色
`PersistenceService` 是 batch 共享对象。
它共享的是:
1. 后端类型。
2. DB 路径或连接串。
3. 连接池或底层连接能力。
它不共享的是:
1. 当前项目的内存缓存。
2. 当前 scope 的脏状态。
#### 关键语义
建议把 persistence 理解成两层:
1. `PersistenceService`
- 共享后端服务。
2. `ScopedPersistenceView`
- 绑定某个 `scope` 的轻量 view。
所以“scope 属于 persistence 的 scope view”的意思是:
1. 整个 batch 共享一个 persistence service。
2. 每个项目拿一个绑定了当前 `scope` 的轻量操作视图。
不是说每个项目都新建一套 persistence backend。
#### 主要方法伪代码
```python
class PersistenceService:
async def initialize(self) -> None: ...
def view(self, scope: str) -> ScopedPersistenceView: ...
async def close(self) -> None: ...
class ScopedPersistenceView:
async def load_nodes(self, collection_id: str) -> list[dict]: ...
async def save_nodes(self, collection_id: str, nodes: list[SyncNode]) -> None: ...
async def delete_nodes(self, collection_id: str, node_ids: list[str]) -> None: ...
async def load_bindings(self, node_type: str) -> list[dict]: ...
async def save_bindings(self, node_type: str, bindings: dict[str, str | None]) -> None: ...
```
### 3.2 Collection
#### 角色
`Collection` 必须是 scope 级对象,不能做单例。
原因:
1. 它内部天然持有当前 scope 的节点集合。
2. 它持有 data_id 索引。
3. 它持有删除集合和脏状态。
#### 关键语义
project scope 的 `Collection` 只存项目域数据。
对于共享数据,不复制 global,而是支持只读回退:
1. 先查当前 scope。
2. 未命中再查 global 只读 collection。
#### 主要方法伪代码
```python
class DataCollection:
async def load_from_persistence(self) -> None: ...
async def persist(self) -> None: ...
async def unload(self) -> None: ...
def get(self, node_id: str) -> SyncNode | None: ...
def get_by_data_id(self, node_type: str, data_id: str) -> SyncNode | None:
node = self._get_local(node_type, data_id)
if node is not None:
return node
return self._get_global_fallback(node_type, data_id)
```
### 3.3 BindingManager
#### 角色
`BindingManager` 也是 scope 级对象,不应共享。
它负责:
1. 当前 scope 的绑定运行态。
2. 当前 scope 的绑定持久化。
3. 当前 scope 未命中时对 global 只读绑定做回退查询。
#### 主要方法伪代码
```python
class BindingManager:
async def load_from_persistence(self, node_type: str) -> None: ...
async def persist(self) -> None: ...
async def unload(self) -> None: ...
async def get_remote_id(self, node_type: str, local_id: str) -> str | None:
remote_id = self._get_local_remote_id(node_type, local_id)
if remote_id is not None:
return remote_id
return await self._get_global_remote_id(node_type, local_id)
```
### 3.4 DataSource
#### 角色
`local_datasource / remote_datasource / logging` 不应按项目变化,这一点成立。
但按当前实现,`BaseDataSource` 不适合直接做单例,因为它内部持有:
1. `_collection`
2. `_stats`
3. `_action_summary`
4. `_action_by_node_id`
#### 推荐的最小改法
第一阶段不强行拆成两个正式类,而是采用下面的语义:
1. datasource 的静态配置和共享 client 是 batch 级共享的。
2. datasource 的运行态对象仍然是项目级创建的。
也就是说:
1. 配置不按项目变化。
2. runtime 仍然按项目隔离。
这样改动范围最小,也更不容易污染现有 handler 行为。
#### 主要方法伪代码
```python
class ScopedDataSourceRuntime(BaseDataSource):
def set_collection(self, collection: DataCollection) -> None: ...
async def load_all(self, order: list[str]) -> None: ...
async def sync_all(self, order: list[str]) -> None: ...
def reset_runtime_state(self) -> None: ...
async def close(self) -> None: ...
```
## 4. 改进路线图
### 第一阶段:先收敛架构边界
只改基础层,不动业务层:
1. 新建 `ProjectBatchPipeline`
2. 明确 `global``project_id:<local_project_id>` 语义。
3. 引入 `ProjectScopeRuntime` 这个薄的项目运行上下文。
4.`Collection``BindingManager` 增加 global fallback 语义。
5. persistence 侧增加共享 service + scope view 语义。
### 第二阶段:清理生命周期
把项目级清理流程固定下来:
1. `load`
2. `sync`
3. `persist`
4. `unload`
5. `close`
### 第三阶段:再决定 datasource 是否正式拆层
如果第一阶段跑通,再决定是否把 datasource 正式拆成:
1. 共享静态层。
2. scope runtime 层。
这一步不是第一批必须完成。
### 最终落点
最终希望达到的状态:
1. global 数据只同步一次,只读共享。
2. project 数据按 scope 独立运行。
3. 项目跑完即可卸载运行态。
4. 大部分改动都限制在基础设施层。
8. 当前项目统计对象
### 2.3 为什么推荐 `ProjectScopeRuntime`
你问的是:要不要做 session 对象,还是做子 pipeline,还是都放函数里。
建议是:
1. 不做新的复杂 session 框架。
2. 不做 `FullSyncPipeline` 风格的子 pipeline 继承树。
3. 做一个很薄的 `ProjectScopeRuntime`
它的作用只有两个:
1. 把当前项目的一组 scope 对象收拢到一起。
2. 让清理顺序明确,不要把资源散在 `sync_project(...)` 的局部变量里。
如果你坚持更简单,也可以第一版直接写在 `sync_project(...)` 里;但文档层面仍建议用 `ProjectScopeRuntime` 表达这个层次。
### 2.4 伪代码
```python
class ProjectBatchPipeline:
def __init__(self, config: BatchConfig):
self.config = config
self.runtime = build_state_machine_runtime()
self.logger = build_logger(config.logging)
self.persistence = PersistenceService(config.persist)
self.global_runtime = None
self.shared_local_ds = build_shared_local_datasource_resources(config.local_datasource)
self.shared_remote_ds = build_shared_remote_datasource_resources(config.remote_datasource)
async def run(self) -> None:
self.global_runtime = await self.sync_global()
await self.dispatch_projects()
async def sync_global(self) -> "GlobalReadonlyRuntime":
# 只同步 shared_node_types
...
async def dispatch_projects(self) -> None:
# semaphore 控制并发
...
async def sync_project(self, local_project_id: str, remote_project_id: str) -> None:
scope_runtime = await self.create_project_scope_runtime(local_project_id, remote_project_id)
try:
await scope_runtime.load()
await run_project_sync(scope_runtime, self.config.project_node_types)
await scope_runtime.persist()
finally:
await scope_runtime.unload()
await scope_runtime.close()
```
```python
class ProjectScopeRuntime:
def __init__(self, scope: str, persistence_view, global_runtime, shared_local_ds, shared_remote_ds):
self.scope = scope
self.persistence_view = persistence_view
self.local_collection = DataCollection(...)
self.remote_collection = DataCollection(...)
self.binding_manager = BindingManager(...)
self.local_datasource = build_local_datasource_runtime(shared_local_ds, self.local_collection)
self.remote_datasource = build_remote_datasource_runtime(shared_remote_ds, self.remote_collection)
self.global_runtime = global_runtime
async def load(self) -> None: ...
async def persist(self) -> None: ...
async def unload(self) -> None: ...
async def close(self) -> None: ...
```
### 2.5 Global 数据的处理原则
这点现在可以直接定死:
1. `global` 数据不复制到每个 project scope。
2. `global` 数据不在每个项目里重复 bootstrap。
3. `global` 数据不在每个项目里重复 post-check。
项目级运行时如果需要共享节点或绑定,采用:
1. 当前 project scope 先查本地运行态。
2. 未命中时回退到 `global` 只读运行态。
这样可以避免复制大体量 global 数据。
2. `preload_shared_state(...)` 把项目 pipeline 变成了“半冷启动、半热注入”的特殊运行态。
3. `ephemeral_node_types` / `bootstrap_binding_node_types` 这些参数开始承担原本不属于它们的职责。
## 3. 组件说明
### 3.1 Persistence
#### 角色
`PersistenceService` 是 batch 共享对象。
它共享的是:
1. 后端类型。
2. DB 路径或连接串。
3. 连接池或底层连接能力。
它不共享的是:
1. 当前项目的内存缓存。
2. 当前 scope 的脏状态。
#### 关键语义
建议把 `persistence` 理解成两层:
1. `PersistenceService`
- 共享后端服务。
2. `ScopedPersistenceView`
- 绑定某个 `scope` 的轻量 view。
所以“scope 属于 persistence 的 scope view”这句话的意思是:
1. 整个 batch 共享一个 persistence service。
2. 每个项目拿一个绑定了当前 `scope` 的轻量操作视图。
不是说每个项目都新建一套 persistence backend。
#### 主要方法伪代码
```python
class PersistenceService:
async def initialize(self) -> None: ...
def view(self, scope: str) -> ScopedPersistenceView: ...
async def close(self) -> None: ...
class ScopedPersistenceView:
async def load_nodes(self, collection_id: str) -> list[dict]: ...
async def save_nodes(self, collection_id: str, nodes: list[SyncNode]) -> None: ...
async def delete_nodes(self, collection_id: str, node_ids: list[str]) -> None: ...
async def load_bindings(self, node_type: str) -> list[dict]: ...
async def save_bindings(self, node_type: str, bindings: dict[str, str | None]) -> None: ...
```
### 3.2 Collection
#### 角色
`Collection` 必须是 scope 级对象,不能做单例。
原因很简单:
1. 它内部天然持有当前 scope 的节点集合。
2. 它持有 data_id 索引。
3. 它持有删除集合和脏状态。
#### 关键语义
project scope 的 `Collection` 只存项目域数据。
对于共享数据,不复制 global,而是支持只读回退:
1. 先查当前 scope。
2. 未命中再查 global 只读 collection。
#### 主要方法伪代码
```python
class DataCollection:
async def load_from_persistence(self) -> None: ...
async def persist(self) -> None: ...
async def unload(self) -> None: ...
def get(self, node_id: str) -> SyncNode | None: ...
def get_by_data_id(self, node_type: str, data_id: str) -> SyncNode | None:
node = self._get_local(node_type, data_id)
if node is not None:
return node
return self._get_global_fallback(node_type, data_id)
```
### 3.3 BindingManager
#### 角色
`BindingManager` 也是 scope 级对象,不应共享。
它负责:
1. 当前 scope 的绑定运行态。
2. 当前 scope 的绑定持久化。
3. 当前 scope 未命中时对 global 只读绑定做回退查询。
#### 主要方法伪代码
```python
class BindingManager:
async def load_from_persistence(self, node_type: str) -> None: ...
async def persist(self) -> None: ...
async def unload(self) -> None: ...
async def get_remote_id(self, node_type: str, local_id: str) -> str | None:
remote_id = self._get_local_remote_id(node_type, local_id)
if remote_id is not None:
return remote_id
return await self._get_global_remote_id(node_type, local_id)
```
### 3.4 DataSource
#### 角色
`local_datasource / remote_datasource / logging` 不应按项目变化,这一点是对的。
但按当前实现,`BaseDataSource` 也不适合直接做单例,因为它内部持有:
1. `_collection`
2. `_stats`
3. `_action_summary`
4. `_action_by_node_id`
所以第一阶段不建议“直接把当前 `BaseDataSource` 改成单例 + 注入 scope”。
#### 推荐的最小改法
第一阶段先不强行拆成两个正式类,而是采用下面的语义:
1. datasource 的静态配置和共享 client 是 batch 级共享的。
2. datasource 的运行态对象仍然是项目级创建的。
也就是说:
1. 配置不按项目变化。
2. runtime 仍然按项目隔离。
这样改动范围最小,也更不容易污染现有 handler 行为。
#### 主要方法伪代码
```python
class ScopedDataSourceRuntime(BaseDataSource):
def set_collection(self, collection: DataCollection) -> None: ...
async def load_all(self, order: list[str]) -> None: ...
async def sync_all(self, order: list[str]) -> None: ...
def reset_runtime_state(self) -> None: ...
async def close(self) -> None: ...
```
当前多项目配置要求:
1. `global_config`
### 第二阶段:清理生命周期
把项目级清理流程固定下来:
1. `load`
2. `sync`
3. `persist`
4. `unload`
5. `close`
### 第三阶段:再决定 datasource 是否正式拆层
如果第一阶段跑通,可以再决定是否把 datasource 正式拆成:
1. 共享静态层。
2. scope runtime 层。
这一步不是必须第一批完成。
### 最终落点
最终希望达到的状态是:
1. global 数据只同步一次,只读共享。
2. project 数据按 scope 独立运行。
3. 项目跑完即可卸载运行态。
4. 大部分改动都限制在基础设施层。
1. 全局阶段一份配置。
2. 项目阶段一份默认配置。
3. 每个项目还能再覆盖一份配置。
这对于少量项目还能接受,但项目数到了几百个就不可维护了。
如果每个项目都允许自带一份 pipeline 配置,系统就失去“批量同步”的基本前提:
1. 无法判断哪些差异是业务差异,哪些是配置漂移。
2. 无法稳定复用并发调度逻辑。
3. 无法保证结果可复现。
结论是:多项目批量同步不应允许“每项目一套 pipeline 定义”。
+9
View File
@@ -37,6 +37,10 @@
| `ABNORMAL` | 数据损坏或核心绑定异常 | 否 |
| `DEPENDENCY_ERROR` | 依赖不可满足 | 否 |
bootstrap 特殊语义:
- 如果节点在持久化恢复后存在绑定,但 bootstrap datasource refresh 未返回最新数据,则节点进入 `S17` 语义并被 `bootstrap_blocked` 隔离。
- 如果节点仅残留在 persistence 中、没有绑定,且 datasource refresh 也未返回,则节点按 stale zombie 清理,不再进入 bind。
## 3. create / update 语义
### create
@@ -64,6 +68,11 @@
- 状态落地:`Decision.to_state` 对应 YAML `states[Sxx]`
- 不变量:每次落地后执行 `runtime.check_invariants()`
bootstrap 额外输入:
- `is_create_zombie`
- `bootstrap_source_present`
- `has_binding_record`
## 6. 关键一致性约束
- `UNCHECKED` 不允许带动作(`INV-1_UNCHECKED_ACTION_NONE`)。
+68
View File
@@ -0,0 +1,68 @@
# ECM Sync Quality Cleanup Execution Plan
## Goal
This cleanup pass keeps behavior unchanged and prioritizes deleting, merging, and unifying duplicated code on the single-pipeline path.
Acceptance remains unchanged:
- `backend/local_tests/sync_system_tests/depm_to_ecm_bootstrap_pull_then_project_push.pipeline.yaml` must still run successfully.
- Local Summary and Remote Summary must keep the same structure and stable output shape.
## Constraints
- Do not modify `schemas/`.
- Do not modify `filtered_datasource/`.
- Do not bypass schema validation.
- Do not add fallback logic that changes behavior.
- Prefer helper extraction, deletion, and merge over new abstraction layers.
## Execution Order
### Batch 1: Entry + Factory
Status: completed
- Turn `run.py` into a thin wrapper around `sync_state_machine.cli.main`.
- Merge duplicated datasource lifecycle and resolved-config logging in `sync_state_machine/pipeline/factory.py`.
- Verify with focused unit tests and the real DEPM to ECM profile.
### Batch 2: Pipeline Flow + Backend Runner
Status: completed
- Reduce repeated phase-completion, strategy-loop, reload-skip, and close-handling code in `sync_state_machine/pipeline/full_sync_pipeline.py`.
- Reduce repeated orchestration code in `backend/app/thirdparty/ecm_sync/scripts/depm_to_ecm/run_pipeline.py` without changing CLI behavior or fail-fast rules.
- Keep log text stable where practical.
### Batch 3: Datasource + Handler Base Cleanup
Status: completed
- Reduce duplicated handler registration and context injection between datasource base classes and subclasses.
- Reduce duplicated node metadata initialization across handler subclasses by pushing safe defaults into handler base classes.
- Delete domain-level constructor boilerplate where subclasses only repeat node type, node class, schema, or no-op `_create_node` wrappers.
### Batch 4: Domain Repeat Cleanup + Docs
Status: in progress
- Inspect domain handlers for repeated load/update helper logic and merge only stable repeated fragments into shared helpers.
- Update docs so they only describe the real maintained execution path.
- Keep backend runner documentation aligned with current script behavior.
- Remove stale references only after code paths are confirmed stable.
## Validation Plan
1. Run targeted `ecm_sync_system` unit tests for the touched pipeline code.
2. Run the real backend profile with the backend interpreter.
3. Confirm the output still ends with pipeline completion plus Local Summary and Remote Summary tables.
## Current Focus
Current implementation focus is Batch 4.
- repeated load/update helpers under `sync_state_machine/domain/`
- maintained execution-path docs under `docs/`
The intent is code reduction only, not behavior redesign.
+469 -104
View File
@@ -1,143 +1,508 @@
# 运行配置业务说明
# 运行配置写法说明
本文档从业务视角说明配置字段含义、对阶段行为的影响,以及推荐的覆盖方式
这份文档只讲一件事:现在 `ecm_sync_system` 的配置文件应该怎么写
## 1. 推荐使用方式(先 default,再覆盖)
重点不是把所有默认值抄一遍,而是说明当前真实生效的写法、哪些字段是内置能力、哪些字段来自业务侧扩展,以及单项目和多项目文件各自长什么样。
1. 先使用基线配置运行:
- `run_profiles/preset/jsonl_to_jsonl.default.yaml`
2. 若需项目定制,再复制为:
- `run_profiles/jsonl_to_jsonl.local.yaml`
3. 仅改有业务差异的字段,避免一次改太多导致排障困难。
## 1. 先分清两类配置文件
占位符
- `${PROJECT_ROOT}` 会在运行时替换为项目根目录绝对路径。
当前有两类 YAML
## 2. 顶层字段(你在改什么)
1. 单 pipeline 配置
- 对应 `PipelineRunConfig`
- 用来描述一条完整同步链路
- 顶层通常包含:`node_types``local_datasource``remote_datasource``persist``logging``strategies`
2. 多项目编排配置
- 对应 `MultiProjectRunConfig`
- 只负责描述“先跑全局,再按项目并发跑哪些项目”
- 顶层包含:`global_config``default_project_config``max_concurrency``project_bootstrap_node_types``projects`
判断规则很简单:
- 如果文件里有 `global_config``default_project_config``projects`,它就是多项目编排配置。
- 否则按单 pipeline 配置解析。
## 2. 配置加载约定
当前约定如下:
- 跟踪在仓库里的模板和示例,只写必要覆盖项,不重复展开默认值。
- pipeline 启动后会打印一份 `Resolved Full Config`,它才是最终生效的完整配置。
- 排障优先看 `Resolved Full Config`,不要只盯着输入 YAML。
## 3. 占位符规则
`ecm_sync_system` 核心库内置只认识一个占位符:
- `${PROJECT_ROOT}`:运行时替换为 `ecm_sync_system` 仓库根目录绝对路径。
例如:
```yaml
persist:
backend: sqlite
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
```
需要特别区分的是:
- `${BACKEND_ROOT}``${ECM_SYNC_LOG_DIR}` 这类占位符不是核心库通用语义。
- 它们通常是 backend 集成脚本在读取 profile 时额外替换的调用方语义。
- 如果你不是通过 backend 的联调脚本运行,而是直接在 `ecm_sync_system` 内跑 profile,就不要假设这些占位符一定可用。
## 4. 单 pipeline 配置的当前写法
### 4.1 最小骨架
```yaml
node_types:
- company
- supplier
- project
local_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
remote_datasource:
type: api
api_base_url: http://127.0.0.1:8000/api/third/v2
api_uid: your_uid
api_secret: your_secret
persist:
backend: sqlite
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
logging:
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
file_name_pattern: demo_{timestamp}.log
strategies:
project:
config:
auto_bind: true
auto_bind_fields:
- name
local_orphan_action: none
remote_orphan_action: none
update_direction: push
```
### 4.2 顶层字段含义
- `node_types`
- 本轮参与同步的业务类型列表。
- 顺序会影响依赖链执行时机(当前按 node_type 串行执行 bind/create/update
- `target_project_ids`
- 兼容字段。作为两侧 datasource 的项目白名单兜底值。
- 若 datasource 自身配置了 `target_project_ids`,则优先使用 datasource 级配置。
- 本轮参与同步的节点类型列表。
- 顺序会影响执行顺序,依赖链通常要把上游类型放前面
- `local_datasource` / `remote_datasource`
- 分别定义两侧数据来源与执行端
- `strategies`
- 每个业务类型的行为策略(依赖、自动绑定、创建方向、更新方向、跳过等)。
- 定义两侧数据源类型和对应参数
- `persist`
- 持久化开关与数据库位置。
- 定义持久化后端及目标位置。
- `logging`
- 运行日志输出位置级别。
- 定义日志初始化、落盘位置级别。
- `strategies`
- 定义每个 `node_type` 的同步策略。
## 3. 数据源字段(DataSource
## 5. datasource 字段怎么写
### 3.1 JSONL 数据源
### 5.1 内置 datasource type
- `type=jsonl`:使用本地 JSONL 文件作为数据源。
- `jsonl_dir`JSONL 目录路径。
- `read_only=false`:允许该侧执行 create/update 回写;若为 true,通常用于只读加载。
- `target_project_ids=[]`:该侧项目白名单;为空时不进行项目筛选。
- `handler_configs`:预留给各业务 handler 的扩展参数。
核心库内置注册了两种 datasource type
### 3.2 API 数据源
1. `jsonl`
2. `api`
- `type=api`:对接 HTTP API。
- `api_base_url` / `api_uid` / `api_secret`:接口访问参数。
- `api_debug`:开启后输出更多调试信息。
- `poll_max_retries` / `poll_interval`:异步任务轮询次数与间隔。
- `target_project_ids`:**必填**,该侧项目白名单;至少需要指定一个项目ID,以避免加载过多远程数据。
- `handler_configs`:业务 handler 扩展参数。
#### `jsonl` 写法
## 4. 策略字段(Strategy
```yaml
local_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
read_only: false
handler_configs:
project:
data_id_filter:
- project-id-1
```
每个 node_type 都建议显式写出以下字段,便于审阅与排障。
字段说明:
### 4.1 `update_direction`
- `jsonl_dir`JSONL 数据目录。
- `read_only`:是否只读。
- `handler_configs`:按 `node_type` 下发给 handler 的扩展参数。
- `push`:本地变更推送到远端。
- `pull`:远端变更拉取到本地。
- `bidirectional`:双向更新(需要明确冲突策略,谨慎使用)。
- `none`:关闭 update 阶段。
#### `api` 写法
### 4.2 `local_orphan_action` / `remote_orphan_action`
```yaml
remote_datasource:
type: api
api_base_url: http://127.0.0.1:8000/api/third/v2
api_uid: your_uid
api_secret: your_secret
api_debug: true
poll_max_retries: 10
poll_interval: 0.5
api_rate_limit_max_requests: 30
api_rate_limit_window_seconds: 10.0
handler_configs:
project:
data_id_filter:
- remote-project-id-1
```
- `create_remote`:本地孤儿创建到远端。
- `create_local`:远端孤儿创建到本地。
- `none`:不自动创建。
- `delete_local` / `delete_remote`:当前主流程未作为默认路径,谨慎使用。
字段说明:
### 4.3 `auto_bind` / `auto_bind_fields` / `pre_bind_data_id`
- `api_debug`:开启后输出更详细的请求日志。
- `poll_max_retries` / `poll_interval`:异步任务轮询控制。
- `api_rate_limit_max_requests` / `api_rate_limit_window_seconds`:全局限流窗口。
- `handler_configs`:同样按 `node_type` 下发业务参数。
- `auto_bind=true`:在 `S02(MISSING)` 上尝试自动匹配。
- `auto_bind=false`:不会做自动匹配,也不会走自动创建识别路径;通常会进入阻断态等待人工处理。
- `auto_bind_fields`:自动匹配使用的业务键字段。
- `pre_bind_data_id`:显式 data_id 预绑定清单(通用能力),格式为:
- `- local_data_id: "..."`
` remote_data_id: "..."`
- 规则:若两侧数据源都存在这对 data_id 对应节点,且当前无绑定记录,则直接绑定。
- 不依赖 `auto_bind/auto_bind_fields`,即使 `auto_bind=false` 也会执行。
- 不做业务键一对一匹配检查,按你给定的映射对优先执行。
### 5.2 业务侧注册的 datasource type
### 4.4 `depend_fields`
核心库允许业务系统在启动阶段注册自己的 datasource type。
- 定义依赖字段到依赖 node_type 的映射,例如:`company_id -> company`
- 依赖节点状态非 `NORMAL` 时,会在 bind 的依赖检查阶段进入 `DEPENDENCY_ERROR`
因此,配置里不只可以写 `jsonl``api`,还可以写业务侧注册出来的类型,例如 backend 里的 `depm_db`
### 4.5 `skip_sync`
当前 backend 的 `depm_db` 真实写法是:
- 运行配置中已不建议使用 `skip_sync`
- 推荐使用显式组合:
- `local_orphan_action: none`
- `remote_orphan_action: none`
- `update_direction: none`
- 当三者同时为 `none/none/none` 时,pipeline 会跳过该 node_type 的 bind/create/update。
```yaml
local_datasource:
type: depm_db
tenant_id: "019976df-0628-7c46-a896-81d3f290a7df"
app:
config_file: ${BACKEND_ROOT}/config.yaml
overrides:
database:
url: mysql+aiomysql://root:123456@127.0.0.1/depm_prod_local_auto_test?charset=utf8mb4
handler_configs:
project:
data_id_filter:
- 019a62a5-2323-7be2-99c8-82534f8befa4
```
### 4.6 Handler 级参数(`handler_configs`
这个例子说明两点:
- 建议将 handler 特有参数放在 datasource 的 `handler_configs.<node_type>`
- 示例:
- `remote_datasource.handler_configs.supplier.load_mode: persisted_only`
- `persisted_only` 表示只使用已持久化数据,不主动从外部拉新数据(具体以对应 handler 实现为准)。
1. datasource 的可写字段由对应注册模型决定,不是所有 type 都共用同一套字段
2. 如果 `type` 没有先被注册,配置校验会直接报错,不会做降级处理。
## 5. 阶段行为与配置关系
`depm_db` 这类业务扩展字段需要以业务侧实现为准。对 backend 当前实现来说:
- bind 阶段
- `depend_fields` 决定是否做依赖校验
- `pre_bind_data_id` 先按显式 local/remote data_id 映射做预绑定
- `auto_bind/auto_bind_fields` 决定是否继续进行业务键自动绑定。
- create 阶段
- `local_orphan_action` / `remote_orphan_action` 决定孤儿是否创建。
- update 阶段
- `update_direction` 决定是否和向哪侧更新。
- 写入后 reloadPipeline 内置)
- 当某个 node_type 的 create 或 update 发生实际写入成功后,pipeline 会自动对该 node_type 重新执行两侧 `load`
- 目的:让后续 update 判断基于最新远端/本地数据,支持“create 后再补一次 update”的链路。
- 同步后一致性检查(Pipeline 内置)
- 每个 node_type 在本轮同步后会基于绑定对执行数据一致性校验(比较时忽略 `id/_id` 字段)。
- 若发现差异,会在日志中打印差异样本,并在汇总表新增 `Consistent` 列展示 `Y` / `N(差异数)`
- 全局跳过
- `local_orphan_action=none` + `remote_orphan_action=none` + `update_direction=none`
可直接跳过该类型的 bind/create/update。
- `tenant_id`DEPM 本地 datasource 的租户过滤 ID。
- `app.config_file`:backend 配置文件路径,默认可落回 `backend/config.yaml`
- `app.overrides`:直接传给 backend `Settings()` 的覆盖项
## 6. 覆盖配置建议
## 6. strategy 写法
- 先在 default 跑通一轮,再做 local 覆盖。
- 每次只改一类问题相关字段(例如只改 `project` 的依赖与自动绑定)。
- 覆盖后优先看 `sync_log` 是否出现预期状态迁移。
### 6.1 当前推荐写法
## 7. 持久化与日志字段
`strategies` 现在推荐写成“按 `node_type` 为键”的映射,而不是手工写一长串列表。
### `persist`
```yaml
strategies:
supplier:
skip_post_check: true
config:
auto_bind: true
auto_bind_fields:
- credit_code
depend_fields: {}
local_orphan_action: none
remote_orphan_action: create_local
update_direction: none
project:
domain_option:
pull_group_plan_production_date: true
config:
auto_bind: false
auto_bind_fields: []
pre_bind_data_id:
- local_data_id: local-project-id
remote_data_id: remote-project-id
depend_fields:
company_id: company
local_orphan_action: none
remote_orphan_action: none
update_direction: push
```
- `enable`:是否开启持久化。
- `db_path`SQLite 路径。
- `wipe_on_start`:启动是否清理历史持久化。
### 6.2 常用字段
### `logging`
- `auto_bind`
- 是否开启自动绑定。
- `auto_bind_fields`
- 自动绑定使用的业务键字段。
- `pre_bind_data_id`
- 显式 local/remote data_id 绑定对。
- `depend_fields`
- 依赖字段到依赖 `node_type` 的映射。
- `local_orphan_action` / `remote_orphan_action`
- 孤儿节点创建策略。
- `update_direction`
- `push` / `pull` / `none`
- `post_check_ignore_fields`
- 只影响最终一致性检查,不影响 create/update 差异检测。
- `compare_ignore_fields`
- schema diff / validate 时忽略的顶层字段。
- `skip_sync`
- 只跳过 create/update,不跳过 load/bind。
- `skip_post_check`
- 跳过该类型的 post-check reload 与一致性校验。
- `domain_option`
- 领域策略自己的扩展参数,由对应 domain 自己解释。
- `file_dir`:日志目录。
- `file_name_pattern`:文件命名模板。
- `level`:日志级别。
### 6.3 `preset` 仍然可用
排障建议:运行失败时优先看日志中的 `状态机流转``reason=`,再对照 `docs/state_machine.md``docs/node_state_definition.md` 理解含义。
如果只是想套一组预设策略,可以这样写:
```yaml
strategies:
project:
preset: pull_only
```
当前内置预设包括:
- `first_sync`
- `daily_sync`
- `repair_sync`
- `pull_only`
- `push_only`
### 6.4 已废弃的旧字段不要再写
以下 legacy key 已明确拒绝:
- `force_sync`
- `load_mode`
如果你在 strategy 里写这两个字段,配置会直接报错。
当前替代方式是:
- 是否跳过同步,用 `skip_sync`
- handler 特有加载参数,写到 `datasource.handler_configs.<node_type>`
## 7. persist 的当前写法
### 7.1 校验规则
当前 `persist` 配置遵循下面的规则:
1. `db_path``url` 至少要有一个。
2.`backend != sqlite` 时,必须显式提供 `url`
3. `wipe_on_start=true` 只会尝试删除可映射到文件路径的持久化文件;对 MySQL 这类远程数据库不会帮你删库。
### 7.2 SQLite 文件持久化
```yaml
persist:
backend: sqlite
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/simple_pipeline_sm.db
enable: true
wipe_on_start: false
```
### 7.3 SQLAlchemy 持久化到 SQLite URL
```yaml
persist:
backend: sqlalchemy
url: sqlite+aiosqlite:////abs/path/to/pipeline.db
enable: true
wipe_on_start: false
```
### 7.4 SQLAlchemy 持久化到 MySQL
```yaml
persist:
backend: sqlalchemy
url: mysql+aiomysql://root:123456@127.0.0.1/ecm_sync_system?charset=utf8mb4
enable: true
wipe_on_start: false
```
当前语义:
- `sqlalchemy` backend 会自动创建它自己的 `nodes``bindings` 表及索引。
- 它不会替你创建 MySQL 数据库本身,所以 `ecm_sync_system` 这个库需要提前存在。
- 现有表结构已经兼容 MySQL,不需要再额外给主键字段手工指定前缀长度。
### 7.5 `backend_options` 什么时候用
`backend_options` 仍然保留,但现在优先使用显式字段:
- `db_path`
- `url`
- `enable`
- `wipe_on_start`
除非某个自定义 backend 明确要求,否则不要把主连接信息再塞进 `backend_options`
## 8. logging 的当前写法
```yaml
logging:
initialize: true
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
file_name_pattern: simple_pipeline_sm_{timestamp}.log
level: 20
suppress_node_logs: true
```
字段说明:
- `initialize`
- 是否初始化应用日志。
- `file_path`
- 直接指定日志文件绝对路径。
- `file_dir`
- 只指定目录时,实际文件名由 `file_name_pattern` 生成。
- `file_name_pattern`
- 支持 `{timestamp}` 占位。
- `level`
- 默认 `20`,即 `INFO`
- `suppress_node_logs`
- 是否压缩逐节点日志噪音。
## 9. 多项目编排文件怎么写
### 9.1 结构
多项目文件不是把所有字段都堆在一起,而是引用两份单 pipeline 配置:
```yaml
global_config: run_profiles/jsonl_to_jsonl.multi_project.global.yaml
default_project_config: run_profiles/jsonl_to_jsonl.multi_project.project.yaml
project_bootstrap_node_types:
- company
- supplier
max_concurrency: 8
projects:
- local_project_id: local-project-1
remote_project_id: remote-project-1
- local_project_id: local-project-2
remote_project_id: remote-project-2
```
字段说明:
- `global_config`
- 全局公共节点要跑的单 pipeline 配置。
- `default_project_config`
- 每个项目默认使用的单 pipeline 配置。
- `project_bootstrap_node_types`
- 需要从 global scope 继承到项目 scope 的公共节点类型,例如 `company``supplier`
- `max_concurrency`
- 项目并发数上限,默认 `20`
- `projects`
- 项目映射清单。
### 9.2 项目项支持单独切配置
如果某个项目需要特殊策略,可以在项目项里覆盖 `config_path`
```yaml
projects:
- local_project_id: local-project-1
remote_project_id: remote-project-1
- local_project_id: local-project-2
remote_project_id: remote-project-2
config_path: run_profiles/special_project.pipeline.yaml
```
此时该项目不会再用 `default_project_config`,而是改用自己的单 pipeline 配置文件。
### 9.3 当前多项目 profile 的实际风格
以当前 backend 两项目推送链路为例,常见组合是:
- `local_datasource.type: depm_db`
- `remote_datasource.type: api`
- `persist.backend: sqlalchemy`
- `persist.url: mysql+aiomysql://...`
- `strategies.project.config.pre_bind_data_id` 明确写每个项目的 local/remote 映射
这类 profile 本质上仍然是“多项目编排文件 + 两份单 pipeline 文件”的组合,不要把它理解成第三种配置格式。
## 10. 当前推荐的写法习惯
### 10.1 只写必要覆盖项
推荐:
- 只写与你当前环境、项目过滤、持久化目标、策略差异直接相关的字段。
不推荐:
- 把默认值全部展开复制一遍。
原因很直接:
- 默认值一旦调整,你的大段本地 YAML 会悄悄过时。
- 排障时也更难看出你到底改了什么。
### 10.2 handler 特有参数放 datasource 里
比如:
```yaml
remote_datasource:
type: api
handler_configs:
contract_settlement_detail:
load_method: aggregate
```
这类参数应该写在 `datasource.handler_configs.<node_type>`,不要错写到 strategy 里。
### 10.3 项目过滤显式下沉到 handler
例如:
```yaml
handler_configs:
project:
data_id_filter:
- project-id-1
- project-id-2
```
不要在 pipeline 外层再做一层临时过滤逻辑。项目范围应尽量体现在配置里,并由对应 handler 解释。
### 10.4 看到报错先修配置,不要做降级
当前系统对配置保持 fail-fast:
- datasource type 未注册会报错
- legacy strategy key 会报错
- `persist` 缺失目标会报错
- profile 结构不符合 schema 会报错
这属于预期行为。优先修正配置,而不是在文档或调用侧加 fallback。
## 11. 参考文件
如果你想直接对照真实样例,优先看这些文件:
- `run_profiles/jsonl_to_api.yaml`
- `run_profiles/jsonl_to_jsonl.multi_project.yaml`
- backend 中的 `local_tests/sync_system_tests/depm_to_ecm_bootstrap_pull_then_project_push.two_projects.sqlalchemy.pipeline.yaml`
如果你想看配置模型定义,直接看:
- `sync_state_machine/config/pipeline_config.py`
- `sync_state_machine/config/datasource_config.py`
- `sync_state_machine/config/persist_config.py`
- `sync_state_machine/config/multi_project_config.py`
- `sync_state_machine/config/strategy_config.py`
排障时优先顺序:
1. 看输入 YAML。
2. 看启动日志里的 `Resolved Full Config`
3. 看对应 datasource type 是否真的已经注册。
4. 再看运行日志和状态机结果。
+14 -11
View File
@@ -21,21 +21,22 @@
- `S12C/S12U/S12D`:按动作拆分的执行失败状态
- `S13`:对侧创建等待态(`PEER_CREATING/NONE/PENDING`
- `S14`UPDATE 运行时降级为 `NONE` 并跳过
- `S17`:bootstrap 加载异常隔离(存在绑定,但 datasource refresh 未返回最新节点)
## 3. 事件到代码映射
### bootstrap
- `E01``engine/events.py::e01_bootstrap()`
- `E01``sync_state_machine/engine/events.py::e01_bootstrap()`
### bind
- `E10``engine/events.py::e10_bind_core()`
- `E15``engine/events.py::e15_bind_dependency()`
- `E16``engine/events.py::e16_bind_auto()`(自动绑定判定;`ZERO+create_enabled=true` 时进入 `S13`,等待创建副作用提交)
- `E10``sync_state_machine/engine/events.py::e10_bind_core()`
- `E15``sync_state_machine/engine/events.py::e15_bind_dependency()`
- `E16``sync_state_machine/engine/events.py::e16_bind_auto()`(自动绑定判定;`ZERO+create_enabled=true` 时进入 `S13`,等待创建副作用提交)
### create/update prepare
- `E20``engine/events.py::e20_create_prepare()`spawn 成功从 `S13` 提交到 `S01`;失败保持 `S13`
- `E21``engine/events.py::e21_create_prepare()`
- `E30``engine/events.py::e30_update_prepare()`
- `E20``sync_state_machine/engine/events.py::e20_create_prepare()`spawn 成功从 `S13` 提交到 `S01`;失败保持 `S13`
- `E21``sync_state_machine/engine/events.py::e21_create_prepare()`
- `E30``sync_state_machine/engine/events.py::e30_update_prepare()`
### sync execute
- `E40C_START/E40C`CREATE 执行入口与结果回写
@@ -43,14 +44,15 @@
- `E40D_START/E40D`DELETE 执行入口与结果回写
## 4. strategy 调用链(当前)
- bind 主入口:`sync_system/strategy.py::DefaultSyncStrategy.bind()` -> `strategy_ops/bind_ops.py::run_bind()`
- create 主入口:`sync_system/strategy.py::DefaultSyncStrategy.create()` -> `strategy_ops/create_ops.py::run_create()`
- update 主入口:`sync_system/strategy.py::DefaultSyncStrategy.update()` -> `strategy_ops/update_ops.py::run_update()`
- reset 主入口:`sync_system/strategy.py::BaseSyncStrategy.run_reset()` -> `strategy_ops/reset_ops.py::run_phase2_cleanup()`
- bind 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.bind()`
- create 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.create()`
- update 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.update()`
- reset 主入口:`sync_state_machine/pipeline/full_sync_pipeline.py::_load_persistence_state()` -> `sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`
## 5. context 字段(以 YAML 为准)
### bind 相关
- `bootstrap_source_present`
- `has_binding_record`
- `core_binding_result` / `peer_core_binding_result` (`NORMAL|WARNING|ABNORMAL`)
- `data_present`
@@ -75,6 +77,7 @@
## 6. 语义边界
- strategy 负责业务语义计算(如依赖是否满足、自动绑定结果、差异检测)。
- 状态机只负责迁移判定与不变量校验,不直接访问数据源。
- bootstrap 阶段的数据源刷新由 pipeline 完成,状态机只消费 `bootstrap_source_present / has_binding_record / is_create_zombie` 这些已计算上下文。
## 7. 不变量
- `INV-1_UNCHECKED_ACTION_NONE`
+46 -21
View File
@@ -2,49 +2,74 @@
## 1. 全流程主线
1. pipeline 装配组件并加载持久化。
2. pipeline 调用 `strategy.bind()`
3. pipeline 调用 `strategy.create()` / `strategy.update()` 生成动作节点
4. pipeline 调用 `datasource.sync_all()` 执行动作并回写执行状态
5. pipeline 输出统计并持久化
2. pipeline 为 datasource 绑定 collection,但不在 bootstrap 阶段全量拉取业务数据
3. pipeline 在每个 `node_type` 开始执行前,单独加载该 `node_type` 的最新数据
4. pipeline 调用 `strategy.bind()`
5. pipeline 调用 `strategy.create()` / `strategy.update()` 生成动作节点
6. pipeline 调用 `datasource.sync_all()` 执行动作并回写执行状态。
7. pipeline 输出统计并持久化。
## 2. reset 时序
### 2.1 加载时 resetphase1 语义)
-`common/collection.py::load_from_persistence()` 执行。
- 语义:仅恢复持久化状态不在此阶段做状态归并
### 2.1 加载时恢复
-`sync_state_machine/common/collection.py::load_from_persistence()` 执行。
- 语义:仅恢复持久化状态,并记录本轮 `sync_log``persistence` 恢复来源;不在此阶段做状态归并。
- 恢复后保留 `data/origin_data`,等待 bootstrap 阶段的 datasource 刷新覆盖最新数据。
### 2.2 加载后 cleanupphase2 语义)
- pipeline 调用 `BaseSyncStrategy.run_reset()`
- 实际实现:`strategy_ops/reset_ops.py::run_phase2_cleanup()`
### 2.2 bootstrap datasource refresh
- pipeline 在持久化恢复后、`E01` 之前,按已恢复到内存的 `node_type` 做一次 datasource 刷新
- 实际实现:`sync_state_machine/pipeline/full_sync_pipeline.py::_bootstrap_refresh_latest_data()`
- 语义:
- 如果 datasource 命中节点,则用最新数据覆盖持久化快照,并清除 `_loaded_from_persistence` 标记。
- 如果 datasource 未命中节点,则节点保留持久化快照,供 `E01` 判断它是 stale zombie 还是 bootstrap 异常隔离对象。
### 2.3 加载后 cleanup
- pipeline 在 datasource refresh 完成后调用 `run_phase2_cleanup()`
- 实际实现:`sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`
- 核心事件:`E01``S16 -> S00/S15`)。
- `S16` 表示“持久化加载输入态”:本轮从持久化恢复出的节点集合(可包含上轮遗留节点,也可包含上轮已稳定节点)
- `S16` 表示“bootstrap 输入态”:持久化恢复出的节点,叠加 datasource refresh 后仍待 `E01` 判定的节点集合
- 判定:
- `is_create_zombie=true` -> `S15`,节点删除(并尝试解绑);
- `is_create_zombie=false` -> `S00`节点归并为 `UNCHECKED/NONE/PENDING` 并清空 `error`
- `bootstrap_source_present=false && has_binding_record=false` -> `S15`说明该节点仅残留在 persistence 中且未绑定,按 stale zombie 清理;
- `bootstrap_source_present=false && has_binding_record=true` -> `S17`,说明该节点存在绑定,但 datasource refresh 未返回,进入 bootstrap 加载异常隔离;
- 其余情况 -> `S00`,节点归并为 `UNCHECKED/NONE/PENDING` 并清空 `error`
### 2.2.1 `enable_persistence` 与 bootstrap 关系
- `enable_persistence=true`:先 `load_from_persistence()`,再对加载出的节点执行 `E01``S16` 域内处理)。
- `enable_persistence=true`:先 `load_from_persistence()`,再做 bootstrap datasource refresh,最后对输入节点执行 `E01``S16` 域内处理)。
- `enable_persistence=false`:本轮没有持久化输入,`S16/E01` 不参与;随后 Stage1 新加载节点直接进入本轮 bind/create/update/sync 主流程。
### 2.3 加载异常隔离(S17
### 2.4 加载异常隔离(S17
- 持久化中若出现非法枚举值(`action/status/binding_status`),节点标记为 `bootstrap_blocked``S17` 语义)。
- 持久化恢复后若节点存在绑定,但 bootstrap datasource refresh 未返回该节点,也会进入 `S17` 语义并隔离。
- 这类节点保留 `error`,用于人工排查。
- 这类节点不会进入 `E01`,也不会参与 bind/create/update/sync 自动流程。
## 3. bind 流程(strategy_ops
- 入口:`strategy_ops/bind_ops.py::run_bind()`
- 入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.bind()`
- 入口状态:`S00`
### bind 编排步骤
- strategy 先通过 `bind_ops.collect_bind_entry_nodes()` 收集两侧 `S00` 节点,作为 bind 入口工作集。
- `phase_core_state()` 对入口工作集触发 `E10`,把“已有绑定记录”和“无绑定记录”的节点分流到 `S01/S03/S04/S02`
- `phase_dependency_check()` 对仍需继续处理的缺绑节点触发 `E15`,把缺数据、依赖不满足、语义错误的节点拦到 `S03/S05/S04`
- strategy 再通过 `bind_ops.collect_auto_bind_ready_nodes()` 收窄工作集,只保留当前 `S02 + binding_status=MISSING` 的节点进入自动绑定判定。
- strategy 在顶层显式构造 orphan create 控制参数,并传给 `bind_ops.plan_auto_bind_updates()` 生成自动绑定/自动创建决策。
- `apply_auto_bind_updates()` 对这些决策逐个触发 `E16`,把节点迁移到 `S01/S04/S05/S13`
- `refresh_bind_data_id()` 最后按绑定关系回填两侧节点的 `context.bind_data_id`
### 核心绑定判定
- `phase_core_state()` 内触发 `E10`
- 有绑定记录:根据本端/对端有效性进入 `S01/S03/S04`
- 无绑定记录:进入 `S02`
注意:如果节点在 bootstrap 阶段已被隔离为 `S17` 语义(`bootstrap_blocked=true`),默认不会进入 bind 工作集。
### 依赖检查
- `phase_dependency_check()` -> `check_dependencies_for_nodes()` 内触发 `E15`
- `data_present=true``dep_satisfied=true` -> 不迁移,节点保持在 `S02`,继续进入自动绑定判定。
- `data_present=false` -> `S03`
- `dep_satisfied=false` -> `S05`
- `depend_fields` 为空 -> `S04`(无法自动处理,阻断后续自动创建)
- `depend_fields` 为空本身不会触发阻断;若节点数据存在,则等价于“当前依赖满足”,继续留在 `S02`
### 自动绑定
- `phase_auto_binding()` 内触发 `E16`
@@ -55,7 +80,7 @@
## 4. create/update 准备
### create
- 入口:`strategy_ops/create_ops.py::run_create()`
- 入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.create()`
- 运行时事件:`E20``E21/E22` 为配置保留分支,当前 strategy 未接入自动执行)
- 运行时入口状态:`S13`
- 说明:`E16` 负责识别“需自动创建”并将源节点置为 `S13`;业务代码执行对侧 spawn+bind 后,再通过 `E20``S13` 提交到 `S01`(成功);若 spawn 失败,`E20` 不迁移并保留在 `S13` 等待人工/重试。
@@ -64,19 +89,19 @@
- 新创建目标节点在 `S06/S10C` 阶段允许 `data_id` 为空;`CREATE SUCCESS``S11C`)必须携带非空 `result_data_id`,否则事件直接报错。
### update
- 入口:`strategy_ops/update_ops.py::run_update()`
- 入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.update()`
- 事件:`E30`
- 入口状态(source 节点):`S01`
- `target_has_data_id=false``update_id_resolve_ok=false` -> `S08`
- `has_diff=true` -> `S07`
### post-create 归并
- 入口:`pipeline/full_sync_pipeline.py::normalize_create_success_to_update_entry()`
- 入口:`sync_state_machine/pipeline/full_sync_pipeline.py::normalize_create_success_to_update_entry()`
- 事件:`E41`
- 语义:`CREATE` 执行成功后,节点先落 `S11C`SUCCESS),再归并 `S11C -> S01`,作为同轮 update 起点。
## 5. 执行回写(datasource
- 入口:`datasource/datasource.py`
- 入口:`sync_state_machine/datasource/datasource.py`
- 事件:`E40C_START/E40C``E40U_START/E40U``E40D_START/E40D`
- 当前约束:delete stage 尚未接入 strategy 主流程;若运行时检测到 `S09/DELETE` 节点,DataSource 会显式抛错阻断(避免散落删除执行)。
- 执行准备入口状态:
@@ -103,5 +128,5 @@
| 隔离(人工) | `S17` | 不参与自动阶段,仅人工处理 |
## 7. 不变量保障
- 每次事件落地后,`engine/events.py::_apply_decision()` 会执行 `runtime.check_invariants()`
- 每次事件落地后,`sync_state_machine/engine/events.py::_apply_decision()` 会执行 `runtime.check_invariants()`
- 违反 invariant 直接抛错,终止当前路径。
+195
View File
@@ -0,0 +1,195 @@
# 测试指南
本文档说明三件事:
- 单元测试怎么跑。
- 集成测试怎么组织。
- 测试辅助工具如何使用,尤其是远程 ECM 环境初始化工具。
## 1. 测试分层
### 1.1 单元测试
位置:`tests/unit/`
目标:
- 校验状态机不变量、事件约束、策略辅助逻辑。
- 校验 datasource/handler 工具链、配置严格性、导入面稳定性。
- 对具体 domain 的 update/create 行为做快速回归。
单元测试取舍原则:
- 优先测文档里承诺的行为语义,不优先测单纯转发、setter、空包装器。
- 如果一个测试只是在断言“函数 A 调了函数 B”,但不验证最终行为,通常应该删掉或改写成行为测试。
- 如果需要 test double,优先让它承载真实行为断言,不保留仅为凑接口而存在的空壳类。
常用命令:
- 全量单元测试:`pytest tests/unit -q`
- 指定文件:`pytest tests/unit/test_update_ops.py -q`
适用场景:
- 修改 `sync_state_machine/` 公共逻辑。
- 修改某个 domain 的 handler/strategy。
- 校验 schema 相关逻辑没有引入行为回退。
### 1.2 集成测试
位置:`tests/integration/`
目标:
- 覆盖 pipeline 级行为,包括 load/bind/create/update/poll/persist。
- 校验多节点、多轮次、mock datasource 与 UI smoke 场景。
- 验证运行配置、状态机配置和工具链连通性。
常用命令:
- 全量集成测试:`pytest tests/integration -q`
- UI 烟测:`pytest tests/integration/test_ui_api_smoke.py -q`
- 工具链专项:`pytest tests/integration/test_toolchain_config_and_graph.py -q`
适用场景:
- 修改 pipeline、collection、状态机执行路径。
- 修改持久化或跨 domain 依赖行为。
- 修改运行配置解析、注册表装配等入口逻辑。
### 1.3 脚本化大规模回归
位置:`scripts/auto_test_domains/`
目标:
- 用“多轮数据集 + 报告”的方式覆盖复杂业务 domain。
- 适合真实接口联调、回归复现、差异收敛分析。
特点:
- 这类脚本不直接纳入 pytest。
- 它们更偏向测试编排与问题定位,而不是轻量级快速回归。
相关文档:
- `docs/auto_test_spec/agent_auto_test_method.md`
## 2. 远程环境初始化工具
位置:`tools/remote_env.py`
这个工具是给真实 ECM 环境联调用的最小控制器,提供 4 个命令:
- `clear`:停止服务、恢复 `.env`、清空自动测试专用 CouchDB 数据库。
- `init`:停止服务,按 JSONL 数据目录执行 `run.py restore-database` 恢复测试数据库。
- `run`:启动远程 ECM 服务,不阻塞当前命令。
- `stop`:关闭远程 ECM 服务,并恢复原始 `.env`
它不是 pipeline 执行器,只负责测试环境的基础生命周期管理。
### 2.1 配置项
示例配置:`tools/remote_env.example.yaml`
核心配置:
- `couchdb.base_url` / `couchdb.database` / `couchdb.username` / `couchdb.password`
- `service.ecm_root`
- `service.python_path`
- `data.data_dir`
- `env_file.path`
- `env_file.overrides`
按命令的最低要求:
- `run` / `stop`:只要求 `service.*` 可用。
- `clear`:要求 `service.*``couchdb.*` 可用。
- `init`:要求 `service.*``couchdb.*``data.*` 全部可用。
可选配置:
- `service.start_command`
- `service.restore_command`
- `service.ready_url`
- `service.log_file`
- `service.env`
- `env_file.restore_on_stop`
支持占位符:
- `${PROJECT_ROOT}`:当前仓库根目录。
- `${CONFIG_DIR}`:配置文件所在目录。
- `${ECM_ROOT}``service.ecm_root` 的值。
- `${PYTHON}``service.python_path` 的值。
- `${DATA_DIR}``data.data_dir` 的值。
- `${COUCHDB_DATABASE}``couchdb.database` 的值。
- `${COUCHDB_USERNAME}``couchdb.username` 的值。
- `${COUCHDB_PASSWORD}``couchdb.password` 的值。
- `${COUCHDB_URL}``couchdb.url` 的值。
- `${COUCHDB_PORT}``couchdb.port` 的值。
### 2.2 基本用法
1. 复制示例配置并填入真实路径。
2. 先执行 `init` 恢复测试数据库。
3. 再执行 `run` 启动远程服务。
4. 测试结束后执行 `stop`
命令示例:
```bash
python tools/remote_env.py --config tools/remote_env.example.yaml init
python tools/remote_env.py --config tools/remote_env.example.yaml run
python tools/remote_env.py --config tools/remote_env.example.yaml stop
python tools/remote_env.py --config tools/remote_env.example.yaml clear
```
### 2.3 命令行覆盖
如果只想临时覆盖少数字段,可以直接传命令行参数:
```bash
python tools/remote_env.py \
--config tools/remote_env.example.yaml \
--db-name test_push_auto_tmp \
--ecm-root /data/ecm_app \
--python-path /data/venv/bin/python \
init
```
支持覆盖的字段:
- `--db-name`
- `--db-user`
- `--db-password`
- `--couchdb-base-url`
- `--couchdb-url`
- `--couchdb-port`
- `--ecm-root`
- `--python-path`
- `--data-dir`
- `--ready-url`
### 2.4 行为约束
- `clear``init` 会先执行停服,避免服务仍占用 CouchDB 连接。
- `init` 不会自动执行 `run`;数据库恢复和服务启动分开控制。
- `run` 如果检测到远端仓库下已有 `run.py` 进程在运行,会直接返回 `already_running`
- `.env` 只覆盖 `env_file.overrides` 中声明的键,未声明的键会原样保留。
- 第一次覆盖 `.env` 时会自动备份,`stop``clear` 默认会恢复原始 `.env`
- 进程状态写到 `runtime/remote_env/state.json`,不需要手工配置 pid 文件。
## 3. 推荐测试流程
### 3.1 本地开发回归
1. 先跑 `pytest tests/unit -q`
2. 如果改动影响 pipeline,再跑目标集成测试。
3. 如果改动影响真实接口流程,再使用 `tools/remote_env.py` 管理远程环境。
4. 如果修改了配置语义或 pipeline 控制流,优先补一条直接覆盖设计语义的 pytest,而不是只补装配/转发测试。
### 3.2 真实接口联调
1.`init` 恢复 ECM 测试库。
2.`run` 拉起 ECM 服务。
3. 执行同步脚本或联调脚本。
4. 查看 `runtime/remote_env/` 下的状态文件与日志。
5. 完成后执行 `stop`
### 3.3 复杂 domain 回归
1.`scripts/auto_test_domains/` 中的 case 脚本准备多轮数据。
2. 先缩小到单个 domain 复现。
3. 确认差异与日志后,再扩大到聚合回归。
## 4. 相关文件
- `tests/README.md`
- `tools/remote_env.py`
- `tools/remote_env.example.yaml`
- `docs/operations.md`
- `docs/auto_test_spec/agent_auto_test_method.md`
@@ -1 +1 @@
{"_id": "a1d69bf1-b95e-4832-9258-5bc35c629e9a", "_rev": "2-e265a6cb3674c71d6957883177b270cb", "type": "company", "name": "金上公司", "short_name": "金上公司", "code": "028B", "company_type": "region", "parent_id": "5da30aa7-b560-416e-9223-284290692351", "path": ["5da30aa7-b560-416e-9223-284290692351", "a1d69bf1-b95e-4832-9258-5bc35c629e9a"], "path_name": ["集团公司", "金上公司"], "province": "510000", "city": "510100", "status": "active", "sort": 27, "created_by": "system", "created_by_name": "system", "created_at": "2025-05-30 09:00:09", "updated_by": "system", "updated_by_name": "system", "updated_at": "2025-09-25 20:08:42", "company_nature": ["owner"], "id": "a1d69bf1-b95e-4832-9258-5bc35c629e9a", "credit_code": null, "city_name": "成都市", "province_name": "四川省", "parent_name": "集团公司"}
{"_id": "a1d69bf1-b95e-4832-9258-5bc35c629e9a", "_rev": "2-e265a6cb3674c71d6957883177b270cb", "type": "company", "name": "金上公司", "short_name": "金上公司", "code": "028B", "company_type": "region", "parent_id": "5da30aa7-b560-416e-9223-284290692351", "path": ["5da30aa7-b560-416e-9223-284290692351", "a1d69bf1-b95e-4832-9258-5bc35c629e9a"], "path_name": ["集团公司", "金上公司"], "province": "510000", "city": "510100", "status": "active", "sort": 27, "created_by": "system", "created_by_name": "system", "created_at": "2025-05-30 09:00:09", "updated_by": "system", "updated_by_name": "system", "updated_at": "2025-09-25 20:08:42", "company_nature": ["owner"], "id": "a1d69bf1-b95e-4832-9258-5bc35c629e9a", "credit_code": null, "city_name": "成都市", "province_name": "四川省", "parent_name": ""}
@@ -1,28 +1,28 @@
{"_id": "0798431d-115a-4b1c-b95a-78470160fa9e", "_rev": "3-18cce2b2f9854c79aeb4329c31e6028c", "type": "supplier", "name": "沈阳中变电气有限责任公司", "short_name": "沈阳中变电气有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "210000", "city": "210100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91210100564677033W", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "210000", "210100"]}
{"_id": "1098875c-e949-47b1-9f40-237eb507f890", "_rev": "3-0ef325ac17b9232224f5c9f924e2cf6f", "type": "supplier", "name": "中国水利水电第九工程局有限公司", "short_name": "中国水利水电第九工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "520000", "city": "520100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91520000214428387H", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "520000", "520100"]}
{"_id": "1115a7f3-b2c1-4bfb-8c89-605250c402dd", "_rev": "3-6e0a5e272f35cd473d6cd82a1ad6cfa3", "type": "supplier", "name": "中国安能集团第三工程局有限公司", "short_name": "中国安能集团第三工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915101006331302278", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
{"_id": "1ff5b102-2f9e-496c-8591-1a8e32407c7f", "_rev": "3-2857c4610fd9e9a9b8313dee83454969", "type": "supplier", "name": "中国葛洲坝集团三峡建设工程有限公司", "short_name": "中国葛洲坝集团三峡建设工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "420000", "city": "420500", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91420000615573462P", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "420000", "420500"]}
{"_id": "4be9e1da-9ebf-4a2a-b6bd-14a41ac31858", "_rev": "5-b54a288a89b9e55cee26d42b08bb69a0", "type": "supplier", "name": "中国水利水电第七工程局有限公司", "short_name": "中国水利水电第七工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510000201827469M", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
{"_id": "357c7eaf-5a33-4999-be94-79c42bfbd955", "_rev": "3-35879fa40a8fbe379f03c7a4fa8696ad", "type": "supplier", "name": "广州擎天实业有限公司", "short_name": "广州擎天实业有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "440300", "city": "440100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91440101231240749H", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "440000", "440100"]}
{"_id": "2ca09c22-68b4-4e43-b9fa-619dcda5f325", "_rev": "3-ac12e05cedcda85821240a9066aa91ef", "type": "supplier", "name": "江苏苏博特新材料股份有限公司", "short_name": "江苏苏博特新材料股份有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "320000", "city": "320100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91320000768299302G", "created_at": "2025-09-25 20:10:43", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "320000", "320100"]}
{"_id": "31b6b6d7-989b-4abe-a7e1-6d518f3a4020", "_rev": "3-bedf089583a4645867c7036dba8e7a9e", "type": "supplier", "name": "中国水利水电第十工程局有限公司", "short_name": "中国水利水电第十工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9151018120276341XK", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
{"_id": "34db820f-860c-4c6e-9326-5c7cb385bf50", "_rev": "3-e475374124da6df3a40f38043efb7237", "type": "supplier", "name": "特变电工中发上海高压开关有限公司", "short_name": "特变电工中发上海高压开关有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "310000", "city": "310000", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9131012078784664XA", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "310000", "310000"]}
{"_id": "4b069ce4-3687-4263-a1f2-4735c22c3f8e", "_rev": "3-9cccdfa9f992ae4f0ec07068c479487c", "type": "supplier", "name": "四川二滩国际工程咨询有限责任公司", "short_name": "四川二滩国际工程咨询有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510115201873799G", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
{"_id": "2e465bd8-b240-4046-bf00-187d22823f7b", "_rev": "3-2629f9e1646b7e398fd8bb41a7fc47a2", "type": "supplier", "name": "哈尔滨电机厂有限责任公司", "short_name": "哈尔滨电机厂有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "230000", "city": "230100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "912301991270479655", "created_at": "2025-09-26 17:00:05", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "230000", "230100"]}
{"_id": "3ab47229-6da8-4efb-aa44-b96866a2ce13", "_rev": "3-998c447f59be1475e690d1416384fb64", "type": "supplier", "name": "四川蜀府贸易有限公司", "short_name": "四川蜀府贸易有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915100005632853965", "created_at": "2025-09-26 17:00:10", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
{"_id": "48a5f1a7-5d5e-41cf-8676-042f1d5316f1", "_rev": "2-d43e58afff0c5317401a22d075866c56", "field_metadata": null, "created_by": "a775915a-0bed-4a17-94f0-47b29916be90", "created_by_name": "集团公司管理员", "created_at": "2025-11-27 11:32:41", "updated_by": "a775915a-0bed-4a17-94f0-47b29916be90", "updated_by_name": "集团公司管理员", "updated_at": "2025-12-25 08:59:53", "deleted_at": null, "deleted_by": null, "deleted_by_name": null, "type": "supplier", "code": "czyy", "company_type": "other", "name": "常州液压成套设备厂有限公司", "short_name": "常州液压", "parent_id": null, "path": [], "path_name": [], "province": "320000", "city": "320400", "sort": 0, "status": "active", "company_nature": ["supplier"], "credit_code": "91320412250851852D", "country": "100000", "region_path": ["100000", "320000", "320400"]}
{"_id": "67fa779d-2ce9-468f-b94f-a3d29f1f22c9", "_rev": "3-2c6e66d316216cfc460662bf4a40e612", "type": "supplier", "name": "杭州华新机电工程有限公司", "short_name": "杭州华新机电工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "330000", "city": "330100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "913301087227885306", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "330000", "330100"]}
{"_id": "594a09af-8944-476d-b098-7aebe96ad4e8", "_rev": "3-fc6edc3d3384c95def0be02ee734a5b2", "type": "supplier", "name": "云南水电十四局东华建筑工程有限公司", "short_name": "云南水电十四局东华建筑工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "530000", "city": "530100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91530000713408946K", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "530000", "530100"]}
{"_id": "6db1ed5b-f52a-4142-867c-d8b2731ead71", "_rev": "3-f769878777781eb98071dddc7c7c4968", "type": "supplier", "name": "中国水利水电第十四工程局有限公司", "short_name": "中国水利水电第十四工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "530000", "city": "530100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91530100216579074C", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "530000", "530100"]}
{"_id": "538e01cd-85cb-42b3-b313-16977f7a5224", "_rev": "3-d9acfbf91ba6d83dc33c94b84127f2e3", "type": "supplier", "name": "中国水利水电第五工程局有限公司", "short_name": "中国水利水电第五工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510000205804264E", "created_at": "2025-09-26 17:00:12", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
{"_id": "97be34dc-0a17-4ce5-baea-0f3ceee61103", "_rev": "4-4fbf33c81fff4a4a93ba8da7f99b786d", "type": "supplier", "name": "国电南京自动化股份有限公司", "short_name": "国电南京自动化股份有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "320000", "city": "320100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "913201007162522468", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "320000", "320100"]}
{"_id": "77316fb4-6167-432b-94e5-89c3d8a6b900", "_rev": "3-2112fbd151033d97a2c90123a02c70a3", "type": "supplier", "name": "中国水利水电夹江水工机械有限公司", "short_name": "中国水利水电夹江水工机械有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "511100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91511126207500117U", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "511100"]}
{"_id": "936c6316-8d87-4681-a3b5-b90ed4804a06", "_rev": "3-a9a9b85f9cdd45dc3646e66c9a1a41aa", "type": "supplier", "name": "中国电建集团成都勘测设计研究院有限公司", "short_name": "中国电建集团成都勘测设计研究院有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915100004507513971", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]}
{"_id": "98892419-6a71-4d7e-a0ac-ad5a0a15f1d6", "_rev": "3-f5bb2f9abe284037054df3b323f2482f", "type": "supplier", "name": "中国电建集团贵阳勘测设计研究院有限公司", "short_name": "中国电建集团贵阳勘测设计研究院有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "520000", "city": "520100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9152000070966703X2", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "520000", "520100"]}
{"_id": "9c80aace-ba73-4550-8fef-3080b51027d9", "_rev": "3-0dddd7f05fccce78e17fdf4ae9975f0a", "type": "supplier", "name": "中国水利水电第四工程局有限公司", "short_name": "中国水利水电第四工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "630000", "city": "630100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9163000022658124XK", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "630000", "630100"]}
{"_id": "930dfaf9-708e-4ec0-95f4-521b10bf1c00", "_rev": "2-5c464399a48505836497072601a60159", "field_metadata": null, "created_by": "a775915a-0bed-4a17-94f0-47b29916be90", "created_by_name": "集团公司管理员", "created_at": "2025-11-27 11:33:18", "updated_by": "a775915a-0bed-4a17-94f0-47b29916be90", "updated_by_name": "集团公司管理员", "updated_at": "2025-12-25 08:59:53", "deleted_at": null, "deleted_by": null, "deleted_by_name": null, "type": "supplier", "code": "hnwz", "company_type": "other", "name": "某公司海南物资有限公司", "short_name": "海南物资", "parent_id": null, "path": [], "path_name": [], "province": "460000", "city": "460100", "sort": 0, "status": "active", "company_nature": ["supplier"], "credit_code": "91469007MA5TW", "country": "100000", "region_path": ["100000", "460000", "460100"]}
{"_id": "ab5ab605-f27a-461b-9a88-73d1eafde5ed", "_rev": "3-a463c974fbcb84a524d1ee6ed72e64ec", "type": "supplier", "name": "东方电气集团东方电机有限公司", "short_name": "东方电气集团东方电机有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510600", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510600671415848H", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "510000", "510600"]}
{"_id": "b4656ed6-5329-4673-93c8-d11e15d6ece5", "_rev": "3-0a11b989849efb63378b8a8fddb989e4", "type": "supplier", "name": "中国水利水电建设工程咨询西北有限公司", "short_name": "中国水利水电建设工程咨询西北有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "610000", "city": "610100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91610000220530812K", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "610000", "610100"]}
{"_id": "e0ceec90-0ad7-417f-a669-fd4f5db07eff", "_rev": "4-71df8660a58c5628bb55f2d39a9264fb", "type": "supplier", "name": "西安西电变压器有限责任公司", "short_name": "西安西电变压器有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "610000", "city": "610100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91610104220601553B", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "610000", "610100"]}
{"_id": "cd9e9224-a08f-4bed-81a6-b289b74506e8", "_rev": "3-572570411c03424fb3370e74b2f7f496", "type": "supplier", "name": "乐山一拉得电网自动化有限公司", "short_name": "乐山一拉得电网自动化有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "511100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91511100MA6281NC0N", "created_at": "2025-09-25 20:10:43", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "510000", "511100"]}
{"_id": "e88c8c18-bc49-4bab-acc5-6fb78f4c7f8f", "_rev": "3-269fcaf6630593cb44ee3e6a1ffd4ef9", "type": "supplier", "name": "中国水利水电第六工程局有限公司", "short_name": "中国水利水电第六工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "210000", "city": "210100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9121011211756300XA", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "210000", "210100"]}
{"_id": "0798431d-115a-4b1c-b95a-78470160fa9e", "_rev": "3-18cce2b2f9854c79aeb4329c31e6028c", "type": "supplier", "name": "沈阳中变电气有限责任公司", "short_name": "沈阳中变电气有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "210000", "city": "210100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91210100564677033W", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "210000", "210100"], "id": "0798431d-115a-4b1c-b95a-78470160fa9e", "city_name": "沈阳市", "province_name": "辽宁省", "country_name": "中国"}
{"_id": "1098875c-e949-47b1-9f40-237eb507f890", "_rev": "3-0ef325ac17b9232224f5c9f924e2cf6f", "type": "supplier", "name": "中国水利水电第九工程局有限公司", "short_name": "中国水利水电第九工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "520000", "city": "520100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91520000214428387H", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "520000", "520100"], "id": "1098875c-e949-47b1-9f40-237eb507f890", "city_name": "贵阳市", "province_name": "贵州省", "country_name": "中国"}
{"_id": "1115a7f3-b2c1-4bfb-8c89-605250c402dd", "_rev": "3-6e0a5e272f35cd473d6cd82a1ad6cfa3", "type": "supplier", "name": "中国安能集团第三工程局有限公司", "short_name": "中国安能集团第三工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915101006331302278", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "1115a7f3-b2c1-4bfb-8c89-605250c402dd", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "1ff5b102-2f9e-496c-8591-1a8e32407c7f", "_rev": "3-2857c4610fd9e9a9b8313dee83454969", "type": "supplier", "name": "中国葛洲坝集团三峡建设工程有限公司", "short_name": "中国葛洲坝集团三峡建设工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "420000", "city": "420500", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91420000615573462P", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "420000", "420500"], "id": "1ff5b102-2f9e-496c-8591-1a8e32407c7f", "city_name": "宜昌市", "province_name": "湖北省", "country_name": "中国"}
{"_id": "4be9e1da-9ebf-4a2a-b6bd-14a41ac31858", "_rev": "5-b54a288a89b9e55cee26d42b08bb69a0", "type": "supplier", "name": "中国水利水电第七工程局有限公司", "short_name": "中国水利水电第七工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510000201827469M", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "4be9e1da-9ebf-4a2a-b6bd-14a41ac31858", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "357c7eaf-5a33-4999-be94-79c42bfbd955", "_rev": "3-35879fa40a8fbe379f03c7a4fa8696ad", "type": "supplier", "name": "广州擎天实业有限公司", "short_name": "广州擎天实业有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "440300", "city": "440100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91440101231240749H", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "440000", "440100"], "id": "357c7eaf-5a33-4999-be94-79c42bfbd955", "city_name": "广州市", "province_name": "广东省", "country_name": "中国"}
{"_id": "2ca09c22-68b4-4e43-b9fa-619dcda5f325", "_rev": "3-ac12e05cedcda85821240a9066aa91ef", "type": "supplier", "name": "江苏苏博特新材料股份有限公司", "short_name": "江苏苏博特新材料股份有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "320000", "city": "320100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91320000768299302G", "created_at": "2025-09-25 20:10:43", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "320000", "320100"], "id": "2ca09c22-68b4-4e43-b9fa-619dcda5f325", "city_name": "南京市", "province_name": "江苏省", "country_name": "中国"}
{"_id": "31b6b6d7-989b-4abe-a7e1-6d518f3a4020", "_rev": "3-bedf089583a4645867c7036dba8e7a9e", "type": "supplier", "name": "中国水利水电第十工程局有限公司", "short_name": "中国水利水电第十工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9151018120276341XK", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "31b6b6d7-989b-4abe-a7e1-6d518f3a4020", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "34db820f-860c-4c6e-9326-5c7cb385bf50", "_rev": "3-e475374124da6df3a40f38043efb7237", "type": "supplier", "name": "特变电工中发上海高压开关有限公司", "short_name": "特变电工中发上海高压开关有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "310000", "city": "310000", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9131012078784664XA", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "310000", "310000"], "id": "34db820f-860c-4c6e-9326-5c7cb385bf50", "city_name": "直辖", "province_name": "上海市", "country_name": "中国"}
{"_id": "4b069ce4-3687-4263-a1f2-4735c22c3f8e", "_rev": "3-9cccdfa9f992ae4f0ec07068c479487c", "type": "supplier", "name": "四川二滩国际工程咨询有限责任公司", "short_name": "四川二滩国际工程咨询有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510115201873799G", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "4b069ce4-3687-4263-a1f2-4735c22c3f8e", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "2e465bd8-b240-4046-bf00-187d22823f7b", "_rev": "3-2629f9e1646b7e398fd8bb41a7fc47a2", "type": "supplier", "name": "哈尔滨电机厂有限责任公司", "short_name": "哈尔滨电机厂有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "230000", "city": "230100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "912301991270479655", "created_at": "2025-09-26 17:00:05", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "230000", "230100"], "id": "2e465bd8-b240-4046-bf00-187d22823f7b", "city_name": "哈尔滨市", "province_name": "黑龙江省", "country_name": "中国"}
{"_id": "3ab47229-6da8-4efb-aa44-b96866a2ce13", "_rev": "3-998c447f59be1475e690d1416384fb64", "type": "supplier", "name": "四川蜀府贸易有限公司", "short_name": "四川蜀府贸易有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915100005632853965", "created_at": "2025-09-26 17:00:10", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "3ab47229-6da8-4efb-aa44-b96866a2ce13", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "48a5f1a7-5d5e-41cf-8676-042f1d5316f1", "_rev": "2-d43e58afff0c5317401a22d075866c56", "field_metadata": null, "created_by": "a775915a-0bed-4a17-94f0-47b29916be90", "created_by_name": "集团公司管理员", "created_at": "2025-11-27 11:32:41", "updated_by": "a775915a-0bed-4a17-94f0-47b29916be90", "updated_by_name": "集团公司管理员", "updated_at": "2025-12-25 08:59:53", "deleted_at": null, "deleted_by": null, "deleted_by_name": null, "type": "supplier", "code": "czyy", "company_type": "other", "name": "常州液压成套设备厂有限公司", "short_name": "常州液压", "parent_id": null, "path": [], "path_name": [], "province": "320000", "city": "320400", "sort": 0, "status": "active", "company_nature": ["supplier"], "credit_code": "91320412250851852D", "country": "100000", "region_path": ["100000", "320000", "320400"], "id": "48a5f1a7-5d5e-41cf-8676-042f1d5316f1", "city_name": "常州市", "province_name": "江苏省", "country_name": "中国"}
{"_id": "67fa779d-2ce9-468f-b94f-a3d29f1f22c9", "_rev": "3-2c6e66d316216cfc460662bf4a40e612", "type": "supplier", "name": "杭州华新机电工程有限公司", "short_name": "杭州华新机电工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "330000", "city": "330100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "913301087227885306", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "330000", "330100"], "id": "67fa779d-2ce9-468f-b94f-a3d29f1f22c9", "city_name": "杭州市", "province_name": "浙江省", "country_name": "中国"}
{"_id": "594a09af-8944-476d-b098-7aebe96ad4e8", "_rev": "3-fc6edc3d3384c95def0be02ee734a5b2", "type": "supplier", "name": "云南水电十四局东华建筑工程有限公司", "short_name": "云南水电十四局东华建筑工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "530000", "city": "530100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91530000713408946K", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "530000", "530100"], "id": "594a09af-8944-476d-b098-7aebe96ad4e8", "city_name": "昆明市", "province_name": "云南省", "country_name": "中国"}
{"_id": "6db1ed5b-f52a-4142-867c-d8b2731ead71", "_rev": "3-f769878777781eb98071dddc7c7c4968", "type": "supplier", "name": "中国水利水电第十四工程局有限公司", "short_name": "中国水利水电第十四工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "530000", "city": "530100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91530100216579074C", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "530000", "530100"], "id": "6db1ed5b-f52a-4142-867c-d8b2731ead71", "city_name": "昆明市", "province_name": "云南省", "country_name": "中国"}
{"_id": "538e01cd-85cb-42b3-b313-16977f7a5224", "_rev": "3-d9acfbf91ba6d83dc33c94b84127f2e3", "type": "supplier", "name": "中国水利水电第五工程局有限公司", "short_name": "中国水利水电第五工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510000205804264E", "created_at": "2025-09-26 17:00:12", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "538e01cd-85cb-42b3-b313-16977f7a5224", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "97be34dc-0a17-4ce5-baea-0f3ceee61103", "_rev": "4-4fbf33c81fff4a4a93ba8da7f99b786d", "type": "supplier", "name": "国电南京自动化股份有限公司", "short_name": "国电南京自动化股份有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "320000", "city": "320100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "913201007162522468", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "320000", "320100"], "id": "97be34dc-0a17-4ce5-baea-0f3ceee61103", "city_name": "南京市", "province_name": "江苏省", "country_name": "中国"}
{"_id": "77316fb4-6167-432b-94e5-89c3d8a6b900", "_rev": "3-2112fbd151033d97a2c90123a02c70a3", "type": "supplier", "name": "中国水利水电夹江水工机械有限公司", "short_name": "中国水利水电夹江水工机械有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "511100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91511126207500117U", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "511100"], "id": "77316fb4-6167-432b-94e5-89c3d8a6b900", "city_name": "乐山市", "province_name": "四川省", "country_name": "中国"}
{"_id": "936c6316-8d87-4681-a3b5-b90ed4804a06", "_rev": "3-a9a9b85f9cdd45dc3646e66c9a1a41aa", "type": "supplier", "name": "中国电建集团成都勘测设计研究院有限公司", "short_name": "中国电建集团成都勘测设计研究院有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915100004507513971", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "936c6316-8d87-4681-a3b5-b90ed4804a06", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "98892419-6a71-4d7e-a0ac-ad5a0a15f1d6", "_rev": "3-f5bb2f9abe284037054df3b323f2482f", "type": "supplier", "name": "中国电建集团贵阳勘测设计研究院有限公司", "short_name": "中国电建集团贵阳勘测设计研究院有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "520000", "city": "520100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9152000070966703X2", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "520000", "520100"], "id": "98892419-6a71-4d7e-a0ac-ad5a0a15f1d6", "city_name": "贵阳市", "province_name": "贵州省", "country_name": "中国"}
{"_id": "9c80aace-ba73-4550-8fef-3080b51027d9", "_rev": "3-0dddd7f05fccce78e17fdf4ae9975f0a", "type": "supplier", "name": "中国水利水电第四工程局有限公司", "short_name": "中国水利水电第四工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "630000", "city": "630100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9163000022658124XK", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "630000", "630100"], "id": "9c80aace-ba73-4550-8fef-3080b51027d9", "city_name": "西宁市", "province_name": "青海省", "country_name": "中国"}
{"_id": "930dfaf9-708e-4ec0-95f4-521b10bf1c00", "_rev": "2-5c464399a48505836497072601a60159", "field_metadata": null, "created_by": "a775915a-0bed-4a17-94f0-47b29916be90", "created_by_name": "集团公司管理员", "created_at": "2025-11-27 11:33:18", "updated_by": "a775915a-0bed-4a17-94f0-47b29916be90", "updated_by_name": "集团公司管理员", "updated_at": "2025-12-25 08:59:53", "deleted_at": null, "deleted_by": null, "deleted_by_name": null, "type": "supplier", "code": "hnwz", "company_type": "other", "name": "华电海南物资有限公司", "short_name": "海南物资", "parent_id": null, "path": [], "path_name": [], "province": "460000", "city": "460100", "sort": 0, "status": "active", "company_nature": ["supplier"], "credit_code": "91469007MA5TW", "country": "100000", "region_path": ["100000", "460000", "460100"], "id": "930dfaf9-708e-4ec0-95f4-521b10bf1c00", "city_name": "海口市", "province_name": "海南省", "country_name": "中国"}
{"_id": "ab5ab605-f27a-461b-9a88-73d1eafde5ed", "_rev": "3-a463c974fbcb84a524d1ee6ed72e64ec", "type": "supplier", "name": "东方电气集团东方电机有限公司", "short_name": "东方电气集团东方电机有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510600", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510600671415848H", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "510000", "510600"], "id": "ab5ab605-f27a-461b-9a88-73d1eafde5ed", "city_name": "德阳市", "province_name": "四川省", "country_name": "中国"}
{"_id": "b4656ed6-5329-4673-93c8-d11e15d6ece5", "_rev": "3-0a11b989849efb63378b8a8fddb989e4", "type": "supplier", "name": "中国水利水电建设工程咨询西北有限公司", "short_name": "中国水利水电建设工程咨询西北有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "610000", "city": "610100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91610000220530812K", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "610000", "610100"], "id": "b4656ed6-5329-4673-93c8-d11e15d6ece5", "city_name": "西安市", "province_name": "陕西省", "country_name": "中国"}
{"_id": "e0ceec90-0ad7-417f-a669-fd4f5db07eff", "_rev": "4-71df8660a58c5628bb55f2d39a9264fb", "type": "supplier", "name": "西安西电变压器有限责任公司", "short_name": "西安西电变压器有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "610000", "city": "610100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91610104220601553B", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "610000", "610100"], "id": "e0ceec90-0ad7-417f-a669-fd4f5db07eff", "city_name": "西安市", "province_name": "陕西省", "country_name": "中国"}
{"_id": "cd9e9224-a08f-4bed-81a6-b289b74506e8", "_rev": "3-572570411c03424fb3370e74b2f7f496", "type": "supplier", "name": "乐山一拉得电网自动化有限公司", "short_name": "乐山一拉得电网自动化有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "511100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91511100MA6281NC0N", "created_at": "2025-09-25 20:10:43", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "510000", "511100"], "id": "cd9e9224-a08f-4bed-81a6-b289b74506e8", "city_name": "乐山市", "province_name": "四川省", "country_name": "中国"}
{"_id": "e88c8c18-bc49-4bab-acc5-6fb78f4c7f8f", "_rev": "3-269fcaf6630593cb44ee3e6a1ffd4ef9", "type": "supplier", "name": "中国水利水电第六工程局有限公司", "short_name": "中国水利水电第六工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "210000", "city": "210100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9121011211756300XA", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "210000", "210100"], "id": "e88c8c18-bc49-4bab-acc5-6fb78f4c7f8f", "city_name": "沈阳市", "province_name": "辽宁省", "country_name": "中国"}
+2
View File
@@ -10,6 +10,8 @@ dependencies = [
"PyYAML>=6.0,<7",
"httpx>=0.27,<1",
"aiosqlite>=0.20,<1",
"SQLAlchemy>=2.0,<3",
"greenlet>=3,<4",
"pytz>=2024.1,<2027",
"fastapi>=0.115,<1",
"uvicorn>=0.30,<1",
-1
View File
@@ -10,4 +10,3 @@ filterwarnings =
ignore:'asyncio.set_event_loop_policy' is deprecated and slated for removal in Python 3.16:DeprecationWarning:pytest_asyncio\.plugin
# 也可以设置异步测试的默认 loop scope
asyncio_default_fixture_loop_scope = function
addopts = --ignore=scripts/test_api_sync.py --ignore=scripts/test_api_sync_top10.py --ignore=scripts/test_api_write_real_sample.py
+1 -51
View File
@@ -1,8 +1,5 @@
from __future__ import annotations
import argparse
import asyncio
import logging
import sys
from pathlib import Path
@@ -10,54 +7,7 @@ PROJECT_ROOT = Path(__file__).resolve().parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
from sync_state_machine.config import load_overrides_from_file
from sync_state_machine.pipeline import build_config, run_pipeline_from_config
def _resolve_config_path(raw_path: str) -> Path:
path = Path(raw_path)
if not path.is_absolute():
path = PROJECT_ROOT / path
return path.resolve()
async def _main() -> int:
parser = argparse.ArgumentParser(description="Run sync pipeline from config override file")
parser.add_argument(
"--config_path",
required=True,
help="Path to run profile file (JSON/YAML, supports relative path, e.g. run_profiles/jsonl_to_jsonl.local.json)",
)
args = parser.parse_args()
cfg_path = _resolve_config_path(args.config_path)
if not cfg_path.exists() or not cfg_path.is_file():
raise FileNotFoundError(f"Config file not found: {cfg_path}")
overrides = load_overrides_from_file(cfg_path, PROJECT_ROOT)
config = build_config(project_root=PROJECT_ROOT, overrides=overrides)
mode = f"{config.local_datasource.type}->{config.remote_datasource.type}"
print(f"🧩 Loaded config file: {cfg_path}")
await run_pipeline_from_config(
config,
title=f"sync_state_machine pipeline ({mode})",
print_summary=True,
)
return 0
def main() -> int:
try:
return asyncio.run(_main())
except KeyboardInterrupt:
print("\n⚠️ Interrupted by user")
return 130
except Exception as exc:
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
return 1
finally:
logging.shutdown()
from sync_state_machine.cli import main
if __name__ == "__main__":
+19 -3
View File
@@ -3,19 +3,35 @@
- 路径:项目根目录 `run_profiles/`
- 命名:建议使用 `*.local.yaml`(该目录下 YAML 默认会被 git 忽略)
当前约定:
- 仓库里跟踪的配置文件只写必要字段,不重复抄默认值。
- pipeline 启动时会打印 **Resolved Full Config**,把省略字段补齐。
- 如果你要看完整字段和字段含义,不看模板,直接看 `docs/runtime_config_guide.md`
模板来源:
- `run_profiles/preset/jsonl_to_jsonl.default.yaml`
- `run_profiles/preset/jsonl_to_api.default.yaml`
- `run_profiles/preset/jsonl_to_jsonl.multi_project.default.yaml`
推荐流程:
1. 从模板复制到本目录,例如:
- `run_profiles/jsonl_to_jsonl.local.yaml`
- `run_profiles/jsonl_to_api.local.yaml`
- `run_profiles/jsonl_to_jsonl.local.yaml`
- `run_profiles/jsonl_to_api.local.yaml`
2. 按需修改后运行:
- `ecm-sync-run --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
- `ecm-sync-run --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
兼容旧入口:
- `python run.py --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
多项目示例:
- `run_profiles/jsonl_to_jsonl.multi_project.yaml`
- `run_profiles/jsonl_to_jsonl.multi_project.global.yaml`
- `run_profiles/jsonl_to_jsonl.multi_project.project.yaml`
支持占位符:
- `${PROJECT_ROOT}`:运行时替换为项目根目录绝对路径。
现成示例:
- `run_profiles/jsonl_to_api.yaml`:单项目 JSONL -> API 联调示例
- `run_profiles/jsonl_to_api.test.yaml`auto-test 使用的 JSONL -> API 示例
- `run_profiles/jsonl_to_jsonl.multi_project.yaml`:多项目 JSONL -> JSONL 编排入口
+17 -2
View File
@@ -2,20 +2,35 @@
该目录存放版本管理的运行配置模板(default)。
模板约定:
- 这里只保留必要覆盖项。
- 省略的字段会在加载时从默认 datasource 配置和各 domain 的 `default_config` 自动补齐。
- 真正生效的全量配置,以启动日志里的 **Resolved Full Config** 为准。
当前模板:
- `jsonl_to_jsonl.default.yaml`
- `jsonl_to_api.default.yaml`
- `jsonl_to_jsonl.multi_project.default.yaml`
- `jsonl_to_jsonl.multi_project.global.default.yaml`
- `jsonl_to_jsonl.multi_project.project.default.yaml`
使用方式:
1. 复制模板到项目根目录 `run_profiles/`,并重命名为你自己的文件(如 `*.local.yaml`)。
2. 修改后通过入口运行:
2. 只改和当前环境/业务差异相关的字段。
3. 修改后通过入口运行:
- `ecm-sync-run --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
兼容旧入口:
- `python run.py --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
多项目配置说明:
- `jsonl_to_jsonl.multi_project.default.yaml` 是多 pipeline 编排入口。
- `jsonl_to_jsonl.multi_project.global.default.yaml` 定义全局公共 domain 的 pipeline。
- `jsonl_to_jsonl.multi_project.project.default.yaml` 定义单项目 pipeline 默认配置。
- `project_bootstrap_node_types` 用于显式声明项目级 pipeline 需要从 global scope 继承到项目 scope 的公共节点类型,例如 `company``supplier`
说明:
- `run_profiles/*.yaml` / `run_profiles/*.yml` 默认被 `.gitignore` 忽略,用于本地私有配置。
- 模板内可使用 `${PROJECT_ROOT}` 占位符。
- 模板已显式展开 datasource 与 strategies 关键字段,便于集中审查
- 完整字段定义与字段含义见 `docs/runtime_config_guide.md`
- API 数据源支持全局限流配置:`api_rate_limit_max_requests``api_rate_limit_window_seconds`
+41 -261
View File
@@ -1,31 +1,33 @@
node_types:
- company
- supplier
- project
- contract
- contract_change
- construction_task
- construction_task_detail
- material
- material_detail
- contract_settlement
- contract_settlement_detail
- personnel
- personnel_detail
- preparation
- production
- lar
- lar_change
- project_detail_data
- units
- company
- supplier
- project
- contract
- contract_change
- construction_task
- construction_task_detail
- material
- material_detail
- contract_settlement
- contract_settlement_detail
- personnel
- personnel_detail
- preparation
- production
- lar
- lar_change
- project_detail_data
- units
local_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
read_only: false
target_project_ids:
- "0f9c3e22-f4bb-4803-825a-4f238eeb06d5"
- "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
handler_configs: {}
handler_configs:
project:
data_id_filter:
- "0f9c3e22-f4bb-4803-825a-4f238eeb06d5"
- "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
remote_datasource:
type: api
api_base_url: http://localhost:8000/api/third/v2
@@ -35,256 +37,34 @@ remote_datasource:
api_rate_limit_max_requests: 30
api_rate_limit_window_seconds: 10.0
poll_max_retries: 1
poll_interval: 0.5
target_project_ids:
# 请替换为实际的项目ID,至少需要一个
- "0f9c3e22-f4bb-4803-825a-4f238eeb06d5"
- "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
handler_configs:
supplier: {}
project:
data_id_filter:
- "0f9c3e22-f4bb-4803-825a-4f238eeb06d5"
- "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
persist:
backend: sqlite
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/simple_pipeline_sm.db
enable: true
wipe_on_start: false
logging:
initialize: true
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
file_name_pattern: simple_pipeline_sm_{timestamp}.log
level: 20
strategies:
company:
skip_sync: false
preset: pull_only
config:
auto_bind: true
auto_bind_fields:
- credit_code
depend_fields: {}
local_orphan_action: none
remote_orphan_action: create_local
update_direction: pull
- credit_code
supplier:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- credit_code
depend_fields: {}
local_orphan_action: none
remote_orphan_action: create_local
update_direction: pull
preset: pull_only
project:
skip_sync: false
config:
auto_bind: false
auto_bind_fields: []
pre_bind_data_id: []
depend_fields:
company_id: company
local_orphan_action: none
remote_orphan_action: none
update_direction: push
contract:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- company_id
- code
depend_fields:
project_id: project
company_id: supplier
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract_change:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
- change_time
- title
depend_fields:
project_id: project
contract_id: contract
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
construction_task:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
- task_type
depend_fields:
contract_id: contract
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
construction_task_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: construction_task
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
material:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
- material_type
depend_fields:
contract_id: contract
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
material_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: material
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract_settlement:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
depend_fields:
contract_id: contract
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract_settlement_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: contract_settlement
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
personnel:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- code
depend_fields:
project_id: project
contract_id: contract
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
personnel_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: personnel
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
preparation:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- code
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
production:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- code
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
lar:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
lar_change:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- change_time
- title
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
project_detail_data:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- key
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push # 未在 field_direction_overrides 中列出的 key 默认 push
field_direction_key: key # 以节点 data.key 字段值作为方向查表键
field_direction_overrides:
ensure_production: pull # 远程 → 本地
climb_production: pull # 远程 → 本地
challenge_production: pull # 远程 → 本地
units:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- serial_number
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
post_check_ignore_fields:
- is_exist
+2 -219
View File
@@ -22,263 +22,46 @@ node_types:
local_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
read_only: false
target_project_ids: []
handler_configs: {}
remote_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/remote_datasource/datasource
read_only: false
target_project_ids: []
handler_configs: {}
persist:
backend: sqlite
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/full_sync_pipeline_jsonl_sm.db
enable: true
wipe_on_start: false
logging:
initialize: true
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
file_name_pattern: jsonl_pipeline_sm_{timestamp}.log
level: 20
strategies:
company:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- credit_code
depend_fields: {}
- credit_code
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
user:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- username
depend_fields: {}
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
supplier:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- credit_code
depend_fields: {}
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
project:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- name
depend_fields:
company_id: company
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- company_id
- code
depend_fields:
project_id: project
company_id: supplier
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract_change:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
- change_time
- title
depend_fields:
project_id: project
contract_id: contract
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
construction_task:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
- task_type
depend_fields:
contract_id: contract
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
construction_task_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: construction_task
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
material:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
- material_type
depend_fields:
contract_id: contract
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
material_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: material
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract_settlement:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
depend_fields:
contract_id: contract
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract_settlement_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: contract_settlement
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
personnel:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- code
depend_fields:
project_id: project
contract_id: contract
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
personnel_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: personnel
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
preparation:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- code
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
production:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- code
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
- name
lar:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
depend_fields:
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
lar_change:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- change_time
- title
depend_fields:
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
project_detail_data:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- key
depend_fields:
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
units:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- serial_number
depend_fields:
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
@@ -0,0 +1,12 @@
global_config: run_profiles/preset/jsonl_to_jsonl.multi_project.global.default.yaml
default_project_config: run_profiles/preset/jsonl_to_jsonl.multi_project.project.default.yaml
project_bootstrap_node_types:
- company
- supplier
projects:
- local_project_id: f3e02e84-e81c-4aa6-88d4-f5aa108c8227
remote_project_id: f3e02e84-e81c-4aa6-88d4-f5aa108c8227
- local_project_id: 0f9c3e22-f4bb-4803-825a-4f238eeb06d5
remote_project_id: 0f9c3e22-f4bb-4803-825a-4f238eeb06d5
@@ -0,0 +1,29 @@
node_types:
- company
- supplier
local_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/working_local_datasource/20260302-020003
remote_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/runtime/tmp/multi_project_jsonl_remote
persist:
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/multi_project_jsonl.db
wipe_on_start: true
logging:
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
file_name_pattern: multi_project_global_{timestamp}.log
strategies:
company:
config:
auto_bind_fields:
- code
- name
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
supplier:
config:
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
@@ -0,0 +1,47 @@
node_types:
- project
- contract
- contract_change
- construction_task
- construction_task_detail
- material
- material_detail
- contract_settlement
- contract_settlement_detail
- personnel
- personnel_detail
- preparation
- production
- lar
- lar_change
- project_detail_data
- units
local_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/working_local_datasource/20260302-020003
remote_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/runtime/tmp/multi_project_jsonl_remote
persist:
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/multi_project_jsonl.db
logging:
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
file_name_pattern: multi_project_project_{timestamp}.log
strategies:
project:
config:
auto_bind: true
auto_bind_fields:
- name
lar:
config:
local_orphan_action: create_remote
lar_change:
config:
local_orphan_action: create_remote
project_detail_data:
config:
local_orphan_action: create_remote
units:
config:
local_orphan_action: create_remote
+33
View File
@@ -59,6 +59,39 @@ class BaseResponseWithID(BaseModel):
# 默认返回(已有 id
return data
def _get_plan_node_date(node: Any) -> Any:
if isinstance(node, dict):
return node.get("date")
return getattr(node, "date", None)
def trim_plan_nodes_by_boundary_dates(
plan_nodes: Optional[List[Any]],
start_date: Optional[Any] = None,
end_date: Optional[Any] = None,
) -> List[Any]:
if not plan_nodes:
return []
cleaned_nodes = list(plan_nodes)
while (
start_date is not None
and cleaned_nodes
and str(_get_plan_node_date(cleaned_nodes[0])) == str(start_date)
):
cleaned_nodes.pop(0)
while (
end_date is not None
and cleaned_nodes
and str(_get_plan_node_date(cleaned_nodes[-1])) == str(end_date)
):
cleaned_nodes.pop()
return cleaned_nodes
class RequestApprovalMixin(BaseModel):
"""
请求审批相关的基础模型
+12 -1
View File
@@ -24,7 +24,18 @@
from pydantic import BaseModel, Field
from typing import TypeVar, Generic, Optional, List, Dict, Any
from ..project.enums import ProjectType
from datetime import datetime
# from backend.modules.admin.approval.models import ApprovalStep
class ApprovalStep(BaseModel):
"""审批流程中的一个步骤"""
step_id: str = Field(description="步骤的唯一ID")
operator_id: Optional[str] = Field("",description="操作人ID")
operator_name: str = Field(description="操作人姓名")
action: str = Field(description="执行的动作,如 'submit', 'approve', 'reject'")
comment: Optional[str] = Field(None, description="审批意见")
created_at: datetime = Field(default_factory=datetime.utcnow, description="步骤的创建时间")
# 定义泛型
T = TypeVar('T', bound='BasePushData')
@@ -52,7 +63,7 @@ class SignatureMixin(BaseModel):
class PushRequestMixin(BaseModel):
"""推送ID字段混入"""
push_id: str = Field(..., description="推送ID,客户端生成的唯一标识(UUIDv7,支持时间排序)")
steps : List[ApprovalStep] =Field(default_factory=list,description="当前数据的审批流程步骤") # 加入字段step用来记录审批流程
# ==================== 响应相关 ====================
+10 -1
View File
@@ -2,7 +2,7 @@ from pydantic import BaseModel, Field, field_validator, model_validator
from fastapi import HTTPException
from typing import List, Optional, Literal
from ..common.validators import DateStr
from ..common.base import BaseResponseWithID
from ..common.base import BaseResponseWithID, trim_plan_nodes_by_boundary_dates
class PlanNode(BaseModel):
date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"])
@@ -40,6 +40,15 @@ class ConstructionResponse(BaseResponseWithID):
# 2025-10-20新增字段,延迟时间
delay_days: Optional[int] = Field(None, description="延迟时间(天)", examples=[0])
@model_validator(mode='after')
def clean_response_plan_nodes(self):
self.plan_node = trim_plan_nodes_by_boundary_dates(
self.plan_node,
start_date=self.plan_start_date,
end_date=self.plan_complete_date,
)
return self
class ConstructionCreate(BaseModel):
"""
施工任务创建模型
@@ -2,7 +2,7 @@ from pydantic import Field, BaseModel,field_validator,model_validator
from fastapi import HTTPException
from typing import List, Optional, Literal
from ..common.validators import DateStr
from ..common.base import BaseResponseWithID
from ..common.base import BaseResponseWithID, trim_plan_nodes_by_boundary_dates
class PlanNode(BaseModel):
date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"])
@@ -39,6 +39,15 @@ class ContractSettlementResponse(BaseResponseWithID):
None, description="统计节点截止日期", examples=["2024-05-31"]
)
@model_validator(mode='after')
def clean_response_plan_nodes(self):
self.plan_node = trim_plan_nodes_by_boundary_dates(
self.plan_node,
start_date=self.plan_start_date,
end_date=self.plan_complete_date,
)
return self
class ContractSettlementCreate(BaseModel):
"""
合同结算创建模型
+10 -1
View File
@@ -2,7 +2,7 @@ from pydantic import Field, BaseModel,field_validator,model_validator
from fastapi import HTTPException
from typing import List, Optional, Literal
from ..common.validators import DateStr
from ..common.base import BaseResponseWithID
from ..common.base import BaseResponseWithID, trim_plan_nodes_by_boundary_dates
class PlanNode(BaseModel):
date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"])
@@ -52,6 +52,15 @@ class MaterialResponse(BaseResponseWithID):
None, description="统计节点截止日期", examples=["2024-05-31"]
)
@model_validator(mode='after')
def clean_response_plan_nodes(self):
self.plan_node = trim_plan_nodes_by_boundary_dates(
self.plan_node,
start_date=self.plan_start_date,
end_date=self.plan_complete_date,
)
return self
class MaterialCreate(BaseModel):
"""
创建物资
+10 -1
View File
@@ -2,7 +2,7 @@ from pydantic import Field, BaseModel,field_validator,model_validator
from fastapi import HTTPException
from typing import List, Optional, Literal
from ..common.validators import DateStr
from ..common.base import BaseResponseWithID
from ..common.base import BaseResponseWithID, trim_plan_nodes_by_boundary_dates
class PlanNode(BaseModel):
date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"])
@@ -61,6 +61,15 @@ class PowerResponse(BaseResponseWithID):
None, description="实际首批到场人数", examples=[4]
)
@model_validator(mode='after')
def clean_response_plan_nodes(self):
self.plan_node = trim_plan_nodes_by_boundary_dates(
self.plan_node,
start_date=self.plan_start_date,
end_date=self.plan_exit_date,
)
return self
class PowerCreate(BaseModel):
"""
创建力能
+9
View File
@@ -74,3 +74,12 @@ class ProjectInfoResponse(BaseModel):
# 里程碑节点
preparation_list: list[PreparationDetail] = Field(default_factory=list, description="里程碑节点列表")
class ProjectListResponse(BaseModel):
"""
项目列表模型:分页
"""
page: int
page_size: int
total: int
list: list[ProjectInfoResponse]
+150
View File
@@ -0,0 +1,150 @@
"""
一个接口按 domain 返回项目聚合数据。
"""
from typing import Literal
from pydantic import BaseModel, Field
from ..construction.construction import ConstructionResponse
from ..construction.construction_detail import ConstructionDetailResponse
from ..contract.contract import ContractResponse
from ..contract_settlement.contract_settlement import (
ContractSettlementResponse,
)
from ..contract_settlement.contract_settlement_detail import (
ContractSettlementDetailResponse,
)
from ..material.material import MaterialResponse
from ..material.material_detail import MaterialDetailResponse
from ..power.power import PowerResponse
from ..power.power_detail import PowerDetail
from ..project.project_base import ProjectResponseBase
from ..project_extensions.contract_change import ContractChangeResponse
from ..project_extensions.generator_unit import GeneratorUnitProject
from ..project_extensions.lar import (
LarChangeDetailSchema,
LarDetailSchema,
)
from ..project_extensions.preparation import PreparationDetail
from ..project_extensions.production import ProductionDetail
from ..project_extensions.project_detail import ProjectDetailProject
from ..project_multi_type import (
GasTurbineExtension,
HydropowerExtension,
OffShoreWindExtension,
OnShoreWindExtension,
PhotovoltaicExtension,
PumpedStorageExtension,
ThermalPowerExtension,
)
ProjectAggregateDomain = Literal[
"project",
"project_detail",
"generator_units",
"lar",
"lar_change",
"contract_changes",
"photovoltaic_extension",
"onshore_wind_extension",
"offshore_wind_extension",
"hydropower_extension",
"pumped_storage_extension",
"gas_turbine_extension",
"thermal_power_extension",
"production_list",
"preparation_list",
"contracts",
"materials",
"material_details",
"constructions",
"construction_details",
"contract_settlements",
"contract_settlement_details",
"power",
"power_details",
]
ALL_PROJECT_AGGREGATE_DOMAINS: list[ProjectAggregateDomain] = [
"project",
"project_detail",
"generator_units",
"lar",
"lar_change",
"contract_changes",
"photovoltaic_extension",
"onshore_wind_extension",
"offshore_wind_extension",
"hydropower_extension",
"pumped_storage_extension",
"gas_turbine_extension",
"thermal_power_extension",
"production_list",
"preparation_list",
"contracts",
"materials",
"material_details",
"constructions",
"construction_details",
"contract_settlements",
"contract_settlement_details",
"power",
"power_details",
]
class ProjectAggregateData(BaseModel):
"""按 domain 组织的项目聚合数据。"""
project: ProjectResponseBase | None = Field(None, description="项目基本信息")
project_detail: list[ProjectDetailProject] = Field(default_factory=list, description="项目容量信息")
generator_units: list[GeneratorUnitProject] = Field(default_factory=list, description="发电单元信息")
lar: LarDetailSchema | None = Field(None, description="移民征地信息")
lar_change: list[LarChangeDetailSchema] = Field(default_factory=list, description="移民变更信息")
contract_changes: list[ContractChangeResponse] = Field(default_factory=list, description="合同变更信息")
photovoltaic_extension: PhotovoltaicExtension | None = Field(None, description="光伏项目扩展信息")
onshore_wind_extension: OnShoreWindExtension | None = Field(None, description="陆风项目扩展信息")
offshore_wind_extension: OffShoreWindExtension | None = Field(None, description="海风项目扩展信息")
hydropower_extension: HydropowerExtension | None = Field(None, description="水电项目扩展信息")
pumped_storage_extension: PumpedStorageExtension | None = Field(None, description="抽蓄项目扩展信息")
gas_turbine_extension: GasTurbineExtension | None = Field(None, description="燃气轮机项目扩展信息")
thermal_power_extension: ThermalPowerExtension | None = Field(None, description="火电项目扩展信息")
production_list: list[ProductionDetail] = Field(default_factory=list, description="投产节点列表")
preparation_list: list[PreparationDetail] = Field(default_factory=list, description="里程碑节点列表")
contracts: list[ContractResponse] = Field(default_factory=list, description="合同列表")
materials: list[MaterialResponse] = Field(default_factory=list, description="物资主表列表")
material_details: list[MaterialDetailResponse] = Field(default_factory=list, description="物资明细列表")
constructions: list[ConstructionResponse] = Field(default_factory=list, description="施工主表列表")
construction_details: list[ConstructionDetailResponse] = Field(default_factory=list, description="施工明细列表")
contract_settlements: list[ContractSettlementResponse] = Field(default_factory=list, description="合同结算主表列表")
contract_settlement_details: list[ContractSettlementDetailResponse] = Field(default_factory=list, description="合同结算明细列表")
power: list[PowerResponse] = Field(default_factory=list, description="力能主表列表")
power_details: list[PowerDetail] = Field(default_factory=list, description="力能明细列表")
class ProjectAggregateMeta(BaseModel):
"""聚合请求回显和截断信息。"""
project_id: str = Field(..., description="项目ID")
requested_domains: list[ProjectAggregateDomain] = Field(
default_factory=lambda: list(ALL_PROJECT_AGGREGATE_DOMAINS),
description="本次请求的 domain 列表",
)
domain_limit: int = Field(10000, description="每个 domain 的最大返回条数")
truncated_domains: list[ProjectAggregateDomain] = Field(default_factory=list, description="被截断的 domain 列表")
class ProjectAggregateResponse(BaseModel):
"""项目聚合同步响应。"""
meta: ProjectAggregateMeta = Field(..., description="请求元信息")
data: ProjectAggregateData = Field(
default_factory=lambda: ProjectAggregateData.model_construct(),
description="按 domain 分组的数据",
)
+31 -5
View File
@@ -5,7 +5,7 @@
from typing import List, Optional
from fastapi import HTTPException
from pydantic import Field, BaseModel, field_validator
from pydantic import Field, BaseModel, field_validator, model_validator
from ..common.push_system import BaseUpdateRequestSchema
from ..common.base import BaseResponseWithID
from ..common.datetime import get_current_date
@@ -70,14 +70,15 @@ class ProjectResponseBase(BaseResponseWithID):
complete_investment: float = Field(0.0, description="完成投资", examples=[80000.0])
# 其他信息
progress_type: ProgressType = Field(..., description="推进分类", examples=[ProgressType.A])
progress_type: ProgressType | None = Field(None, description="推进分类", examples=[ProgressType.A])
key_constraints: KeyConstraints = Field(KeyConstraints.NONE, description="关键制约", examples=[KeyConstraints.NONE])
key_constraints_remark: Optional[str] = Field("", description="关键制约备注", examples=[""])
status: Optional[str] = Field(None, description="项目状态")
# status: Optional[str] = Field(None, description="项目状态")
apply_file_ids: List[str] = Field(default_factory=list, description="申请表附件ID列表", examples=[[]])
# 拓展信息
dbtc_score_list: Optional[list] = Field(default_factory=list, description='达标投产评分列表') # 这里项目用的是list而不是List
# approval_status: Optional[str] = Field(None, description="审核状态", examples=["pending"])
@@ -265,11 +266,35 @@ class ProjectProductionCapacity(BaseModel):
"""
plan_production_date: DateStr | None = Field(None, description="集团公司计划投产日期", examples=["2024-08-01"])
plan_full_production_date: DateStr | None = Field(None, description="集团公司计划全部投产日期", examples=["2024-09-01"])
ic_plan_first_production_date: DateStr | None = Field(None, description="区域公司计划首批投产日期(只有新能源需要)",examples=["2024-08-01"])
ic_plan_full_production_date: DateStr | None = Field(None, description="区域公司计划全投日期(只有新能源需要)",examples=["2024-09-01"])
actual_production_date: DateStr | None = Field(None, description="实际投产日期(只有新能源需要)", examples=["2024-08-15"])
actual_full_production_date: DateStr | None = Field(None, description="实际全投日期(只有新能源需要)", examples=["2024-09-10"])
actual_production_list : list[_ActualProductionNode] | None = Field(None, description="实际投产节点列表(每次必须填写全部)(只有新能源需要)")
@model_validator(mode="before")
@classmethod
def reject_ic_plan_dates(cls, data):
if not isinstance(data, dict):
return data
blocked_fields = [
field for field in ("ic_plan_first_production_date", "ic_plan_full_production_date")
if field in data
]
if blocked_fields:
raise HTTPException(
status_code=400,
detail=f"/production_capacity 接口不支持更新字段: {', '.join(blocked_fields)}"
)
return data
@property
def ic_plan_first_production_date(self) -> str:
return ""
@property
def ic_plan_full_production_date(self) -> str:
return ""
@field_validator("actual_production_date")
def validate_actual_production_date(cls, v):
if v is None:
@@ -304,6 +329,7 @@ class ProjectSettlementInvestment(BaseModel):
"""
PROJECT_SETTLEMENT_INVESTMENT
合同结算投资情况
只有抽蓄-水电-煤电-气电项目类型可操作
"""
implementation_estimate: float = Field(..., description="执行概算", examples=[80000.0])
+2 -2
View File
@@ -21,8 +21,8 @@ class PostGeneratorUnits(BaseModel):
@field_validator('actual_production_date', mode="before")
def validate_actual_date_not_future(cls, v: str) -> str:
if v == "":
return ""
# if v == "":
# return ""
try:
actual = datetime_date(v)
except ValueError:
+2 -2
View File
@@ -33,7 +33,7 @@ class PreparationDetail(BaseResponseWithID):
code: Optional[str] = Field(None, description="里程碑节点编码")
acquire_date: Optional[DateStr] = Field(None, description="取得时间")
is_exist: Optional[bool] = Field(None, description="是否取得")
approval_status: Optional[str] = Field(None, description="审批状态")
# approval_status: Optional[str] = Field(None, description="审批状态")
name: Optional[str] = Field(None, description="里程碑名称")
group: Optional[str] = Field(None, description="分组条件", examples=['KGTJLS'])
is_acquired: Optional[int] = Field(0, description="是否取得")
# is_acquired: Optional[int] = Field(0, description="是否取得")
+2 -2
View File
@@ -34,6 +34,6 @@ class ProductionDetail(BaseResponseWithID):
code: Optional[str] = Field(None, description="投产节点编码")
acquire_date: Optional[DateStr] = Field(None, description="取得时间")
is_exist: Optional[bool] = Field(None, description="是否取得")
approval_status: Optional[str] = Field(None, description="审批状态")
# approval_status: Optional[str] = Field(None, description="审批状态")
name: Optional[str] = Field(None, description="投产节点名称")
is_acquired: Optional[int] = Field(0, description="是否取得")
# is_acquired: Optional[int] = Field(0, description="是否取得")
+96 -8
View File
@@ -1,8 +1,39 @@
from pydantic import BaseModel, Field
from fastapi import HTTPException
from pydantic import BaseModel, Field, model_validator, field_validator
from typing import List, Optional
from enum import Enum
from ..common.base import BaseResponseWithID
class SupplierType(str, Enum):
"""类型枚举"""
PROJECT = "project" # 项目类型
OTHER = "other" # 其他类型
@classmethod
def supplier_list(cls):
return [cls.PROJECT.value, cls.OTHER.value]
class SupplierNature(str, Enum):
"""
性质枚举
"""
CONSTRUCTION = "construction" # 施工单位
DESIGN = "design" # 设计单位
SUPERVISORY = "supervisory" # 监理单位
SUPPLIER = "supplier" # 供应商
SURVEY = "survey" # 2026.3.11 推送系统测试时发现与项目侧 org_role 不一致,已补充勘察单位
GOVERMENT = "goverment" # 政府
@classmethod
def supplier_type_list(cls):
return [member.value for member in cls]
class SupplierStatus(str, Enum):
"""状态枚举"""
ACTIVE = "active" # 正常
INACTIVE = "inactive" # 禁用
class SupplierDetailResponse(BaseResponseWithID):
"""
供应商详情模型
@@ -17,15 +48,9 @@ class SupplierDetailResponse(BaseResponseWithID):
status: str = Field(..., description="供应商状态", examples=["active"])
sort: int = Field(..., description="排序", examples=[1])
credit_code: Optional[str] = Field(None, description="信用代码", examples=["914403007109310121"])
created_by: str = Field(..., description="创建者ID", examples=["user:001"])
created_by_name: str = Field(..., description="创建者名称", examples=["张三"])
created_at: str = Field(..., description="创建时间", examples=["2023-10-01T12:00:00Z"])
updated_by: str = Field(..., description="更新者ID", examples=["user:001"])
updated_by_name: str = Field(..., description="更新者名称", examples=["张三"])
updated_at: str = Field(..., description="更新时间", examples=["2023-10-01T12:00:00Z"])
province: Optional[str] = Field(None, description="省份编码", examples=[None])
city: Optional[str] = Field(None, description="城市编码", examples=[None])
company_nature: List[Optional[str]] = Field(None, description="供应商性质数组", examples=[['supplier']])
company_nature: List[Optional[str]] = Field([], description="供应商性质数组", examples=[['supplier']])
city_name: Optional[str] = Field(None, description="城市名称", examples=[None])
province_name: Optional[str] = Field(None, description="省份名称", examples=[None])
region_path: Optional[list] = Field(default_factory=list, description="供应商国家城市地址编码列表")
@@ -41,3 +66,66 @@ class SupplierListResponse(BaseModel):
page_size: int
total: int
list: List[SupplierDetailResponse] = []
class SupplierCreateRequest(BaseModel):
"""
供应商推送创建请求模型
"""
name: str = Field(..., description="供应商名称", examples=["腾讯集团"])
short_name: str = Field(..., description="供应商简称", examples=["腾讯"])
code: str = Field(..., description="供应商编码", examples=["TX001"])
company_type: SupplierType = Field(..., description="供应商类型", examples=["project"])
parent_id: Optional[str] = Field(None, description="上级供应商ID", examples=[None])
status: SupplierStatus = Field(..., description="供应商状态", examples=["active"])
path: Optional[List[str]] = Field(default_factory=list, description="供应商路径 [..., 上级id, 本身id]")
path_name: Optional[List[str]] = Field(default_factory=list, description="供应商路径名称 [..., 上级名称, 本身名称]")
credit_code: str = Field(..., description="供应商信用代码", examples=["914403007109310121"])
region_path: Optional[list] = Field(default_factory=list, description="供应商国家城市地址编码列表, 需对接集团侧地图组件, 没有对接可以不传递", examples=[[
"100000",
"420000",
"420800"
]])
company_nature: List[SupplierNature] = Field(..., description="供应商性质数组 可多选",
examples=[['construction',
'supplier',
'design',
'survey',
'supervisory',
'goverment']])
@model_validator(mode='after')
def validate_company_type_and_nature(self):
data = self.model_dump()
company_nature = data.get("company_nature")
if not company_nature:
raise HTTPException(detail=f"供应商:company_nature为空", status_code=400)
for cn in company_nature:
if cn not in SupplierNature.supplier_type_list():
raise HTTPException(detail=f"供应商:company_nature范围:{SupplierNature.supplier_type_list()}",
status_code=400)
company_type = data.get("company_type")
if not company_type:
raise HTTPException(detail=f"供应商:company_type为空", status_code=400)
if company_type not in SupplierType.supplier_list():
raise HTTPException(
detail=f"供应商:company_type范围({SupplierType.supplier_list()})", status_code=400)
for field in ["name", "short_name", "code", "credit_code", "status"]:
field_value = data.get(field)
if not field_value:
raise HTTPException(detail=f"字段{field}为空", status_code=400)
return self
@field_validator(
"credit_code",
mode="after"
)
def clean_credit_code(cls, field_value):
if field_value is None:
return None
import re
return re.sub(r'\s+', '', str(field_value))
+7 -6
View File
@@ -17,16 +17,17 @@ class UserListDetailItem(BaseResponseWithID):
status: Optional[str] = Field(None, description="状态")
email: Optional[EmailStr] = Field(None, description="邮箱地址")
user_type: str = Field(..., description="用户类型")
last_login_at: Optional[str] = Field(None, description="最后登录时间")
updated_at: Optional[str] = Field(None, description="更新时间")
# last_login_at: Optional[str] = Field(None, description="最后登录时间")
# updated_at: Optional[str] = Field(None, description="更新时间")
company_id: Optional[str] = Field(None, description="公司id")
company_name: Optional[str] = Field(None, description="公司名称")
created_at: Optional[str] = Field(None, description="创建时间")
approval_status: Optional[str] = Field(None, description="审批状态")
approval_time: Optional[str] = Field(None, description="审批通过时间")
last_login_ip: Optional[str] = Field(None, description="最后登录ip")
# created_at: Optional[str] = Field(None, description="创建时间")
# approval_status: Optional[str] = Field(None, description="审批状态") # 不再区分审批状态和审批时间
# approval_time: Optional[str] = Field(None, description="审批通过时间")
# last_login_ip: Optional[str] = Field(None, description="最后登录ip")
project_ids: Optional[List[str]] = Field([], description="权限内项目id")
contract_ids: Optional[List[str]] = Field([], description="权限内合同id")
phone_hash: Optional[str] = Field(None, description="手机号hash")
apply_files: Optional[List[Dict]] = Field([],
examples=[
{"id": "file:001", "name": "文件1",
+32 -35
View File
@@ -1,44 +1,41 @@
# 脚本目录
## 文件说明
本目录当前主要放两类脚本:
- 自动化测试编排脚本
- 本地数据集构建/分析脚本
### test_api_sync.py
主要的API数据同步测试脚本。
旧的 `scripts/test_api_sync.py``API_SYNC_GUIDE.md` 已不存在,不再作为当前入口。
**功能:**
- 从远程API获取所有GET/LIST接口数据
- 自动分页获取全量数据
- 保存为JSONL格式到 `remote_datasource` 目录
- 支持单项目模式或全量同步
## 当前常用入口
### 自动化测试
- `scripts/auto_test_init.py`
- 初始化 auto-test 运行环境,负责清理状态、恢复测试数据、拉起后端。
- `scripts/auto_test_clear.py`
- 清理 auto-test 运行环境。
- `scripts/run_auto_test.py`
- 运行指定 domain 的多轮自动回归并输出报告。
- `scripts/run_auto_test_all_domains.py`
- 聚合运行多个 domain 的自动回归。
- `scripts/run_large_scale_auto_test.py`
- 大规模自动回归入口。
### 数据集与分析
- `scripts/build_filtered_datasource_for_projects.py`
- 从完整 datasource 中抽取指定项目,生成可复现的精简 JSONL 数据集。
- `scripts/analyze_state_signatures.py`
- 分析状态签名与差异。
## 推荐命令
这些命令目前可直接使用:
**运行方式:**
```bash
python scripts/test_api_sync.py
python scripts/build_filtered_datasource_for_projects.py --help
python scripts/run_auto_test.py --help
```
**数据类型覆盖:**
- 基础数据: company, user, supplier, project
- 项目数据: contract, construction_task, material, contract_settlement, personnel
- 项目扩展: preparation, production, lar, units (从all_info获取)
- 明细数据: construction_task_detail, material_detail, contract_settlement_detail, personnel_detail
## 进一步说明
**配置说明:**
在文件中修改以下配置:
```python
BASE_URL = "http://localhost:8000/api/third/v2"
UID = "your-uid"
SECRET_KEY = "your-secret-key"
TARGET_PROJECT_ID = "xxx-xxx-xxx" # 单项目模式,设为None则全量同步
```
## 输出结果
数据保存在 `remote_datasource/datasource/` 目录下:
- `company_1.jsonl` - 公司数据
- `user_1.jsonl` - 用户数据
- `project_1.jsonl` - 项目数据
- ... 其他数据类型
## 更多文档
详细文档请查看项目根目录的 [API_SYNC_GUIDE.md](../API_SYNC_GUIDE.md)
- 自动化测试总览:`docs/testing_guide.md`
- 大规模自动回归方法:`docs/auto_test_spec/agent_auto_test_method.md`
+8 -8
View File
@@ -292,8 +292,8 @@ def build_temp_run_profile_for_project_auto_bind(
project_detail_push_keys: set[str] | None = None,
) -> tuple[str, list[str]]:
profile = load_yaml_config(config["pipeline"]["run_profile"])
profile.setdefault("local_datasource", {})["target_project_ids"] = [local_project_id]
profile.setdefault("remote_datasource", {})["target_project_ids"] = [remote_project_id]
profile.setdefault("local_datasource", {}).setdefault("handler_configs", {}).setdefault("project", {})["data_id_filter"] = [local_project_id]
profile.setdefault("remote_datasource", {}).setdefault("handler_configs", {}).setdefault("project", {})["data_id_filter"] = [remote_project_id]
project_strategy = profile.setdefault("strategies", {}).setdefault("project", {}).setdefault("config", {})
project_strategy["auto_bind"] = False
@@ -309,11 +309,11 @@ def build_temp_run_profile_for_project_auto_bind(
project_strategy["update_direction"] = project_update_direction or "none"
if project_detail_push_keys:
project_detail_config = profile.setdefault("strategies", {}).setdefault("project_detail_data", {}).setdefault("config", {})
overrides = dict(project_detail_config.get("field_direction_overrides", {}))
for key in project_detail_push_keys:
overrides.pop(key, None)
project_detail_config["field_direction_overrides"] = overrides
fixed_pull_keys = {"ensure_production", "climb_production", "challenge_production"}
if fixed_pull_keys.intersection(project_detail_push_keys):
project_detail_runtime = profile.setdefault("strategies", {}).setdefault("project_detail_data", {})
domain_option = project_detail_runtime.setdefault("domain_option", {})
domain_option["pull_group_production_plans"] = False
runtime_dir = Path(report_root) / "runtime_configs"
runtime_dir.mkdir(parents=True, exist_ok=True)
@@ -365,7 +365,7 @@ def run_pipeline_round(
merge_tree_contents(common_dataset_dir, paths["temp_data_dir"])
enrich_supplier_display_fields(
paths["temp_data_dir"],
f"{config['project_root']}/filtered_datasource/datasource/filtered/region_1.jsonl",
f"{config['project_root']}/working_local_datasource/filtered/region_1.jsonl",
)
normalize_contract_settlement_records(paths["temp_data_dir"])
normalize_construction_task_plan_nodes(paths["temp_data_dir"])
+56 -11
View File
@@ -15,6 +15,8 @@ from urllib import error, request
import yaml
from sync_state_machine.common.persistence import PersistenceBackend
PROJECT_ROOT = Path(__file__).resolve().parents[1]
@@ -312,26 +314,69 @@ def sqlite_rows(db_path: str | Path, sql: str, params: tuple[Any, ...] = ()) ->
return [{key: row[key] for key in row.keys()} for row in rows]
def persistence_summary(db_path: str | Path, node_type: str | None = None) -> dict[str, Any]:
def persistence_rows(
*,
backend: str,
sql: str,
db_path: str | Path | None = None,
url: str | None = None,
limit: int = 200,
) -> list[dict[str, Any]]:
_, rows = PersistenceBackend.query_records(
backend=backend,
db_path=str(db_path) if db_path is not None else None,
url=url,
sql=sql,
limit=limit,
)
return rows
def persistence_summary(
db_path: str | Path | None,
node_type: str | None = None,
*,
backend: str = "sqlite",
url: str | None = None,
) -> dict[str, Any]:
records = PersistenceBackend.records_summary(
backend=backend,
db_path=str(db_path) if db_path is not None else None,
url=url,
)
summary: dict[str, Any] = {
"backend": backend,
"db_path": str(db_path),
"db_exists": Path(db_path).exists(),
"url": url,
"db_exists": records.get("db_exists", False),
"bindings": [],
"node_status": [],
}
if not Path(db_path).exists():
if not summary["db_exists"]:
return summary
target_db_path = str(db_path) if db_path is not None else None
if node_type:
summary["bindings"] = sqlite_rows(
db_path,
"SELECT node_type, local_id, remote_id, last_updated FROM bindings WHERE node_type = ? ORDER BY last_updated DESC LIMIT 20",
(node_type,),
escaped_node_type = node_type.replace("'", "''")
summary["bindings"] = persistence_rows(
backend=backend,
db_path=target_db_path,
url=url,
sql=(
"SELECT node_type, local_id, remote_id, last_updated "
f"FROM bindings WHERE node_type = '{escaped_node_type}' ORDER BY last_updated DESC LIMIT 20"
),
)
summary["node_status"] = sqlite_rows(
db_path,
"SELECT collection_id, node_type, action, status, binding_status, COUNT(1) AS count FROM nodes WHERE node_type = ? GROUP BY collection_id, node_type, action, status, binding_status ORDER BY collection_id, action, status",
(node_type,),
summary["node_status"] = persistence_rows(
backend=backend,
db_path=target_db_path,
url=url,
sql=(
"SELECT collection_id, node_type, action, status, binding_status, COUNT(1) AS count "
f"FROM nodes WHERE node_type = '{escaped_node_type}' "
"GROUP BY collection_id, node_type, action, status, binding_status "
"ORDER BY collection_id, action, status"
),
)
return summary
@@ -1,208 +0,0 @@
"""
Run FullSyncPipeline: JSONL -> JSONL (filtered_datasource -> remote_datasource)
直接运行脚本,不使用 pytest。
"""
from __future__ import annotations
import asyncio
import sys
from pathlib import Path
from typing import List
# 确保能导入项目模块
PROJECT_ROOT = Path(__file__).resolve().parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
# 触发 DomainRegistry 注册
import sync_system_new.domain # noqa: F401, E402
from sync_system_new.common.binding import BindingManager # noqa: E402
from sync_system_new.common.collection import DataCollection # noqa: E402
from sync_system_new.common.persistence import PersistenceBackend # noqa: E402
from sync_system_new.common.registry import DomainRegistry # noqa: E402
from sync_system_new.datasource.jsonl.datasource import JsonlDataSource # noqa: E402
from sync_system_new.pipeline import FullSyncPipeline # noqa: E402
from sync_system_new.sync_system.config import OrphanAction, UpdateDirection # noqa: E402
LOCAL_DIR = PROJECT_ROOT / "filtered_datasource" / "datasource" / "ecm-2025-12-02"
REMOTE_DIR = PROJECT_ROOT / "remote_datasource" / "datasource"
PERSISTENCE_DB = PROJECT_ROOT / "test_results" / "full_sync_pipeline_jsonl_run.db"
WIPE_DB_ON_START = False
NODE_TYPES: List[str] = [
"company",
# "supplier",
"user",
"project",
"project_detail_data",
"contract",
"contract_change",
"construction_task",
"construction_task_detail",
"material",
"material_detail",
"contract_settlement",
"contract_settlement_detail",
"personnel",
"personnel_detail",
"preparation",
"production",
"lar",
"units",
]
class StrategyLogger:
"""包装策略,打印执行完成信息。"""
def __init__(self, inner):
self.inner = inner
self.node_type = inner.node_type
async def bind(self):
return await self.inner.bind()
async def create(self):
created = await self.inner.create()
print(f"✅ Strategy create: {self.node_type} (created={len(created)})")
return created
async def update(self):
updated = await self.inner.update()
print(f"✅ Strategy update: {self.node_type} (updated={len(updated)})")
return updated
async def main() -> None:
PERSISTENCE_DB.parent.mkdir(parents=True, exist_ok=True)
if WIPE_DB_ON_START and PERSISTENCE_DB.exists():
PERSISTENCE_DB.unlink()
persistence = PersistenceBackend(str(PERSISTENCE_DB))
await persistence.initialize()
local_ds = JsonlDataSource(LOCAL_DIR, read_only=True)
remote_ds = JsonlDataSource(REMOTE_DIR, read_only=False)
for node_type in NODE_TYPES:
handler_class = DomainRegistry.get_jsonl_handler(node_type)
if handler_class is None:
raise RuntimeError(f"Missing JSONL handler for node_type: {node_type}")
local_ds.register_handler(handler_class(local_ds))
remote_ds.register_handler(handler_class(remote_ds))
local_collection = DataCollection("local", persistence, auto_persist=False)
remote_collection = DataCollection("remote", persistence, auto_persist=False)
binding_manager = BindingManager(persistence, auto_persist=False)
strategies = []
for node_type in NODE_TYPES:
strategy_class = DomainRegistry.get_strategy_class(node_type)
if strategy_class is None:
raise RuntimeError(f"Missing strategy for node_type: {node_type}")
strategy = strategy_class(node_type, local_collection, remote_collection, binding_manager)
# 运行时强制只推不拉(不覆盖原有 depend_fields/auto_bind_fields
strategy.update_config(
local_orphan_action=OrphanAction.CREATE_REMOTE,
remote_orphan_action=OrphanAction.NONE,
update_direction=UpdateDirection.PUSH,
)
if node_type == "contract":
# contract 仅依赖 project_id,避免 company_id 为空导致依赖阻断
strategy.update_config(depend_fields={"project_id": "project"})
strategies.append(StrategyLogger(strategy))
pipeline = FullSyncPipeline(
local_datasource=local_ds,
remote_datasource=remote_ds,
strategies=strategies,
node_types=NODE_TYPES,
load_order=NODE_TYPES,
sync_order=NODE_TYPES,
persistence=persistence,
local_collection=local_collection,
remote_collection=remote_collection,
binding_manager=binding_manager,
enable_persistence=True,
fail_fast_on_precheck=False,
)
stats = await pipeline.run()
print(f"Pipeline finished. Stats: {stats}")
binding_counts = {}
for node_type in NODE_TYPES:
records = await binding_manager.get_all_records(node_type)
binding_counts[node_type] = len(records)
def format_action(summary, key: str) -> str:
action = summary.get(key, {})
return f"{action.get('total', 0)}({action.get('success', 0)}/{action.get('failed', 0)})"
def format_total(summary) -> str:
total = summary.get("total", {})
return f"{total.get('total', 0)}({total.get('success', 0)}/{total.get('failed', 0)})"
def print_summary(title: str, action_summary: dict) -> None:
print(f"\n=== {title} summary ===")
print("doc_type: bindings | create(s/f) | update(s/f) | delete(s/f) | none | total(s/f)")
total_bindings = 0
total_create = {"total": 0, "success": 0, "failed": 0}
total_update = {"total": 0, "success": 0, "failed": 0}
total_delete = {"total": 0, "success": 0, "failed": 0}
total_none = 0
total_all = {"total": 0, "success": 0, "failed": 0}
for node_type in NODE_TYPES:
summary = action_summary.get(node_type, {})
bindings = binding_counts.get(node_type, 0)
create_str = format_action(summary, "create")
update_str = format_action(summary, "update")
delete_str = format_action(summary, "delete")
none_count = summary.get("none", {}).get("total", 0)
total_str = format_total(summary)
print(
f"{node_type}: bindings={bindings} "
f"create={create_str} update={update_str} delete={delete_str} "
f"none={none_count} total={total_str}"
)
total_bindings += bindings
for bucket, key in (
(total_create, "create"),
(total_update, "update"),
(total_delete, "delete"),
):
action = summary.get(key, {})
bucket["total"] += action.get("total", 0)
bucket["success"] += action.get("success", 0)
bucket["failed"] += action.get("failed", 0)
total_none += none_count
total_all["total"] += summary.get("total", {}).get("total", 0)
total_all["success"] += summary.get("total", {}).get("success", 0)
total_all["failed"] += summary.get("total", {}).get("failed", 0)
print(
"TOTAL: "
f"bindings={total_bindings} "
f"create={total_create['total']}({total_create['success']}/{total_create['failed']}) "
f"update={total_update['total']}({total_update['success']}/{total_update['failed']}) "
f"delete={total_delete['total']}({total_delete['success']}/{total_delete['failed']}) "
f"none={total_none} "
f"total={total_all['total']}({total_all['success']}/{total_all['failed']})"
)
print_summary("Local", local_ds.get_action_summary())
print_summary("Remote", remote_ds.get_action_summary())
await persistence.close()
if __name__ == "__main__":
asyncio.run(main())
@@ -1,153 +0,0 @@
"""
run_full_sync_pipeline_jsonl_sm.py
独立执行:基于 sync_state_machine 的 JSONL -> JSONL 全量同步脚本。
用途:纯本地可复现,不依赖远端 API;用于验证重构后 pipeline 主流程可跑通。
"""
from __future__ import annotations
import asyncio
import os
import sys
from datetime import datetime
from pathlib import Path
from typing import Dict, List
from pydantic import BaseModel, ConfigDict, Field
PROJECT_ROOT = Path(__file__).resolve().parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
import sync_state_machine.domain # noqa: F401, E402
from sync_state_machine.pipeline import create_jsonl_to_jsonl_pipeline # noqa: E402
class JsonlSmConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
local_jsonl_dir: str = str(PROJECT_ROOT / "filtered_datasource" / "datasource" / "ecm-2025-12-02")
remote_jsonl_dir: str = str(PROJECT_ROOT / "remote_datasource" / "datasource")
persistence_db: str = str(PROJECT_ROOT / "test_results" / "sync_state_machine" / "full_sync_pipeline_jsonl_sm.db")
wipe_persistence_on_start: bool = False
target_project_ids: List[str] = Field(default_factory=lambda: ["f3e02e84-e81c-4aa6-88d4-f5aa108c8227"])
project_auto_bind_fields: List[str] = Field(default_factory=lambda: ["name"])
project_id_bindings: Dict[str, str] = Field(default_factory=dict)
node_types: List[str] = Field(default_factory=lambda: [
"company",
"user",
"project",
"contract",
"contract_change",
"construction_task",
"construction_task_detail",
"material",
"material_detail",
"contract_settlement",
"contract_settlement_detail",
"personnel",
"personnel_detail",
"preparation",
"production",
"lar",
"units",
])
def _load_dotenv() -> Dict[str, str]:
dotenv_path = PROJECT_ROOT / ".env"
if not dotenv_path.exists():
return {}
values: Dict[str, str] = {}
for raw_line in dotenv_path.read_text(encoding="utf-8").splitlines():
line = raw_line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
values[key.strip()] = value.strip().strip('"').strip("'")
return values
def build_config() -> JsonlSmConfig:
cfg = JsonlSmConfig()
env = _load_dotenv()
cfg.wipe_persistence_on_start = (
os.getenv("SPS_JSONL_WIPE_DB", env.get("SPS_JSONL_WIPE_DB", str(cfg.wipe_persistence_on_start))).lower()
in {"1", "true", "yes"}
)
return cfg
def _print_config_snapshot(config: JsonlSmConfig) -> None:
snapshot = {
"local_jsonl_dir": config.local_jsonl_dir,
"remote_jsonl_dir": config.remote_jsonl_dir,
"persistence_db": config.persistence_db,
"wipe_persistence_on_start": config.wipe_persistence_on_start,
"target_project_ids": config.target_project_ids,
"project_auto_bind_fields": config.project_auto_bind_fields,
"project_id_bindings": config.project_id_bindings,
"node_types": config.node_types,
}
print(f"🔧 Resolved Config: {snapshot}")
async def main() -> int:
pipeline = None
log_path = None
config = build_config()
try:
log_dir = PROJECT_ROOT / "test_results" / "sync_state_machine"
log_dir.mkdir(parents=True, exist_ok=True)
log_path = log_dir / f"jsonl_pipeline_sm_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
print("\n" + "=" * 80)
print("🚀 Creating sync_state_machine JSONL -> JSONL pipeline")
print(f"📝 Log file: {log_path}")
_print_config_snapshot(config)
print("=" * 80)
Path(config.persistence_db).parent.mkdir(parents=True, exist_ok=True)
pipeline = await create_jsonl_to_jsonl_pipeline(
local_jsonl_dir=config.local_jsonl_dir,
remote_jsonl_dir=config.remote_jsonl_dir,
persistence_db=config.persistence_db,
node_types=config.node_types,
load_order=config.node_types,
sync_order=config.node_types,
enable_persistence=True,
wipe_persistence_on_start=config.wipe_persistence_on_start,
target_project_ids=config.target_project_ids,
project_id_bindings=config.project_id_bindings,
project_auto_bind_fields=config.project_auto_bind_fields,
enable_project_bind_override=True,
enable_project_filter_override=True,
initialize_logger=True,
logger_file_path=str(log_path),
)
stats = await pipeline.run()
print(f"\n✅ Pipeline completed: stats={stats}")
print(f"🗃️ Persistence DB: {config.persistence_db}")
return 0
except KeyboardInterrupt:
print("\n⚠️ Interrupted by user")
return 130
except Exception as exc:
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
import traceback
traceback.print_exc()
return 1
finally:
if pipeline is not None:
await pipeline.close()
if __name__ == "__main__":
raise SystemExit(asyncio.run(main()))
@@ -1,179 +0,0 @@
"""
run_full_sync_pipeline_simple.py
极简测试脚本:演示使用工厂函数创建 Pipeline
- 只需配置参数
- 调用工厂函数
- 执行 pipeline.run()
"""
import asyncio
import sys
from pathlib import Path
# 项目路径
PROJECT_ROOT = Path(__file__).resolve().parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
# 触发 Domain 注册
import sync_system_new.domain # noqa: F401, E402
from sync_system_new.pipeline import create_jsonl_to_api_pipeline # noqa: E402
# ========== 配置参数 ==========
CONFIG = {
# 路径
"local_jsonl_dir": str(PROJECT_ROOT / "filtered_datasource" / "datasource" / "ecm-2025-12-02"),
"persistence_db": str(PROJECT_ROOT / "_runtime" / "test_simple.db"),
# API
"api_base_url": "http://localhost:8000/api/third/v2",
"api_uid": "6b18c681-92d5-4c7f-802c-df413b6504c8",
"api_secret": "1a28Az5dBD1Gc3URpmHwiR0XfzW7UtScQnqJQzgpzdg",
# 节点类型
"node_types": [
"company",
"supplier",
# "user",
"project",
"project_detail_data",
"preparation",
"production",
"lar",
"lar_change",
"units",
"contract",
"contract_change",
"construction_task",
"construction_task_detail",
"material",
"material_detail",
"contract_settlement",
"contract_settlement_detail",
"personnel",
"personnel_detail",
],
# 项目过滤和预绑定
"target_project_ids": ["f3e02e84-e81c-4aa6-88d4-f5aa108c8227"],
"project_id_bindings": {
"f3e02e84-e81c-4aa6-88d4-f5aa108c8227": "f3e02e84-e81c-4aa6-88d4-f5aa108c8227",
},
# Handler 自定义配置
"handler_configs": {
# supplier: 从持久化加载,不调用 API(后续运行时启用)
"supplier": {"load_mode": "persisted_only"},
},
# Skip/Force Sync 控制
# skip_sync_types: 跳过同步(只执行 bind,不执行 create/update
# 后续运行时可配置: "skip_sync_types": ["supplier"]
"skip_sync_types": ["supplier"],
# 行为
"wipe_persistence_on_start": False, # 改为 False,否则持久化数据会被清空
"enable_persistence": True,
# API 调试
"api_debug": True,
"poll_max_retries": 1,
"poll_interval": 2.0,
}
async def main():
"""主函数 - 返回 True 表示成功,False 表示被中断"""
from datetime import datetime
import sys
from pathlib import Path
pipeline = None
interrupted = False
log_file = None
original_stdout = None
original_stderr = None
# 日志设置
log_dir = Path(PROJECT_ROOT) / "test_results"
log_dir.mkdir(parents=True, exist_ok=True)
log_path = log_dir / f"simple_pipeline_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
try:
# 重定向输出到日志文件
log_file = open(log_path, "w", encoding="utf-8")
class Tee:
def __init__(self, *streams):
self._streams = streams
def write(self, data):
for stream in self._streams:
stream.write(data)
stream.flush()
return len(data)
def flush(self):
for stream in self._streams:
stream.flush()
original_stdout = sys.stdout
original_stderr = sys.stderr
sys.stdout = Tee(original_stdout, log_file)
sys.stderr = Tee(original_stderr, log_file)
print("\n" + "=" * 60)
print("🚀 Creating Pipeline...")
print(f"📝 Log file: {log_path}")
print("=" * 60)
# 一行创建完整 Pipeline
pipeline = await create_jsonl_to_api_pipeline(**CONFIG)
print(f"✅ Pipeline created!")
print(f" - Node types: {len(pipeline.node_types)}")
print(f" - Strategies: {len(pipeline.strategies)}")
print("\n" + "=" * 60)
print("🏃 Running Pipeline...")
print("=" * 60)
# 执行同步(print_summary 已整合到 stage4_post_check 中)
stats = await pipeline.run()
print("\n✅ Done!")
except KeyboardInterrupt:
print("\n\n⚠️ KeyboardInterrupt received, shutting down gracefully...")
interrupted = True
except Exception as e:
print(f"\n\n❌ Error: {type(e).__name__}: {e}")
import traceback
traceback.print_exc()
finally:
if pipeline is not None:
print("🧹 Cleaning up resources...")
await pipeline.close()
print("✅ Resources cleaned up")
# 恢复原始输出
if original_stdout is not None:
sys.stdout = original_stdout
if original_stderr is not None:
sys.stderr = original_stderr
if log_file is not None:
log_file.close()
if not interrupted:
print(f"\n📝 Log saved to: {log_path}")
return not interrupted # 返回 False 表示被中断
if __name__ == "__main__":
try:
success = asyncio.run(main())
sys.exit(0 if success else 130) # 130 是标准的 Ctrl+C 退出码
except KeyboardInterrupt:
# asyncio.run() 可能会重新抛出 KeyboardInterrupt
print("\n⚠️ Program interrupted")
sys.exit(130)
@@ -1,143 +0,0 @@
"""
run_full_sync_pipeline_simple_sm.py
独立执行:基于 sync_state_machine 的 JSONL -> API Pipeline。
用途:保留线上接口参数,直接执行重构后的推送链路(非 pytest)。
"""
from __future__ import annotations
import asyncio
import os
import sys
from datetime import datetime
from pathlib import Path
from typing import Dict, List
from pydantic import BaseModel, ConfigDict, Field
PROJECT_ROOT = Path(__file__).resolve().parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
import sync_state_machine.domain # noqa: F401, E402
from sync_state_machine.pipeline import create_jsonl_to_api_pipeline # noqa: E402
class SimpleSmConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
local_jsonl_dir: str = str(PROJECT_ROOT / "filtered_datasource" / "datasource" / "ecm-2025-12-02")
persistence_db: str = str(PROJECT_ROOT / "test_results" / "sync_state_machine" / "simple_pipeline_sm.db")
api_base_url: str = "http://localhost:8000/api/third/v2"
api_uid: str = "6b18c681-92d5-4c7f-802c-df413b6504c8"
api_secret: str = "1a28Az5dBD1Gc3URpmHwiR0XfzW7UtScQnqJQzgpzdg"
node_types: List[str] = Field(default_factory=lambda: [
"project",
"contract",
"contract_settlement",
])
target_project_ids: List[str] = Field(default_factory=lambda: ["f3e02e84-e81c-4aa6-88d4-f5aa108c8227"])
project_id_bindings: Dict[str, str] = Field(
default_factory=lambda: {
"f3e02e84-e81c-4aa6-88d4-f5aa108c8227": "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
}
)
skip_sync_types: List[str] = Field(default_factory=lambda: ["supplier"])
wipe_persistence_on_start: bool = False
enable_persistence: bool = True
api_debug: bool = True
poll_max_retries: int = 1
poll_interval: float = 2.0
def _load_dotenv() -> Dict[str, str]:
dotenv_path = PROJECT_ROOT / ".env"
if not dotenv_path.exists():
return {}
values: Dict[str, str] = {}
for raw_line in dotenv_path.read_text(encoding="utf-8").splitlines():
line = raw_line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
values[key.strip()] = value.strip().strip('"').strip("'")
return values
def build_config() -> SimpleSmConfig:
cfg = SimpleSmConfig()
env = _load_dotenv()
cfg.api_base_url = os.getenv("SPS_API_BASE_URL", env.get("SPS_API_BASE_URL", cfg.api_base_url))
cfg.api_uid = os.getenv("SPS_API_UID", env.get("SPS_API_UID", cfg.api_uid))
cfg.api_secret = os.getenv("SPS_API_SECRET", env.get("SPS_API_SECRET", cfg.api_secret))
return cfg
def _print_config_snapshot(config: SimpleSmConfig) -> None:
masked_secret = "***" if config.api_secret else ""
snapshot = {
"local_jsonl_dir": config.local_jsonl_dir,
"persistence_db": config.persistence_db,
"api_base_url": config.api_base_url,
"api_uid": config.api_uid,
"api_secret": masked_secret,
"node_types": config.node_types,
"target_project_ids": config.target_project_ids,
"project_id_bindings": config.project_id_bindings,
"skip_sync_types": config.skip_sync_types,
"wipe_persistence_on_start": config.wipe_persistence_on_start,
"enable_persistence": config.enable_persistence,
"api_debug": config.api_debug,
"poll_max_retries": config.poll_max_retries,
"poll_interval": config.poll_interval,
}
print(f"🔧 Resolved Config: {snapshot}")
async def main() -> int:
pipeline = None
log_path = None
config = build_config()
try:
log_dir = PROJECT_ROOT / "test_results" / "sync_state_machine"
log_dir.mkdir(parents=True, exist_ok=True)
log_path = log_dir / f"simple_pipeline_sm_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
print("\n" + "=" * 80)
print("🚀 Creating sync_state_machine JSONL -> API pipeline")
print(f"📝 Log file: {log_path}")
_print_config_snapshot(config)
print("=" * 80)
pipeline = await create_jsonl_to_api_pipeline(
**config.model_dump(),
initialize_logger=True,
logger_file_path=str(log_path),
)
stats = await pipeline.run()
print(f"\n✅ Pipeline completed: stats={stats}")
print(f"🗃️ Persistence DB: {config.persistence_db}")
return 0
except KeyboardInterrupt:
print("\n⚠️ Interrupted by user")
return 130
except Exception as exc:
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
import traceback
traceback.print_exc()
return 1
finally:
if pipeline is not None:
await pipeline.close()
if __name__ == "__main__":
raise SystemExit(asyncio.run(main()))
+6 -5
View File
@@ -15,6 +15,7 @@ if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
import sync_state_machine.domain # noqa: F401, E402
from sync_state_machine.logging import configure_app_logging, get_logger # noqa: E402
from sync_state_machine.pipeline import ( # noqa: E402
build_config,
run_pipeline_from_config,
@@ -23,15 +24,17 @@ from sync_state_machine.config import load_named_preset_overrides # noqa: E402
PRESET_NAME = "jsonl_sm"
logger = get_logger(__name__)
async def main() -> int:
try:
overrides, preset_file, used_default = load_named_preset_overrides(PROJECT_ROOT, PRESET_NAME)
source_label = "default" if used_default else "custom"
print(f"🧩 Loaded preset ({source_label}): {preset_file}")
config = build_config(project_root=PROJECT_ROOT, overrides=overrides)
configure_app_logging(config.logging, replace_handlers=True)
logger.info("🧩 Loaded preset (%s): %s", source_label, preset_file)
await run_pipeline_from_config(
config,
title="sync_state_machine JSONL -> JSONL pipeline",
@@ -40,12 +43,10 @@ async def main() -> int:
return 0
except KeyboardInterrupt:
print("\n⚠️ Interrupted by user")
logger.warning("⚠️ Interrupted by user")
return 130
except Exception as exc:
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
import traceback
traceback.print_exc()
logger.exception("❌ Pipeline failed: %s: %s", type(exc).__name__, exc)
return 1
+6 -5
View File
@@ -15,6 +15,7 @@ if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
import sync_state_machine.domain # noqa: F401, E402
from sync_state_machine.logging import configure_app_logging, get_logger # noqa: E402
from sync_state_machine.pipeline import ( # noqa: E402
build_config,
run_pipeline_from_config,
@@ -23,15 +24,17 @@ from sync_state_machine.config import load_named_preset_overrides # noqa: E402
PRESET_NAME = "simple_sm"
logger = get_logger(__name__)
async def main() -> int:
try:
overrides, preset_file, used_default = load_named_preset_overrides(PROJECT_ROOT, PRESET_NAME)
source_label = "default" if used_default else "custom"
print(f"🧩 Loaded preset ({source_label}): {preset_file}")
config = build_config(project_root=PROJECT_ROOT, overrides=overrides)
configure_app_logging(config.logging, replace_handlers=True)
logger.info("🧩 Loaded preset (%s): %s", source_label, preset_file)
await run_pipeline_from_config(
config,
title="sync_state_machine JSONL -> API pipeline",
@@ -40,12 +43,10 @@ async def main() -> int:
return 0
except KeyboardInterrupt:
print("\n⚠️ Interrupted by user")
logger.warning("⚠️ Interrupted by user")
return 130
except Exception as exc:
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
import traceback
traceback.print_exc()
logger.exception("❌ Pipeline failed: %s: %s", type(exc).__name__, exc)
return 1
+13 -7
View File
@@ -15,22 +15,25 @@ from .common.registry import DomainRegistry
from .config import (
ApiDataSourceConfig,
BaseDataSourceConfig,
DbDataSourceConfig,
JsonlDataSourceConfig,
PipelineRunConfig,
build_config,
build_config_from_file,
build_default_config,
get_datasource_config_model,
get_datasource_factory,
list_registered_datasource_types,
load_named_preset_overrides,
load_overrides_from_file,
MultiProjectRunConfig,
build_multi_project_config_from_file,
is_multi_project_payload,
register_datasource_config_model,
register_datasource_factory,
register_datasource_type,
)
from .datasource import ApiClient, ApiDataSource, BaseApiHandler, BaseDbHandler, BaseJsonlHandler, DbDataSource, JsonlDataSource
from .pipeline import FullSyncPipeline, create_pipeline_from_config, run_pipeline_from_config
from .datasource import ApiClient, ApiDataSource, BaseApiHandler, BaseJsonlHandler, JsonlDataSource
from .pipeline import FullSyncPipeline, MultiProjectPipeline, create_pipeline_from_config, run_pipeline_from_config, run_profile_from_file
try:
__version__ = version("ecm-sync-system")
@@ -40,7 +43,7 @@ except PackageNotFoundError:
def ensure_domain_registry_loaded() -> None:
"""Ensure built-in domain registrations are imported."""
_domain # keep module import for side effects
_ = _domain
__all__ = [
@@ -50,26 +53,29 @@ __all__ = [
"PipelineRunConfig",
"ApiDataSourceConfig",
"BaseDataSourceConfig",
"DbDataSourceConfig",
"JsonlDataSourceConfig",
"build_config",
"build_config_from_file",
"build_default_config",
"get_datasource_config_model",
"get_datasource_factory",
"list_registered_datasource_types",
"load_overrides_from_file",
"load_named_preset_overrides",
"MultiProjectRunConfig",
"build_multi_project_config_from_file",
"is_multi_project_payload",
"register_datasource_config_model",
"register_datasource_factory",
"register_datasource_type",
"ApiClient",
"ApiDataSource",
"DbDataSource",
"JsonlDataSource",
"BaseApiHandler",
"BaseDbHandler",
"BaseJsonlHandler",
"FullSyncPipeline",
"MultiProjectPipeline",
"create_pipeline_from_config",
"run_pipeline_from_config",
"run_profile_from_file",
]
+12 -12
View File
@@ -5,8 +5,11 @@ import asyncio
import logging
from pathlib import Path
from .config import build_config, load_overrides_from_file
from .pipeline import run_pipeline_from_config
from .logging import get_logger, init_app_logger
from .pipeline import run_profile_from_file
logger = get_logger(__name__)
def package_project_root() -> Path:
@@ -21,6 +24,7 @@ def resolve_config_path(raw_path: str, *, project_root: Path) -> Path:
async def _main() -> int:
init_app_logger(replace_handlers=True)
project_root = package_project_root()
parser = argparse.ArgumentParser(description="Run sync pipeline from config override file")
@@ -35,14 +39,10 @@ async def _main() -> int:
if not cfg_path.exists() or not cfg_path.is_file():
raise FileNotFoundError(f"Config file not found: {cfg_path}")
overrides = load_overrides_from_file(cfg_path, project_root)
config = build_config(project_root=project_root, overrides=overrides)
mode = f"{config.local_datasource.type}->{config.remote_datasource.type}"
print(f"🧩 Loaded config file: {cfg_path}")
await run_pipeline_from_config(
config,
title=f"sync_state_machine pipeline ({mode})",
logger.info("🧩 Loaded config file: %s", cfg_path)
await run_profile_from_file(
project_root=project_root,
config_path=cfg_path,
print_summary=True,
)
return 0
@@ -52,10 +52,10 @@ def main() -> int:
try:
return asyncio.run(_main())
except KeyboardInterrupt:
print("\n⚠️ Interrupted by user")
logger.warning("⚠️ Interrupted by user")
return 130
except Exception as exc:
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
logger.exception("❌ Pipeline failed: %s: %s", type(exc).__name__, exc)
return 1
finally:
logging.shutdown()
+12 -1
View File
@@ -1,6 +1,12 @@
from .types import SyncAction, SyncStatus, BindingStatus
from .sync_node import SyncNode
from .persistence import PersistenceBackend
from .persistence import (
PersistenceBackend,
create_persistence_backend,
get_registered_persistence_backends,
register_persistence_backend,
)
from .persistence_backends import SQLAlchemyPersistenceBackend, SQLitePersistenceBackend
from .collection import DataCollection
from .binding import BindingManager
@@ -10,6 +16,11 @@ __all__ = [
"BindingStatus",
"SyncNode",
"PersistenceBackend",
"SQLitePersistenceBackend",
"SQLAlchemyPersistenceBackend",
"create_persistence_backend",
"register_persistence_backend",
"get_registered_persistence_backends",
"DataCollection",
"BindingManager",
]
+114 -12
View File
@@ -1,10 +1,13 @@
import json
import logging
from typing import Optional, Dict, Any, List
from ..logging import get_logger, get_scope_display_name
from .persistence import PersistenceBackend
from .persistence_service import ScopedPersistenceView
from .types import BindingStatus, BindingRecord
logger = logging.getLogger(__name__)
logger = get_logger(__name__)
class BindingManager:
@@ -12,13 +15,53 @@ class BindingManager:
BindingManager 负责维护不同 DataCollection 间节点的绑定关系。
所有查询均在内存中进行,持久化仅作为备份。
"""
def __init__(self, persistence: PersistenceBackend, auto_persist: bool = True):
def __init__(
self,
persistence: PersistenceBackend | ScopedPersistenceView,
auto_persist: bool = False,
scope: str = "global",
global_fallback_manager: Optional["BindingManager"] = None,
):
self.persistence = persistence
self.auto_persist = auto_persist
self.scope = scope
self._global_fallback_manager = global_fallback_manager
# { node_type: { local_id: remote_id } }
self._bindings: Dict[str, Dict[str, Optional[str]]] = {}
self._deleted: Dict[str, set[str]] = {}
def set_global_fallback_manager(self, manager: Optional["BindingManager"]) -> None:
self._global_fallback_manager = manager
async def _persistence_save_binding(self, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.save_binding(node_type, local_id, payload_dict)
return
await self.persistence.save_binding(self.scope, node_type, local_id, payload_dict)
async def _persistence_delete_binding(self, node_type: str, local_id: str) -> None:
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.delete_binding(node_type, local_id)
return
await self.persistence.delete_binding(self.scope, node_type, local_id)
async def _persistence_delete_bindings_bulk(self, node_type: str, local_ids: List[str]) -> None:
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.delete_bindings_bulk(node_type, local_ids)
return
await self.persistence.delete_bindings_bulk(self.scope, node_type, local_ids)
async def _persistence_save_bindings_bulk(self, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.save_bindings_bulk(node_type, bindings)
return
await self.persistence.save_bindings_bulk(self.scope, node_type, bindings)
async def _persistence_load_bindings(self, node_type: str) -> List[Dict[str, Any]]:
if isinstance(self.persistence, ScopedPersistenceView):
return await self.persistence.load_bindings(node_type)
return await self.persistence.load_bindings(self.scope, node_type)
def _get_type_bindings(self, node_type: str) -> Dict[str, Optional[str]]:
if node_type not in self._bindings:
self._bindings[node_type] = {}
@@ -55,7 +98,7 @@ class BindingManager:
logger.debug(log_line)
if self.auto_persist:
await self.persistence.save_binding(node_type, local_id, {"remote_id": remote_id})
await self._persistence_save_binding(node_type, local_id, {"remote_id": remote_id})
async def unbind(self, node_type: str, local_id: str, manual: bool = False):
"""
@@ -80,14 +123,19 @@ class BindingManager:
)
if self.auto_persist:
await self.persistence.delete_binding(node_type, local_id)
await self._persistence_delete_binding(node_type, local_id)
else:
self._get_deleted(node_type).add(local_id)
async def get_remote_id(self, node_type: str, local_id: str) -> Optional[str]:
"""查询本地节点对应的远程 ID"""
bindings = self._get_type_bindings(node_type)
return bindings.get(local_id)
remote_id = bindings.get(local_id)
if remote_id is not None or local_id in bindings:
return remote_id
if self._global_fallback_manager is None:
return None
return await self._global_fallback_manager.get_remote_id(node_type, local_id)
async def get_local_id(self, node_type: str, remote_id: str) -> Optional[str]:
"""查询远程节点对应的本地 ID (内存遍历)"""
@@ -95,11 +143,13 @@ class BindingManager:
for lid, rid in bindings.items():
if rid == remote_id:
return lid
return None
if self._global_fallback_manager is None:
return None
return await self._global_fallback_manager.get_local_id(node_type, remote_id)
async def get_record(self, node_type: str, local_id: str) -> Optional[BindingRecord]:
"""获取结构化绑定记录"""
remote_id = self._get_type_bindings(node_type).get(local_id)
remote_id = await self.get_remote_id(node_type, local_id)
if remote_id is None:
return None
return BindingRecord(
@@ -119,25 +169,71 @@ class BindingManager:
async def load_from_persistence(self, node_type: str):
"""从持久化层加载特定类型的绑定关系到内存"""
raw_bindings = await self.persistence.load_bindings(node_type)
raw_bindings = await self._persistence_load_bindings(node_type)
bindings = self._get_type_bindings(node_type)
bindings.clear()
self._get_deleted(node_type).clear()
for rb in raw_bindings:
payload = rb.get("payload") or {}
bindings[rb["local_id"]] = payload.get("remote_id")
async def persist(self) -> None:
def import_bindings(self, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
"""导入既有绑定到当前 manager,仅写内存,不触发持久化。"""
target = self._get_type_bindings(node_type)
target.clear()
target.update(dict(bindings))
self._get_deleted(node_type).clear()
async def prune_missing_nodes(
self,
*,
local_collection,
remote_collection,
node_types: Optional[List[str]] = None,
) -> int:
target_types = list(node_types) if node_types is not None else list(self._bindings.keys())
deleted_count = 0
for node_type in target_types:
bindings = self._get_type_bindings(node_type)
if not bindings:
continue
valid_local_ids = {node.node_id for node in local_collection.filter(node_type=node_type)}
valid_remote_ids = {node.node_id for node in remote_collection.filter(node_type=node_type)}
stale_local_ids = [
local_id
for local_id, remote_id in bindings.items()
if local_id not in valid_local_ids or (remote_id is not None and remote_id not in valid_remote_ids)
]
for local_id in stale_local_ids:
del bindings[local_id]
deleted_count += 1
if self.auto_persist:
await self._persistence_delete_binding(node_type, local_id)
else:
self._get_deleted(node_type).add(local_id)
return deleted_count
async def persist(self, *, exclude_node_types: Optional[List[str]] = None) -> None:
"""将所有绑定关系写回持久化层"""
if not self._bindings and not self._deleted:
logger.info("🔍 [BindingManager.persist] 跳过:无绑定变更")
return
excluded = set(node_type for node_type in (exclude_node_types or []) if node_type)
type_count = 0
saved_total = 0
deleted_total = 0
touched_types: List[str] = []
for node_type, bindings in self._bindings.items():
if node_type in excluded:
continue
deleted = list(self._get_deleted(node_type))
binding_count = len(bindings)
deleted_count = len(deleted)
@@ -149,13 +245,19 @@ class BindingManager:
touched_types.append(node_type)
if deleted:
await self.persistence.delete_bindings_bulk(node_type, deleted)
await self._persistence_delete_bindings_bulk(node_type, deleted)
self._get_deleted(node_type).clear()
if bindings:
await self.persistence.save_bindings_bulk(node_type, bindings)
await self._persistence_save_bindings_bulk(node_type, bindings)
touched_text = ",".join(touched_types) if touched_types else "none"
logger.info(
f"🔍 [BindingManager.persist] 完成: types={type_count}, saved={saved_total}, deleted={deleted_total}, touched=[{touched_text}]"
f"🔍 [BindingManager.persist] 完成: scope={get_scope_display_name(self.scope)}, types={type_count}, saved={saved_total}, deleted={deleted_total}, touched=[{touched_text}]"
)
async def unload(self, *, node_types: Optional[List[str]] = None) -> None:
target_types = list(node_types) if node_types is not None else list(self._bindings.keys())
for node_type in target_types:
self._bindings.pop(node_type, None)
self._deleted.pop(node_type, None)
+281 -27
View File
@@ -1,9 +1,11 @@
import uuid
import copy
from enum import Enum
from typing import Dict, List, Optional, Any, Set, Callable, Literal
from .sync_node import SyncNode
from .types import SyncStatus
from .persistence import PersistenceBackend
from .persistence_service import ScopedPersistenceView
class DataCollection:
@@ -22,17 +24,23 @@ class DataCollection:
def __init__(
self,
collection_id: str,
persistence: Optional[PersistenceBackend] = None,
scope: str = "global",
persistence: Optional[PersistenceBackend | ScopedPersistenceView] = None,
auto_persist: bool = False,
sm_runtime: Optional[Any] = None,
global_fallback_collection: Optional["DataCollection"] = None,
):
self.collection_id = collection_id
self.scope = scope
self.persistence = persistence
self.auto_persist = auto_persist
self._sm_runtime = sm_runtime
self._global_fallback_collection = global_fallback_collection
self._nodes: Dict[str, SyncNode] = {} # 内存缓存
# data_id 全局唯一(按 collection 维度),用于 O(1) 反查 node_id
self._data_id_to_node_id: Dict[str, str] = {}
# node_id -> data_id 反向索引,用于 O(1) 维护 data_id 更新
self._node_id_to_data_id: Dict[str, str] = {}
# auto_persist=False 时,delete() 不会立即落库;这里记录删除集合,persist() 时统一写回
self._deleted_node_ids: set[str] = set()
@@ -48,6 +56,83 @@ class DataCollection:
"""设置 Collection 默认状态机 runtime。"""
self._sm_runtime = runtime
def set_global_fallback_collection(self, collection: Optional["DataCollection"]) -> None:
self._global_fallback_collection = collection
def _get_global_fallback(self) -> Optional["DataCollection"]:
return self._global_fallback_collection
async def _persistence_save_node(self, node: SyncNode) -> None:
if not self.persistence:
return
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.save_node(self.collection_id, node)
return
await self.persistence.save_node(self.collection_id, self.scope, node)
async def _persistence_delete_node(self, node_id: str) -> None:
if not self.persistence:
return
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.delete_node(self.collection_id, node_id)
return
await self.persistence.delete_node(self.collection_id, self.scope, node_id)
async def _persistence_delete_nodes_bulk(self, node_ids: List[str]) -> None:
if not self.persistence:
return
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.delete_nodes_bulk(self.collection_id, node_ids)
return
await self.persistence.delete_nodes_bulk(self.collection_id, self.scope, node_ids)
async def _persistence_save_nodes_bulk(self, nodes: List[SyncNode]) -> None:
if not self.persistence:
return
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.save_nodes_bulk(self.collection_id, nodes)
return
await self.persistence.save_nodes_bulk(self.collection_id, self.scope, nodes)
async def _persistence_load_nodes(
self,
*,
exclude_node_types: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
if not self.persistence:
return []
if isinstance(self.persistence, ScopedPersistenceView):
return await self.persistence.load_nodes(
self.collection_id,
exclude_node_types=exclude_node_types,
)
return await self.persistence.load_nodes(
self.collection_id,
self.scope,
exclude_node_types=exclude_node_types,
)
async def _persistence_load_bind_data_id_overrides(self, node_type: str) -> Dict[str, Optional[str]]:
if not self.persistence:
return {}
if isinstance(self.persistence, ScopedPersistenceView):
return await self.persistence.load_bind_data_id_overrides(node_type)
return await self.persistence.load_bind_data_id_overrides(node_type, self.scope)
def _remove_data_id_index(self, node_id: str, data_id: Optional[str] = None) -> None:
target_data_id = data_id
if target_data_id is None:
target_data_id = self._node_id_to_data_id.pop(node_id, None)
else:
self._node_id_to_data_id.pop(node_id, None)
if target_data_id and self._data_id_to_node_id.get(target_data_id) == node_id:
self._data_id_to_node_id.pop(target_data_id, None)
def _set_data_id_index(self, node_id: str, data_id: str) -> None:
self._data_id_to_node_id[data_id] = node_id
self._node_id_to_data_id[node_id] = data_id
async def add(self, node: SyncNode):
# 检查 node_id 唯一性
if node.node_id in self._nodes:
@@ -62,18 +147,17 @@ class DataCollection:
raise ValueError(
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
)
self._data_id_to_node_id[node.data_id] = node.node_id
self._set_data_id_index(node.node_id, node.data_id)
self._nodes[node.node_id] = node
self._deleted_node_ids.discard(node.node_id)
if self.persistence and self.auto_persist:
await self.persistence.save_node(self.collection_id, node)
await self._persistence_save_node(node)
async def update(self, node: SyncNode):
old_node = self._nodes.get(node.node_id)
if old_node and old_node.data_id and old_node.data_id != "" and old_node.data_id != node.data_id:
# node_id 不变但 data_id 变更,清理旧索引
self._data_id_to_node_id.pop(old_node.data_id, None)
old_data_id = self._node_id_to_data_id.get(node.node_id)
if old_data_id and old_data_id != node.data_id:
self._remove_data_id_index(node.node_id, old_data_id)
if node.data_id and node.data_id != "":
existing_node_id = self._data_id_to_node_id.get(node.data_id)
@@ -81,27 +165,33 @@ class DataCollection:
raise ValueError(
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
)
self._data_id_to_node_id[node.data_id] = node.node_id
self._set_data_id_index(node.node_id, node.data_id)
else:
self._node_id_to_data_id.pop(node.node_id, None)
self._nodes[node.node_id] = node
self._deleted_node_ids.discard(node.node_id)
if self.persistence and self.auto_persist:
await self.persistence.save_node(self.collection_id, node)
await self._persistence_save_node(node)
async def delete(self, node_id: str):
old_node = self._nodes.get(node_id)
if old_node and old_node.data_id and old_node.data_id != "":
self._data_id_to_node_id.pop(old_node.data_id, None)
self._remove_data_id_index(node_id)
if node_id in self._nodes:
del self._nodes[node_id]
if self.persistence:
if self.auto_persist:
await self.persistence.delete_node(self.collection_id, node_id)
await self._persistence_delete_node(node_id)
else:
self._deleted_node_ids.add(node_id)
def get(self, node_id: str) -> Optional[SyncNode]:
return self._nodes.get(node_id)
node = self._nodes.get(node_id)
if node is not None:
return node
fallback = self._get_global_fallback()
if fallback is None:
return None
return fallback.get(node_id)
def get_by_node_id(self, node_id: str) -> Optional[SyncNode]:
"""别名方法,与 get() 功能相同"""
@@ -113,14 +203,23 @@ class DataCollection:
def get_node_id_by_data_id(self, data_id: str) -> Optional[str]:
"""按 data_id 反查 node_idO(1))。"""
return self._data_id_to_node_id.get(data_id)
node_id = self._data_id_to_node_id.get(data_id)
if node_id is not None:
return node_id
fallback = self._get_global_fallback()
if fallback is None:
return None
return fallback.get_node_id_by_data_id(data_id)
def get_by_data_id_any(self, data_id: str) -> Optional[SyncNode]:
"""按 data_id 查找节点(不要求 node_type)(O(1))。"""
node_id = self._data_id_to_node_id.get(data_id)
if not node_id:
if node_id:
return self._nodes.get(node_id)
fallback = self._get_global_fallback()
if fallback is None:
return None
return self._nodes.get(node_id)
return fallback.get_by_data_id_any(data_id)
def get_by_data_id(self, node_type: str, data_id: str) -> Optional[SyncNode]:
"""按 node_type 和 data_id 查找节点"""
@@ -359,7 +458,7 @@ class DataCollection:
- data/origin_data 保留(由 DataSource 覆盖)
**初始化归并(E01)**:
- 由 Pipeline 在加载后调用 run_reset() 触发 E01
- 由 Pipeline 在加载后调用 `run_phase2_cleanup()` 触发 E01
- CREATE 僵尸节点进入 S15 并删除
- 其余节点进入 S00UNCHECKED/NONE/PENDING
"""
@@ -368,9 +467,10 @@ class DataCollection:
self._nodes = {}
self._data_id_to_node_id = {}
self._node_id_to_data_id = {}
self._deleted_node_ids.clear()
node_dicts = await self.persistence.load_nodes(self.collection_id)
node_dicts = await self._persistence_load_nodes()
from .types import SyncAction, SyncStatus, BindingStatus
from .registry import DomainRegistry
@@ -469,6 +569,7 @@ class DataCollection:
context=context_value,
)
node.context["_loaded_from_persistence"] = True
node.append_log("加载来源: persistence 恢复节点")
except Exception:
import logging
logger = logging.getLogger(__name__)
@@ -487,11 +588,11 @@ class DataCollection:
raise ValueError(
f"Duplicate data_id detected while loading: {node.data_id} already bound to node_id={existing_node_id}"
)
self._data_id_to_node_id[node.data_id] = node.node_id
self._set_data_id_index(node.node_id, node.data_id)
node_types = {node.node_type for node in self._nodes.values()}
for node_type in node_types:
override_map = await self.persistence.load_bind_data_id_overrides(node_type)
override_map = await self._persistence_load_bind_data_id_overrides(node_type)
for node in self.filter(node_type=node_type):
bind_data_id = override_map.get(node.node_id)
if bind_data_id:
@@ -499,24 +600,177 @@ class DataCollection:
else:
node.context.pop("bind_data_id", None)
async def persist(self) -> None:
async def load_from_persistence_excluding(self, node_types: Optional[List[str]] = None):
"""从持久化恢复数据,但跳过指定 node_type。"""
if not self.persistence:
return
self._nodes = {}
self._data_id_to_node_id = {}
self._node_id_to_data_id = {}
self._deleted_node_ids.clear()
node_dicts = await self._persistence_load_nodes(exclude_node_types=node_types)
from .types import SyncAction, SyncStatus, BindingStatus
from .registry import DomainRegistry
for node_dict in node_dicts:
node_type = node_dict["node_type"]
node_class = DomainRegistry.get_node_class(node_type)
assert issubclass(node_class, SyncNode)
parse_errors: list[str] = []
raw_action = node_dict.get("action", SyncAction.NONE)
if isinstance(raw_action, str):
try:
action_value = SyncAction(raw_action)
except ValueError:
parse_errors.append(f"invalid action={raw_action}")
action_value = SyncAction.NONE
elif isinstance(raw_action, SyncAction):
action_value = raw_action
else:
if raw_action is None:
action_value = SyncAction.NONE
else:
parse_errors.append(f"invalid action type={type(raw_action).__name__}")
action_value = SyncAction.NONE
raw_status = node_dict.get("status", SyncStatus.PENDING)
if isinstance(raw_status, str):
try:
status_value = SyncStatus(raw_status)
except ValueError:
parse_errors.append(f"invalid status={raw_status}")
status_value = SyncStatus.FAILED
elif isinstance(raw_status, SyncStatus):
status_value = raw_status
else:
if raw_status is None:
status_value = SyncStatus.PENDING
else:
parse_errors.append(f"invalid status type={type(raw_status).__name__}")
status_value = SyncStatus.FAILED
raw_binding_status = node_dict.get("binding_status", BindingStatus.UNCHECKED)
if isinstance(raw_binding_status, str):
try:
binding_status_value = BindingStatus(raw_binding_status)
except ValueError:
parse_errors.append(f"invalid binding_status={raw_binding_status}")
binding_status_value = BindingStatus.ABNORMAL
elif isinstance(raw_binding_status, BindingStatus):
binding_status_value = raw_binding_status
else:
if raw_binding_status is None:
binding_status_value = BindingStatus.UNCHECKED
else:
parse_errors.append(
f"invalid binding_status type={type(raw_binding_status).__name__}"
)
binding_status_value = BindingStatus.ABNORMAL
raw_context = node_dict.get("context", {})
context_value = raw_context if isinstance(raw_context, dict) else {}
error_value = node_dict.get("error")
if parse_errors:
context_value = dict(context_value)
context_value["bootstrap_blocked"] = True
context_value["bootstrap_block_reason"] = "invalid_persisted_enum"
context_value["bootstrap_block_errors"] = parse_errors
binding_status_value = BindingStatus.ABNORMAL
action_value = SyncAction.NONE
status_value = SyncStatus.FAILED
details = "; ".join(parse_errors)
error_value = f"LOAD_INVALID_ENUM: {details}"
node = node_class(
node_id=node_dict["node_id"],
data_id=node_dict.get("data_id", ""),
depend_ids=node_dict.get("depend_ids", []),
data=node_dict.get("data"),
origin_data=node_dict.get("origin_data"),
action=action_value,
status=status_value,
binding_status=binding_status_value,
error=error_value,
sync_log=None,
context=context_value,
)
node.context["_loaded_from_persistence"] = True
node.append_log("加载来源: persistence 恢复节点")
self._nodes[node.node_id] = node
if node.data_id and node.data_id != "":
existing_node_id = self._data_id_to_node_id.get(node.data_id)
if existing_node_id is not None and existing_node_id != node.node_id:
raise ValueError(
f"Duplicate data_id detected while loading: {node.data_id} already bound to node_id={existing_node_id}"
)
self._set_data_id_index(node.node_id, node.data_id)
loaded_node_types = {node.node_type for node in self._nodes.values()}
for node_type in loaded_node_types:
override_map = await self._persistence_load_bind_data_id_overrides(node_type)
for node in self.filter(node_type=node_type):
bind_data_id = override_map.get(node.node_id)
if bind_data_id:
node.context["bind_data_id"] = bind_data_id
else:
node.context.pop("bind_data_id", None)
async def import_nodes(self, nodes: List[SyncNode]) -> None:
"""导入现有节点到当前 collection,仅写内存,不触发持久化。"""
for source_node in nodes:
node = copy.deepcopy(source_node)
if node.node_id in self._nodes:
continue
if node.data_id and node.data_id != "":
existing_node_id = self._data_id_to_node_id.get(node.data_id)
if existing_node_id is not None and existing_node_id != node.node_id:
raise ValueError(
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
)
self._set_data_id_index(node.node_id, node.data_id)
self._nodes[node.node_id] = node
self._deleted_node_ids.discard(node.node_id)
async def persist(self, *, exclude_node_types: Optional[List[str]] = None) -> None:
"""将当前 collection 全量持久化到后端"""
if not self.persistence:
return
# 先落库删除(用于 auto_persist=False 的场景)
if self._deleted_node_ids:
await self.persistence.delete_nodes_bulk(self.collection_id, list(self._deleted_node_ids))
await self._persistence_delete_nodes_bulk(list(self._deleted_node_ids))
self._deleted_node_ids.clear()
if not self._nodes:
return
await self.persistence.save_nodes_bulk(self.collection_id, list(self._nodes.values()))
excluded = set(node_type for node_type in (exclude_node_types or []) if node_type)
nodes_to_persist = [
node for node in self._nodes.values() if node.node_type not in excluded
]
if not nodes_to_persist:
return
async def filter_by_project_ids(self, target_project_ids: List[str]) -> int:
"""按目标项目集合过滤当前 collection,返回删除数量。"""
target_set = set(target_project_ids or [])
await self._persistence_save_nodes_bulk(nodes_to_persist)
async def unload(self) -> None:
self._nodes.clear()
self._data_id_to_node_id.clear()
self._node_id_to_data_id.clear()
self._deleted_node_ids.clear()
async def filter_by_project_ids(self, data_id_filter: List[str]) -> int:
"""按目标项目 data_id 集合过滤当前 collection,返回删除数量。"""
target_set = set(data_id_filter or [])
if not target_set:
return 0
+246 -345
View File
@@ -1,21 +1,23 @@
import aiosqlite
import json
from __future__ import annotations
import os
import sqlite3
from typing import Optional, Dict, List, Any, TYPE_CHECKING
from abc import ABC, abstractmethod
from typing import Any, Callable, ClassVar, Dict, List, Optional, TYPE_CHECKING
from sqlalchemy import create_engine, inspect, text
from sqlalchemy.engine import URL, make_url
from ..config.persist_config import PersistConfig
if TYPE_CHECKING:
from .sync_node import SyncNode
class PersistenceBackend:
"""
Persistence 负责同步系统运行时数据的持久化存储(aiosqlite 实现)。
主要作为 DataCollection 和 BindingManager 的后端存储。
"""
def __init__(self, db_path: str = ":memory:", backend: str = "sqlite"):
self.db_path = db_path
self.backend = backend
self.conn: Optional[aiosqlite.Connection] = None
class PersistenceBackend(ABC):
"""同步系统持久化契约。"""
backend_name: ClassVar[str] = "abstract"
@staticmethod
def _is_read_only_sql(sql: str) -> bool:
@@ -27,385 +29,284 @@ class PersistenceBackend:
cls,
*,
backend: str,
db_path: str,
db_path: str | None = None,
url: str | None = None,
sql: str,
limit: int = 200,
) -> tuple[List[str], List[Dict[str, Any]]]:
if backend != "sqlite":
raise NotImplementedError(f"records query backend not supported yet: {backend}")
if not cls._is_read_only_sql(sql):
raise ValueError("Only read-only SQL is allowed")
path = db_path
if not os.path.exists(path):
raise FileNotFoundError(f"DB not found: {path}")
if backend == "sqlite":
path = db_path
if not path:
raise ValueError("db_path is required for backend=sqlite")
if not os.path.exists(path):
raise FileNotFoundError(f"DB not found: {path}")
with sqlite3.connect(path) as conn:
conn.row_factory = sqlite3.Row
cursor = conn.execute(sql)
rows = cursor.fetchmany(max(1, min(limit, 2000)))
columns = [item[0] for item in (cursor.description or [])]
data = [{k: row[k] for k in columns} for row in rows]
return columns, data
with sqlite3.connect(path) as conn:
conn.row_factory = sqlite3.Row
cursor = conn.execute(sql)
rows = cursor.fetchmany(max(1, min(limit, 2000)))
columns = [item[0] for item in (cursor.description or [])]
data = [{k: row[k] for k in columns} for row in rows]
return columns, data
if backend == "sqlalchemy":
if not url:
raise ValueError("url is required for backend=sqlalchemy")
engine = create_engine(cls._to_sync_sqlalchemy_url(url))
try:
with engine.connect() as conn:
result = conn.execute(text(sql))
rows = result.fetchmany(max(1, min(limit, 2000)))
columns = list(result.keys())
data = [{column: row[index] for index, column in enumerate(columns)} for row in rows]
return columns, data
finally:
engine.dispose()
raise NotImplementedError(f"records query backend not supported yet: {backend}")
@classmethod
def records_summary(
cls,
*,
backend: str,
db_path: str,
db_path: str | None = None,
url: str | None = None,
) -> Dict[str, Any]:
db_exists = cls._target_exists(backend=backend, db_path=db_path, url=url)
summary: Dict[str, Any] = {
"backend": backend,
"db_path": db_path,
"db_exists": os.path.exists(db_path),
"url": url,
"db_exists": db_exists,
"tables": [],
"sample_bindings": [],
}
if not os.path.exists(db_path):
return summary
if backend != "sqlite":
summary["error"] = f"records summary backend not supported yet: {backend}"
if not db_exists:
return summary
with sqlite3.connect(db_path) as conn:
conn.row_factory = sqlite3.Row
tables = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name"
).fetchall()
table_names = [str(r["name"]) for r in tables]
for name in table_names:
count = conn.execute(f"SELECT COUNT(1) AS c FROM {name}").fetchone()["c"]
summary["tables"].append({"name": name, "count": int(count)})
if "bindings" in table_names:
sample = conn.execute(
"SELECT node_type, local_id, remote_id FROM bindings ORDER BY last_updated DESC LIMIT 50"
if backend == "sqlite":
assert db_path is not None
with sqlite3.connect(db_path) as conn:
conn.row_factory = sqlite3.Row
tables = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name"
).fetchall()
summary["sample_bindings"] = [
{
"node_type": row["node_type"],
"local_id": row["local_id"],
"remote_id": row["remote_id"],
}
for row in sample
]
table_names = [str(r["name"]) for r in tables]
for name in table_names:
count = conn.execute(f"SELECT COUNT(1) AS c FROM {name}").fetchone()["c"]
summary["tables"].append({"name": name, "count": int(count)})
if "bindings" in table_names:
sample = conn.execute(
"SELECT node_type, local_id, remote_id FROM bindings ORDER BY last_updated DESC LIMIT 50"
).fetchall()
summary["sample_bindings"] = [
{
"node_type": row["node_type"],
"local_id": row["local_id"],
"remote_id": row["remote_id"],
}
for row in sample
]
return summary
if backend == "sqlalchemy":
if not url:
summary["error"] = "url is required for backend=sqlalchemy"
return summary
engine = create_engine(cls._to_sync_sqlalchemy_url(url))
try:
with engine.connect() as conn:
inspector = inspect(conn)
table_names = sorted(inspector.get_table_names())
for name in table_names:
count = conn.execute(text(f"SELECT COUNT(1) AS c FROM {name}")).scalar_one()
summary["tables"].append({"name": name, "count": int(count)})
if "bindings" in table_names:
sample = conn.execute(
text("SELECT node_type, local_id, remote_id FROM bindings ORDER BY last_updated DESC LIMIT 50")
).fetchall()
summary["sample_bindings"] = [
{
"node_type": row[0],
"local_id": row[1],
"remote_id": row[2],
}
for row in sample
]
finally:
engine.dispose()
return summary
summary["error"] = f"records summary backend not supported yet: {backend}"
return summary
async def initialize(self):
"""异步初始化数据库连接和表结构"""
if self.backend != "sqlite":
raise NotImplementedError(f"Persistence backend not supported yet: {self.backend}")
@staticmethod
def _target_exists(*, backend: str, db_path: str | None, url: str | None) -> bool:
if backend == "sqlite":
return bool(db_path) and os.path.exists(db_path)
if backend == "sqlalchemy":
if not url:
return False
parsed = make_url(url)
if parsed.get_backend_name() == "sqlite":
database = parsed.database
if not database or database == ":memory:":
return True
return os.path.exists(database)
return True
return False
# 确保目录存在
if self.db_path != ":memory:":
dir_path = os.path.dirname(self.db_path)
if dir_path:
os.makedirs(dir_path, exist_ok=True)
@staticmethod
def _to_sync_sqlalchemy_url(url: str) -> str:
parsed = make_url(url)
async_to_sync = {
"sqlite+aiosqlite": "sqlite+pysqlite",
"postgresql+asyncpg": "postgresql+psycopg",
"mysql+aiomysql": "mysql+pymysql",
}
drivername = async_to_sync.get(parsed.drivername, parsed.drivername)
sync_url: URL = parsed.set(drivername=drivername)
return str(sync_url)
self.conn = await aiosqlite.connect(self.db_path)
self.conn.row_factory = aiosqlite.Row
await self._init_db()
@abstractmethod
async def initialize(self) -> None:
raise NotImplementedError
async def _init_db(self):
# 1. 节点表: 仅用于持久化存储 DataCollection 中的 SyncNode
# 注意: depend_ids不持久化,每次从data实时计算
await self.conn.execute("""
CREATE TABLE IF NOT EXISTS nodes (
collection_id TEXT NOT NULL,
node_id TEXT NOT NULL,
node_type TEXT NOT NULL,
data_id TEXT,
bind_data_id TEXT,
data TEXT,
origin_data TEXT,
action TEXT,
status TEXT,
binding_status TEXT,
error TEXT,
sync_log TEXT,
context TEXT,
PRIMARY KEY (collection_id, node_id)
)
""")
await self._ensure_nodes_schema()
@abstractmethod
async def save_node(self, collection_id: str, scope: str, node: "SyncNode") -> None:
raise NotImplementedError
# 2. 绑定表: 存储本地到远程的映射关系
# 只保留 local_id / remote_id 两字段(remote_id 可空)
await self._ensure_bindings_schema()
await self.conn.commit()
@abstractmethod
async def save_nodes_bulk(self, collection_id: str, scope: str, nodes: List["SyncNode"]) -> None:
raise NotImplementedError
async def _ensure_nodes_schema(self) -> None:
cursor = await self.conn.execute("PRAGMA table_info(nodes)")
rows = await cursor.fetchall()
columns = [r[1] for r in rows] if rows else []
if "bind_data_id" not in columns:
await self.conn.execute("ALTER TABLE nodes ADD COLUMN bind_data_id TEXT")
@abstractmethod
async def delete_node(self, collection_id: str, scope: str, node_id: str) -> None:
raise NotImplementedError
async def _ensure_bindings_schema(self) -> None:
"""确保 bindings 表为 (node_type, local_id, remote_id) 结构。
@abstractmethod
async def delete_nodes_bulk(self, collection_id: str, scope: str, node_ids: List[str]) -> None:
raise NotImplementedError
若检测到旧表含 payload 列,则进行迁移。
"""
cursor = await self.conn.execute("PRAGMA table_info(bindings)")
rows = await cursor.fetchall()
columns = [r[1] for r in rows] if rows else []
@abstractmethod
async def load_nodes(
self,
collection_id: str,
scope: str,
*,
exclude_node_types: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
raise NotImplementedError
if not columns:
await self.conn.execute("""
CREATE TABLE IF NOT EXISTS bindings (
node_type TEXT NOT NULL,
local_id TEXT NOT NULL,
remote_id TEXT,
last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (node_type, local_id)
)
""")
return
@abstractmethod
async def save_binding(self, scope: str, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
raise NotImplementedError
if "payload" in columns and "remote_id" not in columns:
# 迁移旧结构 payload -> remote_id
await self.conn.execute("""
CREATE TABLE IF NOT EXISTS bindings_new (
node_type TEXT NOT NULL,
local_id TEXT NOT NULL,
remote_id TEXT,
last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (node_type, local_id)
)
""")
@abstractmethod
async def save_bindings_bulk(self, scope: str, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
raise NotImplementedError
async with self.conn.execute("SELECT node_type, local_id, payload FROM bindings") as cur:
old_rows = await cur.fetchall()
for row in old_rows:
try:
payload_dict = json.loads(row[2]) if row[2] else {}
except json.JSONDecodeError:
payload_dict = {}
remote_id = payload_dict.get("remote_id")
await self.conn.execute(
"INSERT OR REPLACE INTO bindings_new (node_type, local_id, remote_id) VALUES (?, ?, ?)",
(row[0], row[1], remote_id),
)
@abstractmethod
async def delete_binding(self, scope: str, node_type: str, local_id: str) -> None:
raise NotImplementedError
await self.conn.execute("DROP TABLE bindings")
await self.conn.execute("ALTER TABLE bindings_new RENAME TO bindings")
@abstractmethod
async def delete_bindings_bulk(self, scope: str, node_type: str, local_ids: List[str]) -> None:
raise NotImplementedError
# --- DataCollection 支持 ---
@abstractmethod
async def load_bindings(self, scope: str, node_type: str) -> List[Dict[str, Any]]:
raise NotImplementedError
async def save_node(self, collection_id: str, node: "SyncNode"):
"""保存或更新单个节点(depend_ids不持久化,运行时从data实时计算)"""
bind_data_id = None
if isinstance(node.context, dict):
value = node.context.get("bind_data_id")
if value is not None and value != "":
bind_data_id = str(value)
await self.conn.execute("""
INSERT OR REPLACE INTO nodes (
collection_id, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (
collection_id,
node.node_id,
node.node_type,
node.data_id,
bind_data_id,
json.dumps(node.data) if node.data else None,
json.dumps(node.origin_data) if node.origin_data else None,
node.action.value,
node.status.value,
node.binding_status.value,
node.error,
node.sync_log,
json.dumps(node.context) if node.context else None
))
await self.conn.commit()
@abstractmethod
async def load_bind_data_id_overrides(self, node_type: str, scope: str) -> Dict[str, Optional[str]]:
raise NotImplementedError
async def save_nodes_bulk(self, collection_id: str, nodes: List["SyncNode"]):
"""批量保存或更新节点 (单次提交,depend_ids不持久化)"""
if not nodes:
return
rows = []
for node in nodes:
bind_data_id = None
if isinstance(node.context, dict):
value = node.context.get("bind_data_id")
if value is not None and value != "":
bind_data_id = str(value)
rows.append((
collection_id,
node.node_id,
node.node_type,
node.data_id,
bind_data_id,
json.dumps(node.data) if node.data else None,
json.dumps(node.origin_data) if node.origin_data else None,
node.action.value,
node.status.value,
node.binding_status.value,
node.error,
node.sync_log,
json.dumps(node.context) if node.context else None
))
@abstractmethod
async def delete_scope_node_types(self, *, scope: str, node_types: List[str]) -> None:
raise NotImplementedError
await self.conn.executemany("""
INSERT OR REPLACE INTO nodes (
collection_id, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", rows)
await self.conn.commit()
@abstractmethod
async def copy_nodes_between_scopes(
self,
*,
collection_id: str,
source_scope: str,
target_scope: str,
node_types: List[str],
) -> None:
raise NotImplementedError
async def delete_node(self, collection_id: str, node_id: str):
"""删除单个节点"""
await self.conn.execute("DELETE FROM nodes WHERE collection_id = ? AND node_id = ?", (collection_id, node_id))
await self.conn.commit()
@abstractmethod
async def copy_bindings_between_scopes(
self,
*,
source_scope: str,
target_scope: str,
node_types: List[str],
) -> None:
raise NotImplementedError
async def delete_nodes_bulk(self, collection_id: str, node_ids: List[str]):
"""批量删除节点 (单次提交)"""
if not node_ids:
return
rows = [(collection_id, node_id) for node_id in node_ids]
await self.conn.executemany(
"DELETE FROM nodes WHERE collection_id = ? AND node_id = ?",
rows,
@abstractmethod
async def dump_to_runtime(self, filename: str) -> None:
raise NotImplementedError
@abstractmethod
async def close(self) -> None:
raise NotImplementedError
PersistenceBackendFactory = Callable[[PersistConfig], PersistenceBackend]
_PERSISTENCE_BACKEND_REGISTRY: dict[str, PersistenceBackendFactory] = {}
def register_persistence_backend(name: str, factory: PersistenceBackendFactory, *, override: bool = False) -> None:
backend_name = name.strip()
if not backend_name:
raise ValueError("backend name cannot be empty")
if not override and backend_name in _PERSISTENCE_BACKEND_REGISTRY:
raise ValueError(f"persistence backend already registered: {backend_name}")
_PERSISTENCE_BACKEND_REGISTRY[backend_name] = factory
def get_registered_persistence_backends() -> Dict[str, PersistenceBackendFactory]:
return dict(_PERSISTENCE_BACKEND_REGISTRY)
def create_persistence_backend(
persist_config: PersistConfig,
) -> PersistenceBackend:
backend_name = (persist_config.backend or "").strip()
if not backend_name:
raise ValueError("persist.backend cannot be empty")
if not _PERSISTENCE_BACKEND_REGISTRY:
register_persistence_backend(
"sqlite",
lambda config: SQLitePersistenceBackend(config),
)
await self.conn.commit()
async def load_nodes(self, collection_id: str) -> List[Dict[str, Any]]:
"""从特定 collection 中加载所有节点(depend_ids初始化为空,稍后从data实时计算)"""
async with self.conn.execute("SELECT * FROM nodes WHERE collection_id = ?", (collection_id,)) as cursor:
rows = await cursor.fetchall()
results = []
for row in rows:
results.append({
"node_id": row["node_id"],
"node_type": row["node_type"],
"data_id": row["data_id"],
"bind_data_id": row["bind_data_id"],
"depend_ids": [], # 不从数据库加载,运行时从data实时计算
"data": json.loads(row["data"]) if row["data"] else None,
"origin_data": json.loads(row["origin_data"]) if row["origin_data"] else None,
"action": row["action"],
"status": row["status"],
"binding_status": row["binding_status"],
"error": row["error"],
"sync_log": row["sync_log"],
"context": json.loads(row["context"]) if row["context"] else {}
})
if results[-1]["bind_data_id"] and isinstance(results[-1]["context"], dict):
results[-1]["context"]["bind_data_id"] = results[-1]["bind_data_id"]
return results
factory = _PERSISTENCE_BACKEND_REGISTRY.get(backend_name)
if factory is None:
available = ", ".join(sorted(_PERSISTENCE_BACKEND_REGISTRY)) or "none"
raise NotImplementedError(f"Persistence backend not supported yet: {backend_name}. Available: {available}")
return factory(persist_config)
# --- BindingManager 支持 ---
async def save_binding(self, node_type: str, local_id: str, payload_dict: Dict[str, Any]):
"""保存或更新绑定关系及其元数据"""
remote_id = payload_dict.get("remote_id") if payload_dict else None
await self.conn.execute("""
INSERT OR REPLACE INTO bindings (node_type, local_id, remote_id)
VALUES (?, ?, ?)
""", (node_type, local_id, remote_id))
await self.conn.commit()
def _register_builtin_persistence_backends() -> None:
from .persistence_backends.sqlalchemy_backend import SQLAlchemyPersistenceBackend
from .persistence_backends.sqlite_backend import SQLitePersistenceBackend
async def save_bindings_bulk(self, node_type: str, bindings: Dict[str, Optional[str]]):
"""批量保存或更新绑定关系 (单次提交)"""
if not bindings:
return
rows = []
for local_id, remote_id in bindings.items():
rows.append((node_type, local_id, remote_id))
await self.conn.executemany(
"INSERT OR REPLACE INTO bindings (node_type, local_id, remote_id) VALUES (?, ?, ?)",
rows,
)
await self.conn.commit()
register_persistence_backend("sqlite", lambda config: SQLitePersistenceBackend(config), override=True)
register_persistence_backend("sqlalchemy", lambda config: SQLAlchemyPersistenceBackend(config), override=True)
async def delete_binding(self, node_type: str, local_id: str):
"""删除绑定关系"""
await self.conn.execute(
"DELETE FROM bindings WHERE node_type = ? AND local_id = ?",
(node_type, local_id)
)
await self.conn.commit()
async def delete_bindings_bulk(self, node_type: str, local_ids: List[str]):
"""批量删除绑定关系 (单次提交)"""
if not local_ids:
return
rows = [(node_type, local_id) for local_id in local_ids]
await self.conn.executemany(
"DELETE FROM bindings WHERE node_type = ? AND local_id = ?",
rows,
)
await self.conn.commit()
async def load_bindings(self, node_type: str) -> List[Dict[str, Any]]:
"""回写该类型下的所有绑定关系到内存"""
async with self.conn.execute(
"SELECT local_id, remote_id FROM bindings WHERE node_type = ?",
(node_type,)
) as cursor:
rows = await cursor.fetchall()
return [
{"local_id": r["local_id"], "payload": {"remote_id": r["remote_id"]}}
for r in rows
]
async def load_bind_data_id_overrides(self, node_type: str) -> Dict[str, Optional[str]]:
"""基于 bindings 表计算 node_id -> bind_data_id 覆盖映射。
规则:
- local 节点的 bind_data_id 取其绑定 remote 节点的 data_id
- remote 节点的 bind_data_id 取其绑定 local 节点的 data_id
"""
sql = """
SELECT b.local_id AS src_node_id, rn.data_id AS bind_data_id
FROM bindings b
LEFT JOIN nodes rn
ON rn.node_id = b.remote_id
AND rn.node_type = b.node_type
WHERE b.node_type = ?
UNION ALL
SELECT b.remote_id AS src_node_id, ln.data_id AS bind_data_id
FROM bindings b
LEFT JOIN nodes ln
ON ln.node_id = b.local_id
AND ln.node_type = b.node_type
WHERE b.node_type = ?
AND b.remote_id IS NOT NULL
"""
async with self.conn.execute(sql, (node_type, node_type)) as cursor:
rows = await cursor.fetchall()
result: Dict[str, Optional[str]] = {}
for row in rows:
src_node_id = row["src_node_id"]
if not src_node_id:
continue
result[str(src_node_id)] = row["bind_data_id"]
return result
async def dump_to_runtime(self, filename: str):
"""将当前状态转储到 _runtime 文件夹 (使用 VACUUM INTO)"""
runtime_path = os.path.join("_runtime", filename)
if os.path.exists(runtime_path):
os.remove(runtime_path)
# VACUUM INTO 是备份 SQLite 的推荐方式 (SQLite 3.27+)
await self.conn.execute("VACUUM INTO ?", (runtime_path,))
async def close(self):
"""关闭数据库连接(包含异常处理)"""
if self.conn:
try:
await self.conn.close()
except Exception as e:
print(f"⚠️ Error closing persistence: {e}")
finally:
self.conn = None
_register_builtin_persistence_backends()
@@ -0,0 +1,4 @@
from .sqlalchemy_backend import SQLAlchemyPersistenceBackend
from .sqlite_backend import SQLitePersistenceBackend
__all__ = ["SQLitePersistenceBackend", "SQLAlchemyPersistenceBackend"]
@@ -0,0 +1,428 @@
from __future__ import annotations
import json
import os
from typing import Any, Dict, List, Optional, TYPE_CHECKING
from sqlalchemy import Column, Index, MetaData, PrimaryKeyConstraint, String, Table, Text, delete, insert, select
from sqlalchemy.engine import make_url
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncConnection, create_async_engine
from ...config.persist_config import PersistConfig
from ..persistence import PersistenceBackend
if TYPE_CHECKING:
from ..sync_node import SyncNode
class SQLAlchemyPersistenceBackend(PersistenceBackend):
"""Generic SQLAlchemy persistence backend with explicit schema bootstrap."""
backend_name = "sqlalchemy"
bulk_delete_batch_size = 1000
collection_id_length = 255
scope_length = 255
node_id_length = 255
node_type_length = 128
data_id_length = 255
bind_data_id_length = 255
action_length = 32
status_length = 32
binding_status_length = 32
remote_id_length = 255
last_updated_length = 64
def __init__(self, persist_config: PersistConfig | str | None = None, **_: Any) -> None:
if isinstance(persist_config, PersistConfig):
self.persist_config = persist_config
self.database_url = persist_config.resolved_url()
self.db_path = persist_config.resolved_db_path()
self.backend_options = dict(persist_config.backend_options)
elif isinstance(persist_config, str):
self.persist_config = None
self.database_url = persist_config
parsed = make_url(persist_config)
self.db_path = str(parsed.database) if parsed.get_backend_name() == "sqlite" and parsed.database else None
self.backend_options = {}
else:
raise ValueError("SQLAlchemyPersistenceBackend requires PersistConfig or database URL")
self.engine: Optional[AsyncEngine] = None
self.metadata = MetaData()
self.nodes = Table(
"nodes",
self.metadata,
Column("collection_id", String(self.collection_id_length), nullable=False),
Column("scope", String(self.scope_length), nullable=False, default="global"),
Column("node_id", String(self.node_id_length), nullable=False),
Column("node_type", String(self.node_type_length), nullable=False),
Column("data_id", String(self.data_id_length), nullable=True),
Column("bind_data_id", String(self.bind_data_id_length), nullable=True),
Column("data", Text, nullable=True),
Column("origin_data", Text, nullable=True),
Column("action", String(self.action_length), nullable=True),
Column("status", String(self.status_length), nullable=True),
Column("binding_status", String(self.binding_status_length), nullable=True),
Column("error", Text, nullable=True),
Column("sync_log", Text, nullable=True),
Column("context", Text, nullable=True),
PrimaryKeyConstraint("collection_id", "scope", "node_id", name="pk_nodes"),
Index("idx_nodes_collection_scope_type", "collection_id", "scope", "node_type"),
Index("idx_nodes_collection_scope_data_id", "collection_id", "scope", "data_id"),
Index("idx_nodes_scope_type_node_id", "scope", "node_type", "node_id"),
)
self.bindings = Table(
"bindings",
self.metadata,
Column("scope", String(self.scope_length), nullable=False, default="global"),
Column("node_type", String(self.node_type_length), nullable=False),
Column("local_id", String(self.node_id_length), nullable=False),
Column("remote_id", String(self.remote_id_length), nullable=True),
Column("last_updated", String(self.last_updated_length), nullable=True),
PrimaryKeyConstraint("scope", "node_type", "local_id", name="pk_bindings"),
Index("idx_bindings_scope_type_remote", "scope", "node_type", "remote_id"),
)
async def initialize(self) -> None:
if self.db_path and self.db_path != ":memory:":
dir_path = os.path.dirname(self.db_path)
if dir_path:
os.makedirs(dir_path, exist_ok=True)
engine_options = {"future": True}
engine_options.update(self.backend_options)
self.engine = create_async_engine(self.database_url, **engine_options)
async with self.engine.begin() as conn:
await conn.run_sync(self.metadata.create_all)
async def save_node(self, collection_id: str, scope: str, node: "SyncNode") -> None:
row = self._node_row(collection_id=collection_id, scope=scope, node=node)
async with self._begin() as conn:
await self._replace_node_rows(conn, collection_id=collection_id, scope=scope, node_ids=[node.node_id])
await conn.execute(insert(self.nodes), [row])
async def save_nodes_bulk(self, collection_id: str, scope: str, nodes: List["SyncNode"]) -> None:
if not nodes:
return
rows = [self._node_row(collection_id=collection_id, scope=scope, node=node) for node in nodes]
node_ids = [node.node_id for node in nodes]
async with self._begin() as conn:
await self._replace_node_rows(conn, collection_id=collection_id, scope=scope, node_ids=node_ids)
await conn.execute(insert(self.nodes), rows)
async def delete_node(self, collection_id: str, scope: str, node_id: str) -> None:
async with self._begin() as conn:
await conn.execute(
delete(self.nodes).where(
self.nodes.c.collection_id == collection_id,
self.nodes.c.scope == scope,
self.nodes.c.node_id == node_id,
)
)
async def delete_nodes_bulk(self, collection_id: str, scope: str, node_ids: List[str]) -> None:
if not node_ids:
return
async with self._begin() as conn:
for batch in self._chunked(node_ids):
await conn.execute(
delete(self.nodes).where(
self.nodes.c.collection_id == collection_id,
self.nodes.c.scope == scope,
self.nodes.c.node_id.in_(batch),
)
)
async def load_nodes(
self,
collection_id: str,
scope: str,
*,
exclude_node_types: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
stmt = select(self.nodes).where(
self.nodes.c.collection_id == collection_id,
self.nodes.c.scope == scope,
)
filtered_types = [node_type for node_type in (exclude_node_types or []) if node_type]
if filtered_types:
stmt = stmt.where(self.nodes.c.node_type.not_in(filtered_types))
async with self._connect() as conn:
rows = (await conn.execute(stmt)).mappings().all()
return [self._deserialize_node_row(row) for row in rows]
async def save_binding(self, scope: str, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
row = self._binding_row(scope=scope, node_type=node_type, local_id=local_id, remote_id=payload_dict.get("remote_id") if payload_dict else None)
async with self._begin() as conn:
await self._replace_binding_rows(conn, scope=scope, node_type=node_type, local_ids=[local_id])
await conn.execute(insert(self.bindings), [row])
async def save_bindings_bulk(self, scope: str, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
if not bindings:
return
rows = [
self._binding_row(scope=scope, node_type=node_type, local_id=local_id, remote_id=remote_id)
for local_id, remote_id in bindings.items()
]
async with self._begin() as conn:
await self._replace_binding_rows(conn, scope=scope, node_type=node_type, local_ids=list(bindings.keys()))
await conn.execute(insert(self.bindings), rows)
async def delete_binding(self, scope: str, node_type: str, local_id: str) -> None:
async with self._begin() as conn:
await conn.execute(
delete(self.bindings).where(
self.bindings.c.scope == scope,
self.bindings.c.node_type == node_type,
self.bindings.c.local_id == local_id,
)
)
async def delete_bindings_bulk(self, scope: str, node_type: str, local_ids: List[str]) -> None:
if not local_ids:
return
async with self._begin() as conn:
for batch in self._chunked(local_ids):
await conn.execute(
delete(self.bindings).where(
self.bindings.c.scope == scope,
self.bindings.c.node_type == node_type,
self.bindings.c.local_id.in_(batch),
)
)
async def load_bindings(self, scope: str, node_type: str) -> List[Dict[str, Any]]:
stmt = select(self.bindings.c.local_id, self.bindings.c.remote_id).where(
self.bindings.c.scope == scope,
self.bindings.c.node_type == node_type,
)
async with self._connect() as conn:
rows = (await conn.execute(stmt)).mappings().all()
return [{"local_id": row["local_id"], "payload": {"remote_id": row["remote_id"]}} for row in rows]
async def load_bind_data_id_overrides(self, node_type: str, scope: str) -> Dict[str, Optional[str]]:
local_join = (
select(
self.bindings.c.local_id.label("src_node_id"),
self.nodes.c.data_id.label("bind_data_id"),
)
.select_from(
self.bindings.outerjoin(
self.nodes,
(self.nodes.c.node_id == self.bindings.c.remote_id)
& (self.nodes.c.node_type == self.bindings.c.node_type)
& (self.nodes.c.scope == self.bindings.c.scope),
)
)
.where(
self.bindings.c.node_type == node_type,
self.bindings.c.scope == scope,
)
)
remote_join = (
select(
self.bindings.c.remote_id.label("src_node_id"),
self.nodes.c.data_id.label("bind_data_id"),
)
.select_from(
self.bindings.outerjoin(
self.nodes,
(self.nodes.c.node_id == self.bindings.c.local_id)
& (self.nodes.c.node_type == self.bindings.c.node_type)
& (self.nodes.c.scope == self.bindings.c.scope),
)
)
.where(
self.bindings.c.node_type == node_type,
self.bindings.c.scope == scope,
self.bindings.c.remote_id.is_not(None),
)
)
async with self._connect() as conn:
rows = (await conn.execute(local_join.union_all(remote_join))).mappings().all()
result: Dict[str, Optional[str]] = {}
for row in rows:
src_node_id = row["src_node_id"]
if src_node_id:
result[str(src_node_id)] = row["bind_data_id"]
return result
async def delete_scope_node_types(self, *, scope: str, node_types: List[str]) -> None:
filtered_types = [node_type for node_type in node_types if node_type]
if not filtered_types:
return
async with self._begin() as conn:
await conn.execute(delete(self.nodes).where(self.nodes.c.scope == scope, self.nodes.c.node_type.in_(filtered_types)))
await conn.execute(delete(self.bindings).where(self.bindings.c.scope == scope, self.bindings.c.node_type.in_(filtered_types)))
async def copy_nodes_between_scopes(
self,
*,
collection_id: str,
source_scope: str,
target_scope: str,
node_types: List[str],
) -> None:
if source_scope == target_scope or not node_types:
return
stmt = select(self.nodes).where(
self.nodes.c.collection_id == collection_id,
self.nodes.c.scope == source_scope,
self.nodes.c.node_type.in_(node_types),
)
async with self._begin() as conn:
rows = (await conn.execute(stmt)).mappings().all()
target_rows = [dict(row, scope=target_scope) for row in rows]
if not target_rows:
return
await self._replace_node_rows(
conn,
collection_id=collection_id,
scope=target_scope,
node_ids=[str(row["node_id"]) for row in target_rows],
)
await conn.execute(insert(self.nodes), target_rows)
async def copy_bindings_between_scopes(
self,
*,
source_scope: str,
target_scope: str,
node_types: List[str],
) -> None:
if source_scope == target_scope or not node_types:
return
stmt = select(self.bindings).where(
self.bindings.c.scope == source_scope,
self.bindings.c.node_type.in_(node_types),
)
async with self._begin() as conn:
rows = (await conn.execute(stmt)).mappings().all()
target_rows = [dict(row, scope=target_scope) for row in rows]
if not target_rows:
return
by_type: Dict[str, List[str]] = {}
for row in target_rows:
by_type.setdefault(str(row["node_type"]), []).append(str(row["local_id"]))
for node_type, local_ids in by_type.items():
await self._replace_binding_rows(conn, scope=target_scope, node_type=node_type, local_ids=local_ids)
await conn.execute(insert(self.bindings), target_rows)
async def dump_to_runtime(self, filename: str) -> None:
raise NotImplementedError("dump_to_runtime is only supported by the sqlite backend")
async def close(self) -> None:
if self.engine is None:
return
await self.engine.dispose()
self.engine = None
async def _replace_node_rows(
self,
conn: AsyncConnection,
*,
collection_id: str,
scope: str,
node_ids: List[str],
) -> None:
if not node_ids:
return
for batch in self._chunked(node_ids):
await conn.execute(
delete(self.nodes).where(
self.nodes.c.collection_id == collection_id,
self.nodes.c.scope == scope,
self.nodes.c.node_id.in_(batch),
)
)
async def _replace_binding_rows(
self,
conn: AsyncConnection,
*,
scope: str,
node_type: str,
local_ids: List[str],
) -> None:
if not local_ids:
return
for batch in self._chunked(local_ids):
await conn.execute(
delete(self.bindings).where(
self.bindings.c.scope == scope,
self.bindings.c.node_type == node_type,
self.bindings.c.local_id.in_(batch),
)
)
def _chunked(self, values: List[str]) -> List[List[str]]:
if not values:
return []
batch_size = max(1, int(self.bulk_delete_batch_size))
return [values[index : index + batch_size] for index in range(0, len(values), batch_size)]
def _node_row(self, *, collection_id: str, scope: str, node: "SyncNode") -> Dict[str, Any]:
bind_data_id = None
if isinstance(node.context, dict):
value = node.context.get("bind_data_id")
if value is not None and value != "":
bind_data_id = str(value)
return {
"collection_id": collection_id,
"scope": scope,
"node_id": node.node_id,
"node_type": node.node_type,
"data_id": node.data_id,
"bind_data_id": bind_data_id,
"data": json.dumps(node.data) if node.data else None,
"origin_data": json.dumps(node.origin_data) if node.origin_data else None,
"action": node.action.value,
"status": node.status.value,
"binding_status": node.binding_status.value,
"error": node.error,
"sync_log": node.sync_log,
"context": json.dumps(node.context) if node.context else None,
}
@staticmethod
def _binding_row(*, scope: str, node_type: str, local_id: str, remote_id: Optional[str]) -> Dict[str, Any]:
return {
"scope": scope,
"node_type": node_type,
"local_id": local_id,
"remote_id": remote_id,
"last_updated": None,
}
@staticmethod
def _deserialize_node_row(row: Dict[str, Any]) -> Dict[str, Any]:
item = {
"node_id": row["node_id"],
"node_type": row["node_type"],
"data_id": row["data_id"],
"bind_data_id": row["bind_data_id"],
"depend_ids": [],
"data": json.loads(row["data"]) if row["data"] else None,
"origin_data": json.loads(row["origin_data"]) if row["origin_data"] else None,
"action": row["action"],
"status": row["status"],
"binding_status": row["binding_status"],
"error": row["error"],
"sync_log": row["sync_log"],
"context": json.loads(row["context"]) if row["context"] else {},
}
if item["bind_data_id"] and isinstance(item["context"], dict):
item["context"]["bind_data_id"] = item["bind_data_id"]
return item
def _require_engine(self) -> AsyncEngine:
if self.engine is None:
raise RuntimeError("SQLAlchemyPersistenceBackend is not initialized")
return self.engine
def _connect(self):
return self._require_engine().connect()
def _begin(self):
return self._require_engine().begin()
@@ -0,0 +1,470 @@
from __future__ import annotations
import aiosqlite
import json
import os
from typing import Any, Dict, List, Optional, TYPE_CHECKING
from ...config.persist_config import PersistConfig
from ..persistence import PersistenceBackend
if TYPE_CHECKING:
from ..sync_node import SyncNode
class SQLitePersistenceBackend(PersistenceBackend):
"""SQLite implementation of the persistence contract."""
backend_name = "sqlite"
def __init__(self, persist_config: PersistConfig | str | None = None, **_: Any) -> None:
if isinstance(persist_config, PersistConfig):
self.persist_config = persist_config
resolved_db_path = persist_config.resolved_db_path()
if not resolved_db_path:
raise ValueError("sqlite persistence requires a resolvable db_path")
self.db_path = resolved_db_path
elif isinstance(persist_config, str):
self.persist_config = None
self.db_path = persist_config
else:
self.persist_config = None
self.db_path = ":memory:"
self.conn: Optional[aiosqlite.Connection] = None
async def initialize(self) -> None:
if self.db_path != ":memory:":
dir_path = os.path.dirname(self.db_path)
if dir_path:
os.makedirs(dir_path, exist_ok=True)
self.conn = await aiosqlite.connect(self.db_path)
self.conn.row_factory = aiosqlite.Row
await self._init_db()
async def _init_db(self) -> None:
assert self.conn is not None
await self.conn.execute("""
CREATE TABLE IF NOT EXISTS nodes (
collection_id TEXT NOT NULL,
scope TEXT NOT NULL DEFAULT 'global',
node_id TEXT NOT NULL,
node_type TEXT NOT NULL,
data_id TEXT,
bind_data_id TEXT,
data TEXT,
origin_data TEXT,
action TEXT,
status TEXT,
binding_status TEXT,
error TEXT,
sync_log TEXT,
context TEXT,
PRIMARY KEY (collection_id, scope, node_id)
)
""")
await self._ensure_nodes_schema()
await self._ensure_bindings_schema()
await self.conn.commit()
async def _ensure_nodes_schema(self) -> None:
assert self.conn is not None
cursor = await self.conn.execute("PRAGMA table_info(nodes)")
rows = await cursor.fetchall()
columns = [r[1] for r in rows] if rows else []
if "scope" not in columns:
await self.conn.execute("""
CREATE TABLE IF NOT EXISTS nodes_new (
collection_id TEXT NOT NULL,
scope TEXT NOT NULL DEFAULT 'global',
node_id TEXT NOT NULL,
node_type TEXT NOT NULL,
data_id TEXT,
bind_data_id TEXT,
data TEXT,
origin_data TEXT,
action TEXT,
status TEXT,
binding_status TEXT,
error TEXT,
sync_log TEXT,
context TEXT,
PRIMARY KEY (collection_id, scope, node_id)
)
""")
await self.conn.execute("""
INSERT INTO nodes_new (
collection_id, scope, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
)
SELECT
collection_id, 'global', node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
FROM nodes
""")
await self.conn.execute("DROP TABLE nodes")
await self.conn.execute("ALTER TABLE nodes_new RENAME TO nodes")
cursor = await self.conn.execute("PRAGMA table_info(nodes)")
rows = await cursor.fetchall()
columns = [r[1] for r in rows] if rows else []
if "bind_data_id" not in columns:
await self.conn.execute("ALTER TABLE nodes ADD COLUMN bind_data_id TEXT")
await self.conn.execute(
"CREATE INDEX IF NOT EXISTS idx_nodes_collection_scope_type ON nodes (collection_id, scope, node_type)"
)
await self.conn.execute(
"CREATE INDEX IF NOT EXISTS idx_nodes_collection_scope_data_id ON nodes (collection_id, scope, data_id)"
)
await self.conn.execute(
"CREATE INDEX IF NOT EXISTS idx_nodes_scope_type_node_id ON nodes (scope, node_type, node_id)"
)
async def _ensure_bindings_schema(self) -> None:
assert self.conn is not None
cursor = await self.conn.execute("PRAGMA table_info(bindings)")
rows = await cursor.fetchall()
columns = [r[1] for r in rows] if rows else []
if not columns:
await self.conn.execute("""
CREATE TABLE IF NOT EXISTS bindings (
scope TEXT NOT NULL DEFAULT 'global',
node_type TEXT NOT NULL,
local_id TEXT NOT NULL,
remote_id TEXT,
last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (scope, node_type, local_id)
)
""")
return
if "scope" not in columns or ("payload" in columns and "remote_id" not in columns):
await self.conn.execute("""
CREATE TABLE IF NOT EXISTS bindings_new (
scope TEXT NOT NULL DEFAULT 'global',
node_type TEXT NOT NULL,
local_id TEXT NOT NULL,
remote_id TEXT,
last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (scope, node_type, local_id)
)
""")
if "payload" in columns and "remote_id" not in columns:
async with self.conn.execute("SELECT node_type, local_id, payload FROM bindings") as cur:
old_rows = await cur.fetchall()
for row in old_rows:
try:
payload_dict = json.loads(row[2]) if row[2] else {}
except json.JSONDecodeError:
payload_dict = {}
remote_id = payload_dict.get("remote_id")
await self.conn.execute(
"INSERT OR REPLACE INTO bindings_new (scope, node_type, local_id, remote_id) VALUES (?, ?, ?, ?)",
("global", row[0], row[1], remote_id),
)
else:
async with self.conn.execute("SELECT node_type, local_id, remote_id FROM bindings") as cur:
old_rows = await cur.fetchall()
for row in old_rows:
await self.conn.execute(
"INSERT OR REPLACE INTO bindings_new (scope, node_type, local_id, remote_id) VALUES (?, ?, ?, ?)",
("global", row[0], row[1], row[2]),
)
await self.conn.execute("DROP TABLE bindings")
await self.conn.execute("ALTER TABLE bindings_new RENAME TO bindings")
await self.conn.execute(
"CREATE INDEX IF NOT EXISTS idx_bindings_scope_type_remote ON bindings (scope, node_type, remote_id)"
)
async def save_node(self, collection_id: str, scope: str, node: "SyncNode") -> None:
assert self.conn is not None
bind_data_id = None
if isinstance(node.context, dict):
value = node.context.get("bind_data_id")
if value is not None and value != "":
bind_data_id = str(value)
await self.conn.execute("""
INSERT OR REPLACE INTO nodes (
collection_id, scope, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (
collection_id,
scope,
node.node_id,
node.node_type,
node.data_id,
bind_data_id,
json.dumps(node.data) if node.data else None,
json.dumps(node.origin_data) if node.origin_data else None,
node.action.value,
node.status.value,
node.binding_status.value,
node.error,
node.sync_log,
json.dumps(node.context) if node.context else None,
))
await self.conn.commit()
async def save_nodes_bulk(self, collection_id: str, scope: str, nodes: List["SyncNode"]) -> None:
assert self.conn is not None
if not nodes:
return
rows = []
for node in nodes:
bind_data_id = None
if isinstance(node.context, dict):
value = node.context.get("bind_data_id")
if value is not None and value != "":
bind_data_id = str(value)
rows.append((
collection_id,
scope,
node.node_id,
node.node_type,
node.data_id,
bind_data_id,
json.dumps(node.data) if node.data else None,
json.dumps(node.origin_data) if node.origin_data else None,
node.action.value,
node.status.value,
node.binding_status.value,
node.error,
node.sync_log,
json.dumps(node.context) if node.context else None,
))
await self.conn.executemany("""
INSERT OR REPLACE INTO nodes (
collection_id, scope, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", rows)
await self.conn.commit()
async def delete_node(self, collection_id: str, scope: str, node_id: str) -> None:
assert self.conn is not None
await self.conn.execute(
"DELETE FROM nodes WHERE collection_id = ? AND scope = ? AND node_id = ?",
(collection_id, scope, node_id),
)
await self.conn.commit()
async def delete_nodes_bulk(self, collection_id: str, scope: str, node_ids: List[str]) -> None:
assert self.conn is not None
if not node_ids:
return
rows = [(collection_id, scope, node_id) for node_id in node_ids]
await self.conn.executemany(
"DELETE FROM nodes WHERE collection_id = ? AND scope = ? AND node_id = ?",
rows,
)
await self.conn.commit()
async def load_nodes(
self,
collection_id: str,
scope: str,
*,
exclude_node_types: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
assert self.conn is not None
sql = "SELECT * FROM nodes WHERE collection_id = ? AND scope = ?"
params: List[Any] = [collection_id, scope]
filtered_types = [node_type for node_type in (exclude_node_types or []) if node_type]
if filtered_types:
placeholders = ", ".join("?" for _ in filtered_types)
sql += f" AND node_type NOT IN ({placeholders})"
params.extend(filtered_types)
async with self.conn.execute(sql, tuple(params)) as cursor:
rows = await cursor.fetchall()
results = []
for row in rows:
item = {
"node_id": row["node_id"],
"node_type": row["node_type"],
"data_id": row["data_id"],
"bind_data_id": row["bind_data_id"],
"depend_ids": [],
"data": json.loads(row["data"]) if row["data"] else None,
"origin_data": json.loads(row["origin_data"]) if row["origin_data"] else None,
"action": row["action"],
"status": row["status"],
"binding_status": row["binding_status"],
"error": row["error"],
"sync_log": row["sync_log"],
"context": json.loads(row["context"]) if row["context"] else {},
}
if item["bind_data_id"] and isinstance(item["context"], dict):
item["context"]["bind_data_id"] = item["bind_data_id"]
results.append(item)
return results
async def save_binding(self, scope: str, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
assert self.conn is not None
remote_id = payload_dict.get("remote_id") if payload_dict else None
await self.conn.execute("""
INSERT OR REPLACE INTO bindings (scope, node_type, local_id, remote_id)
VALUES (?, ?, ?, ?)
""", (scope, node_type, local_id, remote_id))
await self.conn.commit()
async def save_bindings_bulk(self, scope: str, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
assert self.conn is not None
if not bindings:
return
rows = [(scope, node_type, local_id, remote_id) for local_id, remote_id in bindings.items()]
await self.conn.executemany(
"INSERT OR REPLACE INTO bindings (scope, node_type, local_id, remote_id) VALUES (?, ?, ?, ?)",
rows,
)
await self.conn.commit()
async def delete_binding(self, scope: str, node_type: str, local_id: str) -> None:
assert self.conn is not None
await self.conn.execute(
"DELETE FROM bindings WHERE scope = ? AND node_type = ? AND local_id = ?",
(scope, node_type, local_id),
)
await self.conn.commit()
async def delete_bindings_bulk(self, scope: str, node_type: str, local_ids: List[str]) -> None:
assert self.conn is not None
if not local_ids:
return
rows = [(scope, node_type, local_id) for local_id in local_ids]
await self.conn.executemany(
"DELETE FROM bindings WHERE scope = ? AND node_type = ? AND local_id = ?",
rows,
)
await self.conn.commit()
async def load_bindings(self, scope: str, node_type: str) -> List[Dict[str, Any]]:
assert self.conn is not None
async with self.conn.execute(
"SELECT local_id, remote_id FROM bindings WHERE scope = ? AND node_type = ?",
(scope, node_type),
) as cursor:
rows = await cursor.fetchall()
return [{"local_id": r["local_id"], "payload": {"remote_id": r["remote_id"]}} for r in rows]
async def load_bind_data_id_overrides(self, node_type: str, scope: str) -> Dict[str, Optional[str]]:
assert self.conn is not None
sql = """
SELECT b.local_id AS src_node_id, rn.data_id AS bind_data_id
FROM bindings b
LEFT JOIN nodes rn
ON rn.node_id = b.remote_id
AND rn.node_type = b.node_type
AND rn.scope = b.scope
WHERE b.node_type = ?
AND b.scope = ?
UNION ALL
SELECT b.remote_id AS src_node_id, ln.data_id AS bind_data_id
FROM bindings b
LEFT JOIN nodes ln
ON ln.node_id = b.local_id
AND ln.node_type = b.node_type
AND ln.scope = b.scope
WHERE b.node_type = ?
AND b.scope = ?
AND b.remote_id IS NOT NULL
"""
async with self.conn.execute(sql, (node_type, scope, node_type, scope)) as cursor:
rows = await cursor.fetchall()
result: Dict[str, Optional[str]] = {}
for row in rows:
src_node_id = row["src_node_id"]
if not src_node_id:
continue
result[str(src_node_id)] = row["bind_data_id"]
return result
async def delete_scope_node_types(self, *, scope: str, node_types: List[str]) -> None:
assert self.conn is not None
filtered_types = [node_type for node_type in node_types if node_type]
if not filtered_types:
return
placeholders = ", ".join("?" for _ in filtered_types)
await self.conn.execute(
f"DELETE FROM nodes WHERE scope = ? AND node_type IN ({placeholders})",
(scope, *filtered_types),
)
await self.conn.execute(
f"DELETE FROM bindings WHERE scope = ? AND node_type IN ({placeholders})",
(scope, *filtered_types),
)
await self.conn.commit()
async def copy_nodes_between_scopes(
self,
*,
collection_id: str,
source_scope: str,
target_scope: str,
node_types: List[str],
) -> None:
assert self.conn is not None
if source_scope == target_scope or not node_types:
return
placeholders = ", ".join("?" for _ in node_types)
sql = f"""
INSERT OR REPLACE INTO nodes (
collection_id, scope, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
)
SELECT
collection_id, ?, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
FROM nodes
WHERE collection_id = ?
AND scope = ?
AND node_type IN ({placeholders})
"""
await self.conn.execute(sql, (target_scope, collection_id, source_scope, *node_types))
await self.conn.commit()
async def copy_bindings_between_scopes(
self,
*,
source_scope: str,
target_scope: str,
node_types: List[str],
) -> None:
assert self.conn is not None
if source_scope == target_scope or not node_types:
return
placeholders = ", ".join("?" for _ in node_types)
sql = f"""
INSERT OR REPLACE INTO bindings (scope, node_type, local_id, remote_id)
SELECT ?, node_type, local_id, remote_id
FROM bindings
WHERE scope = ?
AND node_type IN ({placeholders})
"""
await self.conn.execute(sql, (target_scope, source_scope, *node_types))
await self.conn.commit()
async def dump_to_runtime(self, filename: str) -> None:
assert self.conn is not None
runtime_path = os.path.join("_runtime", filename)
if os.path.exists(runtime_path):
os.remove(runtime_path)
await self.conn.execute("VACUUM INTO ?", (runtime_path,))
async def close(self) -> None:
if self.conn:
try:
await self.conn.close()
except Exception as exc:
print(f"⚠️ Error closing persistence: {exc}")
finally:
self.conn = None
@@ -0,0 +1,85 @@
from __future__ import annotations
from typing import Any, Dict, List, Optional
from .persistence import PersistenceBackend
class PersistenceService:
"""Shared persistence service that owns the backend lifecycle."""
def __init__(self, backend: PersistenceBackend):
self._backend = backend
@property
def backend(self) -> PersistenceBackend:
return self._backend
async def initialize(self) -> None:
await self._backend.initialize()
def view(self, scope: str) -> "ScopedPersistenceView":
return ScopedPersistenceView(self._backend, scope=scope)
async def close(self) -> None:
await self._backend.close()
class ScopedPersistenceView:
"""Thin scope-bound persistence view used by project runtimes."""
def __init__(self, backend: PersistenceBackend, *, scope: str):
self._backend = backend
self.scope = scope
@property
def backend(self) -> PersistenceBackend:
return self._backend
async def load_nodes(
self,
collection_id: str,
*,
exclude_node_types: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
return await self._backend.load_nodes(
collection_id,
self.scope,
exclude_node_types=exclude_node_types,
)
async def save_node(self, collection_id: str, node) -> None:
await self._backend.save_node(collection_id, self.scope, node)
async def save_nodes_bulk(self, collection_id: str, nodes) -> None:
await self._backend.save_nodes_bulk(collection_id, self.scope, nodes)
async def delete_node(self, collection_id: str, node_id: str) -> None:
await self._backend.delete_node(collection_id, self.scope, node_id)
async def delete_nodes_bulk(self, collection_id: str, node_ids: List[str]) -> None:
await self._backend.delete_nodes_bulk(collection_id, self.scope, node_ids)
async def load_bindings(self, node_type: str) -> List[Dict[str, Any]]:
return await self._backend.load_bindings(self.scope, node_type)
async def save_binding(self, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
await self._backend.save_binding(self.scope, node_type, local_id, payload_dict)
async def save_bindings_bulk(self, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
await self._backend.save_bindings_bulk(self.scope, node_type, bindings)
async def delete_binding(self, node_type: str, local_id: str) -> None:
await self._backend.delete_binding(self.scope, node_type, local_id)
async def delete_bindings_bulk(self, node_type: str, local_ids: List[str]) -> None:
await self._backend.delete_bindings_bulk(self.scope, node_type, local_ids)
async def load_bind_data_id_overrides(self, node_type: str) -> Dict[str, Optional[str]]:
return await self._backend.load_bind_data_id_overrides(node_type, self.scope)
async def delete_scope_node_types(self, *, node_types: List[str]) -> None:
await self._backend.delete_scope_node_types(scope=self.scope, node_types=node_types)
async def close(self) -> None:
return None
+2 -13
View File
@@ -17,7 +17,7 @@ Domain 注册中心
node_type="contract",
schema=ContractResponse,
node_class=ContractSyncNode,
jsonl_handler_class=ContractJsonlHandler
jsonl_handler_class=ContractJsonlHandler,
)
# Collection 查询 SyncNode 类
@@ -47,7 +47,6 @@ class DomainRegistration:
strategy_class: Optional[Type[Any]] = None # 可选,部分 domain 可能只读
jsonl_handler_class: Optional[Type[Any]] = None
api_handler_class: Optional[Type[Any]] = None
db_handler_class: Optional[Type[Any]] = None
handler_classes: Dict[str, Type[Any]] = field(default_factory=dict)
metadata: Dict[str, Any] = field(default_factory=dict) # 额外元数据
@@ -69,7 +68,6 @@ class DomainRegistry:
_BUILTIN_HANDLER_ATTRS: Dict[str, str] = {
"jsonl": "jsonl_handler_class",
"api": "api_handler_class",
"db": "db_handler_class",
}
@classmethod
@@ -81,7 +79,6 @@ class DomainRegistry:
strategy_class: Optional[Type[Any]] = None,
jsonl_handler_class: Optional[Type[Any]] = None,
api_handler_class: Optional[Type[Any]] = None,
db_handler_class: Optional[Type[Any]] = None,
**metadata
) -> None:
"""
@@ -94,7 +91,6 @@ class DomainRegistry:
strategy_class: Strategy 子类(可选)
jsonl_handler_class: JSONL Handler 类(可选)
api_handler_class: API Handler 类(可选)
db_handler_class: DB Handler 类(可选)
**metadata: 额外的元数据
Raises:
@@ -110,7 +106,6 @@ class DomainRegistry:
strategy_class=strategy_class,
jsonl_handler_class=jsonl_handler_class,
api_handler_class=api_handler_class,
db_handler_class=db_handler_class,
metadata=metadata
)
for datasource_type, attr_name in cls._BUILTIN_HANDLER_ATTRS.items():
@@ -187,10 +182,6 @@ class DomainRegistry:
setattr(reg, attr_name, handler_class)
@classmethod
def register_db_handler(cls, node_type: str, db_handler_class: Type[Any]) -> None:
"""为已注册 domain 补充 DB Handler。"""
cls.register_handler(node_type, "db", db_handler_class)
@classmethod
def get_schema(cls, node_type: str) -> Type[BaseModel]:
"""获取 Schema 类"""
@@ -245,7 +236,6 @@ class DomainRegistry:
print(f" Strategy Class: {reg.strategy_class.__name__ if reg.strategy_class else '❌ None'}")
print(f" JSONL Handler: {reg.jsonl_handler_class.__name__ if reg.jsonl_handler_class else '❌ None'}")
print(f" API Handler: {reg.api_handler_class.__name__ if reg.api_handler_class else '❌ None'}")
print(f" DB Handler: {reg.db_handler_class.__name__ if reg.db_handler_class else '❌ None'}")
extra_handler_types = sorted(
handler_type for handler_type in reg.handler_classes if handler_type not in cls._BUILTIN_HANDLER_ATTRS
)
@@ -272,7 +262,7 @@ class DomainRegistry:
"has_strategy": True,
"has_jsonl_handler": True,
"has_api_handler": True,
"has_db_handler": False
"handler_types": []
},
...
}
@@ -284,7 +274,6 @@ class DomainRegistry:
"has_strategy": reg.strategy_class is not None,
"has_jsonl_handler": reg.jsonl_handler_class is not None,
"has_api_handler": reg.api_handler_class is not None,
"has_db_handler": reg.db_handler_class is not None,
"handler_types": sorted(reg.handler_classes.keys()),
}
return summary
@@ -0,0 +1,3 @@
from .persistence_backends.sqlalchemy_backend import SQLAlchemyPersistenceBackend
__all__ = ["SQLAlchemyPersistenceBackend"]
@@ -0,0 +1,3 @@
from .persistence_backends.sqlite_backend import SQLitePersistenceBackend
__all__ = ["SQLitePersistenceBackend"]
+13 -4
View File
@@ -1,4 +1,4 @@
from .datasource_config import ApiDataSourceConfig, BaseDataSourceConfig, DbDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
from .datasource_config import ApiDataSourceConfig, BaseDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
from .persist_config import PersistConfig
from .logging_config import LoggingConfig, resolve_log_file_path
from ..datasource.type_registry import (
@@ -24,18 +24,23 @@ from .strategy_config import (
ConfigPresets,
)
from .pipeline_config import PipelineRunConfig, DEFAULT_NODE_TYPES
from .multi_project_config import (
MultiProjectItemConfig,
MultiProjectRunConfig,
build_multi_project_config_from_file,
is_multi_project_payload,
)
from .builder import (
build_default_config,
build_config,
build_config_from_file,
apply_config_overrides,
apply_env_overrides,
config_snapshot,
)
__all__ = [
"ApiDataSourceConfig",
"BaseDataSourceConfig",
"DbDataSourceConfig",
"JsonlDataSourceConfig",
"PipelineRunConfig",
"DataSourceConfig",
@@ -59,9 +64,13 @@ __all__ = [
"StrategyRuntimeConfig",
"ConfigPresets",
"DEFAULT_NODE_TYPES",
"MultiProjectItemConfig",
"MultiProjectRunConfig",
"build_multi_project_config_from_file",
"is_multi_project_payload",
"build_default_config",
"build_config",
"build_config_from_file",
"apply_config_overrides",
"apply_env_overrides",
"config_snapshot",
]
+85 -43
View File
@@ -9,7 +9,7 @@ from ..common.registry import DomainRegistry
from .datasource_config import DataSourceConfig
from .pipeline_config import PipelineRunConfig
from .run_presets import load_overrides_from_file
from .strategy_config import ConfigPresets, StrategyRuntimeConfig
from .strategy_config import ConfigPresets, StrategyRuntimeConfig, resolve_domain_option_config
def _deep_merge(base: Mapping[str, Any], override: Mapping[str, Any]) -> Dict[str, Any]:
@@ -28,19 +28,76 @@ def _deep_merge(base: Mapping[str, Any], override: Mapping[str, Any]) -> Dict[st
return merged
def _build_default_strategy_runtime_config(node_type: str) -> StrategyRuntimeConfig | None:
strategy_class = DomainRegistry.get_strategy_class(node_type)
if strategy_class is None:
return None
config = strategy_class.default_config.model_copy(deep=True)
config.domain_option = resolve_domain_option_config(
getattr(strategy_class, "domain_option_model", None),
config.domain_option,
)
return StrategyRuntimeConfig(
node_type=node_type,
config=config,
)
def _default_strategy_map(node_types: list[str]) -> Dict[str, StrategyRuntimeConfig]:
strategy_map: Dict[str, StrategyRuntimeConfig] = {}
for node_type in node_types:
strategy_class = DomainRegistry.get_strategy_class(node_type)
if strategy_class is None:
runtime_cfg = _build_default_strategy_runtime_config(node_type)
if runtime_cfg is None:
continue
strategy_map[node_type] = StrategyRuntimeConfig(
node_type=node_type,
config=strategy_class.default_config.model_copy(deep=True),
)
strategy_map[node_type] = runtime_cfg
return strategy_map
def _merge_strategy_runtime_config(
runtime_cfg: StrategyRuntimeConfig,
raw_override: Any,
) -> StrategyRuntimeConfig:
override = dict(raw_override or {})
override.pop("node_type", None)
deprecated_keys = [key for key in ("force_sync", "load_mode") if key in override]
if deprecated_keys:
joined = ", ".join(deprecated_keys)
raise ValueError(
f"Strategy override for {runtime_cfg.node_type!r} contains unsupported legacy keys: {joined}. "
"Use skip_sync under strategies.<node_type> or datasource.handler_configs instead."
)
preset = override.pop("preset", None)
if preset:
runtime_cfg.config = runtime_cfg.config.model_validate(
_deep_merge(
runtime_cfg.config.model_dump(),
ConfigPresets.get_preset(str(preset)).model_dump(exclude_defaults=True),
)
)
if "config" in override and isinstance(override["config"], dict):
runtime_cfg.config = runtime_cfg.config.model_validate(
_deep_merge(runtime_cfg.config.model_dump(), override.pop("config"))
)
if override:
runtime_cfg.config = runtime_cfg.config.model_validate(
_deep_merge(runtime_cfg.config.model_dump(), override)
)
strategy_class = DomainRegistry.get_strategy_class(runtime_cfg.node_type)
runtime_cfg.config.domain_option = resolve_domain_option_config(
getattr(strategy_class, "domain_option_model", None) if strategy_class is not None else None,
runtime_cfg.config.domain_option,
)
return runtime_cfg
def _apply_strategy_overrides(
node_types: list[str],
strategy_overrides: Any,
@@ -49,8 +106,16 @@ def _apply_strategy_overrides(
if isinstance(strategy_overrides, list):
for item in strategy_overrides:
runtime_cfg = StrategyRuntimeConfig.model_validate(item)
strategy_map[runtime_cfg.node_type] = runtime_cfg
raw_item = dict(item or {})
node_type = str(raw_item.get("node_type") or "")
if not node_type:
raise ValueError("Strategy override list item missing required node_type")
runtime_cfg = strategy_map.get(node_type)
if runtime_cfg is None:
runtime_cfg = _build_default_strategy_runtime_config(node_type) or StrategyRuntimeConfig(node_type=node_type)
strategy_map[node_type] = _merge_strategy_runtime_config(runtime_cfg, raw_item)
return [strategy_map[n] for n in node_types if n in strategy_map]
if not isinstance(strategy_overrides, dict):
@@ -58,28 +123,10 @@ def _apply_strategy_overrides(
for node_type, raw_override in strategy_overrides.items():
if node_type not in strategy_map:
strategy_map[node_type] = StrategyRuntimeConfig(node_type=node_type)
strategy_map[node_type] = _build_default_strategy_runtime_config(node_type) or StrategyRuntimeConfig(node_type=node_type)
runtime_cfg = strategy_map[node_type]
override = dict(raw_override or {})
preset = override.pop("preset", None)
if preset:
runtime_cfg.config = ConfigPresets.get_preset(str(preset))
override.pop("skip_sync", None)
override.pop("force_sync", None)
override.pop("load_mode", None)
if "config" in override and isinstance(override["config"], dict):
runtime_cfg.config = runtime_cfg.config.model_validate(
_deep_merge(runtime_cfg.config.model_dump(), override.pop("config"))
)
if override:
runtime_cfg.config = runtime_cfg.config.model_validate(
_deep_merge(runtime_cfg.config.model_dump(), override)
)
strategy_map[node_type] = _merge_strategy_runtime_config(runtime_cfg, raw_override)
return [strategy_map[n] for n in node_types if n in strategy_map]
@@ -107,7 +154,6 @@ def build_default_config(project_root: Path) -> PipelineRunConfig:
"read_only": False,
"handler_configs": {},
},
"target_project_ids": [],
"persist": {
"db_path": str(project_root / "test_results" / "sync_state_machine" / "pipeline_default.db"),
"enable": True,
@@ -129,29 +175,17 @@ def build_default_config(project_root: Path) -> PipelineRunConfig:
def apply_config_overrides(config: PipelineRunConfig, overrides: Dict[str, Any]) -> PipelineRunConfig:
merged = _deep_merge(config.model_dump(), overrides)
raw_strategies = merged.pop("strategies", None)
legacy_strategy_overrides = merged.pop("strategy_overrides", None)
resolved = PipelineRunConfig.model_validate(merged)
if raw_strategies is not None:
resolved.strategies = _apply_strategy_overrides(resolved.node_types, raw_strategies)
else:
resolved.strategies = _apply_strategy_overrides(resolved.node_types, legacy_strategy_overrides or {})
resolved.strategies = _apply_strategy_overrides(resolved.node_types, {})
return resolved
def apply_env_overrides(
config: PipelineRunConfig,
project_root: Path,
*,
dotenv_file: Optional[Union[str, Path]] = None,
) -> PipelineRunConfig:
raise NotImplementedError(
"Environment-based overrides are removed. Use config files via overrides_file instead."
)
def build_config(
*,
project_root: Path,
@@ -187,3 +221,11 @@ def build_config(
config = apply_config_overrides(config, overrides)
return config
def build_config_from_file(
*,
project_root: Path,
file_path: Union[str, Path],
) -> PipelineRunConfig:
return build_config(project_root=project_root, overrides_file=Path(file_path))
@@ -11,7 +11,6 @@ class BaseDataSourceConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
type: str
target_project_ids: List[str] = Field(default_factory=list)
handler_configs: Dict[str, Dict[str, Any]] = Field(default_factory=dict)
@@ -22,11 +21,6 @@ class JsonlDataSourceConfig(BaseDataSourceConfig):
read_only: bool = False
class DbDataSourceConfig(BaseDataSourceConfig):
type: Literal["db"] = "db"
class ApiDataSourceConfig(BaseDataSourceConfig):
type: Literal["api"] = "api"
@@ -38,7 +32,6 @@ class ApiDataSourceConfig(BaseDataSourceConfig):
poll_interval: float = 0.5
api_rate_limit_max_requests: int = 10
api_rate_limit_window_seconds: float = 10.0
target_project_ids: List[str] = Field(min_length=1)
DataSourceConfig = Annotated[
@@ -52,5 +45,4 @@ DataSourceConfig = Annotated[
register_datasource_config_model("jsonl", JsonlDataSourceConfig)
register_datasource_config_model("db", DbDataSourceConfig)
register_datasource_config_model("api", ApiDataSourceConfig)
@@ -0,0 +1,40 @@
from __future__ import annotations
from pathlib import Path
from typing import Any, Dict, List, Optional
from pydantic import BaseModel, ConfigDict
from .run_presets import load_overrides_from_file
class MultiProjectItemConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
local_project_id: str
remote_project_id: str
config_path: Optional[str] = None
@property
def scope(self) -> str:
return f"project_id:{self.local_project_id}"
class MultiProjectRunConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
global_config: str
default_project_config: str
max_concurrency: int = 20
project_bootstrap_node_types: List[str] = []
projects: List[MultiProjectItemConfig]
def is_multi_project_payload(payload: Dict[str, Any]) -> bool:
return all(key in payload for key in ("global_config", "default_project_config", "projects"))
def build_multi_project_config_from_file(*, project_root: Path, file_path: str | Path) -> MultiProjectRunConfig:
path = Path(file_path)
overrides = load_overrides_from_file(path, project_root)
return MultiProjectRunConfig.model_validate(overrides)
@@ -17,6 +17,7 @@ meta:
# represented as separate invocations with different inputs.
context:
is_create_zombie: { type: bool, desc: "初始化时判定为 CREATE 僵尸:action=CREATE 且 (status=FAILED 或 data_id 为空)" }
bootstrap_source_present: { type: bool, desc: "bootstrap 阶段 datasource 刷新后,当前节点是否在最新数据源中命中" }
has_binding_record: { type: bool, desc: "binding_manager 里存在绑定记录 (remote_id/local_id 非空)" }
core_binding_result: { type: enum, values: [NORMAL, WARNING, ABNORMAL], desc: "SyncDecisionEngine.determine_core_status() 给本节点的结果(仅在 has_binding_record=true 时有效)" }
@@ -304,19 +305,30 @@ states:
status: SKIPPED
data_id_exist: true
S17:
binding_status: ABNORMAL
action: NONE
status: FAILED
data_id_exist: [false, true]
desc: |
bootstrap 加载异常隔离态
持久化恢复后存在绑定,但 datasource 刷新未命中最新数据
binding_status: ABNORMAL
action: NONE
status: FAILED
data_id_exist: false | true
pseudo_states:
S15:
desc: "节点被清理删除(zombie cleanup"
S16:
desc: "持久化加载输入态:本轮 E01 的输入节点集合(可包含上轮遗留与已稳定节点)"
S17:
desc: "加载异常隔离态:持久化枚举非法,保留错误供人工处理,不进入自动流程"
stages:
bootstrap:
desc: "启动/初始化(仅针对持久化加载出的节点):僵尸清理与归并到 S00"
desc: "启动/初始化:持久化恢复后先做 datasource 刷新,再按 E01 清理僵尸或归并/隔离"
entry_states: [S16]
exit_states: [S00, S15]
exit_states: [S00, S15, S17]
bind:
desc: "绑定判定:core matrix +(可选)依赖检查 +(可选)自动绑定;stage 仅约束边界,不要求内部顺序"
@@ -346,13 +358,23 @@ stages:
transitions:
E01:
stage: bootstrap
desc: "初始化:删CREATE僵尸;其余重置后进入起点"
desc: "初始化:先依据 datasource 刷新结果判定 source 是否存在,再做僵尸清理/隔离/归并"
from: [S16]
outcomes:
- when: { is_create_zombie: true }
to: S15
- when: { is_create_zombie: false }
- when: { is_create_zombie: false, bootstrap_source_present: false, has_binding_record: false }
to: S15
emit:
note: "持久化孤儿节点未命中最新数据源,按僵尸清理"
- when: { is_create_zombie: false, bootstrap_source_present: false, has_binding_record: true }
to: S17
emit:
note: "存在绑定但最新数据源未返回节点,进入加载异常隔离"
- when: { is_create_zombie: false, bootstrap_source_present: true }
to: S00
emit:
note: "bootstrap 已刷新到最新数据,归并到起点"
# --- Bind phase1 core ---
E10:
+61 -4
View File
@@ -1,13 +1,70 @@
from typing import Any, Dict, Literal
from typing import Any, Dict, Optional
from pydantic import BaseModel, ConfigDict, Field
from pydantic import BaseModel, ConfigDict, Field, model_validator
class PersistConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
backend: Literal["sqlite", "sqlalchemy", "custom"] = "sqlite"
db_path: str
backend: str = "sqlite"
db_path: Optional[str] = None
url: Optional[str] = None
backend_options: Dict[str, Any] = Field(default_factory=dict)
enable: bool = True
wipe_on_start: bool = False
@model_validator(mode="before")
@classmethod
def _validate_target(cls, data: Any) -> Any:
if not isinstance(data, dict):
return data
normalized = dict(data)
normalized["backend"] = (str(normalized.get("backend") or "sqlite").strip().lower() or "sqlite")
normalized["db_path"] = cls._normalize_optional_text(normalized.get("db_path"))
normalized["url"] = cls._normalize_optional_text(normalized.get("url"))
if not normalized["db_path"] and not normalized["url"]:
raise ValueError("persist requires either db_path or url")
if normalized["backend"] != "sqlite" and not normalized["url"]:
raise ValueError(f"persist.url is required for backend={normalized['backend']}")
return normalized
@staticmethod
def _normalize_optional_text(value: Optional[str]) -> Optional[str]:
if value is None:
return None
normalized = str(value).strip()
return normalized or None
def resolved_url(self) -> str:
if self.url:
return self.url
if self.backend == "sqlite" and self.db_path:
return f"sqlite+aiosqlite:///{self.db_path}"
raise ValueError(f"persist.url is required for backend={self.backend}")
def resolved_db_path(self) -> Optional[str]:
if self.db_path:
return self.db_path
if not self.url:
return None
from sqlalchemy.engine import make_url
parsed = make_url(self.url)
if parsed.get_backend_name() != "sqlite":
return None
database = parsed.database
if not database:
return None
return str(database)
def wipe_file_path(self) -> Optional[str]:
db_path = self.resolved_db_path()
if not db_path or db_path == ":memory:":
return None
return db_path
def display_target(self) -> str:
return self.url or self.db_path or "<unset>"
@@ -38,7 +38,6 @@ class PipelineRunConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
node_types: List[str] = Field(default_factory=lambda: list(DEFAULT_NODE_TYPES))
target_project_ids: List[str] = Field(default_factory=list)
local_datasource: DataSourceConfig
remote_datasource: DataSourceConfig
+2 -39
View File
@@ -1,7 +1,6 @@
from __future__ import annotations
import json
import importlib
from pathlib import Path
from typing import Any, Dict, List, Tuple
@@ -29,43 +28,6 @@ def _replace_placeholders(value: Any, project_root: Path) -> Any:
return value
def _import_from_path(import_path: str) -> Any:
module_path, sep, attr_path = import_path.partition(":")
if not module_path:
raise ValueError(f"Invalid import path: {import_path}")
module = importlib.import_module(module_path)
if not sep:
return module
obj: Any = module
for attr_name in attr_path.split("."):
obj = getattr(obj, attr_name)
return obj
def _apply_datasource_bootstraps(payload: Dict[str, Any]) -> Dict[str, Any]:
extensions = payload.pop("extensions", None)
if extensions is None:
return payload
if not isinstance(extensions, dict):
raise ValueError("extensions must be an object mapping")
bootstrap_paths = extensions.get("datasource_bootstraps") or []
if not isinstance(bootstrap_paths, list):
raise ValueError("extensions.datasource_bootstraps must be a list")
for item in bootstrap_paths:
if not isinstance(item, str) or not item.strip():
raise ValueError("extensions.datasource_bootstraps entries must be non-empty strings")
loaded = _import_from_path(item)
if not callable(loaded):
raise TypeError(f"datasource bootstrap must resolve to a callable: {item}")
loaded()
return payload
def list_preset_names() -> List[str]:
return sorted(PRESET_FILES.keys())
@@ -115,7 +77,8 @@ def load_overrides_from_file(file_path: Path, project_root: Path) -> Dict[str, A
resolved = _replace_placeholders(payload, project_root)
if not isinstance(resolved, dict):
raise ValueError(f"Preset file must contain object mapping: {file_path}")
return _apply_datasource_bootstraps(resolved)
resolved.pop("extensions", None)
return resolved
def load_named_preset_overrides(project_root: Path, preset_name: str) -> Tuple[Dict[str, Any], Path, bool]:
+13 -4
View File
@@ -3,7 +3,7 @@
Prefer importing from `sync_state_machine.config` directly.
"""
from .datasource_config import ApiDataSourceConfig, DbDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
from .datasource_config import ApiDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
from .persist_config import PersistConfig
from .logging_config import LoggingConfig
from .strategy_config import (
@@ -14,17 +14,22 @@ from .strategy_config import (
ConfigPresets,
)
from .pipeline_config import PipelineRunConfig, DEFAULT_NODE_TYPES
from .multi_project_config import (
MultiProjectItemConfig,
MultiProjectRunConfig,
build_multi_project_config_from_file,
is_multi_project_payload,
)
from .builder import (
build_default_config,
build_config,
build_config_from_file,
apply_config_overrides,
apply_env_overrides,
config_snapshot,
)
__all__ = [
"ApiDataSourceConfig",
"DbDataSourceConfig",
"JsonlDataSourceConfig",
"DataSourceConfig",
"PersistConfig",
@@ -36,9 +41,13 @@ __all__ = [
"ConfigPresets",
"PipelineRunConfig",
"DEFAULT_NODE_TYPES",
"MultiProjectItemConfig",
"MultiProjectRunConfig",
"build_multi_project_config_from_file",
"is_multi_project_payload",
"build_default_config",
"build_config",
"build_config_from_file",
"apply_config_overrides",
"apply_env_overrides",
"config_snapshot",
]
+75 -8
View File
@@ -1,11 +1,21 @@
from __future__ import annotations
import logging
from enum import Enum
from typing import Dict, List, Optional
from typing import Any, Dict, List, Mapping, Type
from pydantic import BaseModel, ConfigDict, Field
def resolve_domain_option_config(
domain_option_model: Type[BaseModel] | None,
domain_option: Mapping[str, Any] | None,
) -> Dict[str, Any]:
if domain_option_model is None:
return dict(domain_option or {})
return domain_option_model.model_validate(domain_option or {}).model_dump()
class OrphanAction(str, Enum):
CREATE_REMOTE = "create_remote"
CREATE_LOCAL = "create_local"
@@ -17,7 +27,6 @@ class OrphanAction(str, Enum):
class UpdateDirection(str, Enum):
PUSH = "push"
PULL = "pull"
BIDIRECTIONAL = "bidirectional"
NONE = "none"
@@ -40,18 +49,76 @@ class StrategyConfig(BaseModel):
remote_orphan_action: OrphanAction = OrphanAction.NONE
update_direction: UpdateDirection = UpdateDirection.PUSH
# 字段级同步方向覆盖
# field_direction_key: 节点 data 中用作查表键的字段名,如 "key"
# field_direction_overrides: 字段值 → UpdateDirection 的映射。
# 未命中映射的节点沿用 update_direction 作为默认方向。
field_direction_key: Optional[str] = None
field_direction_overrides: Dict[str, UpdateDirection] = Field(default_factory=dict)
# post-check 一致性比较时忽略的顶层字段。
# 只影响最终一致性评估,不影响 create/update 的差异检测
post_check_ignore_fields: List[str] = Field(default_factory=list)
# post-check 一致性比较时,忽略列表型字段内各元素的特定子键。
# 格式: {字段名: [子键, ...]},如 {"plan_node": ["is_audit"]}。
# 用于排除后端硬写死、永远与本地不一致的字段,避免误报。
post_check_ignore_list_item_fields: Dict[str, List[str]] = Field(default_factory=dict)
# schema diff / validate 时忽略的顶层字段。
compare_ignore_fields: List[str] = Field(default_factory=list)
# 当 auto-bind 命中“多对一”候选时,允许稳定选择一条进行绑定,剩余候选按 orphan/create 流程继续处理。
# 默认关闭,避免在一般场景下引入隐式匹配。
allow_many_to_one_auto_bind: bool = False
# 是否跳过该类型的 create/update 阶段。
skip_sync: bool = False
# 是否跳过该类型的 post-check reload 与一致性校验。
skip_post_check: bool = False
# 领域策略运行参数,由各 strategy 的 domain_option_model 负责解释与校验。
domain_option: Dict[str, Any] = Field(default_factory=dict)
def log_logic_warnings(self, *, node_type: str, logger: logging.Logger) -> None:
direction = self.update_direction
local_action = self.local_orphan_action
remote_action = self.remote_orphan_action
if direction == UpdateDirection.PUSH:
if local_action == OrphanAction.CREATE_LOCAL:
logger.warning(
f"[{node_type}] Config inconsistency: "
f"update_direction=PUSH but local_orphan_action=CREATE_LOCAL (本地创建)。"
f"应使用 CREATE_REMOTE (推送到远程)"
)
if remote_action == OrphanAction.CREATE_LOCAL:
logger.warning(
f"[{node_type}] Config inconsistency: "
f"update_direction=PUSH but remote_orphan_action=CREATE_LOCAL (本地创建)。"
f"PUSH 模式应设为 NONE 或 DELETE_REMOTE"
)
elif direction == UpdateDirection.PULL:
if local_action == OrphanAction.CREATE_REMOTE:
logger.warning(
f"[{node_type}] Config inconsistency: "
f"update_direction=PULL but local_orphan_action=CREATE_REMOTE (推送到远程)。"
f"PULL 模式应设为 NONE 或 DELETE_LOCAL"
)
if remote_action in {OrphanAction.CREATE_REMOTE, OrphanAction.DELETE_LOCAL}:
logger.warning(
f"[{node_type}] Config inconsistency: "
f"update_direction=PULL but remote_orphan_action={remote_action}"
f"PULL 模式建议使用 CREATE_LOCAL 或 NONE"
)
elif direction == UpdateDirection.NONE:
if local_action != OrphanAction.NONE:
logger.warning(
f"[{node_type}] Config inconsistency: "
f"update_direction=NONE but local_orphan_action={local_action}"
f"应设为 NONE"
)
if remote_action != OrphanAction.NONE:
logger.warning(
f"[{node_type}] Config inconsistency: "
f"update_direction=NONE but remote_orphan_action={remote_action}"
f"应设为 NONE"
)
class StrategyRuntimeConfig(BaseModel):
"""Pipeline 层策略运行配置(按 node_type)。"""
+15 -9
View File
@@ -23,13 +23,22 @@ from .handler import (
from .task_result import TaskResult, TaskStatus
# API DataSource
from .api import ApiClient, ApiDataSource, BaseApiHandler
# DB DataSource
from .db import DbDataSource, BaseDbHandler
from .api import ApiClient, ApiDataSource, BaseApiHandler, register as register_api_datasource
# JSONL DataSource
from .jsonl import JsonlDataSource, BaseJsonlHandler
from .jsonl import JsonlDataSource, BaseJsonlHandler, register as register_jsonl_datasource
_BUILTIN_DATASOURCE_TYPES_REGISTERED = False
def ensure_builtin_datasource_types_registered(*, replace: bool = False) -> None:
global _BUILTIN_DATASOURCE_TYPES_REGISTERED
if _BUILTIN_DATASOURCE_TYPES_REGISTERED and not replace:
return
register_jsonl_datasource(replace=replace)
register_api_datasource(replace=replace)
_BUILTIN_DATASOURCE_TYPES_REGISTERED = True
__all__ = [
# 基础类
@@ -46,16 +55,13 @@ __all__ = [
"TaskResult",
"TaskStatus",
"ValidationResult",
"ensure_builtin_datasource_types_registered",
# API DataSource
"ApiClient",
"ApiDataSource",
"BaseApiHandler",
# DB DataSource
"DbDataSource",
"BaseDbHandler",
# JSONL DataSource
"JsonlDataSource",
"BaseJsonlHandler",
@@ -1,15 +1,50 @@
from __future__ import annotations
"""
API DataSource 模块
提供基于 HTTP API 的数据源实现。
"""
from typing import TYPE_CHECKING
from ..type_registry import register_datasource_type
from .client import ApiClient
from .datasource import ApiDataSource
from .handler import BaseApiHandler
if TYPE_CHECKING:
from ...config.datasource_config import ApiDataSourceConfig
def build_api_datasource(config: ApiDataSourceConfig, endpoint_name: str, **_: object):
del endpoint_name
return ApiDataSource(
base_url=config.api_base_url,
uid=config.api_uid,
secret=config.api_secret,
debug=config.api_debug,
poll_max_retries=config.poll_max_retries,
poll_interval=config.poll_interval,
rate_limit_max_requests=config.api_rate_limit_max_requests,
rate_limit_window_seconds=config.api_rate_limit_window_seconds,
)
def register(*, replace: bool = False) -> None:
from ...config.datasource_config import ApiDataSourceConfig
register_datasource_type(
"api",
config_model=ApiDataSourceConfig,
factory=build_api_datasource,
replace=replace,
)
__all__ = [
"ApiClient",
"ApiDataSource",
"BaseApiHandler",
"build_api_datasource",
"register",
]
+108 -1
View File
@@ -23,6 +23,7 @@ from datetime import datetime
import httpx
from schemas.common.push_system import PushIdsSchema
from ...logging import get_logger, get_scope_display_name
def _build_default_push_steps() -> List[Dict[str, str]]:
@@ -33,6 +34,7 @@ def _build_default_push_steps() -> List[Dict[str, str]]:
class ApiClient:
"""API HTTP 客户端"""
_TRACE_TEXT_MAX_LEN = 1000
_REQUEST_STATS_LOG_PREFIX = "📊 ApiClient request stats"
def __init__(
self,
@@ -41,6 +43,7 @@ class ApiClient:
secret: str,
debug: bool = False,
log_file: Optional[Path] = None,
session_label: Optional[str] = None,
rate_limit_max_requests: int = 30,
rate_limit_window_seconds: float = 10.0,
):
@@ -62,16 +65,58 @@ class ApiClient:
self._client: Optional[httpx.AsyncClient] = None
self._debug = debug
self._log_file = log_file
self._scope = ""
self._session_label = (session_label or "").strip()
self._trace_seq = 0
self._trace_by_push_id: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
self._trace_by_biz_id: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
self._trace_by_op: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
self._load_trace_by_item: Dict[tuple[str, str], deque[Dict[str, Any]]] = defaultdict(deque)
self._logger = logging.getLogger(__name__)
self._logger = get_logger(__name__)
self._rate_limit_max_requests = int(rate_limit_max_requests)
self._rate_limit_window_seconds = float(rate_limit_window_seconds)
self._rate_limit_lock = asyncio.Lock()
self._rate_limit_timestamps: deque[float] = deque()
self._request_total = 0
self._request_success = 0
self._request_failure = 0
def set_scope(self, scope: Optional[str]) -> None:
self._scope = str(scope or "").strip()
def set_session_label(self, label: Optional[str]) -> None:
self._session_label = str(label or "").strip()
def get_log_session_label(self) -> str:
if self._session_label:
return self._session_label
scope_label = get_scope_display_name(self._scope)
return scope_label or self._scope
def _project_name_from_request(self, payload: Optional[Dict[str, Any]]) -> str:
if not isinstance(payload, dict):
return ""
project_id = str(payload.get("project_id") or "").strip()
if not project_id:
return ""
display = get_scope_display_name(f"project_id:{project_id}")
if not display or display == f"project_id:{project_id}":
return ""
return display
def _resolve_request_session_label(
self,
*,
request_params: Optional[Dict[str, Any]] = None,
request_payload: Optional[Dict[str, Any]] = None,
) -> str:
payload_label = self._project_name_from_request(request_payload)
if payload_label:
return payload_label
params_label = self._project_name_from_request(request_params)
if params_label:
return params_label
return self.get_log_session_label()
def _is_rate_limit_enabled(self) -> bool:
return self._rate_limit_max_requests > 0 and self._rate_limit_window_seconds > 0
@@ -105,10 +150,55 @@ class ApiClient:
async def close(self):
"""关闭 HTTP 客户端"""
self.log_request_stats(level="INFO")
if self._client:
await self._client.aclose()
self._client = None
def get_request_stats(self) -> Dict[str, int]:
"""返回当前客户端生命周期内的请求统计。"""
return {
"total": self._request_total,
"success": self._request_success,
"failure": self._request_failure,
}
def format_request_stats(self) -> str:
"""格式化当前客户端生命周期内的请求统计。"""
stats = self.get_request_stats()
session_text = self.get_log_session_label()
session_prefix = f"session={session_text} " if session_text else ""
return (
f"{self._REQUEST_STATS_LOG_PREFIX}: "
f"{session_prefix}"
f"total={stats['total']}, success={stats['success']}, failure={stats['failure']}"
)
def log_request_stats(self, level: str = "DEBUG") -> None:
"""输出当前客户端生命周期内的请求统计。"""
self._log(self.format_request_stats(), level=level)
def _log_request_summary(
self,
*,
method: str,
endpoint: str,
elapsed_seconds: float,
request_params: Optional[Dict[str, Any]] = None,
request_payload: Optional[Dict[str, Any]] = None,
) -> None:
if not self._debug:
return
session_text = self._resolve_request_session_label(
request_params=request_params,
request_payload=request_payload,
)
session_prefix = f"session={session_text} " if session_text else ""
self._log(
f"🔎 ApiClient request: {session_prefix}method={method.upper()} endpoint={endpoint} elapsed={elapsed_seconds:.3f}s",
level="INFO",
)
def _log(self, message: str, level: str = "DEBUG"):
"""
统一日志输出
@@ -382,6 +472,7 @@ class ApiClient:
client = self._get_client()
url = f"{self._base_url}{endpoint}"
await self._acquire_rate_limit_slot()
self._request_total += 1
# 记录请求信息
self._log(f"\n{'='*60}", level="DEBUG")
@@ -433,6 +524,13 @@ class ApiClient:
# 记录响应时间
elapsed = time.time() - start_time
self._log_request_summary(
method=method,
endpoint=endpoint,
elapsed_seconds=elapsed,
request_params=params,
request_payload=data if method.upper() in ['POST', 'PUT', 'DELETE'] else None,
)
self._log(f"Response Time: {elapsed:.3f}s", level="DEBUG")
# 检查 HTTP 状态
@@ -453,14 +551,23 @@ class ApiClient:
response=result,
elapsed_ms=int(elapsed * 1000),
)
self._request_success += 1
return result
except Exception as e:
import traceback
self._request_failure += 1
# 记录错误
elapsed = time.time() - start_time
self._log_request_summary(
method=method,
endpoint=endpoint,
elapsed_seconds=elapsed,
request_params=params,
request_payload=data if method.upper() in ['POST', 'PUT', 'DELETE'] else None,
)
error_msg = "\n========== [ApiClient HTTP Error] ==========\n"
error_msg += f"Request: {method} {url}\n"
error_msg += f"Params: {params}\n"
+30 -18
View File
@@ -12,9 +12,10 @@ import json
from typing import Optional
from pathlib import Path
from ..datasource import BaseDataSource
from ..handler import NodeHandler, ensure_handler_compat
from ..handler import NodeHandler
from ..task_result import TaskResult
from .client import ApiClient
from ...logging import set_scope_display_name
class ApiDataSource(BaseDataSource):
@@ -57,6 +58,29 @@ class ApiDataSource(BaseDataSource):
)
self.poll_mode = poll_mode
def set_collection(self, collection) -> None:
super().set_collection(collection)
self.client.set_scope(getattr(collection, "scope", ""))
def _update_client_session_label_from_project_node(self, node) -> None:
if self._collection is None or getattr(self._collection, "scope", "") == "global":
return
if getattr(node, "node_type", None) != "project":
return
if not hasattr(node, "get_data"):
return
data = node.get_data()
if not isinstance(data, dict):
return
project_name = str(data.get("name") or "").strip()
if not project_name:
return
set_scope_display_name(self._collection.scope, f"project:{project_name}")
self.client.set_session_label(f"project:{project_name}")
async def initialize(self):
"""初始化数据源(初始化 HTTP 客户端)"""
await self.client.initialize()
@@ -65,23 +89,10 @@ class ApiDataSource(BaseDataSource):
"""关闭数据源(关闭 HTTP 客户端)"""
await self.client.close()
def register_handler(self, handler: NodeHandler) -> None:
"""
注册业务 Handler
Args:
handler: API Handler 实例
"""
compatible_handler = ensure_handler_compat(handler)
compatible_handler.set_api_client(self.client)
compatible_handler.set_poll_mode(self.poll_mode)
# 调用基类注册方法
super().register_handler(compatible_handler)
# 如果 collection 已设置,确保新 handler 拿到引用
if self._collection is not None:
compatible_handler.set_collection(self._collection) # type: ignore[arg-type]
def _prepare_handler_for_registration(self, handler: NodeHandler) -> None:
handler.set_api_client(self.client)
handler.set_poll_mode(self.poll_mode)
super()._prepare_handler_for_registration(handler)
@staticmethod
def _fmt_trace_value(value, max_len: int = 600) -> str:
@@ -188,6 +199,7 @@ class ApiDataSource(BaseDataSource):
return None
def _on_node_loaded(self, handler: NodeHandler, node) -> None:
self._update_client_session_label_from_project_node(node)
data_id = node.data_id or ""
if not data_id:
return
+151 -79
View File
@@ -15,11 +15,13 @@ BaseApiHandler - API Handler 基类
import uuid
import asyncio
from abc import abstractmethod
from copy import deepcopy
from typing import ClassVar, Dict, List, Any, Optional, TYPE_CHECKING, TypeVar
from pydantic import BaseModel
from pydantic import BaseModel, ValidationError
from ...common.type_safety import get_pydantic_model_fields
from ..handler import NodeHandler
from ...logging import get_logger
from ..handler import BaseNodeHandler
from ..task_result import TaskResult
from ..handler import ValidationResult # Import ValidationResult
@@ -29,9 +31,10 @@ if TYPE_CHECKING:
from .client import ApiClient
T = TypeVar("T", bound=BaseModel)
logger = get_logger(__name__)
class BaseApiHandler(NodeHandler[T]):
class BaseApiHandler(BaseNodeHandler[T]):
"""API Handler 基类"""
# 子类需要设置这些属性
@@ -39,12 +42,22 @@ class BaseApiHandler(NodeHandler[T]):
_node_class: type['SyncNode']
_schema: Any
def __init__(self):
# 子类应该在调用 super().__init__() 之前或之后设置 _node_type, _node_class, _schema
def __init__(self, **handler_config: Any):
# 子类可直接传入 node_class;少数特殊场景可显式覆盖 schema/node_type。
node_class = handler_config.pop("node_class", None)
node_type = handler_config.pop("node_type", None)
schema = handler_config.pop("schema", None)
if node_class is not None and node_type is None:
node_type = getattr(node_class, "node_type", None)
if node_class is not None and schema is None:
schema = getattr(node_class, "schema", None)
super().__init__(node_type=node_type, node_class=node_class, schema=schema)
self.api_client: 'ApiClient' = None # type: ignore # 由 DataSource 注入
self.poll_mode: str = "async" # async | sync
self._collection: 'DataCollection' = None # type: ignore # 由 DataSource 注入
self.handler_config: Dict[str, Any] = {}
self.handler_config = dict(handler_config)
# 子类在 __init__ 中设置此列表以声明 create/update 中涉及的 Pydantic schema 类;
# post-check 将只比较这些 schema 覆盖的字段,过滤后端只读的派生字段。
self.update_schemas: List[type] = []
@@ -63,14 +76,17 @@ class BaseApiHandler(NodeHandler[T]):
def set_collection(self, collection: 'DataCollection') -> None:
"""设置 Collection(由 DataSource 调用)"""
self._collection = collection
super().set_collection(collection)
def set_handler_config(self, config: Dict[str, Any]) -> None:
self.handler_config = dict(config)
def ensure_api_client(self) -> 'ApiClient':
if self.api_client is None:
raise RuntimeError("API client not set. Call set_api_client() before invoking API handler methods.")
return self.api_client
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""默认忽略项目过滤列表,具体子类按需覆盖。"""
return
def ensure_collection(self) -> 'DataCollection':
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() before invoking handler methods.")
return self._collection
@property
def node_type(self) -> str:
@@ -114,8 +130,7 @@ class BaseApiHandler(NodeHandler[T]):
def _create_basic_node(self, data: Dict[str, Any], depend_ids: Optional[List[str]] = None) -> "SyncNode":
"""创建或复用基础节点(默认使用 data['id'] 或 data['_id']"""
data_id = str(data.get("id") or data.get("_id") or "")
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() before creating nodes.")
self.ensure_collection()
if data_id:
existing_node = self._collection.get_by_data_id(self.node_type, data_id)
@@ -139,6 +154,44 @@ class BaseApiHandler(NodeHandler[T]):
"""默认节点创建(子类可覆盖并补充 context)"""
return self._create_basic_node(data)
def _get_committed_approval_snapshot(self, node: 'SyncNode') -> Dict[str, Any]:
snapshot = node.context.get("approval_snapshot")
return snapshot if isinstance(snapshot, dict) else {}
def _get_pending_approval_snapshot(self, node: 'SyncNode') -> Dict[str, Any]:
snapshot = node.context.get("pending_approval_snapshot")
return snapshot if isinstance(snapshot, dict) else {}
def _get_active_approval_snapshot(self, node: 'SyncNode') -> Dict[str, Any]:
pending_snapshot = self._get_pending_approval_snapshot(node)
if pending_snapshot:
return pending_snapshot
return self._get_committed_approval_snapshot(node)
def _get_request_steps(self, node: 'SyncNode') -> List[Dict[str, Any]]:
snapshot = self._get_active_approval_snapshot(node)
steps = snapshot.get("steps")
if isinstance(steps, list):
return deepcopy(steps)
return []
def _steps_changed(self, node: 'SyncNode') -> bool:
snapshot = self._get_active_approval_snapshot(node)
return bool(snapshot.get("steps_changed"))
def _attach_request_steps(self, request_data: Dict[str, Any], node: Optional['SyncNode']) -> Dict[str, Any]:
payload = dict(request_data)
payload["steps"] = self._get_request_steps(node) if node is not None else []
return payload
def _mark_approval_snapshot_synced(self, node: 'SyncNode') -> None:
snapshot = self._get_active_approval_snapshot(node)
if snapshot:
committed_snapshot = deepcopy(snapshot)
committed_snapshot.pop("steps_changed", None)
node.context["approval_snapshot"] = committed_snapshot
node.context.pop("pending_approval_snapshot", None)
def extract_id(self, data: Dict[str, Any]) -> str:
"""从原始数据提取 ID(默认优先 id,其次 _id)"""
return str(data.get("id") or data.get("_id") or "")
@@ -216,15 +269,60 @@ class BaseApiHandler(NodeHandler[T]):
except Exception as e:
return str(e)
def _get_schema_diff(
def _get_update_schema_payload(
self,
schema: Any,
data: Dict[str, Any],
*,
require_valid: bool,
) -> Dict[str, Any] | None:
"""Update 内部实现:提取指定 schema 下的 payload。
这里明确只服务update schema不负责 response model 一致性校验
这个方法只给 BaseApiHandler update 入口方法复用不作为 handler 入口
处理顺序
1. 先尝试用 update schema 校验成功则直接使用校验后的 model_dump
2. 如果校验失败且 require_valid=False则退化为只抽取 schema 定义的字段
之所以允许退化是因为 update 比较时两边快照常常不完整
某些字段在旧快照缺失但新数据已经有值此时应该继续比较 schema
相关字段而不是把它误判成无差异
空字符串和 None 会被统一过滤掉因为当前 update handler 并不把它们
当成清空字段的显式语义而是当作不参与本次更新
"""
try:
validated = schema.model_validate(data)
payload = validated.model_dump(exclude_unset=True)
except ValidationError:
if require_valid:
return None
schema_fields = get_pydantic_model_fields(schema)
if not schema_fields:
return {}
payload = {
field_name: data.get(field_name)
for field_name in schema_fields.keys()
if field_name in data
}
return {
field_name: value
for field_name, value in payload.items()
if value not in (None, "")
}
def get_update_schema_diff(
self,
schema,
data: Dict[str, Any],
origin_data: Dict[str, Any],
node_id: Optional[str] = None
node_id: Optional[str] = None,
force_full_schema: bool = False,
) -> Dict[str, Any] | None:
"""
获取数据在指定 schema 下的差异字段
"""Update 入口:获取指定 update schema 下的差异字段。
Args:
schema: Pydantic schema
@@ -235,43 +333,37 @@ class BaseApiHandler(NodeHandler[T]):
Returns:
Dict[str, Any] | None: 有差异返回差异字段字典无差异或字段不全返回 None
"""
from pydantic import ValidationError
# 当前待发送的数据必须能构成合法 update payload;允许旧快照不完整。
new_dict = self._get_update_schema_payload(schema, data, require_valid=True)
origin_dict = self._get_update_schema_payload(schema, origin_data, require_valid=False)
try:
new_model = schema.model_validate(data)
origin_model = schema.model_validate(origin_data)
new_dict = new_model.model_dump(exclude_unset=True)
origin_dict = origin_model.model_dump(exclude_unset=True)
if new_dict == origin_dict:
return None
# 返回有差异的字段
diff_fields = {k: v for k, v in new_dict.items() if origin_dict.get(k) != v}
if diff_fields:
# 打印差异详情
node_info = f"[{node_id}]" if node_id else ""
print(f" [DIFF] {self.node_type}{node_info} schema={schema.__name__}")
for k, v in diff_fields.items():
orig_v = origin_dict.get(k)
print(f"{k}: {orig_v!r}{v!r}")
return diff_fields if diff_fields else None
except ValidationError:
# schema 字段不全,视为无差异
if new_dict is None or origin_dict is None:
return None
def _filter_update_data(
if new_dict == origin_dict:
if force_full_schema:
return new_dict
return None
diff_fields = {k: v for k, v in new_dict.items() if origin_dict.get(k) != v}
if diff_fields:
node_info = f"[{node_id}]" if node_id else ""
logger.debug(" [DIFF] %s%s schema=%s", self.node_type, node_info, schema.__name__)
for k, v in diff_fields.items():
orig_v = origin_dict.get(k)
logger.debug("%s: %r%r", k, orig_v, v)
return diff_fields if diff_fields else None
def build_update_request_data(
self,
new_data: Dict[str, Any],
original_data: Dict[str, Any],
schema: Optional[Any] = None
schema: Optional[Any] = None,
force_full_schema: bool = False,
) -> Dict[str, Any]:
"""
过滤出需要更新的字段
"""Update 入口:构造最终要发送给更新接口的请求数据。
规则
1. 提取 schema 定义的所有字段包括必填
@@ -298,29 +390,12 @@ class BaseApiHandler(NodeHandler[T]):
update_data[key] = new_value
return update_data
# 1. 提取 schema 定义的所有字段
schema_fields = set(schema_fields.keys())
schema_data = {}
has_change = False
schema_data = self._get_update_schema_payload(schema, new_data, require_valid=True) or {}
origin_schema_data = self._get_update_schema_payload(schema, original_data, require_valid=False) or {}
for key in schema_fields:
new_value = new_data.get(key)
# 排除空值
if new_value == "" or new_value is None:
continue
schema_data[key] = new_value
# 检查是否变化
original_value = original_data.get(key)
if new_value != original_value:
# 记录具体差异以便排查
# print(f" [DEBUG] Field '{key}' changed: '{original_value}' -> '{new_value}'")
has_change = True
# 2. 只有存在变化时才返回数据
return schema_data if has_change else {}
if force_full_schema or schema_data != origin_schema_data:
return schema_data
return {}
def _schema_changed_fields(
self,
@@ -328,16 +403,12 @@ class BaseApiHandler(NodeHandler[T]):
new_data: Dict[str, Any],
origin_data: Dict[str, Any],
) -> List[str]:
schema_fields = get_pydantic_model_fields(schema)
if not schema_fields:
return []
changed: List[str] = []
for field_name in schema_fields.keys():
if new_data.get(field_name) != origin_data.get(field_name):
changed.append(field_name)
return changed
new_payload = self._get_update_schema_payload(schema, new_data, require_valid=True) or {}
origin_payload = self._get_update_schema_payload(schema, origin_data, require_valid=False) or {}
field_names = sorted(set(new_payload) | set(origin_payload))
return [field_name for field_name in field_names if new_payload.get(field_name) != origin_payload.get(field_name)]
def _build_update_schema_status(
def build_update_schema_status(
self,
*,
schema_name: str,
@@ -347,6 +418,7 @@ class BaseApiHandler(NodeHandler[T]):
will_update: bool,
update_error: Optional[str] = None,
) -> str:
"""Update 入口:生成 update schema 的状态说明文本。"""
new_error = self._validate_schema(new_data, schema)
origin_error = self._validate_schema(origin_data, schema)
changed_fields = self._schema_changed_fields(schema, new_data, origin_data)
+51 -37
View File
@@ -31,16 +31,16 @@ if TYPE_CHECKING:
from ..common.types import SyncStatus
from ..common.type_safety import get_pydantic_model_fields
from .handler import ensure_handler_compat
from .task_result import TaskResult
from ..engine import (
StateMachineConfig,
StateMachineRuntime,
e40_sync_execute,
)
from ..logging import get_logger, get_scope_display_name
logger = logging.getLogger(__name__)
logger = get_logger(__name__)
class BaseDataSource(ABC):
@@ -58,9 +58,9 @@ class BaseDataSource(ABC):
# 1. 初始化 DataSource
datasource = MyDataSource()
# 2. 注册 Handler
datasource.register_handler(project_handler)
datasource.register_handler(contract_handler)
# 2. 添加 Handler
datasource.add_handler(project_handler)
datasource.add_handler(contract_handler)
# 3. 设置 Collection
datasource.set_collection(collection)
@@ -75,7 +75,8 @@ class BaseDataSource(ABC):
def __init__(self, poll_max_retries: int = 1, poll_interval: float = 0.5):
self._handlers: Dict[str, "NodeHandler"] = {}
self._collection: Optional["DataCollection"] = None
self._target_project_ids: List[str] = []
self._endpoint_name: str = "unknown"
self._datasource_type: str = self.__class__.__name__.lower()
# 按节点类型统计信息
# {"company": {"loaded": 10, "synced": 10, "success": 10, "failed": 0}, ...}
@@ -88,6 +89,17 @@ class BaseDataSource(ABC):
self._poll_interval = max(0, poll_interval)
self._sm_runtime: Optional[StateMachineRuntime] = None
def set_runtime_logging_context(self, *, endpoint_name: str, datasource_type: str) -> None:
self._endpoint_name = endpoint_name
self._datasource_type = datasource_type
def _datasource_log_prefix(self) -> str:
scope = self._collection.scope if self._collection is not None else "unbound"
return f"[{get_scope_display_name(scope)}][{self._endpoint_name}:{self._datasource_type}]"
def _node_log_prefix(self, node_type: str) -> str:
return f"{self._datasource_log_prefix()}[{node_type}]"
def _ensure_sm_runtime(self) -> StateMachineRuntime:
if self._sm_runtime is None:
cfg_path = Path(__file__).resolve().parents[1] / "config" / "node_state_machine.yaml"
@@ -164,6 +176,11 @@ class BaseDataSource(ABC):
"""Datasource-specific hook after a FAILED result is applied to node."""
return
def _prepare_handler_for_registration(self, handler: "NodeHandler") -> None:
"""Allow subclasses to inject datasource-specific dependencies before registration."""
if self._collection is not None:
handler.set_collection(self._collection)
async def _handle_success_result(self, node: "SyncNode", result: "TaskResult") -> None:
previous_data_id = node.data_id
ok = self._apply_sync_execute_state(
@@ -362,29 +379,18 @@ class BaseDataSource(ABC):
# ========== Handler 管理 ==========
def register_handler(self, handler: "NodeHandler") -> None:
def add_handler(self, handler: "NodeHandler") -> None:
"""
注册 Handler
添加 Handler 到当前 datasource
Args:
handler: 节点处理器实例
示例
datasource.register_handler(ContractNodeHandler(api_client))
datasource.add_handler(ContractNodeHandler(api_client))
"""
compatible_handler = ensure_handler_compat(handler)
self._handlers[compatible_handler.node_type] = compatible_handler
self._apply_target_project_ids_to_handler(compatible_handler)
def set_target_project_ids(self, target_project_ids: Optional[List[str]]) -> None:
"""设置项目过滤列表,并在注册阶段下发给所有 handler。"""
normalized = [str(pid) for pid in (target_project_ids or []) if str(pid)]
self._target_project_ids = normalized
for handler in self._handlers.values():
self._apply_target_project_ids_to_handler(handler)
def _apply_target_project_ids_to_handler(self, handler: "NodeHandler") -> None:
handler.set_target_project_ids(self._target_project_ids)
self._prepare_handler_for_registration(handler)
self._handlers[handler.node_type] = handler
def get_handler(self, node_type: str) -> "NodeHandler":
"""获取 Handler"""
@@ -432,7 +438,9 @@ class BaseDataSource(ABC):
async def load_all(
self,
order: Optional[List[str]] = None,
data_type: Optional[str] = None
data_type: Optional[str] = None,
*,
raise_on_error: bool = False,
) -> None:
"""
按依赖顺序加载所有数据
@@ -474,7 +482,7 @@ class BaseDataSource(ABC):
# Handler 加载数据并创建节点
nodes = await handler.load()
print(f"Loaded nodes for {node_type}: {len(nodes)}")
logger.debug("%s Loaded nodes for %s: %d", self._datasource_log_prefix(), node_type, len(nodes))
# 将节点写入 Collection:优先按 data_id 命中已有节点并更新
await self._upsert_loaded_nodes(handler, node_type, nodes)
@@ -482,7 +490,9 @@ class BaseDataSource(ABC):
# 统计加载的节点数
self._stats[node_type]["loaded"] += len(nodes)
except Exception as exc:
print(f"❌ [{node_type}] 加载失败: {exc}")
logger.error("%s ❌ 加载失败: %s", self._node_log_prefix(node_type), exc)
if raise_on_error:
raise RuntimeError(f"[{node_type}] 加载失败: {exc}") from exc
continue
# ========== 同步执行 ==========
@@ -631,7 +641,7 @@ class BaseDataSource(ABC):
# 批量创建(传入节点列表)
if create_nodes:
logger.info(f"[{handler.node_type}] create提交开始: nodes={len(create_nodes)}")
logger.debug(f"{self._node_log_prefix(handler.node_type)} create提交开始: nodes={len(create_nodes)}")
try:
create_results = await handler.create_all(create_nodes)
create_success = 0
@@ -648,12 +658,12 @@ class BaseDataSource(ABC):
elif result.status == _TaskStatus.SKIPPED:
create_skipped += 1
await self._handle_task_result(handler, result, async_tasks)
logger.info(
f"[{handler.node_type}] create提交完成: success={create_success}, "
logger.debug(
f"{self._node_log_prefix(handler.node_type)} create提交完成: success={create_success}, "
f"in_progress={create_in_progress}, failed={create_failed}, skipped={create_skipped}"
)
except Exception as e:
print(f"❌ [{handler.node_type}] create_all 异常: {e}")
logger.error("❌ [%s] create_all 异常: %s", handler.node_type, e)
for node in create_nodes:
ok = self._apply_sync_execute_state(
node,
@@ -670,7 +680,7 @@ class BaseDataSource(ABC):
for result in update_results:
await self._handle_task_result(handler, result, async_tasks)
except Exception as e:
print(f"❌ [{handler.node_type}] update_all 异常: {e}")
logger.error("❌ [%s] update_all 异常: %s", handler.node_type, e)
for node in update_nodes:
ok = self._apply_sync_execute_state(
node,
@@ -684,8 +694,8 @@ class BaseDataSource(ABC):
# 轮询异步任务
if async_tasks and poll_async_tasks:
logger.info(
f"[{handler.node_type}] create已提交,push_id检查即将开始: pending={len(async_tasks)}, "
logger.debug(
f"{self._node_log_prefix(handler.node_type)} create已提交,push_id检查即将开始: pending={len(async_tasks)}, "
f"first_wait={self._poll_interval:.3f}s"
)
await self._poll_async_tasks(handler, async_tasks)
@@ -835,14 +845,14 @@ class BaseDataSource(ABC):
return
if async_tasks and poll_interval > 0:
logger.info(
f"[{handler.node_type}] push_id首次检查前等待 {poll_interval:.3f}s, pending={len(async_tasks)}"
logger.debug(
f"{self._node_log_prefix(handler.node_type)} push_id首次检查前等待 {poll_interval:.3f}s, pending={len(async_tasks)}"
)
await asyncio.sleep(poll_interval)
while async_tasks and retry_count < max_retries:
logger.info(
f"[{handler.node_type}] push_id检查 {retry_count + 1}/{max_retries}, pending={len(async_tasks)}"
logger.debug(
f"{self._node_log_prefix(handler.node_type)} push_id检查 {retry_count + 1}/{max_retries}, pending={len(async_tasks)}"
)
# 批量查询
task_ids = list(async_tasks.values())
@@ -957,6 +967,10 @@ class BaseDataSource(ABC):
return summary
async def initialize(self) -> None:
"""初始化数据源资源(子类可覆盖)"""
return None
async def close(self) -> None:
"""关闭数据源资源(子类可覆盖以实现特定的清理逻辑)"""
pass # 基类默认空实现
return None
@@ -1,6 +0,0 @@
"""DB DataSource 模块。"""
from .datasource import DbDataSource
from .handler import BaseDbHandler
__all__ = ["DbDataSource", "BaseDbHandler"]
@@ -1,22 +0,0 @@
"""DbDataSource - 数据库数据源协调层。"""
from __future__ import annotations
from ..datasource import BaseDataSource
class DbDataSource(BaseDataSource):
"""基于数据库 Handler 的数据源实现。"""
async def close(self) -> None:
return None
def _on_node_loaded(self, handler, node) -> None:
node.append_log(
f"DB链路[load] node_type={node.node_type} data_id={node.data_id or '<empty>'}"
)
def _on_in_progress_result(self, handler, node, result) -> None:
node.append_log(
f"DB链路[submit] action={node.action.value} data_id={node.data_id or '<empty>'}"
)
@@ -1,68 +0,0 @@
"""BaseDbHandler - DB Handler 基类。"""
from __future__ import annotations
from typing import Any, Dict, List, Optional, Set, TypeVar, TYPE_CHECKING
from pydantic import BaseModel
from ...common.type_safety import get_pydantic_model_fields
from ..handler import BaseNodeHandler
if TYPE_CHECKING:
from ...common.sync_node import SyncNode
from .datasource import DbDataSource
T = TypeVar("T", bound=BaseModel)
class BaseDbHandler(BaseNodeHandler[T]):
"""数据库 Handler 基类。"""
def __init__(
self,
datasource: "DbDataSource",
node_type: str,
node_class: type["SyncNode[T]"],
schema: type[T],
):
super().__init__(node_type, node_class, schema)
self.datasource = datasource
self.update_schemas: List[type] = []
self._target_project_ids: Optional[Set[str]] = None
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
if target_project_ids:
self._target_project_ids = {str(item) for item in target_project_ids if str(item)}
else:
self._target_project_ids = None
def get_update_fields(self, *, exclude: frozenset = frozenset({"id"})) -> List[str]:
fields: set[str] = set()
for schema in self.update_schemas:
for field_name in get_pydantic_model_fields(schema):
if field_name not in exclude:
fields.add(field_name)
return sorted(fields)
def _create_basic_node(self, data: Dict[str, Any], depend_ids: Optional[List[str]] = None) -> "SyncNode[T]":
data_id = self.extract_id(data)
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() before creating nodes.")
if data_id:
existing_node = self._collection.get_by_data_id(self.node_type, data_id)
if existing_node is not None:
existing_node.depend_ids = list(depend_ids or [])
existing_node.set_data(data.copy())
existing_node.set_origin_data(data.copy())
return existing_node
node_id = self._collection.generate_node_id()
return self.node_class(
node_id=node_id,
data_id=data_id,
data=data.copy(),
depend_ids=depend_ids or [],
origin_data=data.copy(),
)
+28 -93
View File
@@ -74,10 +74,6 @@ class NodeHandler(Protocol[T]):
"""注入 handler 级别配置。默认可忽略。"""
...
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""注入项目过滤列表。默认可忽略。"""
...
def set_api_client(self, client: "ApiClient") -> None:
"""注入 API 客户端。默认可忽略。"""
...
@@ -341,11 +337,11 @@ class BaseNodeHandler(ABC, Generic[T]):
def __init__(
self,
node_type: str,
node_class: Type["SyncNode[T]"],
schema: Type[T]
node_type: str | None = None,
node_class: Type["SyncNode[T]"] | None = None,
schema: Type[T] | None = None,
):
self._node_type = node_type
self._node_type = node_type or ""
self._node_class = node_class
self._schema = schema
self._collection: Optional["DataCollection"] = None
@@ -360,14 +356,34 @@ class BaseNodeHandler(ABC, Generic[T]):
"""
self._collection = collection
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""默认忽略项目过滤能力。"""
return
def set_handler_config(self, config: Dict[str, Any]) -> None:
"""默认保存 handler 级别配置,供子类按需读取。"""
self.handler_config = dict(config)
def get_data_id_filter(self) -> List[str]:
if "data_id_filter" in self.handler_config:
value = self.handler_config.get("data_id_filter", [])
if isinstance(value, str):
return [value] if value else []
if isinstance(value, list):
return [str(item) for item in value if str(item)]
return []
if self.node_type != "project" and self._collection is not None:
return [node.data_id for node in self._collection.filter(node_type="project") if node.data_id]
return []
def should_skip_by_data_id_filter(self, item: Dict[str, Any], item_id: str) -> bool:
data_id_filter = set(self.get_data_id_filter())
if not data_id_filter:
return False
if self.node_type == "project":
return item_id not in data_id_filter
project_id = item.get("project_id")
if project_id:
return str(project_id) not in data_id_filter
return False
def set_api_client(self, client: "ApiClient") -> None:
"""默认忽略 API 客户端注入。"""
return
@@ -479,87 +495,6 @@ class BaseNodeHandler(ABC, Generic[T]):
return self.create_node(data)
class CompatibleNodeHandler(Generic[T]):
"""兼容旧式 Handler,实现扩展后的可选接口并委托给原对象。"""
def __init__(self, handler: NodeHandler[T]):
self._handler = handler
@property
def node_type(self) -> str:
return self._handler.node_type
@property
def node_class(self) -> Type["SyncNode[T]"]:
return self._handler.node_class
@property
def schema(self) -> Type[T]:
return self._handler.schema
def set_collection(self, collection: "DataCollection") -> None:
self._handler.set_collection(collection)
def set_handler_config(self, config: Dict[str, Any]) -> None:
method = getattr(self._handler, "set_handler_config", None)
if callable(method):
method(config)
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
method = getattr(self._handler, "set_target_project_ids", None)
if callable(method):
method(target_project_ids)
def set_api_client(self, client: "ApiClient") -> None:
method = getattr(self._handler, "set_api_client", None)
if callable(method):
method(client)
def set_poll_mode(self, mode: str) -> None:
method = getattr(self._handler, "set_poll_mode", None)
if callable(method):
method(mode)
def get_update_fields(self, *, exclude: frozenset[str] = frozenset({"id"})) -> List[str]:
method = getattr(self._handler, "get_update_fields", None)
if callable(method):
return method(exclude=exclude)
return []
async def load(self) -> List["SyncNode"]:
return await self._handler.load()
async def create_all(self, nodes: List["SyncNode"]) -> List[TaskResult]:
return await self._handler.create_all(nodes)
async def update_all(self, nodes: List["SyncNode"]) -> List[TaskResult]:
return await self._handler.update_all(nodes)
async def delete_all(self, nodes: List["SyncNode"]) -> List[TaskResult]:
return await self._handler.delete_all(nodes)
async def poll_tasks(self, task_ids: List[str]) -> Dict[str, TaskResult]:
return await self._handler.poll_tasks(task_ids)
async def validate(self, data: Dict[str, Any]) -> ValidationResult:
return await self._handler.validate(data)
def extract_id(self, data: Dict[str, Any]) -> str:
return self._handler.extract_id(data)
def _create_node(self, data: Dict[str, Any]) -> "SyncNode[T]":
return self._handler._create_node(data)
def __getattr__(self, item: str) -> Any:
return getattr(self._handler, item)
def ensure_handler_compat(handler: NodeHandler[T]) -> NodeHandler[T]:
if isinstance(handler, BaseNodeHandler) or isinstance(handler, CompatibleNodeHandler):
return handler
return CompatibleNodeHandler(handler)
class NodeHandlerRegistry:
"""节点处理器注册表"""
@@ -1,13 +1,53 @@
from __future__ import annotations
"""
JSONL DataSource 模块
提供基于 JSONL 文件的数据源实现用于测试和本地开发
"""
from pathlib import Path
from typing import TYPE_CHECKING
from ..type_registry import register_datasource_type
from .datasource import JsonlDataSource
from .handler import BaseJsonlHandler
from .path_utils import prepare_remote_jsonl_dir
if TYPE_CHECKING:
from ...config.datasource_config import JsonlDataSourceConfig
def build_jsonl_datasource(
config: JsonlDataSourceConfig,
endpoint_name: str,
*,
project_root: Path | None = None,
):
dir_path = Path(config.jsonl_dir)
if endpoint_name == "remote":
dir_path = prepare_remote_jsonl_dir(dir_path, project_root=project_root or Path.cwd())
elif not dir_path.exists() or not dir_path.is_dir():
raise FileNotFoundError(
f"{endpoint_name}_datasource.jsonl_dir does not exist or is not a directory: {dir_path}"
)
return JsonlDataSource(dir_path, read_only=config.read_only)
def register(*, replace: bool = False) -> None:
from ...config.datasource_config import JsonlDataSourceConfig
register_datasource_type(
"jsonl",
config_model=JsonlDataSourceConfig,
factory=build_jsonl_datasource,
replace=replace,
)
__all__ = [
"JsonlDataSource",
"BaseJsonlHandler",
"build_jsonl_datasource",
"prepare_remote_jsonl_dir",
"register",
]
@@ -10,8 +10,10 @@ JsonlDataSource - JSONL 文件数据源
from __future__ import annotations
import json
import re
from pathlib import Path
from typing import Dict, Any, Set
from typing import Dict, Any, Set, Callable, List, Optional
from ..datasource import BaseDataSource
@@ -30,7 +32,7 @@ class JsonlDataSource(BaseDataSource):
示例
datasource = JsonlDataSource(Path("filtered_datasource/datasource/ecm-2025-12-02"))
datasource.register_handler(ContractJsonlHandler(datasource))
datasource.add_handler(ContractJsonlHandler(datasource))
collection = DataCollection()
datasource.set_collection(collection)
@@ -46,6 +48,7 @@ class JsonlDataSource(BaseDataSource):
# 用于 create/update/delete 的内存缓存
self._data: Dict[str, Dict[str, Dict[str, Any]]] = {} # {node_type: {id: data}}
self._dirty_types: Set[str] = set()
self._node_items_cache: Dict[str, Dict[str, Any]] = {}
def _on_node_loaded(self, handler, node) -> None:
node.append_log(
@@ -62,6 +65,98 @@ class JsonlDataSource(BaseDataSource):
f"JSONL链路[poll] task_id={task_id} status={result.status.value}"
)
def invalidate_node_cache(self, node_type: str) -> None:
self._node_items_cache.pop(node_type, None)
def _matching_files(self, node_type: str) -> List[Path]:
if not self.dir_path.exists():
return []
pattern = re.compile(rf"^{re.escape(node_type)}_\d+\.jsonl$")
return sorted(
file_path
for file_path in self.dir_path.iterdir()
if pattern.match(file_path.name)
)
def _build_cache_entry(
self,
*,
node_type: str,
extract_id: Callable[[Dict[str, Any]], str],
) -> Dict[str, Any]:
files = self._matching_files(node_type)
signature = tuple(
(file_path.name, file_path.stat().st_mtime_ns, file_path.stat().st_size)
for file_path in files
)
cached = self._node_items_cache.get(node_type)
if cached and cached.get("signature") == signature:
return cached
items: List[Dict[str, Any]] = []
data_id_index: Dict[str, List[Dict[str, Any]]] = {}
project_index: Dict[str, List[Dict[str, Any]]] = {}
no_project_items: List[Dict[str, Any]] = []
for file_path in files:
with open(file_path, "r", encoding="utf-8") as f:
for line_num, line in enumerate(f, 1):
line = line.strip()
if not line:
continue
try:
item = json.loads(line)
except json.JSONDecodeError:
print(f"Warning: Invalid JSON in {file_path.name}:{line_num}")
continue
items.append(item)
item_id = extract_id(item)
if item_id:
data_id_index.setdefault(str(item_id), []).append(item)
project_id = item.get("project_id")
if project_id:
project_index.setdefault(str(project_id), []).append(item)
else:
no_project_items.append(item)
entry = {
"signature": signature,
"items": items,
"data_id_index": data_id_index,
"project_index": project_index,
"no_project_items": no_project_items,
}
self._node_items_cache[node_type] = entry
return entry
def get_items_for_load(
self,
*,
node_type: str,
extract_id: Callable[[Dict[str, Any]], str],
data_id_filter: Optional[List[str]] = None,
filter_on_project_id: bool = False,
) -> List[Dict[str, Any]]:
entry = self._build_cache_entry(node_type=node_type, extract_id=extract_id)
filters = [str(item) for item in (data_id_filter or []) if str(item)]
if not filters:
return list(entry["items"])
if not filter_on_project_id:
matched: List[Dict[str, Any]] = []
for data_id in filters:
matched.extend(entry["data_id_index"].get(data_id, []))
return matched
matched = list(entry["no_project_items"])
for project_id in filters:
matched.extend(entry["project_index"].get(project_id, []))
return matched
# load_all 和 sync_all 继承自 BaseDataSource
# 保存逻辑在 BaseJsonlHandler 中实现
+29 -52
View File
@@ -12,7 +12,6 @@ BaseJsonlHandler - JSONL Handler 基类
from __future__ import annotations
import json
import re
from typing import Dict, List, Any, TYPE_CHECKING, Optional, Set
from uuid import uuid4
@@ -44,31 +43,22 @@ class BaseJsonlHandler(BaseNodeHandler):
def __init__(
self,
datasource: JsonlDataSource,
node_type: str,
node_class: type,
schema: type
node_class_or_node_type,
node_class: type | None = None,
schema: type | None = None,
node_type: str | None = None,
):
super().__init__(node_type, node_class, schema)
self.datasource = datasource
self._target_project_ids: Optional[Set[str]] = None
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""注入项目过滤列表,供 load 阶段预过滤使用。"""
if target_project_ids:
self._target_project_ids = {str(pid) for pid in target_project_ids if str(pid)}
if isinstance(node_class_or_node_type, str):
resolved_node_type = node_class_or_node_type
resolved_node_class = node_class
resolved_schema = schema
else:
self._target_project_ids = None
resolved_node_class = node_class_or_node_type
resolved_node_type = node_type or getattr(resolved_node_class, "node_type", None)
resolved_schema = schema or getattr(resolved_node_class, "schema", None)
def _should_skip_item_by_project_filter(self, item: Dict[str, Any], item_id: str) -> bool:
if not self._target_project_ids:
return False
if self.node_type == "project":
return item_id not in self._target_project_ids
project_id = item.get("project_id")
if project_id:
return str(project_id) not in self._target_project_ids
return False
super().__init__(resolved_node_type, resolved_node_class, resolved_schema)
self.datasource = datasource
async def load(self) -> List['SyncNode']:
"""
@@ -84,39 +74,25 @@ class BaseJsonlHandler(BaseNodeHandler):
if not self.datasource.dir_path.exists():
return nodes
# 查找所有匹配的文件:{node_type}_N.jsonl
pattern = re.compile(rf"^{re.escape(self.node_type)}_\d+\.jsonl$")
data_id_filter = self.get_data_id_filter()
items = self.datasource.get_items_for_load(
node_type=self.node_type,
extract_id=self.extract_id,
data_id_filter=data_id_filter,
filter_on_project_id=self.node_type != "project",
)
for file_path in self.datasource.dir_path.iterdir():
if not pattern.match(file_path.name):
for item in items:
item_id = self.extract_id(item)
if self.should_skip_by_data_id_filter(item, item_id):
continue
# 读取文件
with open(file_path, "r", encoding="utf-8") as f:
for line_num, line in enumerate(f, 1):
line = line.strip()
if not line:
continue
node = self.create_node(item)
nodes.append(node)
try:
item = json.loads(line)
item_id = self.extract_id(item)
if self._should_skip_item_by_project_filter(item, item_id):
continue
# 创建节点
node = self.create_node(item)
nodes.append(node)
# 同时加载到缓存
if item_id:
bucket = self.datasource._data.setdefault(self.node_type, {})
bucket[item_id] = item
except json.JSONDecodeError as e:
print(f"Warning: Invalid JSON in {file_path.name}:{line_num}: {e}")
# 继续处理下一行
if item_id:
bucket = self.datasource._data.setdefault(self.node_type, {})
bucket[item_id] = item
return nodes
@@ -295,6 +271,7 @@ class BaseJsonlHandler(BaseNodeHandler):
# 清除该类型的 dirty 标记
self.datasource._dirty_types.discard(self.node_type)
self.datasource.invalidate_node_cache(self.node_type)
except Exception as e:
print(f"Error saving {self.node_type}: {str(e)}")
@@ -0,0 +1,33 @@
from __future__ import annotations
import logging
from pathlib import Path
from ...logging import get_logger
logger = get_logger(__name__)
def prepare_remote_jsonl_dir(remote_dir: Path, *, project_root: Path) -> Path:
project_root_path = project_root.resolve()
resolved_remote_dir = remote_dir.resolve(strict=False)
if resolved_remote_dir.exists():
if not resolved_remote_dir.is_dir():
raise FileNotFoundError(
f"remote_datasource.jsonl_dir exists but is not a directory: {resolved_remote_dir}"
)
return resolved_remote_dir
try:
resolved_remote_dir.relative_to(project_root_path)
except ValueError as exc:
raise FileNotFoundError(
"remote_datasource.jsonl_dir does not exist and auto-create is only allowed "
f"under project root: {resolved_remote_dir} (project_root={project_root_path})"
) from exc
resolved_remote_dir.mkdir(parents=True, exist_ok=True)
logger.info("Created missing remote JSONL directory: %s", resolved_remote_dir)
return resolved_remote_dir
@@ -17,10 +17,7 @@ class AttachmentApiHandler(BaseApiHandler):
"""附件 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "attachment"
self._node_class = AttachmentSyncNode
self._schema = AttachmentSchema
super().__init__(node_class=AttachmentSyncNode, schema=AttachmentSchema)
async def load(self) -> List[SyncNode]:
"""加载附件列表(依赖 Project"""

Some files were not shown because too many files have changed in this diff Show More