strepsiades/ecm_sync_system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: strepsiades/ecm_sync_system:6ae54e3edd42311169912c639eb639d1aa27c368
Branches Tags
strepsiades/ecm_sync_system:main
..
compare: strepsiades/ecm_sync_system:4b81b83fd88320915cf5832949de6d2b2284b1d4
Branches Tags
strepsiades/ecm_sync_system:main

12 Commits
6ae54e3edd .. 4b81b83fd8

Author SHA1 Message Date
strepsiades 4b81b83fd8 更新文档 2026-03-25 10:32:50 +08:00
strepsiades 67370657db 同步用户相关修改 2026-03-24 15:13:46 +08:00
strepsiades 8ee0131d47 修改schema 2026-03-24 10:27:14 +08:00
strepsiades 65bf2346a3 现在depend_fields可以处理列表类id 2026-03-24 10:18:13 +08:00
strepsiades 03a22b0c1c 将persist和后端分离，方便替换为其他数据库 2026-03-24 10:15:16 +08:00
strepsiades 7d92b3f1fe 更新了部分测试 2026-03-24 09:41:08 +08:00
strepsiades b51ed0175b 多pipeline同步支持 2026-03-24 08:40:40 +08:00
strepsiades 0812489f9d 增强validator和打印区别 2026-03-23 15:16:16 +08:00
strepsiades 4c380f1157 修正了preparation production的schema 2026-03-20 15:09:55 +08:00
strepsiades 2c09c61165 调整了pipeline执行顺序。现在按domain顺序读取数据源数据，防止前面domain的更新影响后面domain. 
将target_project_ids替换为更通用的data_id_filter
 2026-03-20 14:38:56 +08:00
strepsiades f9f175ee79 进一步清理代码和注册过程 2026-03-19 11:44:02 +08:00
strepsiades 32fa374e86 清理代码 2026-03-18 17:09:46 +08:00
122 changed files with 4369 additions and 2579 deletions
Expand all files Collapse all files
.gitignore
+1
View File
@@ -96,3 +96,4 @@ run_profiles/*.yml
_runtime/ _runtime/
test_results/ test_results/
*.code-workspace *.code-workspace
runtime/
README.md
+65 -6
View File
@@ -2,6 +2,42 @@
可作为命令行工具运行，也可作为 `pip install -e .` 的本地可编辑安装库接入其他项目。 可作为命令行工具运行，也可作为 `pip install -e .` 的本地可编辑安装库接入其他项目。
## 0. 10 分钟上手
如果你是第一次进入这个仓库，先不要从所有文档开始读，先确认环境和最小工具链可用。
推荐顺序：
1. 准备 Python 3.9+ 环境（当前仓库已在 `pyproject.toml` 中声明 `requires-python >= 3.9`）。
2. 在仓库根目录创建并激活虚拟环境。
3. 以可编辑模式安装本项目。
4. 先跑配置校验，再跑一个轻量集成测试，确认本地环境可用。
5. 只在确认入口可运行后，再去看架构与状态机文档。
最小命令链：
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
python run.py --help
python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml
pytest tests/integration/test_toolchain_config_and_graph.py -q
```
如果你只是想直接运行仓库内现成配置，优先用仓库入口：
```bash
python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
```
如果你已经完成 `pip install -e .`，也可以使用安装后的命令行入口：
```bash
ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
```
面向第一次接触仓库的更完整说明见：`docs/getting_started.md`
## 1. 项目简介 ## 1. 项目简介
本项目用于实现异构系统之间的数据推送：将项目侧管理平台的十余种业务数据，稳定推送到集团侧管理平台。 本项目用于实现异构系统之间的数据推送：将项目侧管理平台的十余种业务数据，稳定推送到集团侧管理平台。
@@ -18,6 +54,7 @@
项目采用：`schema` 严格约束 + 状态机 + 节点模型 + 分层架构 + 注册式 `domain` + 数据源与同步系统解耦。 项目采用：`schema` 严格约束 + 状态机 + 节点模型 + 分层架构 + 注册式 `domain` + 数据源与同步系统解耦。
架构详见： 架构详见：
- `docs/getting_started.md`
- `docs/architecture.md` - `docs/architecture.md`
- `docs/node_state_definition.md` - `docs/node_state_definition.md`
- `docs/state_machine.md` - `docs/state_machine.md`
@@ -58,15 +95,25 @@
## 5. 使用方法 ## 5. 使用方法
先区分两种常见使用方式：
- 仓库内开发与调试：优先使用 `python run.py ...`，因为它不依赖 shell 中已经存在 `ecm-sync-run`。
- 作为库或已安装命令使用：先执行 `pip install -e .`，再使用 `ecm-sync-run ...`。
运行配置遵循一个约定：
- 跟踪在仓库里的 `run_profiles/*.yaml` 与 `run_profiles/preset/*.default.yaml` 只写必要的覆盖项。
- pipeline 启动时会打印并写入一份 **Resolved Full Config**，把省略的默认值全部补齐，便于排障和审阅。
- 字段含义与全量示例见：`docs/runtime_config_guide.md`
### JSONL 同步 ### JSONL 同步
- 基线配置（优先）：`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml` - 仓库内直接运行：`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
- 兼容旧入口：`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml` - 已安装命令行入口：`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
- 使用自定义配置：先从 `run_profiles/preset/jsonl_to_jsonl.default.yaml` 复制到 `run_profiles/jsonl_to_jsonl.local.yaml`，按需修改后运行：`ecm-sync-run --config_path=run_profiles/jsonl_to_jsonl.local.yaml` - 多项目 JSONL 示例：`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.multi_project.default.yaml`
- 自定义配置：从 `run_profiles/preset/` 复制一份到 `run_profiles/*.local.yaml`，只改有差异的字段即可。
### API 同步 ### API 同步
- 基线配置：`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_api.default.yaml` - 仓库内直接运行：`python run.py --config_path=run_profiles/preset/jsonl_to_api.default.yaml`
- 兼容旧入口：`python run.py --config_path=run_profiles/preset/jsonl_to_api.default.yaml` - 已安装命令行入口：`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_api.default.yaml`
- 使用自定义配置：先从 `run_profiles/preset/jsonl_to_api.default.yaml` 复制到 `run_profiles/jsonl_to_api.local.yaml`，按需修改后运行：`ecm-sync-run --config_path=run_profiles/jsonl_to_api.local.yaml` - 如果是单项目联调，优先参考仓库内现成示例：`run_profiles/jsonl_to_api.yaml`
### 库模式使用 ### 库模式使用
- 本地可编辑安装：`pip install -e .` - 本地可编辑安装：`pip install -e .`
@@ -75,6 +122,7 @@
### 配置与校验 ### 配置与校验
- 模板配置：`run_profiles/preset/*.default.yaml` - 模板配置：`run_profiles/preset/*.default.yaml`
- 建议先跑 default，再在 `run_profiles/*.local.yaml` 做覆盖（该目录 YAML 已默认忽略） - 建议先跑 default，再在 `run_profiles/*.local.yaml` 做覆盖（该目录 YAML 已默认忽略）
- 跟踪中的模板与示例配置都采用“只写必要字段”的风格；启动日志里的 `Resolved Full Config` 才是最终生效配置。
- 状态机配置校验：`python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml` - 状态机配置校验：`python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml`
- 状态机图生成：`python tools/render_graph.py --format mermaid --out docs/state_machine.mmd` - 状态机图生成：`python tools/render_graph.py --format mermaid --out docs/state_machine.mmd`
@@ -88,9 +136,20 @@
- UI 调试：`python -m ui.main --host 127.0.0.1 --port 8765` - UI 调试：`python -m ui.main --host 127.0.0.1 --port 8765`
- 运维与排障：`docs/operations.md` - 运维与排障：`docs/operations.md`
- 远程测试环境：`tools/remote_env.py` - 远程测试环境：`tools/remote_env.py`
- 脚本化回归与数据集构建：`scripts/README.md`
## 文档导航 ## 文档导航
如果你是按任务查文档，建议按下面的顺序进入：
- 第一次接手仓库：`docs/getting_started.md`
- 想理解整体分层：`docs/architecture.md`
- 想理解状态与迁移：`docs/node_state_definition.md` + `docs/state_machine.md`
- 想调运行配置：`docs/runtime_config_guide.md` + `run_profiles/README.md`
- 想接新数据源或改 handler：`sync_state_machine/datasource/README.md` + `docs/library_integration_guide.md`
- 想排障：`docs/operations.md`
- 想补测试或跑联调：`docs/testing_guide.md` + `tests/README.md`
- 新人上手：`docs/getting_started.md`
- 总体架构：`docs/architecture.md` - 总体架构：`docs/architecture.md`
- 状态定义：`docs/node_state_definition.md` - 状态定义：`docs/node_state_definition.md`
- 状态机规则：`docs/state_machine.md` - 状态机规则：`docs/state_machine.md`
docs/architecture.md
+8 -8
View File
@@ -3,7 +3,7 @@
## 1. 分层职责（pipeline + strategy + datasource + domain） ## 1. 分层职责（pipeline + strategy + datasource + domain）
### pipeline（编排层） ### pipeline（编排层）
- 入口：`pipeline/full_sync_pipeline.py` - 入口：`sync_state_machine/pipeline/full_sync_pipeline.py`
- 负责组装并驱动整条流程：加载持久化、执行 bind/create/update、调用 datasource 同步、持久化结果。 - 负责组装并驱动整条流程：加载持久化、执行 bind/create/update、调用 datasource 同步、持久化结果。
- reset 语义： - reset 语义：
- 加载阶段仅恢复持久化状态（不做状态重置）； - 加载阶段仅恢复持久化状态（不做状态重置）；
@@ -24,8 +24,8 @@
- 非僵尸节点统一进入 `S00`，再进入本轮 bind/create/update 判定。 - 非僵尸节点统一进入 `S00`，再进入本轮 bind/create/update 判定。
### strategy（决策组织层） ### strategy（决策组织层）
- 外部入口：`sync_system/strategy.py` - 外部入口：`sync_state_machine/sync_system/strategy.py`
- 具体实现下沉：`sync_system/strategy_ops/` - 具体实现下沉：`sync_state_machine/sync_system/strategy_ops/`
- `bind_ops.py`: `run_bind()` + `phase_core_state()/phase_dependency_check()/phase_auto_binding()` - `bind_ops.py`: `run_bind()` + `phase_core_state()/phase_dependency_check()/phase_auto_binding()`
- `create_ops.py`: `run_create()` - `create_ops.py`: `run_create()`
- `update_ops.py`: `run_update()` - `update_ops.py`: `run_update()`
@@ -34,19 +34,19 @@
- strategy 负责收集上下文并调用状态机事件函数，不直接硬编码状态值。 - strategy 负责收集上下文并调用状态机事件函数，不直接硬编码状态值。
### datasource（执行与回写层） ### datasource（执行与回写层）
- 入口：`datasource/datasource.py` - 入口：`sync_state_machine/datasource/datasource.py`
- 执行 handler（create/update/delete），并结合 `E40` 将执行结果落地到节点状态。 - 执行 handler（create/update/delete），并结合 `E40` 将执行结果落地到节点状态。
### domain（业务适配层） ### domain（业务适配层）
- 目录：`domain/*` - 目录：`sync_state_machine/domain/*`
- 负责业务字段映射、handler 细节与 schema 约束，不承担状态迁移判定。 - 负责业务字段映射、handler 细节与 schema 约束，不承担状态迁移判定。
## 2. 运行时契约（Runtime Contract） ## 2. 运行时契约（Runtime Contract）
### 真源 ### 真源
- 状态机配置真源：`config/node_state_machine.yaml` - 状态机配置真源：`sync_state_machine/config/node_state_machine.yaml`
- 事件执行入口：`engine/events.py` - 事件执行入口：`sync_state_machine/engine/events.py`
- 状态落地入口：`engine/state_apply.py` - 状态落地入口：`sync_state_machine/engine/state_apply.py`
### 执行规则 ### 执行规则
1. 业务层构造 `context`。 1. 业务层构造 `context`。
docs/datasource_guide.md
+8 -6
View File
@@ -38,7 +38,7 @@ JSONL 数据源对应实现：
- 从目录中按 `{node_type}_N.jsonl` 规则发现并加载数据 - 从目录中按 `{node_type}_N.jsonl` 规则发现并加载数据
- 批量 create/update/delete 时，先改内存缓存，再统一刷盘 - 批量 create/update/delete 时，先改内存缓存，再统一刷盘
- 支持 `read_only`，在只读模式下禁止写回 - 支持 `read_only`，在只读模式下禁止写回
- 支持按项目过滤（`target_project_ids`）在 load 阶段预过滤 - 支持按项目过滤（`handler_configs.project.data_id_filter`）在 load 阶段预过滤
### 2.2 行为特点 ### 2.2 行为特点
@@ -53,7 +53,7 @@ JSONL 数据源对应实现：
- `type: jsonl` - `type: jsonl`
- `jsonl_dir`: 数据目录 - `jsonl_dir`: 数据目录
- `read_only`: 是否只读 - `read_only`: 是否只读
- `target_project_ids`: 项目过滤列表 - `handler_configs.project.data_id_filter`: 项目过滤列表
- `handler_configs`: 按节点类型的 handler 参数 - `handler_configs`: 按节点类型的 handler 参数
--- ---
@@ -159,8 +159,9 @@ local_datasource:
type: jsonl type: jsonl
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
read_only: false read_only: false
target_project_ids: [] handler_configs:
handler_configs: {} project:
data_id_filter: []
``` ```
### 4.2 API 数据源配置示例（含限流） ### 4.2 API 数据源配置示例（含限流）
@@ -181,9 +182,10 @@ remote_datasource:
api_rate_limit_max_requests: 10 api_rate_limit_max_requests: 10
api_rate_limit_window_seconds: 10.0 api_rate_limit_window_seconds: 10.0
target_project_ids: handler_configs:
project:
data_id_filter:
- "project-id-1" - "project-id-1"
handler_configs: {}
``` ```
### 4.3 配置模型位置 ### 4.3 配置模型位置
docs/ecm_sync_followup_plan.md
+111
View File
@@ -0,0 +1,111 @@
# ECM Sync Follow-up Plan
## 目标
本轮先落两件事：
1. 项目聚合读取接口，降低同步读取请求数。
2. 增强现有 validate，让 strategy 可以直接复用 schema diff validator，并输出聚合日志。
第三件事作为下一步：在 backend 里为 data mapping 增加静态样本测试，绕开 pipeline，直接验证本地样本和远程样本在 sync schema 层是否足够一致。
## 当前结论
### 1. 项目聚合读取接口
现有按业务拆散的读取方式请求数过高，不适合大规模同步。
本轮改为提供统一项目聚合接口，最小聚合范围：
- `project`
- `project_detail_data`
- `preparation`
- `production`
- `contract`
目标：
- 以项目为同步分片单位。
- 减少绝大多数项目级 fan-out 请求。
- 保持写入侧业务流程不变，只优化读取侧接口粒度。
接口返回的数据仍然基于现有 service 组装，不新增业务含义。
### 2. validate 增强方案
现有 validate 主要停留在 schema 校验，缺少可读的聚合 diff 输出。
本轮新增 schema diff validator，原则：
- 仍以现有 sync schema 为比较中心。
- 支持按 schema 聚合。
- 支持忽略字段配置。
- 支持输出字段级聚合统计。
- 可以被 strategy 直接引用，也可以脱离 pipeline 单独调用。
建议输出格式：
- `field_name`
- `mismatch_count / total_count`
- `schema_refs`
- `sample local/remote`
例如：
```text
ProjectDiff
code 12/30 [ProjectResponseBase, ProjectBaseInfoUpdate] sample[project-1]=""/"012A"
```
### 3. data mapping 静态测试
下一步在 depm backend 中做静态 mapping 测试，目标不是验证整条 pipeline，而是验证 mapping 本身。
测试输入：
- 本地数据静态文件
- 远程数据静态文件
要求：
- 本地样本通过本地 `data_mapping` 转成 sync schema。
- 远程样本直接按 sync schema 校验。
- 忽略掉少量必须动态生成或动态获取的字段。
- 通过统一 validator 输出字段级 diff 报告。
这类测试应当绕开推送系统，通过手工配对样本来验证 mapping，而不是混在 pipeline 里排查。
## 已落地项
### 项目聚合接口
backend 中新增项目同步聚合读取接口，按项目统一返回：
- `project`
- `project_detail_data`
- `preparation`
- `production`
- `contract`
### schema diff validator
同步引擎新增可独立调用的 schema diff validator：
- 支持忽略顶层字段
- 支持忽略列表元素内特定子字段
- 支持输出字段级 mismatch 聚合统计
- strategy 在 update diff 阶段直接复用该 validator
## 下一步
### 先做
1. 为 `project` 建第一批静态样本。
2. 用统一 validator 对本地样本和远程样本做 schema 层比较。
3. 只在 validator 配置里维护少量 ignore 字段，不额外引入 contract 文件。
### 暂不做
1. 不引入 schema 外的独立契约系统。
2. 不为所有 domain 一次性补齐静态样本。
3. 不把副作用字段强行纳入完全一致比较。
docs/factory_registry_refactor_plan.md
-1
View File
@@ -93,7 +93,6 @@
- endpoint 创建 handler - endpoint 创建 handler
- endpoint 统一注入： - endpoint 统一注入：
- `handler_config` - `handler_config`
- `target_project_ids`
- datasource-specific runtime object（如 api client） - datasource-specific runtime object（如 api client）
最终让 `factory` 只负责： 最终让 `factory` 只负责：
docs/getting_started.md
+128
View File
@@ -0,0 +1,128 @@
# 快速上手
这份文档只服务第一次接手 `ecm_sync_system` 的开发者。目标不是完整讲架构，而是让你先跑起来、知道从哪里入手，再去看更细的说明。
## 1. 先做什么
第一次进入仓库时，建议按下面顺序执行：
1. 准备 Python 3.9+。
2. 在仓库根目录创建并激活虚拟环境。
3. 执行 `pip install -e .` 安装当前项目。
4. 跑状态机配置校验，确认基础工具链正常。
5. 跑一条轻量测试，确认测试与渲染工具链可用。
6. 再去看架构文档和状态机文档。
推荐命令：
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
python run.py --help
python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml
pytest tests/integration/test_toolchain_config_and_graph.py -q
```
## 2. 仓库里最重要的目录
- `sync_state_machine/`
- 真正的引擎实现目录。大多数“核心代码入口”都在这里。
- `run_profiles/`
- 运行配置与模板。
- `tests/`
- 单元测试与集成测试。
- `tools/`
- 状态机校验、图渲染、远程环境等工具。
- `scripts/`
- 自动化测试编排和数据集相关脚本。
- `docs/`
- 面向开发者的设计、运行和排障文档。
## 3. 先读哪几篇文档
按任务来看：
- 想知道系统整体怎么分层：`docs/architecture.md`
- 想知道状态机怎么工作：`docs/node_state_definition.md` 和 `docs/state_machine.md`
- 想知道一次同步是怎么跑的：`docs/sync_flow.md`
- 想调运行配置：`docs/runtime_config_guide.md`
- 想接新 datasource 或 handler：`sync_state_machine/datasource/README.md` 和 `docs/library_integration_guide.md`
- 想看测试体系：`docs/testing_guide.md` 和 `tests/README.md`
- 想做排障：`docs/operations.md`
## 4. 新人最容易踩的坑
### 4.1 先用仓库入口，不要默认 shell 里已经有命令
如果你还没执行过 `pip install -e .`，那么 `ecm-sync-run` 不一定可用。
首次调试建议优先用：
```bash
python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
```
安装完成后，再切换到：
```bash
ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
```
### 4.2 核心实现路径以 `sync_state_machine/` 为准
阅读历史文档时，如果看到类似下面的写法：
- `pipeline/full_sync_pipeline.py`
- `sync_system/strategy.py`
- `datasource/datasource.py`
- `engine/events.py`
请优先理解为它们在当前仓库中的真实位置分别是：
- `sync_state_machine/pipeline/full_sync_pipeline.py`
- `sync_state_machine/sync_system/strategy.py`
- `sync_state_machine/datasource/datasource.py`
- `sync_state_machine/engine/events.py`
## 5. 如果你要开始改代码
### 5.1 改 domain 行为
优先查看：
- `sync_state_machine/domain/<node_type>/`
- `sync_state_machine/sync_system/strategy_ops/`
- `tests/unit/` 中对应测试
### 5.2 改状态机语义
优先查看：
- `sync_state_machine/config/node_state_machine.yaml`
- `sync_state_machine/engine/events.py`
- `sync_state_machine/engine/state_apply.py`
改完建议至少执行：
```bash
python tools/validate_config.py --config sync_state_machine/config/node_state_machine.yaml
pytest tests/integration/test_toolchain_config_and_graph.py -q
```
### 5.3 改运行配置或 datasource
优先查看：
- `docs/runtime_config_guide.md`
- `run_profiles/README.md`
- `sync_state_machine/datasource/README.md`
## 6. 出错先看哪里
推荐排查顺序：
1. 先看运行日志中的 `Resolved Full Config`。
2. 再看节点 `sync_log / status / error`。
3. 再对照 `docs/operations.md` 和 `docs/state_machine.md`。
4. 如果是配置问题，先让校验报错，不要绕过 schema 或状态机校验。
docs/library_integration_guide.md
+18 -6
View File
@@ -11,7 +11,7 @@
推荐接入方式： 推荐接入方式：
- `sync_state_machine` 负责同步引擎、状态机、pipeline、datasource 抽象。 - `sync_state_machine` 负责同步引擎、状态机、pipeline、datasource 抽象。
- 业务系统负责提供自己的数据源适配层，不要反向让 `sync_state_machine` 依赖业务系统。 - 业务系统负责提供自己的 datasource / handler 适配层，不要反向让 `sync_state_machine` 依赖业务系统。
--- ---
@@ -46,13 +46,18 @@ ecm-sync-run --config_path=run_profiles/preset/jsonl_to_api.default.yaml
- `sync_state_machine.build_config` - `sync_state_machine.build_config`
- `sync_state_machine.build_default_config` - `sync_state_machine.build_default_config`
- `sync_state_machine.run_pipeline_from_config`
- `sync_state_machine.create_pipeline_from_config` - `sync_state_machine.create_pipeline_from_config`
- `sync_state_machine.build_config_from_file`
- `sync_state_machine.PipelineRunConfig` - `sync_state_machine.PipelineRunConfig`
- `sync_state_machine.JsonlDataSourceConfig` - `sync_state_machine.JsonlDataSourceConfig`
- `sync_state_machine.ApiDataSourceConfig` - `sync_state_machine.ApiDataSourceConfig`
- `sync_state_machine.DomainRegistry` - `sync_state_machine.DomainRegistry`
配置加载分两步：
1. 先构造或读取 `PipelineRunConfig`
2. 再交给 `create_pipeline_from_config()` 创建 pipeline
内置 domain 注册会在导入包时自动完成。 内置 domain 注册会在导入包时自动完成。
--- ---
@@ -200,7 +205,7 @@ class DemoJsonlHandler(BaseJsonlHandler):
--- ---
### 6.2 如果 backend 是数据库：建议写“数据库 handler / adapter”，不要让引擎直接依赖 ORM ### 6.2 如果 backend 是数据库：建议写 handler / adapter，不要让引擎直接依赖 ORM
推荐结构： 推荐结构：
@@ -234,6 +239,7 @@ backend/
- 不要在 `sync_state_machine` 包里 import backend 的 model/repository - 不要在 `sync_state_machine` 包里 import backend 的 model/repository
- 不要让 `DomainRegistry` 注册逻辑散落在 backend 业务层之外 - 不要让 `DomainRegistry` 注册逻辑散落在 backend 业务层之外
- 不要在 datasource 之外做额外的 project 过滤；项目范围应通过 `handler_configs.project.data_id_filter` 下沉到具体 handler 解释
--- ---
@@ -268,6 +274,7 @@ backend/
- `strategy_class` - `strategy_class`
- `jsonl_handler_class` - `jsonl_handler_class`
- `api_handler_class` - `api_handler_class`
- 以及任意通过 `register_handler()` 追加的 datasource handler
示例： 示例：
@@ -282,9 +289,11 @@ DomainRegistry.register(
jsonl_handler_class=DemoJsonlHandler, jsonl_handler_class=DemoJsonlHandler,
api_handler_class=DemoApiHandler, api_handler_class=DemoApiHandler,
) )
DomainRegistry.register_handler("demo", "custom_remote", DemoCustomHandler)
``` ```
如果是 backend 接入，建议在 backend 的 integration 启动阶段集中注册，而不是分散在 controller / service 里。 如果是 backend 接入，建议在启动入口集中注册，而不是分散在 controller / service 里。
--- ---
@@ -311,7 +320,9 @@ async def run_demo_sync() -> None:
"api_base_url": "https://example.com", "api_base_url": "https://example.com",
"api_uid": "xxx", "api_uid": "xxx",
"api_secret": "xxx", "api_secret": "xxx",
"target_project_ids": ["demo-project"], "handler_configs": {
"project": {"data_id_filter": ["demo-project"]}
},
}, },
} }
@@ -353,8 +364,9 @@ async def run_demo_sync() -> None:
## 11. 相关文件 ## 11. 相关文件
- 包入口：[sync_state_machine/__init__.py](../sync_state_machine/__init__.py) - 包入口：[sync_state_machine/__init__.py](../sync_state_machine/__init__.py)
- CLI：[sync_state_machine/cli.py](../sync_state_machine/cli.py) - CLI / 运行入口：[sync_state_machine/cli.py](../sync_state_machine/cli.py)
- 配置工厂：[sync_state_machine/config](../sync_state_machine/config) - 配置工厂：[sync_state_machine/config](../sync_state_machine/config)
- datasource 基础层：[sync_state_machine/datasource](../sync_state_machine/datasource) - datasource 基础层：[sync_state_machine/datasource](../sync_state_machine/datasource)
- pipeline 工厂：[sync_state_machine/pipeline/factory.py](../sync_state_machine/pipeline/factory.py) - pipeline 工厂：[sync_state_machine/pipeline/factory.py](../sync_state_machine/pipeline/factory.py)
- domain 注册：[sync_state_machine/common/registry.py](../sync_state_machine/common/registry.py) - domain 注册：[sync_state_machine/common/registry.py](../sync_state_machine/common/registry.py)
- backend 集成入口：[backend/app/thirdparty/ecm_sync/services/sync_runner.py](../backend/app/thirdparty/ecm_sync/services/sync_runner.py)
docs/runtime_config_guide.md
+166 -11
View File
@@ -2,6 +2,93 @@
本文档从业务视角说明配置字段含义、对阶段行为的影响，以及推荐的覆盖方式。 本文档从业务视角说明配置字段含义、对阶段行为的影响，以及推荐的覆盖方式。
## 0. 配置约定
当前运行配置分成两层：
1. 输入配置
- 你写在 `run_profiles/*.yaml` 或 `run_profiles/preset/*.default.yaml` 里的内容。
- 只建议写必要覆盖项，不要把默认值全部展开。
2. Resolved Full Config
- pipeline 启动时打印并写入日志的全量配置。
- 其中已经补齐 datasource 默认值、策略默认值和日志/持久化默认值。
也就是说：
- 模板应当尽量短。
- 排障时看启动日志里的 **Resolved Full Config**。
### 0.1 一个最小输入示例
```yaml
local_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
remote_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/remote_datasource/datasource
persist:
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
backend: sqlite
backend_options: {}
strategies:
project:
config:
auto_bind: true
auto_bind_fields:
- name
```
### 0.2 对应的全量 resolved 结果骨架
```yaml
node_types:
- company
- supplier
- user
- project
...
local_datasource:
type: jsonl
jsonl_dir: /abs/path/filtered_datasource/datasource/filtered
read_only: false
handler_configs: {}
remote_datasource:
type: jsonl
jsonl_dir: /abs/path/remote_datasource/datasource
read_only: false
handler_configs: {}
strategies:
- node_type: project
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- name
pre_bind_data_id: []
depend_fields:
company_id: company
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
field_direction_key: null
field_direction_overrides: {}
post_check_ignore_fields: []
compare_ignore_fields: []
persist:
backend: sqlite
db_path: /abs/path/test_results/sync_state_machine/demo.db
backend_options: {}
enable: true
wipe_on_start: false
logging:
initialize: true
file_path: null
file_dir: null
file_name_pattern: null
level: 20
suppress_node_logs: true
```
## 1. 推荐使用方式（先 default，再覆盖） ## 1. 推荐使用方式（先 default，再覆盖）
1. 先使用基线配置运行： 1. 先使用基线配置运行：
@@ -18,15 +105,51 @@
- `node_types` - `node_types`
- 本轮参与同步的业务类型列表。 - 本轮参与同步的业务类型列表。
- 顺序会影响依赖链执行时机（当前按 node_type 串行执行 bind/create/update）。 - 顺序会影响依赖链执行时机（当前按 node_type 串行执行 bind/create/update）。
- `target_project_ids`
- 兼容字段。作为两侧 datasource 的项目白名单兜底值。
- 若 datasource 自身配置了 `target_project_ids`，则优先使用 datasource 级配置。
- `local_datasource` / `remote_datasource` - `local_datasource` / `remote_datasource`
- 分别定义两侧数据来源与执行端。 - 分别定义两侧数据来源与执行端。
- `strategies` - `strategies`
- 每个业务类型的行为策略（依赖、自动绑定、创建方向、更新方向、跳过等）。 - 每个业务类型的行为策略（依赖、自动绑定、创建方向、更新方向、跳过等）。
- `persist` - `persist`
- 持久化开关与数据库位置。 - 持久化开关与数据库位置。
- `persist.backend`
- 持久化后端注册表里的名称，例如 `sqlite`。
- backend 实现会自行解释 `persist` 的其余字段。
- `persist.backend_options`
- 后端私有配置，后端自己决定如何解释。
### 6. 持久化后端注册
如果你希望在 backend 初始化阶段替换持久化后端，可以先在启动代码里按名称注册，再通过配置选择，不需要改调用方。
#### 6.1 内置 SQLite
```yaml
persist:
backend: sqlite
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
backend_options: {}
```
#### 6.2 注册自定义后端
```python
from sync_state_machine.common.persistence import register_persistence_backend
register_persistence_backend("mysql", lambda config: MySQLPersistenceBackend(config))
```
然后在配置里选择这个名字：
```yaml
persist:
backend: mysql
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
backend_options:
url: mysql+aiomysql://user:pass@127.0.0.1/depm_sync?charset=utf8mb4
pool_size: 5
```
后端实现会收到整份 `PersistConfig`，自行决定是读取 `db_path`、`backend_options`，还是额外扩展别的字段。
- `logging` - `logging`
- 运行日志输出位置与级别。 - 运行日志输出位置与级别。
@@ -37,8 +160,8 @@
- `type=jsonl`：使用本地 JSONL 文件作为数据源。 - `type=jsonl`：使用本地 JSONL 文件作为数据源。
- `jsonl_dir`：JSONL 目录路径。 - `jsonl_dir`：JSONL 目录路径。
- `read_only=false`：允许该侧执行 create/update 回写；若为 true，通常用于只读加载。 - `read_only=false`：允许该侧执行 create/update 回写；若为 true，通常用于只读加载。
- `target_project_ids=[]`：该侧项目白名单；为空时不进行项目筛选。 - `handler_configs.project.data_id_filter=[]`：项目白名单；为空时不进行项目筛选。
- `handler_configs`：预留给各业务 handler 的扩展参数。 - `handler_configs`：按节点类型下发给各业务 handler 的扩展参数。
### 3.2 API 数据源 ### 3.2 API 数据源
@@ -46,12 +169,12 @@
- `api_base_url` / `api_uid` / `api_secret`：接口访问参数。 - `api_base_url` / `api_uid` / `api_secret`：接口访问参数。
- `api_debug`：开启后输出更多调试信息。 - `api_debug`：开启后输出更多调试信息。
- `poll_max_retries` / `poll_interval`：异步任务轮询次数与间隔。 - `poll_max_retries` / `poll_interval`：异步任务轮询次数与间隔。
- `target_project_ids`：**必填**，该侧项目白名单；至少需要指定一个项目ID，以避免加载过多远程数据。 - `handler_configs.project.data_id_filter`：建议显式配置的项目白名单；至少需要指定一个项目ID，以避免加载过多远程数据。
- `handler_configs`：业务 handler 扩展参数。 - `handler_configs`：业务 handler 扩展参数。
## 4. 策略字段（Strategy） ## 4. 策略字段（Strategy）
每个 node_type 都建议显式写出以下字段，便于审阅与排障。 每个 node_type 的 **Resolved Full Config** 都会包含以下字段，但输入配置不需要全部显式写出。
### 4.1 `update_direction` ### 4.1 `update_direction`
@@ -91,7 +214,12 @@
- `local_orphan_action: none` - `local_orphan_action: none`
- `remote_orphan_action: none` - `remote_orphan_action: none`
- `update_direction: none` - `update_direction: none`
- 当三者同时为 `none/none/none` 时，pipeline 会跳过该 node_type 的 bind/create/update。 - `skip_sync=true` 的当前语义是：
- 仍会执行该 node_type 的 load / bind
- 只跳过 create / update 写入阶段
- 适合“保留绑定与依赖可见性，但临时禁止写入”的场景。
- `none/none/none` 是更推荐的输入风格，因为它表达的是“该类型不做创建、不做补本地、不做更新”。
- 这组配置不会绕过前置 load / bind / 依赖检查；它只会把该类型变成无写入策略。
### 4.6 Handler 级参数（`handler_configs`） ### 4.6 Handler 级参数（`handler_configs`）
@@ -100,6 +228,32 @@
- `remote_datasource.handler_configs.supplier.load_mode: persisted_only` - `remote_datasource.handler_configs.supplier.load_mode: persisted_only`
- `persisted_only` 表示只使用已持久化数据，不主动从外部拉新数据（具体以对应 handler 实现为准）。 - `persisted_only` 表示只使用已持久化数据，不主动从外部拉新数据（具体以对应 handler 实现为准）。
### 4.7 `field_direction_key` / `field_direction_overrides`
- 这两个字段只在少数 node_type 上需要，典型例子是 `project_detail_data`。
- `field_direction_key`
- 指定从节点 data 里取哪个字段作为“方向查表键”。
- 例如 `key`，表示用 `data["key"]` 决定这条记录是 push 还是 pull。
- `field_direction_overrides`
- 指定“字段值 -> 更新方向”的映射。
- 没命中的值继续使用默认 `update_direction`。
示例：
```yaml
project_detail_data:
config:
field_direction_key: key
field_direction_overrides:
ensure_production: pull
climb_production: pull
challenge_production: pull
```
含义：
- `key=ensure_production/climb_production/challenge_production` 时，从远端拉回本地。
- 其余 key 继续按默认 `update_direction=push` 处理。
## 5. 阶段行为与配置关系 ## 5. 阶段行为与配置关系
- bind 阶段 - bind 阶段
@@ -111,14 +265,15 @@
- update 阶段 - update 阶段
- `update_direction` 决定是否和向哪侧更新。 - `update_direction` 决定是否和向哪侧更新。
- 写入后 reload（Pipeline 内置） - 写入后 reload（Pipeline 内置）
- 当某个 node_type 的 create 或 update 发生实际写入成功后，pipeline 会自动对该 node_type 重新执行两侧 `load`。 - 对 API / 需要重新拉取权威状态的数据源，create 或 update 成功后仍会按 node_type reload。
- 目的：让后续 update 判断基于最新远端/本地数据，支持“create 后再补一次 update”的链路。 - 对纯 JSONL -> JSONL pipeline，reload 会跳过，直接复用当前内存状态。
- 目的：避免对离线 JSONL 数据做无意义重复扫描，同时保留 API 数据源的权威状态刷新。
- 同步后一致性检查（Pipeline 内置） - 同步后一致性检查（Pipeline 内置）
- 每个 node_type 在本轮同步后会基于绑定对执行数据一致性校验（比较时忽略 `id/_id` 字段）。 - 每个 node_type 在本轮同步后会基于绑定对执行数据一致性校验（比较时忽略 `id/_id` 字段）。
- 若发现差异，会在日志中打印差异样本，并在汇总表新增 `Consistent` 列展示 `Y` / `N(差异数)`。 - 若发现差异，会在日志中打印差异样本，并在汇总表新增 `Consistent` 列展示 `Y` / `N(差异数)`。
- 全局跳过 - 全局跳过
- `local_orphan_action=none` + `remote_orphan_action=none` + `update_direction=none` - `local_orphan_action=none` + `remote_orphan_action=none` + `update_direction=none`
可直接跳过该类型的 bind/create/update。 可将该类型收敛为“只做 load/bind，不做 create/update 写入”的只读同步策略。
## 6. 覆盖配置建议 ## 6. 覆盖配置建议
docs/state_machine.md
+11 -11
View File
@@ -25,17 +25,17 @@
## 3. 事件到代码映射 ## 3. 事件到代码映射
### bootstrap ### bootstrap
- `E01`：`engine/events.py::e01_bootstrap()` - `E01`：`sync_state_machine/engine/events.py::e01_bootstrap()`
### bind ### bind
- `E10`：`engine/events.py::e10_bind_core()` - `E10`：`sync_state_machine/engine/events.py::e10_bind_core()`
- `E15`：`engine/events.py::e15_bind_dependency()` - `E15`：`sync_state_machine/engine/events.py::e15_bind_dependency()`
- `E16`：`engine/events.py::e16_bind_auto()`（自动绑定判定；`ZERO+create_enabled=true` 时进入 `S13`，等待创建副作用提交） - `E16`：`sync_state_machine/engine/events.py::e16_bind_auto()`（自动绑定判定；`ZERO+create_enabled=true` 时进入 `S13`，等待创建副作用提交）
### create/update prepare ### create/update prepare
- `E20`：`engine/events.py::e20_create_prepare()`（spawn 成功从 `S13` 提交到 `S01`；失败保持 `S13`） - `E20`：`sync_state_machine/engine/events.py::e20_create_prepare()`（spawn 成功从 `S13` 提交到 `S01`；失败保持 `S13`）
- `E21`：`engine/events.py::e21_create_prepare()` - `E21`：`sync_state_machine/engine/events.py::e21_create_prepare()`
- `E30`：`engine/events.py::e30_update_prepare()` - `E30`：`sync_state_machine/engine/events.py::e30_update_prepare()`
### sync execute ### sync execute
- `E40C_START/E40C`：CREATE 执行入口与结果回写 - `E40C_START/E40C`：CREATE 执行入口与结果回写
@@ -43,10 +43,10 @@
- `E40D_START/E40D`：DELETE 执行入口与结果回写 - `E40D_START/E40D`：DELETE 执行入口与结果回写
## 4. strategy 调用链（当前） ## 4. strategy 调用链（当前）
- bind 主入口：`sync_system/strategy.py::DefaultSyncStrategy.bind()` -> `strategy_ops/bind_ops.py::run_bind()` - bind 主入口：`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.bind()` -> `sync_state_machine/sync_system/strategy_ops/bind_ops.py::run_bind()`
- create 主入口：`sync_system/strategy.py::DefaultSyncStrategy.create()` -> `strategy_ops/create_ops.py::run_create()` - create 主入口：`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.create()` -> `sync_state_machine/sync_system/strategy_ops/create_ops.py::run_create()`
- update 主入口：`sync_system/strategy.py::DefaultSyncStrategy.update()` -> `strategy_ops/update_ops.py::run_update()` - update 主入口：`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.update()` -> `sync_state_machine/sync_system/strategy_ops/update_ops.py::run_update()`
- reset 主入口：`sync_system/strategy.py::BaseSyncStrategy.run_reset()` -> `strategy_ops/reset_ops.py::run_phase2_cleanup()` - reset 主入口：`sync_state_machine/sync_system/strategy.py::BaseSyncStrategy.run_reset()` -> `sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`
## 5. context 字段（以 YAML 为准） ## 5. context 字段（以 YAML 为准）
docs/sync_flow.md
+14 -12
View File
@@ -2,20 +2,22 @@
## 1. 全流程主线 ## 1. 全流程主线
1. pipeline 装配组件并加载持久化。 1. pipeline 装配组件并加载持久化。
2. pipeline 调用 `strategy.bind()`。 2. pipeline 为 datasource 绑定 collection，但不在 bootstrap 阶段全量拉取业务数据。
3. pipeline 调用 `strategy.create()` / `strategy.update()` 生成动作节点。 3. pipeline 在每个 `node_type` 开始执行前，单独加载该 `node_type` 的最新数据。
4. pipeline 调用 `datasource.sync_all()` 执行动作并回写执行状态。 4. pipeline 调用 `strategy.bind()`。
5. pipeline 输出统计并持久化。 5. pipeline 调用 `strategy.create()` / `strategy.update()` 生成动作节点。
6. pipeline 调用 `datasource.sync_all()` 执行动作并回写执行状态。
7. pipeline 输出统计并持久化。
## 2. reset 时序 ## 2. reset 时序
### 2.1 加载时 reset（phase1 语义） ### 2.1 加载时 reset（phase1 语义）
- 在 `common/collection.py::load_from_persistence()` 执行。 - 在 `sync_state_machine/common/collection.py::load_from_persistence()` 执行。
- 语义：仅恢复持久化状态（不在此阶段做状态归并）。 - 语义：仅恢复持久化状态（不在此阶段做状态归并）。
### 2.2 加载后 cleanup（phase2 语义） ### 2.2 加载后 cleanup（phase2 语义）
- pipeline 调用 `BaseSyncStrategy.run_reset()`。 - pipeline 调用 `BaseSyncStrategy.run_reset()`。
- 实际实现：`strategy_ops/reset_ops.py::run_phase2_cleanup()`。 - 实际实现：`sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`。
- 核心事件：`E01`（`S16 -> S00/S15`）。 - 核心事件：`E01`（`S16 -> S00/S15`）。
- `S16` 表示“持久化加载输入态”：本轮从持久化恢复出的节点集合（可包含上轮遗留节点，也可包含上轮已稳定节点）。 - `S16` 表示“持久化加载输入态”：本轮从持久化恢复出的节点集合（可包含上轮遗留节点，也可包含上轮已稳定节点）。
- 判定： - 判定：
@@ -32,7 +34,7 @@
- 这类节点不会进入 `E01`，也不会参与 bind/create/update/sync 自动流程。 - 这类节点不会进入 `E01`，也不会参与 bind/create/update/sync 自动流程。
## 3. bind 流程（strategy_ops） ## 3. bind 流程（strategy_ops）
- 入口：`strategy_ops/bind_ops.py::run_bind()` - 入口：`sync_state_machine/sync_system/strategy_ops/bind_ops.py::run_bind()`
- 入口状态：`S00` - 入口状态：`S00`
### 核心绑定判定 ### 核心绑定判定
@@ -55,7 +57,7 @@
## 4. create/update 准备 ## 4. create/update 准备
### create ### create
- 入口：`strategy_ops/create_ops.py::run_create()` - 入口：`sync_state_machine/sync_system/strategy_ops/create_ops.py::run_create()`
- 运行时事件：`E20`（`E21/E22` 为配置保留分支，当前 strategy 未接入自动执行） - 运行时事件：`E20`（`E21/E22` 为配置保留分支，当前 strategy 未接入自动执行）
- 运行时入口状态：`S13` - 运行时入口状态：`S13`
- 说明：`E16` 负责识别“需自动创建”并将源节点置为 `S13`；业务代码执行对侧 spawn+bind 后，再通过 `E20` 将 `S13` 提交到 `S01`（成功）；若 spawn 失败，`E20` 不迁移并保留在 `S13` 等待人工/重试。 - 说明：`E16` 负责识别“需自动创建”并将源节点置为 `S13`；业务代码执行对侧 spawn+bind 后，再通过 `E20` 将 `S13` 提交到 `S01`（成功）；若 spawn 失败，`E20` 不迁移并保留在 `S13` 等待人工/重试。
@@ -64,19 +66,19 @@
- 新创建目标节点在 `S06/S10C` 阶段允许 `data_id` 为空；`CREATE SUCCESS`（`S11C`）必须携带非空 `result_data_id`，否则事件直接报错。 - 新创建目标节点在 `S06/S10C` 阶段允许 `data_id` 为空；`CREATE SUCCESS`（`S11C`）必须携带非空 `result_data_id`，否则事件直接报错。
### update ### update
- 入口：`strategy_ops/update_ops.py::run_update()` - 入口：`sync_state_machine/sync_system/strategy_ops/update_ops.py::run_update()`
- 事件：`E30` - 事件：`E30`
- 入口状态（source 节点）：`S01` - 入口状态（source 节点）：`S01`
- `target_has_data_id=false` 或 `update_id_resolve_ok=false` -> `S08` - `target_has_data_id=false` 或 `update_id_resolve_ok=false` -> `S08`
- `has_diff=true` -> `S07` - `has_diff=true` -> `S07`
### post-create 归并 ### post-create 归并
- 入口：`pipeline/full_sync_pipeline.py::normalize_create_success_to_update_entry()` - 入口：`sync_state_machine/pipeline/full_sync_pipeline.py::normalize_create_success_to_update_entry()`
- 事件：`E41` - 事件：`E41`
- 语义：`CREATE` 执行成功后，节点先落 `S11C`（SUCCESS），再归并 `S11C -> S01`，作为同轮 update 起点。 - 语义：`CREATE` 执行成功后，节点先落 `S11C`（SUCCESS），再归并 `S11C -> S01`，作为同轮 update 起点。
## 5. 执行回写（datasource） ## 5. 执行回写（datasource）
- 入口：`datasource/datasource.py` - 入口：`sync_state_machine/datasource/datasource.py`
- 事件：`E40C_START/E40C`、`E40U_START/E40U`、`E40D_START/E40D` - 事件：`E40C_START/E40C`、`E40U_START/E40U`、`E40D_START/E40D`
- 当前约束：delete stage 尚未接入 strategy 主流程；若运行时检测到 `S09/DELETE` 节点，DataSource 会显式抛错阻断（避免散落删除执行）。 - 当前约束：delete stage 尚未接入 strategy 主流程；若运行时检测到 `S09/DELETE` 节点，DataSource 会显式抛错阻断（避免散落删除执行）。
- 执行准备入口状态： - 执行准备入口状态：
@@ -103,5 +105,5 @@
| 隔离（人工） | `S17` | 不参与自动阶段，仅人工处理 | | 隔离（人工） | `S17` | 不参与自动阶段，仅人工处理 |
## 7. 不变量保障 ## 7. 不变量保障
- 每次事件落地后，`engine/events.py::_apply_decision()` 会执行 `runtime.check_invariants()`。 - 每次事件落地后，`sync_state_machine/engine/events.py::_apply_decision()` 会执行 `runtime.check_invariants()`。
- 违反 invariant 直接抛错，终止当前路径。 - 违反 invariant 直接抛错，终止当前路径。
docs/testing_guide.md
+6
View File
@@ -16,6 +16,11 @@
- 校验 datasource/handler 工具链、配置严格性、导入面稳定性。 - 校验 datasource/handler 工具链、配置严格性、导入面稳定性。
- 对具体 domain 的 update/create 行为做快速回归。 - 对具体 domain 的 update/create 行为做快速回归。
单元测试取舍原则：
- 优先测文档里承诺的行为语义，不优先测单纯转发、setter、空包装器。
- 如果一个测试只是在断言“函数 A 调了函数 B”，但不验证最终行为，通常应该删掉或改写成行为测试。
- 如果需要 test double，优先让它承载真实行为断言，不保留仅为凑接口而存在的空壳类。
常用命令： 常用命令：
- 全量单元测试：`pytest tests/unit -q` - 全量单元测试：`pytest tests/unit -q`
- 指定文件：`pytest tests/unit/test_update_ops.py -q` - 指定文件：`pytest tests/unit/test_update_ops.py -q`
@@ -165,6 +170,7 @@ python tools/remote_env.py \
1. 先跑 `pytest tests/unit -q`。 1. 先跑 `pytest tests/unit -q`。
2. 如果改动影响 pipeline，再跑目标集成测试。 2. 如果改动影响 pipeline，再跑目标集成测试。
3. 如果改动影响真实接口流程，再使用 `tools/remote_env.py` 管理远程环境。 3. 如果改动影响真实接口流程，再使用 `tools/remote_env.py` 管理远程环境。
4. 如果修改了配置语义或 pipeline 控制流，优先补一条直接覆盖设计语义的 pytest，而不是只补装配/转发测试。
### 3.2 真实接口联调 ### 3.2 真实接口联调
filtered_datasource/datasource/filtered/company_1.jsonl
+1 -1
View File
@@ -1 +1 @@
{"_id": "a1d69bf1-b95e-4832-9258-5bc35c629e9a", "_rev": "2-e265a6cb3674c71d6957883177b270cb", "type": "company", "name": "金上公司", "short_name": "金上公司", "code": "028B", "company_type": "region", "parent_id": "5da30aa7-b560-416e-9223-284290692351", "path": ["5da30aa7-b560-416e-9223-284290692351", "a1d69bf1-b95e-4832-9258-5bc35c629e9a"], "path_name": ["集团公司", "金上公司"], "province": "510000", "city": "510100", "status": "active", "sort": 27, "created_by": "system", "created_by_name": "system", "created_at": "2025-05-30 09:00:09", "updated_by": "system", "updated_by_name": "system", "updated_at": "2025-09-25 20:08:42", "company_nature": ["owner"], "id": "a1d69bf1-b95e-4832-9258-5bc35c629e9a", "credit_code": null, "city_name": "成都市", "province_name": "四川省", "parent_name": "集团公司"} {"_id": "a1d69bf1-b95e-4832-9258-5bc35c629e9a", "_rev": "2-e265a6cb3674c71d6957883177b270cb", "type": "company", "name": "金上公司", "short_name": "金上公司", "code": "028B", "company_type": "region", "parent_id": "5da30aa7-b560-416e-9223-284290692351", "path": ["5da30aa7-b560-416e-9223-284290692351", "a1d69bf1-b95e-4832-9258-5bc35c629e9a"], "path_name": ["集团公司", "金上公司"], "province": "510000", "city": "510100", "status": "active", "sort": 27, "created_by": "system", "created_by_name": "system", "created_at": "2025-05-30 09:00:09", "updated_by": "system", "updated_by_name": "system", "updated_at": "2025-09-25 20:08:42", "company_nature": ["owner"], "id": "a1d69bf1-b95e-4832-9258-5bc35c629e9a", "credit_code": null, "city_name": "成都市", "province_name": "四川省", "parent_name": ""}
filtered_datasource/datasource/filtered/supplier_1.jsonl
+28 -28
View File
@@ -1,28 +1,28 @@
{"_id": "0798431d-115a-4b1c-b95a-78470160fa9e", "_rev": "3-18cce2b2f9854c79aeb4329c31e6028c", "type": "supplier", "name": "沈阳中变电气有限责任公司", "short_name": "沈阳中变电气有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "210000", "city": "210100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91210100564677033W", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "210000", "210100"]} {"_id": "0798431d-115a-4b1c-b95a-78470160fa9e", "_rev": "3-18cce2b2f9854c79aeb4329c31e6028c", "type": "supplier", "name": "沈阳中变电气有限责任公司", "short_name": "沈阳中变电气有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "210000", "city": "210100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91210100564677033W", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "210000", "210100"], "id": "0798431d-115a-4b1c-b95a-78470160fa9e", "city_name": "沈阳市", "province_name": "辽宁省", "country_name": "中国"}
{"_id": "1098875c-e949-47b1-9f40-237eb507f890", "_rev": "3-0ef325ac17b9232224f5c9f924e2cf6f", "type": "supplier", "name": "中国水利水电第九工程局有限公司", "short_name": "中国水利水电第九工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "520000", "city": "520100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91520000214428387H", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "520000", "520100"]} {"_id": "1098875c-e949-47b1-9f40-237eb507f890", "_rev": "3-0ef325ac17b9232224f5c9f924e2cf6f", "type": "supplier", "name": "中国水利水电第九工程局有限公司", "short_name": "中国水利水电第九工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "520000", "city": "520100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91520000214428387H", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "520000", "520100"], "id": "1098875c-e949-47b1-9f40-237eb507f890", "city_name": "贵阳市", "province_name": "贵州省", "country_name": "中国"}
{"_id": "1115a7f3-b2c1-4bfb-8c89-605250c402dd", "_rev": "3-6e0a5e272f35cd473d6cd82a1ad6cfa3", "type": "supplier", "name": "中国安能集团第三工程局有限公司", "short_name": "中国安能集团第三工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915101006331302278", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]} {"_id": "1115a7f3-b2c1-4bfb-8c89-605250c402dd", "_rev": "3-6e0a5e272f35cd473d6cd82a1ad6cfa3", "type": "supplier", "name": "中国安能集团第三工程局有限公司", "short_name": "中国安能集团第三工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915101006331302278", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "1115a7f3-b2c1-4bfb-8c89-605250c402dd", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "1ff5b102-2f9e-496c-8591-1a8e32407c7f", "_rev": "3-2857c4610fd9e9a9b8313dee83454969", "type": "supplier", "name": "中国葛洲坝集团三峡建设工程有限公司", "short_name": "中国葛洲坝集团三峡建设工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "420000", "city": "420500", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91420000615573462P", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "420000", "420500"]} {"_id": "1ff5b102-2f9e-496c-8591-1a8e32407c7f", "_rev": "3-2857c4610fd9e9a9b8313dee83454969", "type": "supplier", "name": "中国葛洲坝集团三峡建设工程有限公司", "short_name": "中国葛洲坝集团三峡建设工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "420000", "city": "420500", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91420000615573462P", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "420000", "420500"], "id": "1ff5b102-2f9e-496c-8591-1a8e32407c7f", "city_name": "宜昌市", "province_name": "湖北省", "country_name": "中国"}
{"_id": "4be9e1da-9ebf-4a2a-b6bd-14a41ac31858", "_rev": "5-b54a288a89b9e55cee26d42b08bb69a0", "type": "supplier", "name": "中国水利水电第七工程局有限公司", "short_name": "中国水利水电第七工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510000201827469M", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]} {"_id": "4be9e1da-9ebf-4a2a-b6bd-14a41ac31858", "_rev": "5-b54a288a89b9e55cee26d42b08bb69a0", "type": "supplier", "name": "中国水利水电第七工程局有限公司", "short_name": "中国水利水电第七工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510000201827469M", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "4be9e1da-9ebf-4a2a-b6bd-14a41ac31858", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "357c7eaf-5a33-4999-be94-79c42bfbd955", "_rev": "3-35879fa40a8fbe379f03c7a4fa8696ad", "type": "supplier", "name": "广州擎天实业有限公司", "short_name": "广州擎天实业有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "440300", "city": "440100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91440101231240749H", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "440000", "440100"]} {"_id": "357c7eaf-5a33-4999-be94-79c42bfbd955", "_rev": "3-35879fa40a8fbe379f03c7a4fa8696ad", "type": "supplier", "name": "广州擎天实业有限公司", "short_name": "广州擎天实业有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "440300", "city": "440100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91440101231240749H", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "440000", "440100"], "id": "357c7eaf-5a33-4999-be94-79c42bfbd955", "city_name": "广州市", "province_name": "广东省", "country_name": "中国"}
{"_id": "2ca09c22-68b4-4e43-b9fa-619dcda5f325", "_rev": "3-ac12e05cedcda85821240a9066aa91ef", "type": "supplier", "name": "江苏苏博特新材料股份有限公司", "short_name": "江苏苏博特新材料股份有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "320000", "city": "320100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91320000768299302G", "created_at": "2025-09-25 20:10:43", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "320000", "320100"]} {"_id": "2ca09c22-68b4-4e43-b9fa-619dcda5f325", "_rev": "3-ac12e05cedcda85821240a9066aa91ef", "type": "supplier", "name": "江苏苏博特新材料股份有限公司", "short_name": "江苏苏博特新材料股份有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "320000", "city": "320100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91320000768299302G", "created_at": "2025-09-25 20:10:43", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "320000", "320100"], "id": "2ca09c22-68b4-4e43-b9fa-619dcda5f325", "city_name": "南京市", "province_name": "江苏省", "country_name": "中国"}
{"_id": "31b6b6d7-989b-4abe-a7e1-6d518f3a4020", "_rev": "3-bedf089583a4645867c7036dba8e7a9e", "type": "supplier", "name": "中国水利水电第十工程局有限公司", "short_name": "中国水利水电第十工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9151018120276341XK", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]} {"_id": "31b6b6d7-989b-4abe-a7e1-6d518f3a4020", "_rev": "3-bedf089583a4645867c7036dba8e7a9e", "type": "supplier", "name": "中国水利水电第十工程局有限公司", "short_name": "中国水利水电第十工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9151018120276341XK", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "31b6b6d7-989b-4abe-a7e1-6d518f3a4020", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "34db820f-860c-4c6e-9326-5c7cb385bf50", "_rev": "3-e475374124da6df3a40f38043efb7237", "type": "supplier", "name": "特变电工中发上海高压开关有限公司", "short_name": "特变电工中发上海高压开关有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "310000", "city": "310000", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9131012078784664XA", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "310000", "310000"]} {"_id": "34db820f-860c-4c6e-9326-5c7cb385bf50", "_rev": "3-e475374124da6df3a40f38043efb7237", "type": "supplier", "name": "特变电工中发上海高压开关有限公司", "short_name": "特变电工中发上海高压开关有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "310000", "city": "310000", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9131012078784664XA", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "310000", "310000"], "id": "34db820f-860c-4c6e-9326-5c7cb385bf50", "city_name": "直辖", "province_name": "上海市", "country_name": "中国"}
{"_id": "4b069ce4-3687-4263-a1f2-4735c22c3f8e", "_rev": "3-9cccdfa9f992ae4f0ec07068c479487c", "type": "supplier", "name": "四川二滩国际工程咨询有限责任公司", "short_name": "四川二滩国际工程咨询有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510115201873799G", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]} {"_id": "4b069ce4-3687-4263-a1f2-4735c22c3f8e", "_rev": "3-9cccdfa9f992ae4f0ec07068c479487c", "type": "supplier", "name": "四川二滩国际工程咨询有限责任公司", "short_name": "四川二滩国际工程咨询有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510115201873799G", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "4b069ce4-3687-4263-a1f2-4735c22c3f8e", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "2e465bd8-b240-4046-bf00-187d22823f7b", "_rev": "3-2629f9e1646b7e398fd8bb41a7fc47a2", "type": "supplier", "name": "哈尔滨电机厂有限责任公司", "short_name": "哈尔滨电机厂有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "230000", "city": "230100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "912301991270479655", "created_at": "2025-09-26 17:00:05", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "230000", "230100"]} {"_id": "2e465bd8-b240-4046-bf00-187d22823f7b", "_rev": "3-2629f9e1646b7e398fd8bb41a7fc47a2", "type": "supplier", "name": "哈尔滨电机厂有限责任公司", "short_name": "哈尔滨电机厂有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "230000", "city": "230100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "912301991270479655", "created_at": "2025-09-26 17:00:05", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "230000", "230100"], "id": "2e465bd8-b240-4046-bf00-187d22823f7b", "city_name": "哈尔滨市", "province_name": "黑龙江省", "country_name": "中国"}
{"_id": "3ab47229-6da8-4efb-aa44-b96866a2ce13", "_rev": "3-998c447f59be1475e690d1416384fb64", "type": "supplier", "name": "四川蜀府贸易有限公司", "short_name": "四川蜀府贸易有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915100005632853965", "created_at": "2025-09-26 17:00:10", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]} {"_id": "3ab47229-6da8-4efb-aa44-b96866a2ce13", "_rev": "3-998c447f59be1475e690d1416384fb64", "type": "supplier", "name": "四川蜀府贸易有限公司", "short_name": "四川蜀府贸易有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915100005632853965", "created_at": "2025-09-26 17:00:10", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "3ab47229-6da8-4efb-aa44-b96866a2ce13", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "48a5f1a7-5d5e-41cf-8676-042f1d5316f1", "_rev": "2-d43e58afff0c5317401a22d075866c56", "field_metadata": null, "created_by": "a775915a-0bed-4a17-94f0-47b29916be90", "created_by_name": "集团公司管理员", "created_at": "2025-11-27 11:32:41", "updated_by": "a775915a-0bed-4a17-94f0-47b29916be90", "updated_by_name": "集团公司管理员", "updated_at": "2025-12-25 08:59:53", "deleted_at": null, "deleted_by": null, "deleted_by_name": null, "type": "supplier", "code": "czyy", "company_type": "other", "name": "常州液压成套设备厂有限公司", "short_name": "常州液压", "parent_id": null, "path": [], "path_name": [], "province": "320000", "city": "320400", "sort": 0, "status": "active", "company_nature": ["supplier"], "credit_code": "91320412250851852D", "country": "100000", "region_path": ["100000", "320000", "320400"]} {"_id": "48a5f1a7-5d5e-41cf-8676-042f1d5316f1", "_rev": "2-d43e58afff0c5317401a22d075866c56", "field_metadata": null, "created_by": "a775915a-0bed-4a17-94f0-47b29916be90", "created_by_name": "集团公司管理员", "created_at": "2025-11-27 11:32:41", "updated_by": "a775915a-0bed-4a17-94f0-47b29916be90", "updated_by_name": "集团公司管理员", "updated_at": "2025-12-25 08:59:53", "deleted_at": null, "deleted_by": null, "deleted_by_name": null, "type": "supplier", "code": "czyy", "company_type": "other", "name": "常州液压成套设备厂有限公司", "short_name": "常州液压", "parent_id": null, "path": [], "path_name": [], "province": "320000", "city": "320400", "sort": 0, "status": "active", "company_nature": ["supplier"], "credit_code": "91320412250851852D", "country": "100000", "region_path": ["100000", "320000", "320400"], "id": "48a5f1a7-5d5e-41cf-8676-042f1d5316f1", "city_name": "常州市", "province_name": "江苏省", "country_name": "中国"}
{"_id": "67fa779d-2ce9-468f-b94f-a3d29f1f22c9", "_rev": "3-2c6e66d316216cfc460662bf4a40e612", "type": "supplier", "name": "杭州华新机电工程有限公司", "short_name": "杭州华新机电工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "330000", "city": "330100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "913301087227885306", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "330000", "330100"]} {"_id": "67fa779d-2ce9-468f-b94f-a3d29f1f22c9", "_rev": "3-2c6e66d316216cfc460662bf4a40e612", "type": "supplier", "name": "杭州华新机电工程有限公司", "short_name": "杭州华新机电工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "330000", "city": "330100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "913301087227885306", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "330000", "330100"], "id": "67fa779d-2ce9-468f-b94f-a3d29f1f22c9", "city_name": "杭州市", "province_name": "浙江省", "country_name": "中国"}
{"_id": "594a09af-8944-476d-b098-7aebe96ad4e8", "_rev": "3-fc6edc3d3384c95def0be02ee734a5b2", "type": "supplier", "name": "云南水电十四局东华建筑工程有限公司", "short_name": "云南水电十四局东华建筑工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "530000", "city": "530100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91530000713408946K", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "530000", "530100"]} {"_id": "594a09af-8944-476d-b098-7aebe96ad4e8", "_rev": "3-fc6edc3d3384c95def0be02ee734a5b2", "type": "supplier", "name": "云南水电十四局东华建筑工程有限公司", "short_name": "云南水电十四局东华建筑工程有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "530000", "city": "530100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91530000713408946K", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "530000", "530100"], "id": "594a09af-8944-476d-b098-7aebe96ad4e8", "city_name": "昆明市", "province_name": "云南省", "country_name": "中国"}
{"_id": "6db1ed5b-f52a-4142-867c-d8b2731ead71", "_rev": "3-f769878777781eb98071dddc7c7c4968", "type": "supplier", "name": "中国水利水电第十四工程局有限公司", "short_name": "中国水利水电第十四工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "530000", "city": "530100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91530100216579074C", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "530000", "530100"]} {"_id": "6db1ed5b-f52a-4142-867c-d8b2731ead71", "_rev": "3-f769878777781eb98071dddc7c7c4968", "type": "supplier", "name": "中国水利水电第十四工程局有限公司", "short_name": "中国水利水电第十四工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "530000", "city": "530100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91530100216579074C", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "530000", "530100"], "id": "6db1ed5b-f52a-4142-867c-d8b2731ead71", "city_name": "昆明市", "province_name": "云南省", "country_name": "中国"}
{"_id": "538e01cd-85cb-42b3-b313-16977f7a5224", "_rev": "3-d9acfbf91ba6d83dc33c94b84127f2e3", "type": "supplier", "name": "中国水利水电第五工程局有限公司", "short_name": "中国水利水电第五工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510000205804264E", "created_at": "2025-09-26 17:00:12", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]} {"_id": "538e01cd-85cb-42b3-b313-16977f7a5224", "_rev": "3-d9acfbf91ba6d83dc33c94b84127f2e3", "type": "supplier", "name": "中国水利水电第五工程局有限公司", "short_name": "中国水利水电第五工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510000205804264E", "created_at": "2025-09-26 17:00:12", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "538e01cd-85cb-42b3-b313-16977f7a5224", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "97be34dc-0a17-4ce5-baea-0f3ceee61103", "_rev": "4-4fbf33c81fff4a4a93ba8da7f99b786d", "type": "supplier", "name": "国电南京自动化股份有限公司", "short_name": "国电南京自动化股份有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "320000", "city": "320100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "913201007162522468", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "320000", "320100"]} {"_id": "97be34dc-0a17-4ce5-baea-0f3ceee61103", "_rev": "4-4fbf33c81fff4a4a93ba8da7f99b786d", "type": "supplier", "name": "国电南京自动化股份有限公司", "short_name": "国电南京自动化股份有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "320000", "city": "320100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "913201007162522468", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "320000", "320100"], "id": "97be34dc-0a17-4ce5-baea-0f3ceee61103", "city_name": "南京市", "province_name": "江苏省", "country_name": "中国"}
{"_id": "77316fb4-6167-432b-94e5-89c3d8a6b900", "_rev": "3-2112fbd151033d97a2c90123a02c70a3", "type": "supplier", "name": "中国水利水电夹江水工机械有限公司", "short_name": "中国水利水电夹江水工机械有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "511100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91511126207500117U", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "511100"]} {"_id": "77316fb4-6167-432b-94e5-89c3d8a6b900", "_rev": "3-2112fbd151033d97a2c90123a02c70a3", "type": "supplier", "name": "中国水利水电夹江水工机械有限公司", "short_name": "中国水利水电夹江水工机械有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "511100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91511126207500117U", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "511100"], "id": "77316fb4-6167-432b-94e5-89c3d8a6b900", "city_name": "乐山市", "province_name": "四川省", "country_name": "中国"}
{"_id": "936c6316-8d87-4681-a3b5-b90ed4804a06", "_rev": "3-a9a9b85f9cdd45dc3646e66c9a1a41aa", "type": "supplier", "name": "中国电建集团成都勘测设计研究院有限公司", "short_name": "中国电建集团成都勘测设计研究院有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915100004507513971", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"]} {"_id": "936c6316-8d87-4681-a3b5-b90ed4804a06", "_rev": "3-a9a9b85f9cdd45dc3646e66c9a1a41aa", "type": "supplier", "name": "中国电建集团成都勘测设计研究院有限公司", "short_name": "中国电建集团成都勘测设计研究院有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "915100004507513971", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "510000", "510100"], "id": "936c6316-8d87-4681-a3b5-b90ed4804a06", "city_name": "成都市", "province_name": "四川省", "country_name": "中国"}
{"_id": "98892419-6a71-4d7e-a0ac-ad5a0a15f1d6", "_rev": "3-f5bb2f9abe284037054df3b323f2482f", "type": "supplier", "name": "中国电建集团贵阳勘测设计研究院有限公司", "short_name": "中国电建集团贵阳勘测设计研究院有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "520000", "city": "520100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9152000070966703X2", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "520000", "520100"]} {"_id": "98892419-6a71-4d7e-a0ac-ad5a0a15f1d6", "_rev": "3-f5bb2f9abe284037054df3b323f2482f", "type": "supplier", "name": "中国电建集团贵阳勘测设计研究院有限公司", "short_name": "中国电建集团贵阳勘测设计研究院有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "520000", "city": "520100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9152000070966703X2", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "520000", "520100"], "id": "98892419-6a71-4d7e-a0ac-ad5a0a15f1d6", "city_name": "贵阳市", "province_name": "贵州省", "country_name": "中国"}
{"_id": "9c80aace-ba73-4550-8fef-3080b51027d9", "_rev": "3-0dddd7f05fccce78e17fdf4ae9975f0a", "type": "supplier", "name": "中国水利水电第四工程局有限公司", "short_name": "中国水利水电第四工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "630000", "city": "630100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9163000022658124XK", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "630000", "630100"]} {"_id": "9c80aace-ba73-4550-8fef-3080b51027d9", "_rev": "3-0dddd7f05fccce78e17fdf4ae9975f0a", "type": "supplier", "name": "中国水利水电第四工程局有限公司", "short_name": "中国水利水电第四工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "630000", "city": "630100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9163000022658124XK", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:53", "country": "100000", "region_path": ["100000", "630000", "630100"], "id": "9c80aace-ba73-4550-8fef-3080b51027d9", "city_name": "西宁市", "province_name": "青海省", "country_name": "中国"}
{"_id": "930dfaf9-708e-4ec0-95f4-521b10bf1c00", "_rev": "2-5c464399a48505836497072601a60159", "field_metadata": null, "created_by": "a775915a-0bed-4a17-94f0-47b29916be90", "created_by_name": "集团公司管理员", "created_at": "2025-11-27 11:33:18", "updated_by": "a775915a-0bed-4a17-94f0-47b29916be90", "updated_by_name": "集团公司管理员", "updated_at": "2025-12-25 08:59:53", "deleted_at": null, "deleted_by": null, "deleted_by_name": null, "type": "supplier", "code": "hnwz", "company_type": "other", "name": "某公司海南物资有限公司", "short_name": "海南物资", "parent_id": null, "path": [], "path_name": [], "province": "460000", "city": "460100", "sort": 0, "status": "active", "company_nature": ["supplier"], "credit_code": "91469007MA5TW", "country": "100000", "region_path": ["100000", "460000", "460100"]} {"_id": "930dfaf9-708e-4ec0-95f4-521b10bf1c00", "_rev": "2-5c464399a48505836497072601a60159", "field_metadata": null, "created_by": "a775915a-0bed-4a17-94f0-47b29916be90", "created_by_name": "集团公司管理员", "created_at": "2025-11-27 11:33:18", "updated_by": "a775915a-0bed-4a17-94f0-47b29916be90", "updated_by_name": "集团公司管理员", "updated_at": "2025-12-25 08:59:53", "deleted_at": null, "deleted_by": null, "deleted_by_name": null, "type": "supplier", "code": "hnwz", "company_type": "other", "name": "华电海南物资有限公司", "short_name": "海南物资", "parent_id": null, "path": [], "path_name": [], "province": "460000", "city": "460100", "sort": 0, "status": "active", "company_nature": ["supplier"], "credit_code": "91469007MA5TW", "country": "100000", "region_path": ["100000", "460000", "460100"], "id": "930dfaf9-708e-4ec0-95f4-521b10bf1c00", "city_name": "海口市", "province_name": "海南省", "country_name": "中国"}
{"_id": "ab5ab605-f27a-461b-9a88-73d1eafde5ed", "_rev": "3-a463c974fbcb84a524d1ee6ed72e64ec", "type": "supplier", "name": "东方电气集团东方电机有限公司", "short_name": "东方电气集团东方电机有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510600", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510600671415848H", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "510000", "510600"]} {"_id": "ab5ab605-f27a-461b-9a88-73d1eafde5ed", "_rev": "3-a463c974fbcb84a524d1ee6ed72e64ec", "type": "supplier", "name": "东方电气集团东方电机有限公司", "short_name": "东方电气集团东方电机有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "510600", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91510600671415848H", "created_at": "2025-09-25 20:10:42", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "510000", "510600"], "id": "ab5ab605-f27a-461b-9a88-73d1eafde5ed", "city_name": "德阳市", "province_name": "四川省", "country_name": "中国"}
{"_id": "b4656ed6-5329-4673-93c8-d11e15d6ece5", "_rev": "3-0a11b989849efb63378b8a8fddb989e4", "type": "supplier", "name": "中国水利水电建设工程咨询西北有限公司", "short_name": "中国水利水电建设工程咨询西北有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "610000", "city": "610100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91610000220530812K", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "610000", "610100"]} {"_id": "b4656ed6-5329-4673-93c8-d11e15d6ece5", "_rev": "3-0a11b989849efb63378b8a8fddb989e4", "type": "supplier", "name": "中国水利水电建设工程咨询西北有限公司", "short_name": "中国水利水电建设工程咨询西北有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "610000", "city": "610100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91610000220530812K", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "610000", "610100"], "id": "b4656ed6-5329-4673-93c8-d11e15d6ece5", "city_name": "西安市", "province_name": "陕西省", "country_name": "中国"}
{"_id": "e0ceec90-0ad7-417f-a669-fd4f5db07eff", "_rev": "4-71df8660a58c5628bb55f2d39a9264fb", "type": "supplier", "name": "西安西电变压器有限责任公司", "short_name": "西安西电变压器有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "610000", "city": "610100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91610104220601553B", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "610000", "610100"]} {"_id": "e0ceec90-0ad7-417f-a669-fd4f5db07eff", "_rev": "4-71df8660a58c5628bb55f2d39a9264fb", "type": "supplier", "name": "西安西电变压器有限责任公司", "short_name": "西安西电变压器有限责任公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "610000", "city": "610100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91610104220601553B", "created_at": "2025-09-25 20:09:02", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "610000", "610100"], "id": "e0ceec90-0ad7-417f-a669-fd4f5db07eff", "city_name": "西安市", "province_name": "陕西省", "country_name": "中国"}
{"_id": "cd9e9224-a08f-4bed-81a6-b289b74506e8", "_rev": "3-572570411c03424fb3370e74b2f7f496", "type": "supplier", "name": "乐山一拉得电网自动化有限公司", "short_name": "乐山一拉得电网自动化有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "511100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91511100MA6281NC0N", "created_at": "2025-09-25 20:10:43", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "510000", "511100"]} {"_id": "cd9e9224-a08f-4bed-81a6-b289b74506e8", "_rev": "3-572570411c03424fb3370e74b2f7f496", "type": "supplier", "name": "乐山一拉得电网自动化有限公司", "short_name": "乐山一拉得电网自动化有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "510000", "city": "511100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "91511100MA6281NC0N", "created_at": "2025-09-25 20:10:43", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "510000", "511100"], "id": "cd9e9224-a08f-4bed-81a6-b289b74506e8", "city_name": "乐山市", "province_name": "四川省", "country_name": "中国"}
{"_id": "e88c8c18-bc49-4bab-acc5-6fb78f4c7f8f", "_rev": "3-269fcaf6630593cb44ee3e6a1ffd4ef9", "type": "supplier", "name": "中国水利水电第六工程局有限公司", "short_name": "中国水利水电第六工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "210000", "city": "210100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9121011211756300XA", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "210000", "210100"]} {"_id": "e88c8c18-bc49-4bab-acc5-6fb78f4c7f8f", "_rev": "3-269fcaf6630593cb44ee3e6a1ffd4ef9", "type": "supplier", "name": "中国水利水电第六工程局有限公司", "short_name": "中国水利水电第六工程局有限公司", "code": "010A", "company_type": "other", "parent_id": "", "path": [], "path_name": [], "province": "210000", "city": "210100", "status": "active", "sort": 2, "created_by": "system", "created_by_name": "system", "updated_by": "system", "updated_by_name": "system", "company_nature": ["construction", "design", "supervisory", "supplier", "goverment"], "credit_code": "9121011211756300XA", "created_at": "2025-09-25 20:10:47", "updated_at": "2025-12-25 08:59:54", "country": "100000", "region_path": ["100000", "210000", "210100"], "id": "e88c8c18-bc49-4bab-acc5-6fb78f4c7f8f", "city_name": "沈阳市", "province_name": "辽宁省", "country_name": "中国"}
pytest.ini
-1
View File
@@ -10,4 +10,3 @@ filterwarnings =
ignore:'asyncio.set_event_loop_policy' is deprecated and slated for removal in Python 3.16:DeprecationWarning:pytest_asyncio\.plugin ignore:'asyncio.set_event_loop_policy' is deprecated and slated for removal in Python 3.16:DeprecationWarning:pytest_asyncio\.plugin
# 也可以设置异步测试的默认 loop scope # 也可以设置异步测试的默认 loop scope
asyncio_default_fixture_loop_scope = function asyncio_default_fixture_loop_scope = function
addopts = --ignore=scripts/test_api_sync.py --ignore=scripts/test_api_sync_top10.py --ignore=scripts/test_api_write_real_sample.py
run.py
+4 -8
View File
@@ -11,7 +11,7 @@ if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT)) sys.path.insert(0, str(PROJECT_ROOT))
from sync_state_machine.config import load_overrides_from_file from sync_state_machine.config import load_overrides_from_file
from sync_state_machine.pipeline import build_config, run_pipeline_from_config from sync_state_machine.pipeline import run_profile_from_file
def _resolve_config_path(raw_path: str) -> Path: def _resolve_config_path(raw_path: str) -> Path:
@@ -34,14 +34,10 @@ async def _main() -> int:
if not cfg_path.exists() or not cfg_path.is_file(): if not cfg_path.exists() or not cfg_path.is_file():
raise FileNotFoundError(f"Config file not found: {cfg_path}") raise FileNotFoundError(f"Config file not found: {cfg_path}")
overrides = load_overrides_from_file(cfg_path, PROJECT_ROOT)
config = build_config(project_root=PROJECT_ROOT, overrides=overrides)
mode = f"{config.local_datasource.type}->{config.remote_datasource.type}"
print(f"🧩 Loaded config file: {cfg_path}") print(f"🧩 Loaded config file: {cfg_path}")
await run_pipeline_from_config( await run_profile_from_file(
config, project_root=PROJECT_ROOT,
title=f"sync_state_machine pipeline ({mode})", config_path=cfg_path,
print_summary=True, print_summary=True,
) )
return 0 return 0
run_profiles/README.md
+16
View File
@@ -3,9 +3,15 @@
- 路径：项目根目录 `run_profiles/` - 路径：项目根目录 `run_profiles/`
- 命名：建议使用 `*.local.yaml`（该目录下 YAML 默认会被 git 忽略） - 命名：建议使用 `*.local.yaml`（该目录下 YAML 默认会被 git 忽略）
当前约定：
- 仓库里跟踪的配置文件只写必要字段，不重复抄默认值。
- pipeline 启动时会打印 **Resolved Full Config**，把省略字段补齐。
- 如果你要看完整字段和字段含义，不看模板，直接看 `docs/runtime_config_guide.md`。
模板来源： 模板来源：
- `run_profiles/preset/jsonl_to_jsonl.default.yaml` - `run_profiles/preset/jsonl_to_jsonl.default.yaml`
- `run_profiles/preset/jsonl_to_api.default.yaml` - `run_profiles/preset/jsonl_to_api.default.yaml`
- `run_profiles/preset/jsonl_to_jsonl.multi_project.default.yaml`
推荐流程： 推荐流程：
1. 从模板复制到本目录，例如： 1. 从模板复制到本目录，例如：
@@ -17,5 +23,15 @@
兼容旧入口： 兼容旧入口：
- `python run.py --config_path=run_profiles/jsonl_to_jsonl.local.yaml` - `python run.py --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
多项目示例：
- `run_profiles/jsonl_to_jsonl.multi_project.yaml`
- `run_profiles/jsonl_to_jsonl.multi_project.global.yaml`
- `run_profiles/jsonl_to_jsonl.multi_project.project.yaml`
支持占位符： 支持占位符：
- `${PROJECT_ROOT}`：运行时替换为项目根目录绝对路径。 - `${PROJECT_ROOT}`：运行时替换为项目根目录绝对路径。
现成示例：
- `run_profiles/jsonl_to_api.yaml`：单项目 JSONL -> API 联调示例
- `run_profiles/jsonl_to_api.test.yaml`：auto-test 使用的 JSONL -> API 示例
- `run_profiles/jsonl_to_jsonl.multi_project.yaml`：多项目 JSONL -> JSONL 编排入口
run_profiles/preset/README.md
+17 -2
View File
@@ -2,20 +2,35 @@
该目录存放版本管理的运行配置模板（default）。 该目录存放版本管理的运行配置模板（default）。
模板约定：
- 这里只保留必要覆盖项。
- 省略的字段会在加载时从默认 datasource 配置和各 domain 的 `default_config` 自动补齐。
- 真正生效的全量配置，以启动日志里的 **Resolved Full Config** 为准。
当前模板： 当前模板：
- `jsonl_to_jsonl.default.yaml` - `jsonl_to_jsonl.default.yaml`
- `jsonl_to_api.default.yaml` - `jsonl_to_api.default.yaml`
- `jsonl_to_jsonl.multi_project.default.yaml`
- `jsonl_to_jsonl.multi_project.global.default.yaml`
- `jsonl_to_jsonl.multi_project.project.default.yaml`
使用方式： 使用方式：
1. 复制模板到项目根目录 `run_profiles/`，并重命名为你自己的文件（如 `*.local.yaml`）。 1. 复制模板到项目根目录 `run_profiles/`，并重命名为你自己的文件（如 `*.local.yaml`）。
2. 修改后通过入口运行： 2. 只改和当前环境/业务差异相关的字段。
3. 修改后通过入口运行：
- `ecm-sync-run --config_path=run_profiles/jsonl_to_jsonl.local.yaml` - `ecm-sync-run --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
兼容旧入口： 兼容旧入口：
- `python run.py --config_path=run_profiles/jsonl_to_jsonl.local.yaml` - `python run.py --config_path=run_profiles/jsonl_to_jsonl.local.yaml`
多项目配置说明：
- `jsonl_to_jsonl.multi_project.default.yaml` 是多 pipeline 编排入口。
- `jsonl_to_jsonl.multi_project.global.default.yaml` 定义全局公共 domain 的 pipeline。
- `jsonl_to_jsonl.multi_project.project.default.yaml` 定义单项目 pipeline 默认配置。
- `project_bootstrap_node_types` 用于显式声明项目级 pipeline 需要从 global scope 继承到项目 scope 的公共节点类型，例如 `company`、`supplier`。
说明： 说明：
- `run_profiles/*.yaml` / `run_profiles/*.yml` 默认被 `.gitignore` 忽略，用于本地私有配置。 - `run_profiles/*.yaml` / `run_profiles/*.yml` 默认被 `.gitignore` 忽略，用于本地私有配置。
- 模板内可使用 `${PROJECT_ROOT}` 占位符。 - 模板内可使用 `${PROJECT_ROOT}` 占位符。
- 模板已显式展开 datasource 与 strategies 关键字段，便于集中审查。 - 完整字段定义与字段含义见 `docs/runtime_config_guide.md`。
- API 数据源支持全局限流配置：`api_rate_limit_max_requests` 与 `api_rate_limit_window_seconds`。 - API 数据源支持全局限流配置：`api_rate_limit_max_requests` 与 `api_rate_limit_window_seconds`。
run_profiles/preset/jsonl_to_api.default.yaml
+18 -238
View File
@@ -18,14 +18,16 @@ node_types:
- lar_change - lar_change
- project_detail_data - project_detail_data
- units - units
local_datasource: local_datasource:
type: jsonl type: jsonl
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
read_only: false handler_configs:
target_project_ids: project:
data_id_filter:
- "0f9c3e22-f4bb-4803-825a-4f238eeb06d5" - "0f9c3e22-f4bb-4803-825a-4f238eeb06d5"
- "f3e02e84-e81c-4aa6-88d4-f5aa108c8227" - "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
handler_configs: {}
remote_datasource: remote_datasource:
type: api type: api
api_base_url: http://localhost:8000/api/third/v2 api_base_url: http://localhost:8000/api/third/v2
@@ -35,256 +37,34 @@ remote_datasource:
api_rate_limit_max_requests: 30 api_rate_limit_max_requests: 30
api_rate_limit_window_seconds: 10.0 api_rate_limit_window_seconds: 10.0
poll_max_retries: 1 poll_max_retries: 1
poll_interval: 0.5 handler_configs:
target_project_ids: project:
# 请替换为实际的项目ID，至少需要一个 data_id_filter:
- "0f9c3e22-f4bb-4803-825a-4f238eeb06d5" - "0f9c3e22-f4bb-4803-825a-4f238eeb06d5"
- "f3e02e84-e81c-4aa6-88d4-f5aa108c8227" - "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
handler_configs:
supplier: {}
persist: persist:
backend: sqlite
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/simple_pipeline_sm.db db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/simple_pipeline_sm.db
enable: true
wipe_on_start: false
logging: logging:
initialize: true
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
file_name_pattern: simple_pipeline_sm_{timestamp}.log file_name_pattern: simple_pipeline_sm_{timestamp}.log
level: 20
strategies: strategies:
company: company:
skip_sync: false preset: pull_only
config: config:
auto_bind: true
auto_bind_fields: auto_bind_fields:
- credit_code - credit_code
depend_fields: {}
local_orphan_action: none
remote_orphan_action: create_local
update_direction: pull
supplier: supplier:
skip_sync: false preset: pull_only
config:
auto_bind: true
auto_bind_fields:
- credit_code
depend_fields: {}
local_orphan_action: none
remote_orphan_action: create_local
update_direction: pull
project: project:
skip_sync: false
config: config:
auto_bind: false
auto_bind_fields: []
pre_bind_data_id: []
depend_fields:
company_id: company
local_orphan_action: none local_orphan_action: none
remote_orphan_action: none
update_direction: push
contract:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- company_id
- code
depend_fields:
project_id: project
company_id: supplier
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract_change:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
- change_time
- title
depend_fields:
project_id: project
contract_id: contract
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
construction_task:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
- task_type
depend_fields:
contract_id: contract
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
construction_task_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: construction_task
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
material:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
- material_type
depend_fields:
contract_id: contract
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
material_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: material
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract_settlement:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
depend_fields:
contract_id: contract
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract_settlement_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: contract_settlement
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
personnel:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- code
depend_fields:
project_id: project
contract_id: contract
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
personnel_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: personnel
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
preparation:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- code
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
production: production:
skip_sync: false
config: config:
auto_bind: true post_check_ignore_fields:
auto_bind_fields: - is_exist
- project_id
- code
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
lar:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
lar_change:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- change_time
- title
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
project_detail_data:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- key
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push # 未在 field_direction_overrides 中列出的 key 默认 push
field_direction_key: key # 以节点 data.key 字段值作为方向查表键
field_direction_overrides:
ensure_production: pull # 远程 → 本地
climb_production: pull # 远程 → 本地
challenge_production: pull # 远程 → 本地
units:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- serial_number
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
run_profiles/preset/jsonl_to_jsonl.default.yaml
-217
View File
@@ -22,263 +22,46 @@ node_types:
local_datasource: local_datasource:
type: jsonl type: jsonl
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
read_only: false
target_project_ids: []
handler_configs: {}
remote_datasource: remote_datasource:
type: jsonl type: jsonl
jsonl_dir: ${PROJECT_ROOT}/remote_datasource/datasource jsonl_dir: ${PROJECT_ROOT}/remote_datasource/datasource
read_only: false
target_project_ids: []
handler_configs: {}
persist: persist:
backend: sqlite
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/full_sync_pipeline_jsonl_sm.db db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/full_sync_pipeline_jsonl_sm.db
enable: true
wipe_on_start: false
logging: logging:
initialize: true
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
file_name_pattern: jsonl_pipeline_sm_{timestamp}.log file_name_pattern: jsonl_pipeline_sm_{timestamp}.log
level: 20
strategies: strategies:
company: company:
skip_sync: false
config: config:
auto_bind: true
auto_bind_fields: auto_bind_fields:
- credit_code - credit_code
depend_fields: {}
local_orphan_action: create_remote local_orphan_action: create_remote
remote_orphan_action: none remote_orphan_action: none
update_direction: push update_direction: push
user: user:
skip_sync: false
config: config:
auto_bind: true
auto_bind_fields:
- username
depend_fields: {}
local_orphan_action: create_remote local_orphan_action: create_remote
remote_orphan_action: none remote_orphan_action: none
update_direction: push update_direction: push
supplier: supplier:
skip_sync: false
config: config:
auto_bind: true
auto_bind_fields:
- credit_code
depend_fields: {}
local_orphan_action: create_remote local_orphan_action: create_remote
remote_orphan_action: none remote_orphan_action: none
update_direction: push update_direction: push
project: project:
skip_sync: false
config: config:
auto_bind: true auto_bind: true
auto_bind_fields: auto_bind_fields:
- name - name
depend_fields:
company_id: company
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- company_id
- code
depend_fields:
project_id: project
company_id: supplier
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract_change:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
- change_time
- title
depend_fields:
project_id: project
contract_id: contract
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
construction_task:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
- task_type
depend_fields:
contract_id: contract
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
construction_task_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: construction_task
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
material:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
- material_type
depend_fields:
contract_id: contract
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
material_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: material
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract_settlement:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- contract_id
depend_fields:
contract_id: contract
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
contract_settlement_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: contract_settlement
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
personnel:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- code
depend_fields:
project_id: project
contract_id: contract
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
personnel_detail:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- parent_id
- date
depend_fields:
parent_id: personnel
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
preparation:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- code
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
production:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
- code
depend_fields:
project_id: project
local_orphan_action: none
remote_orphan_action: none
update_direction: push
lar: lar:
skip_sync: false
config: config:
auto_bind: true
auto_bind_fields:
- project_id
depend_fields:
project_id: project
local_orphan_action: create_remote local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
lar_change: lar_change:
skip_sync: false
config: config:
auto_bind: true
auto_bind_fields:
- project_id
- change_time
- title
depend_fields:
project_id: project
local_orphan_action: create_remote local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
project_detail_data: project_detail_data:
skip_sync: false
config: config:
auto_bind: true
auto_bind_fields:
- project_id
- key
depend_fields:
project_id: project
local_orphan_action: create_remote local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
units: units:
skip_sync: false
config: config:
auto_bind: true
auto_bind_fields:
- project_id
- serial_number
depend_fields:
project_id: project
local_orphan_action: create_remote local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
run_profiles/preset/jsonl_to_jsonl.multi_project.default.yaml
+12
View File
@@ -0,0 +1,12 @@
global_config: run_profiles/preset/jsonl_to_jsonl.multi_project.global.default.yaml
default_project_config: run_profiles/preset/jsonl_to_jsonl.multi_project.project.default.yaml
project_bootstrap_node_types:
- company
- supplier
projects:
- local_project_id: f3e02e84-e81c-4aa6-88d4-f5aa108c8227
remote_project_id: f3e02e84-e81c-4aa6-88d4-f5aa108c8227
- local_project_id: 0f9c3e22-f4bb-4803-825a-4f238eeb06d5
remote_project_id: 0f9c3e22-f4bb-4803-825a-4f238eeb06d5
run_profiles/preset/jsonl_to_jsonl.multi_project.global.default.yaml
+29
View File
@@ -0,0 +1,29 @@
node_types:
- company
- supplier
local_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/working_local_datasource/20260302-020003
remote_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/runtime/tmp/multi_project_jsonl_remote
persist:
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/multi_project_jsonl.db
wipe_on_start: true
logging:
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
file_name_pattern: multi_project_global_{timestamp}.log
strategies:
company:
config:
auto_bind_fields:
- code
- name
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
supplier:
config:
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
run_profiles/preset/jsonl_to_jsonl.multi_project.project.default.yaml
+47
View File
@@ -0,0 +1,47 @@
node_types:
- project
- contract
- contract_change
- construction_task
- construction_task_detail
- material
- material_detail
- contract_settlement
- contract_settlement_detail
- personnel
- personnel_detail
- preparation
- production
- lar
- lar_change
- project_detail_data
- units
local_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/working_local_datasource/20260302-020003
remote_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/runtime/tmp/multi_project_jsonl_remote
persist:
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/multi_project_jsonl.db
logging:
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
file_name_pattern: multi_project_project_{timestamp}.log
strategies:
project:
config:
auto_bind: true
auto_bind_fields:
- name
lar:
config:
local_orphan_action: create_remote
lar_change:
config:
local_orphan_action: create_remote
project_detail_data:
config:
local_orphan_action: create_remote
units:
config:
local_orphan_action: create_remote
schemas/common/base.py
+33
View File
@@ -59,6 +59,39 @@ class BaseResponseWithID(BaseModel):
# 默认返回（已有 id） # 默认返回（已有 id）
return data return data
def _get_plan_node_date(node: Any) -> Any:
if isinstance(node, dict):
return node.get("date")
return getattr(node, "date", None)
def trim_plan_nodes_by_boundary_dates(
plan_nodes: Optional[List[Any]],
start_date: Optional[Any] = None,
end_date: Optional[Any] = None,
) -> List[Any]:
if not plan_nodes:
return []
cleaned_nodes = list(plan_nodes)
while (
start_date is not None
and cleaned_nodes
and str(_get_plan_node_date(cleaned_nodes[0])) == str(start_date)
):
cleaned_nodes.pop(0)
while (
end_date is not None
and cleaned_nodes
and str(_get_plan_node_date(cleaned_nodes[-1])) == str(end_date)
):
cleaned_nodes.pop()
return cleaned_nodes
class RequestApprovalMixin(BaseModel): class RequestApprovalMixin(BaseModel):
""" """
请求审批相关的基础模型 请求审批相关的基础模型
schemas/common/push_system.py
+12 -1
View File
@@ -24,7 +24,18 @@
from pydantic import BaseModel, Field from pydantic import BaseModel, Field
from typing import TypeVar, Generic, Optional, List, Dict, Any from typing import TypeVar, Generic, Optional, List, Dict, Any
from ..project.enums import ProjectType from ..project.enums import ProjectType
from datetime import datetime
# from backend.modules.admin.approval.models import ApprovalStep
class ApprovalStep(BaseModel):
"""审批流程中的一个步骤"""
step_id: str = Field(description="步骤的唯一ID")
operator_id: Optional[str] = Field("",description="操作人ID")
operator_name: str = Field(description="操作人姓名")
action: str = Field(description="执行的动作，如 'submit', 'approve', 'reject'")
comment: Optional[str] = Field(None, description="审批意见")
created_at: datetime = Field(default_factory=datetime.utcnow, description="步骤的创建时间")
# 定义泛型 # 定义泛型
T = TypeVar('T', bound='BasePushData') T = TypeVar('T', bound='BasePushData')
@@ -52,7 +63,7 @@ class SignatureMixin(BaseModel):
class PushRequestMixin(BaseModel): class PushRequestMixin(BaseModel):
"""推送ID字段混入""" """推送ID字段混入"""
push_id: str = Field(..., description="推送ID，客户端生成的唯一标识（UUIDv7，支持时间排序）") push_id: str = Field(..., description="推送ID，客户端生成的唯一标识（UUIDv7，支持时间排序）")
steps : List[ApprovalStep] =Field(default_factory=list,description="当前数据的审批流程步骤") # 加入字段step用来记录审批流程
# ==================== 响应相关 ==================== # ==================== 响应相关 ====================
schemas/construction/construction.py
+10 -1
View File
@@ -2,7 +2,7 @@ from pydantic import BaseModel, Field, field_validator, model_validator
from fastapi import HTTPException from fastapi import HTTPException
from typing import List, Optional, Literal from typing import List, Optional, Literal
from ..common.validators import DateStr from ..common.validators import DateStr
from ..common.base import BaseResponseWithID from ..common.base import BaseResponseWithID, trim_plan_nodes_by_boundary_dates
class PlanNode(BaseModel): class PlanNode(BaseModel):
date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"]) date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"])
@@ -40,6 +40,15 @@ class ConstructionResponse(BaseResponseWithID):
# 2025-10-20新增字段，延迟时间 # 2025-10-20新增字段，延迟时间
delay_days: Optional[int] = Field(None, description="延迟时间（天）", examples=[0]) delay_days: Optional[int] = Field(None, description="延迟时间（天）", examples=[0])
@model_validator(mode='after')
def clean_response_plan_nodes(self):
self.plan_node = trim_plan_nodes_by_boundary_dates(
self.plan_node,
start_date=self.plan_start_date,
end_date=self.plan_complete_date,
)
return self
class ConstructionCreate(BaseModel): class ConstructionCreate(BaseModel):
""" """
施工任务创建模型 施工任务创建模型
schemas/contract_settlement/contract_settlement.py
+10 -1
View File
@@ -2,7 +2,7 @@ from pydantic import Field, BaseModel,field_validator,model_validator
from fastapi import HTTPException from fastapi import HTTPException
from typing import List, Optional, Literal from typing import List, Optional, Literal
from ..common.validators import DateStr from ..common.validators import DateStr
from ..common.base import BaseResponseWithID from ..common.base import BaseResponseWithID, trim_plan_nodes_by_boundary_dates
class PlanNode(BaseModel): class PlanNode(BaseModel):
date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"]) date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"])
@@ -39,6 +39,15 @@ class ContractSettlementResponse(BaseResponseWithID):
None, description="统计节点截止日期", examples=["2024-05-31"] None, description="统计节点截止日期", examples=["2024-05-31"]
) )
@model_validator(mode='after')
def clean_response_plan_nodes(self):
self.plan_node = trim_plan_nodes_by_boundary_dates(
self.plan_node,
start_date=self.plan_start_date,
end_date=self.plan_complete_date,
)
return self
class ContractSettlementCreate(BaseModel): class ContractSettlementCreate(BaseModel):
""" """
合同结算创建模型 合同结算创建模型
schemas/material/material.py
+10 -1
View File
@@ -2,7 +2,7 @@ from pydantic import Field, BaseModel,field_validator,model_validator
from fastapi import HTTPException from fastapi import HTTPException
from typing import List, Optional, Literal from typing import List, Optional, Literal
from ..common.validators import DateStr from ..common.validators import DateStr
from ..common.base import BaseResponseWithID from ..common.base import BaseResponseWithID, trim_plan_nodes_by_boundary_dates
class PlanNode(BaseModel): class PlanNode(BaseModel):
date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"]) date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"])
@@ -52,6 +52,15 @@ class MaterialResponse(BaseResponseWithID):
None, description="统计节点截止日期", examples=["2024-05-31"] None, description="统计节点截止日期", examples=["2024-05-31"]
) )
@model_validator(mode='after')
def clean_response_plan_nodes(self):
self.plan_node = trim_plan_nodes_by_boundary_dates(
self.plan_node,
start_date=self.plan_start_date,
end_date=self.plan_complete_date,
)
return self
class MaterialCreate(BaseModel): class MaterialCreate(BaseModel):
""" """
创建物资 创建物资
schemas/power/power.py
+10 -1
View File
@@ -2,7 +2,7 @@ from pydantic import Field, BaseModel,field_validator,model_validator
from fastapi import HTTPException from fastapi import HTTPException
from typing import List, Optional, Literal from typing import List, Optional, Literal
from ..common.validators import DateStr from ..common.validators import DateStr
from ..common.base import BaseResponseWithID from ..common.base import BaseResponseWithID, trim_plan_nodes_by_boundary_dates
class PlanNode(BaseModel): class PlanNode(BaseModel):
date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"]) date: DateStr = Field(..., description="计划节点时间", examples=["2024-01-01"])
@@ -61,6 +61,15 @@ class PowerResponse(BaseResponseWithID):
None, description="实际首批到场人数", examples=[4] None, description="实际首批到场人数", examples=[4]
) )
@model_validator(mode='after')
def clean_response_plan_nodes(self):
self.plan_node = trim_plan_nodes_by_boundary_dates(
self.plan_node,
start_date=self.plan_start_date,
end_date=self.plan_exit_date,
)
return self
class PowerCreate(BaseModel): class PowerCreate(BaseModel):
""" """
创建力能 创建力能
schemas/project/project.py
+9
View File
@@ -74,3 +74,12 @@ class ProjectInfoResponse(BaseModel):
# 里程碑节点 # 里程碑节点
preparation_list: list[PreparationDetail] = Field(default_factory=list, description="里程碑节点列表") preparation_list: list[PreparationDetail] = Field(default_factory=list, description="里程碑节点列表")
class ProjectListResponse(BaseModel):
"""
项目列表模型:分页
"""
page: int
page_size: int
total: int
list: list[ProjectInfoResponse]
schemas/project/project_aggregate.py
+150
View File
@@ -0,0 +1,150 @@
"""
一个接口按 domain 返回项目聚合数据。
"""
from typing import Literal
from pydantic import BaseModel, Field
from ..construction.construction import ConstructionResponse
from ..construction.construction_detail import ConstructionDetailResponse
from ..contract.contract import ContractResponse
from ..contract_settlement.contract_settlement import (
ContractSettlementResponse,
)
from ..contract_settlement.contract_settlement_detail import (
ContractSettlementDetailResponse,
)
from ..material.material import MaterialResponse
from ..material.material_detail import MaterialDetailResponse
from ..power.power import PowerResponse
from ..power.power_detail import PowerDetail
from ..project.project_base import ProjectResponseBase
from ..project_extensions.contract_change import ContractChangeResponse
from ..project_extensions.generator_unit import GeneratorUnitProject
from ..project_extensions.lar import (
LarChangeDetailSchema,
LarDetailSchema,
)
from ..project_extensions.preparation import PreparationDetail
from ..project_extensions.production import ProductionDetail
from ..project_extensions.project_detail import ProjectDetailProject
from ..project_multi_type import (
GasTurbineExtension,
HydropowerExtension,
OffShoreWindExtension,
OnShoreWindExtension,
PhotovoltaicExtension,
PumpedStorageExtension,
ThermalPowerExtension,
)
ProjectAggregateDomain = Literal[
"project",
"project_detail",
"generator_units",
"lar",
"lar_change",
"contract_changes",
"photovoltaic_extension",
"onshore_wind_extension",
"offshore_wind_extension",
"hydropower_extension",
"pumped_storage_extension",
"gas_turbine_extension",
"thermal_power_extension",
"production_list",
"preparation_list",
"contracts",
"materials",
"material_details",
"constructions",
"construction_details",
"contract_settlements",
"contract_settlement_details",
"power",
"power_details",
]
ALL_PROJECT_AGGREGATE_DOMAINS: list[ProjectAggregateDomain] = [
"project",
"project_detail",
"generator_units",
"lar",
"lar_change",
"contract_changes",
"photovoltaic_extension",
"onshore_wind_extension",
"offshore_wind_extension",
"hydropower_extension",
"pumped_storage_extension",
"gas_turbine_extension",
"thermal_power_extension",
"production_list",
"preparation_list",
"contracts",
"materials",
"material_details",
"constructions",
"construction_details",
"contract_settlements",
"contract_settlement_details",
"power",
"power_details",
]
class ProjectAggregateData(BaseModel):
"""按 domain 组织的项目聚合数据。"""
project: ProjectResponseBase | None = Field(None, description="项目基本信息")
project_detail: list[ProjectDetailProject] = Field(default_factory=list, description="项目容量信息")
generator_units: list[GeneratorUnitProject] = Field(default_factory=list, description="发电单元信息")
lar: LarDetailSchema | None = Field(None, description="移民征地信息")
lar_change: list[LarChangeDetailSchema] = Field(default_factory=list, description="移民变更信息")
contract_changes: list[ContractChangeResponse] = Field(default_factory=list, description="合同变更信息")
photovoltaic_extension: PhotovoltaicExtension | None = Field(None, description="光伏项目扩展信息")
onshore_wind_extension: OnShoreWindExtension | None = Field(None, description="陆风项目扩展信息")
offshore_wind_extension: OffShoreWindExtension | None = Field(None, description="海风项目扩展信息")
hydropower_extension: HydropowerExtension | None = Field(None, description="水电项目扩展信息")
pumped_storage_extension: PumpedStorageExtension | None = Field(None, description="抽蓄项目扩展信息")
gas_turbine_extension: GasTurbineExtension | None = Field(None, description="燃气轮机项目扩展信息")
thermal_power_extension: ThermalPowerExtension | None = Field(None, description="火电项目扩展信息")
production_list: list[ProductionDetail] = Field(default_factory=list, description="投产节点列表")
preparation_list: list[PreparationDetail] = Field(default_factory=list, description="里程碑节点列表")
contracts: list[ContractResponse] = Field(default_factory=list, description="合同列表")
materials: list[MaterialResponse] = Field(default_factory=list, description="物资主表列表")
material_details: list[MaterialDetailResponse] = Field(default_factory=list, description="物资明细列表")
constructions: list[ConstructionResponse] = Field(default_factory=list, description="施工主表列表")
construction_details: list[ConstructionDetailResponse] = Field(default_factory=list, description="施工明细列表")
contract_settlements: list[ContractSettlementResponse] = Field(default_factory=list, description="合同结算主表列表")
contract_settlement_details: list[ContractSettlementDetailResponse] = Field(default_factory=list, description="合同结算明细列表")
power: list[PowerResponse] = Field(default_factory=list, description="力能主表列表")
power_details: list[PowerDetail] = Field(default_factory=list, description="力能明细列表")
class ProjectAggregateMeta(BaseModel):
"""聚合请求回显和截断信息。"""
project_id: str = Field(..., description="项目ID")
requested_domains: list[ProjectAggregateDomain] = Field(
default_factory=lambda: list(ALL_PROJECT_AGGREGATE_DOMAINS),
description="本次请求的 domain 列表",
)
domain_limit: int = Field(10000, description="每个 domain 的最大返回条数")
truncated_domains: list[ProjectAggregateDomain] = Field(default_factory=list, description="被截断的 domain 列表")
class ProjectAggregateResponse(BaseModel):
"""项目聚合同步响应。"""
meta: ProjectAggregateMeta = Field(..., description="请求元信息")
data: ProjectAggregateData = Field(
default_factory=lambda: ProjectAggregateData.model_construct(),
description="按 domain 分组的数据",
)
schemas/project/project_base.py
+1
View File
@@ -78,6 +78,7 @@ class ProjectResponseBase(BaseResponseWithID):
# 拓展信息 # 拓展信息
dbtc_score_list: Optional[list] = Field(default_factory=list, description='达标投产评分列表') # 这里项目用的是list而不是List dbtc_score_list: Optional[list] = Field(default_factory=list, description='达标投产评分列表') # 这里项目用的是list而不是List
approval_status: Optional[str] = Field(None, description="审核状态", examples=["pending"])
schemas/project_extensions/generator_unit.py
+2 -2
View File
@@ -21,8 +21,8 @@ class PostGeneratorUnits(BaseModel):
@field_validator('actual_production_date', mode="before") @field_validator('actual_production_date', mode="before")
def validate_actual_date_not_future(cls, v: str) -> str: def validate_actual_date_not_future(cls, v: str) -> str:
if v == "": # if v == "":
return "" # return ""
try: try:
actual = datetime_date(v) actual = datetime_date(v)
except ValueError: except ValueError:
schemas/project_extensions/preparation.py
+1 -1
View File
@@ -33,7 +33,7 @@ class PreparationDetail(BaseResponseWithID):
code: Optional[str] = Field(None, description="里程碑节点编码") code: Optional[str] = Field(None, description="里程碑节点编码")
acquire_date: Optional[DateStr] = Field(None, description="取得时间") acquire_date: Optional[DateStr] = Field(None, description="取得时间")
is_exist: Optional[bool] = Field(None, description="是否取得") is_exist: Optional[bool] = Field(None, description="是否取得")
approval_status: Optional[str] = Field(None, description="审批状态") # approval_status: Optional[str] = Field(None, description="审批状态")
name: Optional[str] = Field(None, description="里程碑名称") name: Optional[str] = Field(None, description="里程碑名称")
group: Optional[str] = Field(None, description="分组条件", examples=['KGTJLS']) group: Optional[str] = Field(None, description="分组条件", examples=['KGTJLS'])
is_acquired: Optional[int] = Field(0, description="是否取得") is_acquired: Optional[int] = Field(0, description="是否取得")
schemas/project_extensions/production.py
+1 -1
View File
@@ -34,6 +34,6 @@ class ProductionDetail(BaseResponseWithID):
code: Optional[str] = Field(None, description="投产节点编码") code: Optional[str] = Field(None, description="投产节点编码")
acquire_date: Optional[DateStr] = Field(None, description="取得时间") acquire_date: Optional[DateStr] = Field(None, description="取得时间")
is_exist: Optional[bool] = Field(None, description="是否取得") is_exist: Optional[bool] = Field(None, description="是否取得")
approval_status: Optional[str] = Field(None, description="审批状态") # approval_status: Optional[str] = Field(None, description="审批状态")
name: Optional[str] = Field(None, description="投产节点名称") name: Optional[str] = Field(None, description="投产节点名称")
is_acquired: Optional[int] = Field(0, description="是否取得") is_acquired: Optional[int] = Field(0, description="是否取得")
schemas/supplier/supplier.py
+95 -1
View File
@@ -1,8 +1,39 @@
from pydantic import BaseModel, Field from fastapi import HTTPException
from pydantic import BaseModel, Field, model_validator, field_validator
from typing import List, Optional from typing import List, Optional
from enum import Enum
from ..common.base import BaseResponseWithID from ..common.base import BaseResponseWithID
class SupplierType(str, Enum):
"""类型枚举"""
PROJECT = "project" # 项目类型
OTHER = "other" # 其他类型
@classmethod
def supplier_list(cls):
return [cls.PROJECT.value, cls.OTHER.value]
class SupplierNature(str, Enum):
"""
性质枚举
"""
CONSTRUCTION = "construction" # 施工单位
DESIGN = "design" # 设计单位
SUPERVISORY = "supervisory" # 监理单位
SUPPLIER = "supplier" # 供应商
SURVEY = "survey" # 2026.3.11 推送系统测试时发现与项目侧 org_role 不一致，已补充勘察单位
GOVERMENT = "goverment" # 政府
@classmethod
def supplier_type_list(cls):
return [member.value for member in cls]
class SupplierStatus(str, Enum):
"""状态枚举"""
ACTIVE = "active" # 正常
INACTIVE = "inactive" # 禁用
class SupplierDetailResponse(BaseResponseWithID): class SupplierDetailResponse(BaseResponseWithID):
""" """
供应商详情模型 供应商详情模型
@@ -41,3 +72,66 @@ class SupplierListResponse(BaseModel):
page_size: int page_size: int
total: int total: int
list: List[SupplierDetailResponse] = [] list: List[SupplierDetailResponse] = []
class SupplierCreateRequest(BaseModel):
"""
供应商推送创建请求模型
"""
name: str = Field(..., description="供应商名称", examples=["腾讯集团"])
short_name: str = Field(..., description="供应商简称", examples=["腾讯"])
code: str = Field(..., description="供应商编码", examples=["TX001"])
company_type: SupplierType = Field(..., description="供应商类型", examples=["project"])
parent_id: Optional[str] = Field(None, description="上级供应商ID", examples=[None])
status: SupplierStatus = Field(..., description="供应商状态", examples=["active"])
path: Optional[List[str]] = Field(default_factory=list, description="供应商路径 [..., 上级id, 本身id]")
path_name: Optional[List[str]] = Field(default_factory=list, description="供应商路径名称 [..., 上级名称, 本身名称]")
credit_code: str = Field(..., description="供应商信用代码", examples=["914403007109310121"])
region_path: Optional[list] = Field(default_factory=list, description="供应商国家城市地址编码列表, 需对接集团侧地图组件, 没有对接可以不传递", examples=[[
"100000",
"420000",
"420800"
]])
company_nature: List[SupplierNature] = Field(..., description="供应商性质数组 可多选",
examples=[['construction',
'supplier',
'design',
'survey',
'supervisory',
'goverment']])
@model_validator(mode='after')
def validate_company_type_and_nature(self):
data = self.model_dump()
company_nature = data.get("company_nature")
if not company_nature:
raise HTTPException(detail=f"供应商:company_nature为空", status_code=400)
for cn in company_nature:
if cn not in SupplierNature.supplier_type_list():
raise HTTPException(detail=f"供应商:company_nature范围:{SupplierNature.supplier_type_list()}",
status_code=400)
company_type = data.get("company_type")
if not company_type:
raise HTTPException(detail=f"供应商:company_type为空", status_code=400)
if company_type not in SupplierType.supplier_list():
raise HTTPException(
detail=f"供应商:company_type范围({SupplierType.supplier_list()})", status_code=400)
for field in ["name", "short_name", "code", "credit_code", "status"]:
field_value = data.get(field)
if not field_value:
raise HTTPException(detail=f"字段{field}为空", status_code=400)
return self
@field_validator(
"credit_code",
mode="after"
)
def clean_credit_code(cls, field_value):
if field_value is None:
return None
import re
return re.sub(r'\s+', '', str(field_value))
schemas/user/user.py
+7 -6
View File
@@ -17,16 +17,17 @@ class UserListDetailItem(BaseResponseWithID):
status: Optional[str] = Field(None, description="状态") status: Optional[str] = Field(None, description="状态")
email: Optional[EmailStr] = Field(None, description="邮箱地址") email: Optional[EmailStr] = Field(None, description="邮箱地址")
user_type: str = Field(..., description="用户类型") user_type: str = Field(..., description="用户类型")
last_login_at: Optional[str] = Field(None, description="最后登录时间") # last_login_at: Optional[str] = Field(None, description="最后登录时间")
updated_at: Optional[str] = Field(None, description="更新时间") # updated_at: Optional[str] = Field(None, description="更新时间")
company_id: Optional[str] = Field(None, description="公司id") company_id: Optional[str] = Field(None, description="公司id")
company_name: Optional[str] = Field(None, description="公司名称") company_name: Optional[str] = Field(None, description="公司名称")
created_at: Optional[str] = Field(None, description="创建时间") # created_at: Optional[str] = Field(None, description="创建时间")
approval_status: Optional[str] = Field(None, description="审批状态") # approval_status: Optional[str] = Field(None, description="审批状态") # 不再区分审批状态和审批时间
approval_time: Optional[str] = Field(None, description="审批通过时间") # approval_time: Optional[str] = Field(None, description="审批通过时间")
last_login_ip: Optional[str] = Field(None, description="最后登录ip") # last_login_ip: Optional[str] = Field(None, description="最后登录ip")
project_ids: Optional[List[str]] = Field([], description="权限内项目id") project_ids: Optional[List[str]] = Field([], description="权限内项目id")
contract_ids: Optional[List[str]] = Field([], description="权限内合同id") contract_ids: Optional[List[str]] = Field([], description="权限内合同id")
phone_hash: Optional[str] = Field(None, description="手机号hash")
apply_files: Optional[List[Dict]] = Field([], apply_files: Optional[List[Dict]] = Field([],
examples=[ examples=[
{"id": "file:001", "name": "文件1", {"id": "file:001", "name": "文件1",
scripts/README.md
+32 -35
View File
@@ -1,44 +1,41 @@
# 脚本目录 # 脚本目录
## 文件说明 本目录当前主要放两类脚本：
- 自动化测试编排脚本
- 本地数据集构建/分析脚本
### test_api_sync.py 旧的 `scripts/test_api_sync.py` 与 `API_SYNC_GUIDE.md` 已不存在，不再作为当前入口。
主要的API数据同步测试脚本。
**功能：** ## 当前常用入口
- 从远程API获取所有GET/LIST接口数据
- 自动分页获取全量数据 ### 自动化测试
- 保存为JSONL格式到 `remote_datasource` 目录 - `scripts/auto_test_init.py`
- 支持单项目模式或全量同步 - 初始化 auto-test 运行环境，负责清理状态、恢复测试数据、拉起后端。
- `scripts/auto_test_clear.py`
- 清理 auto-test 运行环境。
- `scripts/run_auto_test.py`
- 运行指定 domain 的多轮自动回归并输出报告。
- `scripts/run_auto_test_all_domains.py`
- 聚合运行多个 domain 的自动回归。
- `scripts/run_large_scale_auto_test.py`
- 大规模自动回归入口。
### 数据集与分析
- `scripts/build_filtered_datasource_for_projects.py`
- 从完整 datasource 中抽取指定项目，生成可复现的精简 JSONL 数据集。
- `scripts/analyze_state_signatures.py`
- 分析状态签名与差异。
## 推荐命令
这些命令目前可直接使用：
**运行方式：**
```bash ```bash
python scripts/test_api_sync.py python scripts/build_filtered_datasource_for_projects.py --help
python scripts/run_auto_test.py --help
``` ```
**数据类型覆盖：** ## 进一步说明
- 基础数据: company, user, supplier, project
- 项目数据: contract, construction_task, material, contract_settlement, personnel
- 项目扩展: preparation, production, lar, units (从all_info获取)
- 明细数据: construction_task_detail, material_detail, contract_settlement_detail, personnel_detail
**配置说明：** - 自动化测试总览：`docs/testing_guide.md`
在文件中修改以下配置： - 大规模自动回归方法：`docs/auto_test_spec/agent_auto_test_method.md`
```python
BASE_URL = "http://localhost:8000/api/third/v2"
UID = "your-uid"
SECRET_KEY = "your-secret-key"
TARGET_PROJECT_ID = "xxx-xxx-xxx" # 单项目模式，设为None则全量同步
```
## 输出结果
数据保存在 `remote_datasource/datasource/` 目录下：
- `company_1.jsonl` - 公司数据
- `user_1.jsonl` - 用户数据
- `project_1.jsonl` - 项目数据
- ... 其他数据类型
## 更多文档
详细文档请查看项目根目录的 [API_SYNC_GUIDE.md](../API_SYNC_GUIDE.md)
scripts/auto_test_domains/base.py
+3 -3
View File
@@ -292,8 +292,8 @@ def build_temp_run_profile_for_project_auto_bind(
project_detail_push_keys: set[str] | None = None, project_detail_push_keys: set[str] | None = None,
) -> tuple[str, list[str]]: ) -> tuple[str, list[str]]:
profile = load_yaml_config(config["pipeline"]["run_profile"]) profile = load_yaml_config(config["pipeline"]["run_profile"])
profile.setdefault("local_datasource", {})["target_project_ids"] = [local_project_id] profile.setdefault("local_datasource", {}).setdefault("handler_configs", {}).setdefault("project", {})["data_id_filter"] = [local_project_id]
profile.setdefault("remote_datasource", {})["target_project_ids"] = [remote_project_id] profile.setdefault("remote_datasource", {}).setdefault("handler_configs", {}).setdefault("project", {})["data_id_filter"] = [remote_project_id]
project_strategy = profile.setdefault("strategies", {}).setdefault("project", {}).setdefault("config", {}) project_strategy = profile.setdefault("strategies", {}).setdefault("project", {}).setdefault("config", {})
project_strategy["auto_bind"] = False project_strategy["auto_bind"] = False
@@ -365,7 +365,7 @@ def run_pipeline_round(
merge_tree_contents(common_dataset_dir, paths["temp_data_dir"]) merge_tree_contents(common_dataset_dir, paths["temp_data_dir"])
enrich_supplier_display_fields( enrich_supplier_display_fields(
paths["temp_data_dir"], paths["temp_data_dir"],
f"{config['project_root']}/filtered_datasource/datasource/filtered/region_1.jsonl", f"{config['project_root']}/working_local_datasource/filtered/region_1.jsonl",
) )
normalize_contract_settlement_records(paths["temp_data_dir"]) normalize_contract_settlement_records(paths["temp_data_dir"])
normalize_construction_task_plan_nodes(paths["temp_data_dir"]) normalize_construction_task_plan_nodes(paths["temp_data_dir"])
scripts/legacy/run_full_sync_pipeline_jsonl.py
-208
View File
@@ -1,208 +0,0 @@
"""
Run FullSyncPipeline: JSONL -> JSONL (filtered_datasource -> remote_datasource)
直接运行脚本，不使用 pytest。
"""
from __future__ import annotations
import asyncio
import sys
from pathlib import Path
from typing import List
# 确保能导入项目模块
PROJECT_ROOT = Path(__file__).resolve().parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
# 触发 DomainRegistry 注册
import sync_system_new.domain # noqa: F401, E402
from sync_system_new.common.binding import BindingManager # noqa: E402
from sync_system_new.common.collection import DataCollection # noqa: E402
from sync_system_new.common.persistence import PersistenceBackend # noqa: E402
from sync_system_new.common.registry import DomainRegistry # noqa: E402
from sync_system_new.datasource.jsonl.datasource import JsonlDataSource # noqa: E402
from sync_system_new.pipeline import FullSyncPipeline # noqa: E402
from sync_system_new.sync_system.config import OrphanAction, UpdateDirection # noqa: E402
LOCAL_DIR = PROJECT_ROOT / "filtered_datasource" / "datasource" / "ecm-2025-12-02"
REMOTE_DIR = PROJECT_ROOT / "remote_datasource" / "datasource"
PERSISTENCE_DB = PROJECT_ROOT / "test_results" / "full_sync_pipeline_jsonl_run.db"
WIPE_DB_ON_START = False
NODE_TYPES: List[str] = [
"company",
# "supplier",
"user",
"project",
"project_detail_data",
"contract",
"contract_change",
"construction_task",
"construction_task_detail",
"material",
"material_detail",
"contract_settlement",
"contract_settlement_detail",
"personnel",
"personnel_detail",
"preparation",
"production",
"lar",
"units",
]
class StrategyLogger:
"""包装策略，打印执行完成信息。"""
def __init__(self, inner):
self.inner = inner
self.node_type = inner.node_type
async def bind(self):
return await self.inner.bind()
async def create(self):
created = await self.inner.create()
print(f"✅ Strategy create: {self.node_type} (created={len(created)})")
return created
async def update(self):
updated = await self.inner.update()
print(f"✅ Strategy update: {self.node_type} (updated={len(updated)})")
return updated
async def main() -> None:
PERSISTENCE_DB.parent.mkdir(parents=True, exist_ok=True)
if WIPE_DB_ON_START and PERSISTENCE_DB.exists():
PERSISTENCE_DB.unlink()
persistence = PersistenceBackend(str(PERSISTENCE_DB))
await persistence.initialize()
local_ds = JsonlDataSource(LOCAL_DIR, read_only=True)
remote_ds = JsonlDataSource(REMOTE_DIR, read_only=False)
for node_type in NODE_TYPES:
handler_class = DomainRegistry.get_jsonl_handler(node_type)
if handler_class is None:
raise RuntimeError(f"Missing JSONL handler for node_type: {node_type}")
local_ds.register_handler(handler_class(local_ds))
remote_ds.register_handler(handler_class(remote_ds))
local_collection = DataCollection("local", persistence, auto_persist=False)
remote_collection = DataCollection("remote", persistence, auto_persist=False)
binding_manager = BindingManager(persistence, auto_persist=False)
strategies = []
for node_type in NODE_TYPES:
strategy_class = DomainRegistry.get_strategy_class(node_type)
if strategy_class is None:
raise RuntimeError(f"Missing strategy for node_type: {node_type}")
strategy = strategy_class(node_type, local_collection, remote_collection, binding_manager)
# 运行时强制只推不拉（不覆盖原有 depend_fields/auto_bind_fields）
strategy.update_config(
local_orphan_action=OrphanAction.CREATE_REMOTE,
remote_orphan_action=OrphanAction.NONE,
update_direction=UpdateDirection.PUSH,
)
if node_type == "contract":
# contract 仅依赖 project_id，避免 company_id 为空导致依赖阻断
strategy.update_config(depend_fields={"project_id": "project"})
strategies.append(StrategyLogger(strategy))
pipeline = FullSyncPipeline(
local_datasource=local_ds,
remote_datasource=remote_ds,
strategies=strategies,
node_types=NODE_TYPES,
load_order=NODE_TYPES,
sync_order=NODE_TYPES,
persistence=persistence,
local_collection=local_collection,
remote_collection=remote_collection,
binding_manager=binding_manager,
enable_persistence=True,
fail_fast_on_precheck=False,
)
stats = await pipeline.run()
print(f"Pipeline finished. Stats: {stats}")
binding_counts = {}
for node_type in NODE_TYPES:
records = await binding_manager.get_all_records(node_type)
binding_counts[node_type] = len(records)
def format_action(summary, key: str) -> str:
action = summary.get(key, {})
return f"{action.get('total', 0)}({action.get('success', 0)}/{action.get('failed', 0)})"
def format_total(summary) -> str:
total = summary.get("total", {})
return f"{total.get('total', 0)}({total.get('success', 0)}/{total.get('failed', 0)})"
def print_summary(title: str, action_summary: dict) -> None:
print(f"\n=== {title} summary ===")
print("doc_type: bindings | create(s/f) | update(s/f) | delete(s/f) | none | total(s/f)")
total_bindings = 0
total_create = {"total": 0, "success": 0, "failed": 0}
total_update = {"total": 0, "success": 0, "failed": 0}
total_delete = {"total": 0, "success": 0, "failed": 0}
total_none = 0
total_all = {"total": 0, "success": 0, "failed": 0}
for node_type in NODE_TYPES:
summary = action_summary.get(node_type, {})
bindings = binding_counts.get(node_type, 0)
create_str = format_action(summary, "create")
update_str = format_action(summary, "update")
delete_str = format_action(summary, "delete")
none_count = summary.get("none", {}).get("total", 0)
total_str = format_total(summary)
print(
f"{node_type}: bindings={bindings} "
f"create={create_str} update={update_str} delete={delete_str} "
f"none={none_count} total={total_str}"
)
total_bindings += bindings
for bucket, key in (
(total_create, "create"),
(total_update, "update"),
(total_delete, "delete"),
):
action = summary.get(key, {})
bucket["total"] += action.get("total", 0)
bucket["success"] += action.get("success", 0)
bucket["failed"] += action.get("failed", 0)
total_none += none_count
total_all["total"] += summary.get("total", {}).get("total", 0)
total_all["success"] += summary.get("total", {}).get("success", 0)
total_all["failed"] += summary.get("total", {}).get("failed", 0)
print(
"TOTAL: "
f"bindings={total_bindings} "
f"create={total_create['total']}({total_create['success']}/{total_create['failed']}) "
f"update={total_update['total']}({total_update['success']}/{total_update['failed']}) "
f"delete={total_delete['total']}({total_delete['success']}/{total_delete['failed']}) "
f"none={total_none} "
f"total={total_all['total']}({total_all['success']}/{total_all['failed']})"
)
print_summary("Local", local_ds.get_action_summary())
print_summary("Remote", remote_ds.get_action_summary())
await persistence.close()
if __name__ == "__main__":
asyncio.run(main())
scripts/legacy/run_full_sync_pipeline_jsonl_sm.py
-153
View File
@@ -1,153 +0,0 @@
"""
run_full_sync_pipeline_jsonl_sm.py
独立执行：基于 sync_state_machine 的 JSONL -> JSONL 全量同步脚本。
用途：纯本地可复现，不依赖远端 API；用于验证重构后 pipeline 主流程可跑通。
"""
from __future__ import annotations
import asyncio
import os
import sys
from datetime import datetime
from pathlib import Path
from typing import Dict, List
from pydantic import BaseModel, ConfigDict, Field
PROJECT_ROOT = Path(__file__).resolve().parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
import sync_state_machine.domain # noqa: F401, E402
from sync_state_machine.pipeline import create_jsonl_to_jsonl_pipeline # noqa: E402
class JsonlSmConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
local_jsonl_dir: str = str(PROJECT_ROOT / "filtered_datasource" / "datasource" / "ecm-2025-12-02")
remote_jsonl_dir: str = str(PROJECT_ROOT / "remote_datasource" / "datasource")
persistence_db: str = str(PROJECT_ROOT / "test_results" / "sync_state_machine" / "full_sync_pipeline_jsonl_sm.db")
wipe_persistence_on_start: bool = False
target_project_ids: List[str] = Field(default_factory=lambda: ["f3e02e84-e81c-4aa6-88d4-f5aa108c8227"])
project_auto_bind_fields: List[str] = Field(default_factory=lambda: ["name"])
project_id_bindings: Dict[str, str] = Field(default_factory=dict)
node_types: List[str] = Field(default_factory=lambda: [
"company",
"user",
"project",
"contract",
"contract_change",
"construction_task",
"construction_task_detail",
"material",
"material_detail",
"contract_settlement",
"contract_settlement_detail",
"personnel",
"personnel_detail",
"preparation",
"production",
"lar",
"units",
])
def _load_dotenv() -> Dict[str, str]:
dotenv_path = PROJECT_ROOT / ".env"
if not dotenv_path.exists():
return {}
values: Dict[str, str] = {}
for raw_line in dotenv_path.read_text(encoding="utf-8").splitlines():
line = raw_line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
values[key.strip()] = value.strip().strip('"').strip("'")
return values
def build_config() -> JsonlSmConfig:
cfg = JsonlSmConfig()
env = _load_dotenv()
cfg.wipe_persistence_on_start = (
os.getenv("SPS_JSONL_WIPE_DB", env.get("SPS_JSONL_WIPE_DB", str(cfg.wipe_persistence_on_start))).lower()
in {"1", "true", "yes"}
)
return cfg
def _print_config_snapshot(config: JsonlSmConfig) -> None:
snapshot = {
"local_jsonl_dir": config.local_jsonl_dir,
"remote_jsonl_dir": config.remote_jsonl_dir,
"persistence_db": config.persistence_db,
"wipe_persistence_on_start": config.wipe_persistence_on_start,
"target_project_ids": config.target_project_ids,
"project_auto_bind_fields": config.project_auto_bind_fields,
"project_id_bindings": config.project_id_bindings,
"node_types": config.node_types,
}
print(f"🔧 Resolved Config: {snapshot}")
async def main() -> int:
pipeline = None
log_path = None
config = build_config()
try:
log_dir = PROJECT_ROOT / "test_results" / "sync_state_machine"
log_dir.mkdir(parents=True, exist_ok=True)
log_path = log_dir / f"jsonl_pipeline_sm_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
print("\n" + "=" * 80)
print("🚀 Creating sync_state_machine JSONL -> JSONL pipeline")
print(f"📝 Log file: {log_path}")
_print_config_snapshot(config)
print("=" * 80)
Path(config.persistence_db).parent.mkdir(parents=True, exist_ok=True)
pipeline = await create_jsonl_to_jsonl_pipeline(
local_jsonl_dir=config.local_jsonl_dir,
remote_jsonl_dir=config.remote_jsonl_dir,
persistence_db=config.persistence_db,
node_types=config.node_types,
load_order=config.node_types,
sync_order=config.node_types,
enable_persistence=True,
wipe_persistence_on_start=config.wipe_persistence_on_start,
target_project_ids=config.target_project_ids,
project_id_bindings=config.project_id_bindings,
project_auto_bind_fields=config.project_auto_bind_fields,
enable_project_bind_override=True,
enable_project_filter_override=True,
initialize_logger=True,
logger_file_path=str(log_path),
)
stats = await pipeline.run()
print(f"\n✅ Pipeline completed: stats={stats}")
print(f"🗃️ Persistence DB: {config.persistence_db}")
return 0
except KeyboardInterrupt:
print("\n⚠️ Interrupted by user")
return 130
except Exception as exc:
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
import traceback
traceback.print_exc()
return 1
finally:
if pipeline is not None:
await pipeline.close()
if __name__ == "__main__":
raise SystemExit(asyncio.run(main()))
scripts/legacy/run_full_sync_pipeline_simple.py
-179
View File
@@ -1,179 +0,0 @@
"""
run_full_sync_pipeline_simple.py
极简测试脚本：演示使用工厂函数创建 Pipeline
- 只需配置参数
- 调用工厂函数
- 执行 pipeline.run()
"""
import asyncio
import sys
from pathlib import Path
# 项目路径
PROJECT_ROOT = Path(__file__).resolve().parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
# 触发 Domain 注册
import sync_system_new.domain # noqa: F401, E402
from sync_system_new.pipeline import create_jsonl_to_api_pipeline # noqa: E402
# ========== 配置参数 ==========
CONFIG = {
# 路径
"local_jsonl_dir": str(PROJECT_ROOT / "filtered_datasource" / "datasource" / "ecm-2025-12-02"),
"persistence_db": str(PROJECT_ROOT / "_runtime" / "test_simple.db"),
# API
"api_base_url": "http://localhost:8000/api/third/v2",
"api_uid": "6b18c681-92d5-4c7f-802c-df413b6504c8",
"api_secret": "1a28Az5dBD1Gc3URpmHwiR0XfzW7UtScQnqJQzgpzdg",
# 节点类型
"node_types": [
"company",
"supplier",
# "user",
"project",
"project_detail_data",
"preparation",
"production",
"lar",
"lar_change",
"units",
"contract",
"contract_change",
"construction_task",
"construction_task_detail",
"material",
"material_detail",
"contract_settlement",
"contract_settlement_detail",
"personnel",
"personnel_detail",
],
# 项目过滤和预绑定
"target_project_ids": ["f3e02e84-e81c-4aa6-88d4-f5aa108c8227"],
"project_id_bindings": {
"f3e02e84-e81c-4aa6-88d4-f5aa108c8227": "f3e02e84-e81c-4aa6-88d4-f5aa108c8227",
},
# Handler 自定义配置
"handler_configs": {
# supplier: 从持久化加载，不调用 API（后续运行时启用）
"supplier": {"load_mode": "persisted_only"},
},
# Skip/Force Sync 控制
# skip_sync_types: 跳过同步（只执行 bind，不执行 create/update）
# 后续运行时可配置: "skip_sync_types": ["supplier"]
"skip_sync_types": ["supplier"],
# 行为
"wipe_persistence_on_start": False, # 改为 False，否则持久化数据会被清空
"enable_persistence": True,
# API 调试
"api_debug": True,
"poll_max_retries": 1,
"poll_interval": 2.0,
}
async def main():
"""主函数 - 返回 True 表示成功，False 表示被中断"""
from datetime import datetime
import sys
from pathlib import Path
pipeline = None
interrupted = False
log_file = None
original_stdout = None
original_stderr = None
# 日志设置
log_dir = Path(PROJECT_ROOT) / "test_results"
log_dir.mkdir(parents=True, exist_ok=True)
log_path = log_dir / f"simple_pipeline_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
try:
# 重定向输出到日志文件
log_file = open(log_path, "w", encoding="utf-8")
class Tee:
def __init__(self, *streams):
self._streams = streams
def write(self, data):
for stream in self._streams:
stream.write(data)
stream.flush()
return len(data)
def flush(self):
for stream in self._streams:
stream.flush()
original_stdout = sys.stdout
original_stderr = sys.stderr
sys.stdout = Tee(original_stdout, log_file)
sys.stderr = Tee(original_stderr, log_file)
print("\n" + "=" * 60)
print("🚀 Creating Pipeline...")
print(f"📝 Log file: {log_path}")
print("=" * 60)
# 一行创建完整 Pipeline
pipeline = await create_jsonl_to_api_pipeline(**CONFIG)
print(f"✅ Pipeline created!")
print(f" - Node types: {len(pipeline.node_types)}")
print(f" - Strategies: {len(pipeline.strategies)}")
print("\n" + "=" * 60)
print("🏃 Running Pipeline...")
print("=" * 60)
# 执行同步（print_summary 已整合到 stage4_post_check 中）
stats = await pipeline.run()
print("\n✅ Done!")
except KeyboardInterrupt:
print("\n\n⚠️ KeyboardInterrupt received, shutting down gracefully...")
interrupted = True
except Exception as e:
print(f"\n\n❌ Error: {type(e).__name__}: {e}")
import traceback
traceback.print_exc()
finally:
if pipeline is not None:
print("🧹 Cleaning up resources...")
await pipeline.close()
print("✅ Resources cleaned up")
# 恢复原始输出
if original_stdout is not None:
sys.stdout = original_stdout
if original_stderr is not None:
sys.stderr = original_stderr
if log_file is not None:
log_file.close()
if not interrupted:
print(f"\n📝 Log saved to: {log_path}")
return not interrupted # 返回 False 表示被中断
if __name__ == "__main__":
try:
success = asyncio.run(main())
sys.exit(0 if success else 130) # 130 是标准的 Ctrl+C 退出码
except KeyboardInterrupt:
# asyncio.run() 可能会重新抛出 KeyboardInterrupt
print("\n⚠️ Program interrupted")
sys.exit(130)
scripts/legacy/run_full_sync_pipeline_simple_sm.py
-143
View File
@@ -1,143 +0,0 @@
"""
run_full_sync_pipeline_simple_sm.py
独立执行：基于 sync_state_machine 的 JSONL -> API Pipeline。
用途：保留线上接口参数，直接执行重构后的推送链路（非 pytest）。
"""
from __future__ import annotations
import asyncio
import os
import sys
from datetime import datetime
from pathlib import Path
from typing import Dict, List
from pydantic import BaseModel, ConfigDict, Field
PROJECT_ROOT = Path(__file__).resolve().parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
import sync_state_machine.domain # noqa: F401, E402
from sync_state_machine.pipeline import create_jsonl_to_api_pipeline # noqa: E402
class SimpleSmConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
local_jsonl_dir: str = str(PROJECT_ROOT / "filtered_datasource" / "datasource" / "ecm-2025-12-02")
persistence_db: str = str(PROJECT_ROOT / "test_results" / "sync_state_machine" / "simple_pipeline_sm.db")
api_base_url: str = "http://localhost:8000/api/third/v2"
api_uid: str = "6b18c681-92d5-4c7f-802c-df413b6504c8"
api_secret: str = "1a28Az5dBD1Gc3URpmHwiR0XfzW7UtScQnqJQzgpzdg"
node_types: List[str] = Field(default_factory=lambda: [
"project",
"contract",
"contract_settlement",
])
target_project_ids: List[str] = Field(default_factory=lambda: ["f3e02e84-e81c-4aa6-88d4-f5aa108c8227"])
project_id_bindings: Dict[str, str] = Field(
default_factory=lambda: {
"f3e02e84-e81c-4aa6-88d4-f5aa108c8227": "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
}
)
skip_sync_types: List[str] = Field(default_factory=lambda: ["supplier"])
wipe_persistence_on_start: bool = False
enable_persistence: bool = True
api_debug: bool = True
poll_max_retries: int = 1
poll_interval: float = 2.0
def _load_dotenv() -> Dict[str, str]:
dotenv_path = PROJECT_ROOT / ".env"
if not dotenv_path.exists():
return {}
values: Dict[str, str] = {}
for raw_line in dotenv_path.read_text(encoding="utf-8").splitlines():
line = raw_line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
values[key.strip()] = value.strip().strip('"').strip("'")
return values
def build_config() -> SimpleSmConfig:
cfg = SimpleSmConfig()
env = _load_dotenv()
cfg.api_base_url = os.getenv("SPS_API_BASE_URL", env.get("SPS_API_BASE_URL", cfg.api_base_url))
cfg.api_uid = os.getenv("SPS_API_UID", env.get("SPS_API_UID", cfg.api_uid))
cfg.api_secret = os.getenv("SPS_API_SECRET", env.get("SPS_API_SECRET", cfg.api_secret))
return cfg
def _print_config_snapshot(config: SimpleSmConfig) -> None:
masked_secret = "***" if config.api_secret else ""
snapshot = {
"local_jsonl_dir": config.local_jsonl_dir,
"persistence_db": config.persistence_db,
"api_base_url": config.api_base_url,
"api_uid": config.api_uid,
"api_secret": masked_secret,
"node_types": config.node_types,
"target_project_ids": config.target_project_ids,
"project_id_bindings": config.project_id_bindings,
"skip_sync_types": config.skip_sync_types,
"wipe_persistence_on_start": config.wipe_persistence_on_start,
"enable_persistence": config.enable_persistence,
"api_debug": config.api_debug,
"poll_max_retries": config.poll_max_retries,
"poll_interval": config.poll_interval,
}
print(f"🔧 Resolved Config: {snapshot}")
async def main() -> int:
pipeline = None
log_path = None
config = build_config()
try:
log_dir = PROJECT_ROOT / "test_results" / "sync_state_machine"
log_dir.mkdir(parents=True, exist_ok=True)
log_path = log_dir / f"simple_pipeline_sm_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
print("\n" + "=" * 80)
print("🚀 Creating sync_state_machine JSONL -> API pipeline")
print(f"📝 Log file: {log_path}")
_print_config_snapshot(config)
print("=" * 80)
pipeline = await create_jsonl_to_api_pipeline(
**config.model_dump(),
initialize_logger=True,
logger_file_path=str(log_path),
)
stats = await pipeline.run()
print(f"\n✅ Pipeline completed: stats={stats}")
print(f"🗃️ Persistence DB: {config.persistence_db}")
return 0
except KeyboardInterrupt:
print("\n⚠️ Interrupted by user")
return 130
except Exception as exc:
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
import traceback
traceback.print_exc()
return 1
finally:
if pipeline is not None:
await pipeline.close()
if __name__ == "__main__":
raise SystemExit(asyncio.run(main()))
sync_state_machine/__init__.py
+10 -4
View File
@@ -25,12 +25,15 @@ from .config import (
list_registered_datasource_types, list_registered_datasource_types,
load_named_preset_overrides, load_named_preset_overrides,
load_overrides_from_file, load_overrides_from_file,
MultiProjectRunConfig,
build_multi_project_config_from_file,
is_multi_project_payload,
register_datasource_config_model, register_datasource_config_model,
register_datasource_factory, register_datasource_factory,
register_datasource_type, register_datasource_type,
) )
from .datasource import ApiClient, ApiDataSource, BaseApiHandler, BaseDbHandler, BaseJsonlHandler, DbDataSource, JsonlDataSource from .datasource import ApiClient, ApiDataSource, BaseApiHandler, BaseJsonlHandler, JsonlDataSource
from .pipeline import FullSyncPipeline, create_pipeline_from_config, run_pipeline_from_config from .pipeline import FullSyncPipeline, MultiProjectPipeline, create_pipeline_from_config, run_pipeline_from_config, run_profile_from_file
try: try:
__version__ = version("ecm-sync-system") __version__ = version("ecm-sync-system")
@@ -59,17 +62,20 @@ __all__ = [
"list_registered_datasource_types", "list_registered_datasource_types",
"load_overrides_from_file", "load_overrides_from_file",
"load_named_preset_overrides", "load_named_preset_overrides",
"MultiProjectRunConfig",
"build_multi_project_config_from_file",
"is_multi_project_payload",
"register_datasource_config_model", "register_datasource_config_model",
"register_datasource_factory", "register_datasource_factory",
"register_datasource_type", "register_datasource_type",
"ApiClient", "ApiClient",
"ApiDataSource", "ApiDataSource",
"DbDataSource",
"JsonlDataSource", "JsonlDataSource",
"BaseApiHandler", "BaseApiHandler",
"BaseDbHandler",
"BaseJsonlHandler", "BaseJsonlHandler",
"FullSyncPipeline", "FullSyncPipeline",
"MultiProjectPipeline",
"create_pipeline_from_config", "create_pipeline_from_config",
"run_pipeline_from_config", "run_pipeline_from_config",
"run_profile_from_file",
] ]
sync_state_machine/cli.py
+4 -9
View File
@@ -5,8 +5,7 @@ import asyncio
import logging import logging
from pathlib import Path from pathlib import Path
from .config import build_config, load_overrides_from_file from .pipeline import run_profile_from_file
from .pipeline import run_pipeline_from_config
def package_project_root() -> Path: def package_project_root() -> Path:
@@ -35,14 +34,10 @@ async def _main() -> int:
if not cfg_path.exists() or not cfg_path.is_file(): if not cfg_path.exists() or not cfg_path.is_file():
raise FileNotFoundError(f"Config file not found: {cfg_path}") raise FileNotFoundError(f"Config file not found: {cfg_path}")
overrides = load_overrides_from_file(cfg_path, project_root)
config = build_config(project_root=project_root, overrides=overrides)
mode = f"{config.local_datasource.type}->{config.remote_datasource.type}"
print(f"🧩 Loaded config file: {cfg_path}") print(f"🧩 Loaded config file: {cfg_path}")
await run_pipeline_from_config( await run_profile_from_file(
config, project_root=project_root,
title=f"sync_state_machine pipeline ({mode})", config_path=cfg_path,
print_summary=True, print_summary=True,
) )
return 0 return 0
sync_state_machine/common/__init__.py
+11 -1
View File
@@ -1,6 +1,12 @@
from .types import SyncAction, SyncStatus, BindingStatus from .types import SyncAction, SyncStatus, BindingStatus
from .sync_node import SyncNode from .sync_node import SyncNode
from .persistence import PersistenceBackend from .persistence import (
PersistenceBackend,
create_persistence_backend,
get_registered_persistence_backends,
register_persistence_backend,
)
from .persistence_backends import SQLitePersistenceBackend
from .collection import DataCollection from .collection import DataCollection
from .binding import BindingManager from .binding import BindingManager
@@ -10,6 +16,10 @@ __all__ = [
"BindingStatus", "BindingStatus",
"SyncNode", "SyncNode",
"PersistenceBackend", "PersistenceBackend",
"SQLitePersistenceBackend",
"create_persistence_backend",
"register_persistence_backend",
"get_registered_persistence_backends",
"DataCollection", "DataCollection",
"BindingManager", "BindingManager",
] ]
sync_state_machine/common/binding.py
+55 -8
View File
@@ -12,9 +12,10 @@ class BindingManager:
BindingManager 负责维护不同 DataCollection 间节点的绑定关系。 BindingManager 负责维护不同 DataCollection 间节点的绑定关系。
所有查询均在内存中进行，持久化仅作为备份。 所有查询均在内存中进行，持久化仅作为备份。
""" """
def __init__(self, persistence: PersistenceBackend, auto_persist: bool = True): def __init__(self, persistence: PersistenceBackend, auto_persist: bool = True, scope: str = "global"):
self.persistence = persistence self.persistence = persistence
self.auto_persist = auto_persist self.auto_persist = auto_persist
self.scope = scope
# { node_type: { local_id: remote_id } } # { node_type: { local_id: remote_id } }
self._bindings: Dict[str, Dict[str, Optional[str]]] = {} self._bindings: Dict[str, Dict[str, Optional[str]]] = {}
self._deleted: Dict[str, set[str]] = {} self._deleted: Dict[str, set[str]] = {}
@@ -55,7 +56,7 @@ class BindingManager:
logger.debug(log_line) logger.debug(log_line)
if self.auto_persist: if self.auto_persist:
await self.persistence.save_binding(node_type, local_id, {"remote_id": remote_id}) await self.persistence.save_binding(self.scope, node_type, local_id, {"remote_id": remote_id})
async def unbind(self, node_type: str, local_id: str, manual: bool = False): async def unbind(self, node_type: str, local_id: str, manual: bool = False):
""" """
@@ -80,7 +81,7 @@ class BindingManager:
) )
if self.auto_persist: if self.auto_persist:
await self.persistence.delete_binding(node_type, local_id) await self.persistence.delete_binding(self.scope, node_type, local_id)
else: else:
self._get_deleted(node_type).add(local_id) self._get_deleted(node_type).add(local_id)
@@ -119,25 +120,71 @@ class BindingManager:
async def load_from_persistence(self, node_type: str): async def load_from_persistence(self, node_type: str):
"""从持久化层加载特定类型的绑定关系到内存""" """从持久化层加载特定类型的绑定关系到内存"""
raw_bindings = await self.persistence.load_bindings(node_type) raw_bindings = await self.persistence.load_bindings(self.scope, node_type)
bindings = self._get_type_bindings(node_type) bindings = self._get_type_bindings(node_type)
bindings.clear()
self._get_deleted(node_type).clear() self._get_deleted(node_type).clear()
for rb in raw_bindings: for rb in raw_bindings:
payload = rb.get("payload") or {} payload = rb.get("payload") or {}
bindings[rb["local_id"]] = payload.get("remote_id") bindings[rb["local_id"]] = payload.get("remote_id")
async def persist(self) -> None: def import_bindings(self, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
"""导入既有绑定到当前 manager，仅写内存，不触发持久化。"""
target = self._get_type_bindings(node_type)
target.clear()
target.update(dict(bindings))
self._get_deleted(node_type).clear()
async def prune_missing_nodes(
self,
*,
local_collection,
remote_collection,
node_types: Optional[List[str]] = None,
) -> int:
target_types = list(node_types) if node_types is not None else list(self._bindings.keys())
deleted_count = 0
for node_type in target_types:
bindings = self._get_type_bindings(node_type)
if not bindings:
continue
valid_local_ids = {node.node_id for node in local_collection.filter(node_type=node_type)}
valid_remote_ids = {node.node_id for node in remote_collection.filter(node_type=node_type)}
stale_local_ids = [
local_id
for local_id, remote_id in bindings.items()
if local_id not in valid_local_ids or (remote_id is not None and remote_id not in valid_remote_ids)
]
for local_id in stale_local_ids:
del bindings[local_id]
deleted_count += 1
if self.auto_persist:
await self.persistence.delete_binding(self.scope, node_type, local_id)
else:
self._get_deleted(node_type).add(local_id)
return deleted_count
async def persist(self, *, exclude_node_types: Optional[List[str]] = None) -> None:
"""将所有绑定关系写回持久化层""" """将所有绑定关系写回持久化层"""
if not self._bindings and not self._deleted: if not self._bindings and not self._deleted:
logger.info("🔍 [BindingManager.persist] 跳过：无绑定变更") logger.info("🔍 [BindingManager.persist] 跳过：无绑定变更")
return return
excluded = set(node_type for node_type in (exclude_node_types or []) if node_type)
type_count = 0 type_count = 0
saved_total = 0 saved_total = 0
deleted_total = 0 deleted_total = 0
touched_types: List[str] = [] touched_types: List[str] = []
for node_type, bindings in self._bindings.items(): for node_type, bindings in self._bindings.items():
if node_type in excluded:
continue
deleted = list(self._get_deleted(node_type)) deleted = list(self._get_deleted(node_type))
binding_count = len(bindings) binding_count = len(bindings)
deleted_count = len(deleted) deleted_count = len(deleted)
@@ -149,13 +196,13 @@ class BindingManager:
touched_types.append(node_type) touched_types.append(node_type)
if deleted: if deleted:
await self.persistence.delete_bindings_bulk(node_type, deleted) await self.persistence.delete_bindings_bulk(self.scope, node_type, deleted)
self._get_deleted(node_type).clear() self._get_deleted(node_type).clear()
if bindings: if bindings:
await self.persistence.save_bindings_bulk(node_type, bindings) await self.persistence.save_bindings_bulk(self.scope, node_type, bindings)
touched_text = ",".join(touched_types) if touched_types else "none" touched_text = ",".join(touched_types) if touched_types else "none"
logger.info( logger.info(
f"🔍 [BindingManager.persist] 完成: types={type_count}, saved={saved_total}, deleted={deleted_total}, touched=[{touched_text}]" f"🔍 [BindingManager.persist] 完成: scope={self.scope}, types={type_count}, saved={saved_total}, deleted={deleted_total}, touched=[{touched_text}]"
) )
sync_state_machine/common/collection.py
+163 -11
View File
@@ -1,4 +1,5 @@
import uuid import uuid
import copy
from enum import Enum from enum import Enum
from typing import Dict, List, Optional, Any, Set, Callable, Literal from typing import Dict, List, Optional, Any, Set, Callable, Literal
from .sync_node import SyncNode from .sync_node import SyncNode
@@ -22,11 +23,13 @@ class DataCollection:
def __init__( def __init__(
self, self,
collection_id: str, collection_id: str,
scope: str = "global",
persistence: Optional[PersistenceBackend] = None, persistence: Optional[PersistenceBackend] = None,
auto_persist: bool = False, auto_persist: bool = False,
sm_runtime: Optional[Any] = None, sm_runtime: Optional[Any] = None,
): ):
self.collection_id = collection_id self.collection_id = collection_id
self.scope = scope
self.persistence = persistence self.persistence = persistence
self.auto_persist = auto_persist self.auto_persist = auto_persist
self._sm_runtime = sm_runtime self._sm_runtime = sm_runtime
@@ -67,7 +70,7 @@ class DataCollection:
self._nodes[node.node_id] = node self._nodes[node.node_id] = node
self._deleted_node_ids.discard(node.node_id) self._deleted_node_ids.discard(node.node_id)
if self.persistence and self.auto_persist: if self.persistence and self.auto_persist:
await self.persistence.save_node(self.collection_id, node) await self.persistence.save_node(self.collection_id, self.scope, node)
async def update(self, node: SyncNode): async def update(self, node: SyncNode):
old_node = self._nodes.get(node.node_id) old_node = self._nodes.get(node.node_id)
@@ -85,7 +88,7 @@ class DataCollection:
self._nodes[node.node_id] = node self._nodes[node.node_id] = node
self._deleted_node_ids.discard(node.node_id) self._deleted_node_ids.discard(node.node_id)
if self.persistence and self.auto_persist: if self.persistence and self.auto_persist:
await self.persistence.save_node(self.collection_id, node) await self.persistence.save_node(self.collection_id, self.scope, node)
async def delete(self, node_id: str): async def delete(self, node_id: str):
old_node = self._nodes.get(node_id) old_node = self._nodes.get(node_id)
@@ -96,7 +99,7 @@ class DataCollection:
del self._nodes[node_id] del self._nodes[node_id]
if self.persistence: if self.persistence:
if self.auto_persist: if self.auto_persist:
await self.persistence.delete_node(self.collection_id, node_id) await self.persistence.delete_node(self.collection_id, self.scope, node_id)
else: else:
self._deleted_node_ids.add(node_id) self._deleted_node_ids.add(node_id)
@@ -370,7 +373,7 @@ class DataCollection:
self._data_id_to_node_id = {} self._data_id_to_node_id = {}
self._deleted_node_ids.clear() self._deleted_node_ids.clear()
node_dicts = await self.persistence.load_nodes(self.collection_id) node_dicts = await self.persistence.load_nodes(self.collection_id, self.scope)
from .types import SyncAction, SyncStatus, BindingStatus from .types import SyncAction, SyncStatus, BindingStatus
from .registry import DomainRegistry from .registry import DomainRegistry
@@ -491,7 +494,7 @@ class DataCollection:
node_types = {node.node_type for node in self._nodes.values()} node_types = {node.node_type for node in self._nodes.values()}
for node_type in node_types: for node_type in node_types:
override_map = await self.persistence.load_bind_data_id_overrides(node_type) override_map = await self.persistence.load_bind_data_id_overrides(node_type, self.scope)
for node in self.filter(node_type=node_type): for node in self.filter(node_type=node_type):
bind_data_id = override_map.get(node.node_id) bind_data_id = override_map.get(node.node_id)
if bind_data_id: if bind_data_id:
@@ -499,24 +502,173 @@ class DataCollection:
else: else:
node.context.pop("bind_data_id", None) node.context.pop("bind_data_id", None)
async def persist(self) -> None: async def load_from_persistence_excluding(self, node_types: Optional[List[str]] = None):
"""从持久化恢复数据，但跳过指定 node_type。"""
if not self.persistence:
return
self._nodes = {}
self._data_id_to_node_id = {}
self._deleted_node_ids.clear()
node_dicts = await self.persistence.load_nodes(
self.collection_id,
self.scope,
exclude_node_types=node_types,
)
from .types import SyncAction, SyncStatus, BindingStatus
from .registry import DomainRegistry
for node_dict in node_dicts:
node_type = node_dict["node_type"]
node_class = DomainRegistry.get_node_class(node_type)
assert issubclass(node_class, SyncNode)
parse_errors: list[str] = []
raw_action = node_dict.get("action", SyncAction.NONE)
if isinstance(raw_action, str):
try:
action_value = SyncAction(raw_action)
except ValueError:
parse_errors.append(f"invalid action={raw_action}")
action_value = SyncAction.NONE
elif isinstance(raw_action, SyncAction):
action_value = raw_action
else:
if raw_action is None:
action_value = SyncAction.NONE
else:
parse_errors.append(f"invalid action type={type(raw_action).__name__}")
action_value = SyncAction.NONE
raw_status = node_dict.get("status", SyncStatus.PENDING)
if isinstance(raw_status, str):
try:
status_value = SyncStatus(raw_status)
except ValueError:
parse_errors.append(f"invalid status={raw_status}")
status_value = SyncStatus.FAILED
elif isinstance(raw_status, SyncStatus):
status_value = raw_status
else:
if raw_status is None:
status_value = SyncStatus.PENDING
else:
parse_errors.append(f"invalid status type={type(raw_status).__name__}")
status_value = SyncStatus.FAILED
raw_binding_status = node_dict.get("binding_status", BindingStatus.UNCHECKED)
if isinstance(raw_binding_status, str):
try:
binding_status_value = BindingStatus(raw_binding_status)
except ValueError:
parse_errors.append(f"invalid binding_status={raw_binding_status}")
binding_status_value = BindingStatus.ABNORMAL
elif isinstance(raw_binding_status, BindingStatus):
binding_status_value = raw_binding_status
else:
if raw_binding_status is None:
binding_status_value = BindingStatus.UNCHECKED
else:
parse_errors.append(
f"invalid binding_status type={type(raw_binding_status).__name__}"
)
binding_status_value = BindingStatus.ABNORMAL
raw_context = node_dict.get("context", {})
context_value = raw_context if isinstance(raw_context, dict) else {}
error_value = node_dict.get("error")
if parse_errors:
context_value = dict(context_value)
context_value["bootstrap_blocked"] = True
context_value["bootstrap_block_reason"] = "invalid_persisted_enum"
context_value["bootstrap_block_errors"] = parse_errors
binding_status_value = BindingStatus.ABNORMAL
action_value = SyncAction.NONE
status_value = SyncStatus.FAILED
details = "; ".join(parse_errors)
error_value = f"LOAD_INVALID_ENUM: {details}"
node = node_class(
node_id=node_dict["node_id"],
data_id=node_dict.get("data_id", ""),
depend_ids=node_dict.get("depend_ids", []),
data=node_dict.get("data"),
origin_data=node_dict.get("origin_data"),
action=action_value,
status=status_value,
binding_status=binding_status_value,
error=error_value,
sync_log=None,
context=context_value,
)
node.context["_loaded_from_persistence"] = True
self._nodes[node.node_id] = node
if node.data_id and node.data_id != "":
existing_node_id = self._data_id_to_node_id.get(node.data_id)
if existing_node_id is not None and existing_node_id != node.node_id:
raise ValueError(
f"Duplicate data_id detected while loading: {node.data_id} already bound to node_id={existing_node_id}"
)
self._data_id_to_node_id[node.data_id] = node.node_id
loaded_node_types = {node.node_type for node in self._nodes.values()}
for node_type in loaded_node_types:
override_map = await self.persistence.load_bind_data_id_overrides(node_type, self.scope)
for node in self.filter(node_type=node_type):
bind_data_id = override_map.get(node.node_id)
if bind_data_id:
node.context["bind_data_id"] = bind_data_id
else:
node.context.pop("bind_data_id", None)
async def import_nodes(self, nodes: List[SyncNode]) -> None:
"""导入现有节点到当前 collection，仅写内存，不触发持久化。"""
for source_node in nodes:
node = copy.deepcopy(source_node)
if node.node_id in self._nodes:
continue
if node.data_id and node.data_id != "":
existing_node_id = self._data_id_to_node_id.get(node.data_id)
if existing_node_id is not None and existing_node_id != node.node_id:
raise ValueError(
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
)
self._data_id_to_node_id[node.data_id] = node.node_id
self._nodes[node.node_id] = node
self._deleted_node_ids.discard(node.node_id)
async def persist(self, *, exclude_node_types: Optional[List[str]] = None) -> None:
"""将当前 collection 全量持久化到后端""" """将当前 collection 全量持久化到后端"""
if not self.persistence: if not self.persistence:
return return
# 先落库删除（用于 auto_persist=False 的场景） # 先落库删除（用于 auto_persist=False 的场景）
if self._deleted_node_ids: if self._deleted_node_ids:
await self.persistence.delete_nodes_bulk(self.collection_id, list(self._deleted_node_ids)) await self.persistence.delete_nodes_bulk(self.collection_id, self.scope, list(self._deleted_node_ids))
self._deleted_node_ids.clear() self._deleted_node_ids.clear()
if not self._nodes: if not self._nodes:
return return
await self.persistence.save_nodes_bulk(self.collection_id, list(self._nodes.values())) excluded = set(node_type for node_type in (exclude_node_types or []) if node_type)
nodes_to_persist = [
node for node in self._nodes.values() if node.node_type not in excluded
]
if not nodes_to_persist:
return
async def filter_by_project_ids(self, target_project_ids: List[str]) -> int: await self.persistence.save_nodes_bulk(self.collection_id, self.scope, nodes_to_persist)
"""按目标项目集合过滤当前 collection，返回删除数量。"""
target_set = set(target_project_ids or []) async def filter_by_project_ids(self, data_id_filter: List[str]) -> int:
"""按目标项目 data_id 集合过滤当前 collection，返回删除数量。"""
target_set = set(data_id_filter or [])
if not target_set: if not target_set:
return 0 return 0
sync_state_machine/common/persistence.py
+126 -311
View File
@@ -1,21 +1,20 @@
import aiosqlite from __future__ import annotations
import json
import os import os
import sqlite3 import sqlite3
from typing import Optional, Dict, List, Any, TYPE_CHECKING from abc import ABC, abstractmethod
from typing import Any, Callable, ClassVar, Dict, List, Optional, TYPE_CHECKING
from ..config.persist_config import PersistConfig
if TYPE_CHECKING: if TYPE_CHECKING:
from .sync_node import SyncNode from .sync_node import SyncNode
class PersistenceBackend:
""" class PersistenceBackend(ABC):
Persistence 负责同步系统运行时数据的持久化存储（aiosqlite 实现）。 """同步系统持久化契约。"""
主要作为 DataCollection 和 BindingManager 的后端存储。
""" backend_name: ClassVar[str] = "abstract"
def __init__(self, db_path: str = ":memory:", backend: str = "sqlite"):
self.db_path = db_path
self.backend = backend
self.conn: Optional[aiosqlite.Connection] = None
@staticmethod @staticmethod
def _is_read_only_sql(sql: str) -> bool: def _is_read_only_sql(sql: str) -> bool:
@@ -93,319 +92,135 @@ class PersistenceBackend:
] ]
return summary return summary
async def initialize(self): @abstractmethod
"""异步初始化数据库连接和表结构""" async def initialize(self) -> None:
if self.backend != "sqlite": raise NotImplementedError
raise NotImplementedError(f"Persistence backend not supported yet: {self.backend}")
# 确保目录存在 @abstractmethod
if self.db_path != ":memory:": async def save_node(self, collection_id: str, scope: str, node: "SyncNode") -> None:
dir_path = os.path.dirname(self.db_path) raise NotImplementedError
if dir_path:
os.makedirs(dir_path, exist_ok=True)
self.conn = await aiosqlite.connect(self.db_path) @abstractmethod
self.conn.row_factory = aiosqlite.Row async def save_nodes_bulk(self, collection_id: str, scope: str, nodes: List["SyncNode"]) -> None:
await self._init_db() raise NotImplementedError
async def _init_db(self): @abstractmethod
# 1. 节点表: 仅用于持久化存储 DataCollection 中的 SyncNode async def delete_node(self, collection_id: str, scope: str, node_id: str) -> None:
# 注意: depend_ids不持久化，每次从data实时计算 raise NotImplementedError
await self.conn.execute("""
CREATE TABLE IF NOT EXISTS nodes (
collection_id TEXT NOT NULL,
node_id TEXT NOT NULL,
node_type TEXT NOT NULL,
data_id TEXT,
bind_data_id TEXT,
data TEXT,
origin_data TEXT,
action TEXT,
status TEXT,
binding_status TEXT,
error TEXT,
sync_log TEXT,
context TEXT,
PRIMARY KEY (collection_id, node_id)
)
""")
await self._ensure_nodes_schema()
# 2. 绑定表: 存储本地到远程的映射关系 @abstractmethod
# 只保留 local_id / remote_id 两字段（remote_id 可空） async def delete_nodes_bulk(self, collection_id: str, scope: str, node_ids: List[str]) -> None:
await self._ensure_bindings_schema() raise NotImplementedError
await self.conn.commit()
async def _ensure_nodes_schema(self) -> None: @abstractmethod
cursor = await self.conn.execute("PRAGMA table_info(nodes)") async def load_nodes(
rows = await cursor.fetchall() self,
columns = [r[1] for r in rows] if rows else [] collection_id: str,
if "bind_data_id" not in columns: scope: str,
await self.conn.execute("ALTER TABLE nodes ADD COLUMN bind_data_id TEXT") *,
exclude_node_types: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
raise NotImplementedError
async def _ensure_bindings_schema(self) -> None: @abstractmethod
"""确保 bindings 表为 (node_type, local_id, remote_id) 结构。 async def save_binding(self, scope: str, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
raise NotImplementedError
若检测到旧表含 payload 列，则进行迁移。 @abstractmethod
""" async def save_bindings_bulk(self, scope: str, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
cursor = await self.conn.execute("PRAGMA table_info(bindings)") raise NotImplementedError
rows = await cursor.fetchall()
columns = [r[1] for r in rows] if rows else []
if not columns: @abstractmethod
await self.conn.execute(""" async def delete_binding(self, scope: str, node_type: str, local_id: str) -> None:
CREATE TABLE IF NOT EXISTS bindings ( raise NotImplementedError
node_type TEXT NOT NULL,
local_id TEXT NOT NULL,
remote_id TEXT,
last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (node_type, local_id)
)
""")
return
if "payload" in columns and "remote_id" not in columns: @abstractmethod
# 迁移旧结构 payload -> remote_id async def delete_bindings_bulk(self, scope: str, node_type: str, local_ids: List[str]) -> None:
await self.conn.execute(""" raise NotImplementedError
CREATE TABLE IF NOT EXISTS bindings_new (
node_type TEXT NOT NULL,
local_id TEXT NOT NULL,
remote_id TEXT,
last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (node_type, local_id)
)
""")
async with self.conn.execute("SELECT node_type, local_id, payload FROM bindings") as cur: @abstractmethod
old_rows = await cur.fetchall() async def load_bindings(self, scope: str, node_type: str) -> List[Dict[str, Any]]:
for row in old_rows: raise NotImplementedError
try:
payload_dict = json.loads(row[2]) if row[2] else {} @abstractmethod
except json.JSONDecodeError: async def load_bind_data_id_overrides(self, node_type: str, scope: str) -> Dict[str, Optional[str]]:
payload_dict = {} raise NotImplementedError
remote_id = payload_dict.get("remote_id")
await self.conn.execute( @abstractmethod
"INSERT OR REPLACE INTO bindings_new (node_type, local_id, remote_id) VALUES (?, ?, ?)", async def delete_scope_node_types(self, *, scope: str, node_types: List[str]) -> None:
(row[0], row[1], remote_id), raise NotImplementedError
@abstractmethod
async def copy_nodes_between_scopes(
self,
*,
collection_id: str,
source_scope: str,
target_scope: str,
node_types: List[str],
) -> None:
raise NotImplementedError
@abstractmethod
async def copy_bindings_between_scopes(
self,
*,
source_scope: str,
target_scope: str,
node_types: List[str],
) -> None:
raise NotImplementedError
@abstractmethod
async def dump_to_runtime(self, filename: str) -> None:
raise NotImplementedError
@abstractmethod
async def close(self) -> None:
raise NotImplementedError
PersistenceBackendFactory = Callable[[PersistConfig], PersistenceBackend]
_PERSISTENCE_BACKEND_REGISTRY: dict[str, PersistenceBackendFactory] = {}
def register_persistence_backend(name: str, factory: PersistenceBackendFactory, *, override: bool = False) -> None:
backend_name = name.strip()
if not backend_name:
raise ValueError("backend name cannot be empty")
if not override and backend_name in _PERSISTENCE_BACKEND_REGISTRY:
raise ValueError(f"persistence backend already registered: {backend_name}")
_PERSISTENCE_BACKEND_REGISTRY[backend_name] = factory
def get_registered_persistence_backends() -> Dict[str, PersistenceBackendFactory]:
return dict(_PERSISTENCE_BACKEND_REGISTRY)
def create_persistence_backend(
persist_config: PersistConfig,
) -> PersistenceBackend:
backend_name = (persist_config.backend or "").strip()
if not backend_name:
raise ValueError("persist.backend cannot be empty")
if not _PERSISTENCE_BACKEND_REGISTRY:
register_persistence_backend(
"sqlite",
lambda config: SQLitePersistenceBackend(config),
) )
await self.conn.execute("DROP TABLE bindings") factory = _PERSISTENCE_BACKEND_REGISTRY.get(backend_name)
await self.conn.execute("ALTER TABLE bindings_new RENAME TO bindings") if factory is None:
available = ", ".join(sorted(_PERSISTENCE_BACKEND_REGISTRY)) or "none"
raise NotImplementedError(f"Persistence backend not supported yet: {backend_name}. Available: {available}")
return factory(persist_config)
# --- DataCollection 支持 ---
async def save_node(self, collection_id: str, node: "SyncNode"): def _register_builtin_persistence_backends() -> None:
"""保存或更新单个节点（depend_ids不持久化，运行时从data实时计算）""" from .persistence_backends.sqlite_backend import SQLitePersistenceBackend
bind_data_id = None
if isinstance(node.context, dict):
value = node.context.get("bind_data_id")
if value is not None and value != "":
bind_data_id = str(value)
await self.conn.execute("""
INSERT OR REPLACE INTO nodes (
collection_id, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (
collection_id,
node.node_id,
node.node_type,
node.data_id,
bind_data_id,
json.dumps(node.data) if node.data else None,
json.dumps(node.origin_data) if node.origin_data else None,
node.action.value,
node.status.value,
node.binding_status.value,
node.error,
node.sync_log,
json.dumps(node.context) if node.context else None
))
await self.conn.commit()
async def save_nodes_bulk(self, collection_id: str, nodes: List["SyncNode"]): register_persistence_backend("sqlite", lambda config: SQLitePersistenceBackend(config), override=True)
"""批量保存或更新节点 (单次提交，depend_ids不持久化)"""
if not nodes:
return
rows = []
for node in nodes:
bind_data_id = None
if isinstance(node.context, dict):
value = node.context.get("bind_data_id")
if value is not None and value != "":
bind_data_id = str(value)
rows.append((
collection_id,
node.node_id,
node.node_type,
node.data_id,
bind_data_id,
json.dumps(node.data) if node.data else None,
json.dumps(node.origin_data) if node.origin_data else None,
node.action.value,
node.status.value,
node.binding_status.value,
node.error,
node.sync_log,
json.dumps(node.context) if node.context else None
))
await self.conn.executemany("""
INSERT OR REPLACE INTO nodes (
collection_id, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", rows)
await self.conn.commit()
async def delete_node(self, collection_id: str, node_id: str): _register_builtin_persistence_backends()
"""删除单个节点"""
await self.conn.execute("DELETE FROM nodes WHERE collection_id = ? AND node_id = ?", (collection_id, node_id))
await self.conn.commit()
async def delete_nodes_bulk(self, collection_id: str, node_ids: List[str]):
"""批量删除节点 (单次提交)"""
if not node_ids:
return
rows = [(collection_id, node_id) for node_id in node_ids]
await self.conn.executemany(
"DELETE FROM nodes WHERE collection_id = ? AND node_id = ?",
rows,
)
await self.conn.commit()
async def load_nodes(self, collection_id: str) -> List[Dict[str, Any]]:
"""从特定 collection 中加载所有节点（depend_ids初始化为空，稍后从data实时计算）"""
async with self.conn.execute("SELECT * FROM nodes WHERE collection_id = ?", (collection_id,)) as cursor:
rows = await cursor.fetchall()
results = []
for row in rows:
results.append({
"node_id": row["node_id"],
"node_type": row["node_type"],
"data_id": row["data_id"],
"bind_data_id": row["bind_data_id"],
"depend_ids": [], # 不从数据库加载，运行时从data实时计算
"data": json.loads(row["data"]) if row["data"] else None,
"origin_data": json.loads(row["origin_data"]) if row["origin_data"] else None,
"action": row["action"],
"status": row["status"],
"binding_status": row["binding_status"],
"error": row["error"],
"sync_log": row["sync_log"],
"context": json.loads(row["context"]) if row["context"] else {}
})
if results[-1]["bind_data_id"] and isinstance(results[-1]["context"], dict):
results[-1]["context"]["bind_data_id"] = results[-1]["bind_data_id"]
return results
# --- BindingManager 支持 ---
async def save_binding(self, node_type: str, local_id: str, payload_dict: Dict[str, Any]):
"""保存或更新绑定关系及其元数据"""
remote_id = payload_dict.get("remote_id") if payload_dict else None
await self.conn.execute("""
INSERT OR REPLACE INTO bindings (node_type, local_id, remote_id)
VALUES (?, ?, ?)
""", (node_type, local_id, remote_id))
await self.conn.commit()
async def save_bindings_bulk(self, node_type: str, bindings: Dict[str, Optional[str]]):
"""批量保存或更新绑定关系 (单次提交)"""
if not bindings:
return
rows = []
for local_id, remote_id in bindings.items():
rows.append((node_type, local_id, remote_id))
await self.conn.executemany(
"INSERT OR REPLACE INTO bindings (node_type, local_id, remote_id) VALUES (?, ?, ?)",
rows,
)
await self.conn.commit()
async def delete_binding(self, node_type: str, local_id: str):
"""删除绑定关系"""
await self.conn.execute(
"DELETE FROM bindings WHERE node_type = ? AND local_id = ?",
(node_type, local_id)
)
await self.conn.commit()
async def delete_bindings_bulk(self, node_type: str, local_ids: List[str]):
"""批量删除绑定关系 (单次提交)"""
if not local_ids:
return
rows = [(node_type, local_id) for local_id in local_ids]
await self.conn.executemany(
"DELETE FROM bindings WHERE node_type = ? AND local_id = ?",
rows,
)
await self.conn.commit()
async def load_bindings(self, node_type: str) -> List[Dict[str, Any]]:
"""回写该类型下的所有绑定关系到内存"""
async with self.conn.execute(
"SELECT local_id, remote_id FROM bindings WHERE node_type = ?",
(node_type,)
) as cursor:
rows = await cursor.fetchall()
return [
{"local_id": r["local_id"], "payload": {"remote_id": r["remote_id"]}}
for r in rows
]
async def load_bind_data_id_overrides(self, node_type: str) -> Dict[str, Optional[str]]:
"""基于 bindings 表计算 node_id -> bind_data_id 覆盖映射。
规则：
- local 节点的 bind_data_id 取其绑定 remote 节点的 data_id
- remote 节点的 bind_data_id 取其绑定 local 节点的 data_id
"""
sql = """
SELECT b.local_id AS src_node_id, rn.data_id AS bind_data_id
FROM bindings b
LEFT JOIN nodes rn
ON rn.node_id = b.remote_id
AND rn.node_type = b.node_type
WHERE b.node_type = ?
UNION ALL
SELECT b.remote_id AS src_node_id, ln.data_id AS bind_data_id
FROM bindings b
LEFT JOIN nodes ln
ON ln.node_id = b.local_id
AND ln.node_type = b.node_type
WHERE b.node_type = ?
AND b.remote_id IS NOT NULL
"""
async with self.conn.execute(sql, (node_type, node_type)) as cursor:
rows = await cursor.fetchall()
result: Dict[str, Optional[str]] = {}
for row in rows:
src_node_id = row["src_node_id"]
if not src_node_id:
continue
result[str(src_node_id)] = row["bind_data_id"]
return result
async def dump_to_runtime(self, filename: str):
"""将当前状态转储到 _runtime 文件夹 (使用 VACUUM INTO)"""
runtime_path = os.path.join("_runtime", filename)
if os.path.exists(runtime_path):
os.remove(runtime_path)
# VACUUM INTO 是备份 SQLite 的推荐方式 (SQLite 3.27+)
await self.conn.execute("VACUUM INTO ?", (runtime_path,))
async def close(self):
"""关闭数据库连接（包含异常处理）"""
if self.conn:
try:
await self.conn.close()
except Exception as e:
print(f"⚠️ Error closing persistence: {e}")
finally:
self.conn = None
sync_state_machine/common/persistence_backends/__init__.py
+3
View File
@@ -0,0 +1,3 @@
from .sqlite_backend import SQLitePersistenceBackend
__all__ = ["SQLitePersistenceBackend"]
sync_state_machine/common/persistence_backends/sqlite_backend.py
+464
View File
@@ -0,0 +1,464 @@
from __future__ import annotations
import aiosqlite
import json
import os
from typing import Any, Dict, List, Optional, TYPE_CHECKING
from ...config.persist_config import PersistConfig
from ..persistence import PersistenceBackend
if TYPE_CHECKING:
from ..sync_node import SyncNode
class SQLitePersistenceBackend(PersistenceBackend):
"""SQLite implementation of the persistence contract."""
backend_name = "sqlite"
def __init__(self, persist_config: PersistConfig | str | None = None, **_: Any) -> None:
if isinstance(persist_config, PersistConfig):
self.persist_config = persist_config
self.db_path = persist_config.db_path
elif isinstance(persist_config, str):
self.persist_config = None
self.db_path = persist_config
else:
self.persist_config = None
self.db_path = ":memory:"
self.conn: Optional[aiosqlite.Connection] = None
async def initialize(self) -> None:
if self.db_path != ":memory:":
dir_path = os.path.dirname(self.db_path)
if dir_path:
os.makedirs(dir_path, exist_ok=True)
self.conn = await aiosqlite.connect(self.db_path)
self.conn.row_factory = aiosqlite.Row
await self._init_db()
async def _init_db(self) -> None:
assert self.conn is not None
await self.conn.execute("""
CREATE TABLE IF NOT EXISTS nodes (
collection_id TEXT NOT NULL,
scope TEXT NOT NULL DEFAULT 'global',
node_id TEXT NOT NULL,
node_type TEXT NOT NULL,
data_id TEXT,
bind_data_id TEXT,
data TEXT,
origin_data TEXT,
action TEXT,
status TEXT,
binding_status TEXT,
error TEXT,
sync_log TEXT,
context TEXT,
PRIMARY KEY (collection_id, scope, node_id)
)
""")
await self._ensure_nodes_schema()
await self._ensure_bindings_schema()
await self.conn.commit()
async def _ensure_nodes_schema(self) -> None:
assert self.conn is not None
cursor = await self.conn.execute("PRAGMA table_info(nodes)")
rows = await cursor.fetchall()
columns = [r[1] for r in rows] if rows else []
if "scope" not in columns:
await self.conn.execute("""
CREATE TABLE IF NOT EXISTS nodes_new (
collection_id TEXT NOT NULL,
scope TEXT NOT NULL DEFAULT 'global',
node_id TEXT NOT NULL,
node_type TEXT NOT NULL,
data_id TEXT,
bind_data_id TEXT,
data TEXT,
origin_data TEXT,
action TEXT,
status TEXT,
binding_status TEXT,
error TEXT,
sync_log TEXT,
context TEXT,
PRIMARY KEY (collection_id, scope, node_id)
)
""")
await self.conn.execute("""
INSERT INTO nodes_new (
collection_id, scope, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
)
SELECT
collection_id, 'global', node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
FROM nodes
""")
await self.conn.execute("DROP TABLE nodes")
await self.conn.execute("ALTER TABLE nodes_new RENAME TO nodes")
cursor = await self.conn.execute("PRAGMA table_info(nodes)")
rows = await cursor.fetchall()
columns = [r[1] for r in rows] if rows else []
if "bind_data_id" not in columns:
await self.conn.execute("ALTER TABLE nodes ADD COLUMN bind_data_id TEXT")
await self.conn.execute(
"CREATE INDEX IF NOT EXISTS idx_nodes_collection_scope_type ON nodes (collection_id, scope, node_type)"
)
await self.conn.execute(
"CREATE INDEX IF NOT EXISTS idx_nodes_collection_scope_data_id ON nodes (collection_id, scope, data_id)"
)
async def _ensure_bindings_schema(self) -> None:
assert self.conn is not None
cursor = await self.conn.execute("PRAGMA table_info(bindings)")
rows = await cursor.fetchall()
columns = [r[1] for r in rows] if rows else []
if not columns:
await self.conn.execute("""
CREATE TABLE IF NOT EXISTS bindings (
scope TEXT NOT NULL DEFAULT 'global',
node_type TEXT NOT NULL,
local_id TEXT NOT NULL,
remote_id TEXT,
last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (scope, node_type, local_id)
)
""")
return
if "scope" not in columns or ("payload" in columns and "remote_id" not in columns):
await self.conn.execute("""
CREATE TABLE IF NOT EXISTS bindings_new (
scope TEXT NOT NULL DEFAULT 'global',
node_type TEXT NOT NULL,
local_id TEXT NOT NULL,
remote_id TEXT,
last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (scope, node_type, local_id)
)
""")
if "payload" in columns and "remote_id" not in columns:
async with self.conn.execute("SELECT node_type, local_id, payload FROM bindings") as cur:
old_rows = await cur.fetchall()
for row in old_rows:
try:
payload_dict = json.loads(row[2]) if row[2] else {}
except json.JSONDecodeError:
payload_dict = {}
remote_id = payload_dict.get("remote_id")
await self.conn.execute(
"INSERT OR REPLACE INTO bindings_new (scope, node_type, local_id, remote_id) VALUES (?, ?, ?, ?)",
("global", row[0], row[1], remote_id),
)
else:
async with self.conn.execute("SELECT node_type, local_id, remote_id FROM bindings") as cur:
old_rows = await cur.fetchall()
for row in old_rows:
await self.conn.execute(
"INSERT OR REPLACE INTO bindings_new (scope, node_type, local_id, remote_id) VALUES (?, ?, ?, ?)",
("global", row[0], row[1], row[2]),
)
await self.conn.execute("DROP TABLE bindings")
await self.conn.execute("ALTER TABLE bindings_new RENAME TO bindings")
await self.conn.execute(
"CREATE INDEX IF NOT EXISTS idx_bindings_scope_type_remote ON bindings (scope, node_type, remote_id)"
)
async def save_node(self, collection_id: str, scope: str, node: "SyncNode") -> None:
assert self.conn is not None
bind_data_id = None
if isinstance(node.context, dict):
value = node.context.get("bind_data_id")
if value is not None and value != "":
bind_data_id = str(value)
await self.conn.execute("""
INSERT OR REPLACE INTO nodes (
collection_id, scope, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (
collection_id,
scope,
node.node_id,
node.node_type,
node.data_id,
bind_data_id,
json.dumps(node.data) if node.data else None,
json.dumps(node.origin_data) if node.origin_data else None,
node.action.value,
node.status.value,
node.binding_status.value,
node.error,
node.sync_log,
json.dumps(node.context) if node.context else None,
))
await self.conn.commit()
async def save_nodes_bulk(self, collection_id: str, scope: str, nodes: List["SyncNode"]) -> None:
assert self.conn is not None
if not nodes:
return
rows = []
for node in nodes:
bind_data_id = None
if isinstance(node.context, dict):
value = node.context.get("bind_data_id")
if value is not None and value != "":
bind_data_id = str(value)
rows.append((
collection_id,
scope,
node.node_id,
node.node_type,
node.data_id,
bind_data_id,
json.dumps(node.data) if node.data else None,
json.dumps(node.origin_data) if node.origin_data else None,
node.action.value,
node.status.value,
node.binding_status.value,
node.error,
node.sync_log,
json.dumps(node.context) if node.context else None,
))
await self.conn.executemany("""
INSERT OR REPLACE INTO nodes (
collection_id, scope, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", rows)
await self.conn.commit()
async def delete_node(self, collection_id: str, scope: str, node_id: str) -> None:
assert self.conn is not None
await self.conn.execute(
"DELETE FROM nodes WHERE collection_id = ? AND scope = ? AND node_id = ?",
(collection_id, scope, node_id),
)
await self.conn.commit()
async def delete_nodes_bulk(self, collection_id: str, scope: str, node_ids: List[str]) -> None:
assert self.conn is not None
if not node_ids:
return
rows = [(collection_id, scope, node_id) for node_id in node_ids]
await self.conn.executemany(
"DELETE FROM nodes WHERE collection_id = ? AND scope = ? AND node_id = ?",
rows,
)
await self.conn.commit()
async def load_nodes(
self,
collection_id: str,
scope: str,
*,
exclude_node_types: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
assert self.conn is not None
sql = "SELECT * FROM nodes WHERE collection_id = ? AND scope = ?"
params: List[Any] = [collection_id, scope]
filtered_types = [node_type for node_type in (exclude_node_types or []) if node_type]
if filtered_types:
placeholders = ", ".join("?" for _ in filtered_types)
sql += f" AND node_type NOT IN ({placeholders})"
params.extend(filtered_types)
async with self.conn.execute(sql, tuple(params)) as cursor:
rows = await cursor.fetchall()
results = []
for row in rows:
item = {
"node_id": row["node_id"],
"node_type": row["node_type"],
"data_id": row["data_id"],
"bind_data_id": row["bind_data_id"],
"depend_ids": [],
"data": json.loads(row["data"]) if row["data"] else None,
"origin_data": json.loads(row["origin_data"]) if row["origin_data"] else None,
"action": row["action"],
"status": row["status"],
"binding_status": row["binding_status"],
"error": row["error"],
"sync_log": row["sync_log"],
"context": json.loads(row["context"]) if row["context"] else {},
}
if item["bind_data_id"] and isinstance(item["context"], dict):
item["context"]["bind_data_id"] = item["bind_data_id"]
results.append(item)
return results
async def save_binding(self, scope: str, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
assert self.conn is not None
remote_id = payload_dict.get("remote_id") if payload_dict else None
await self.conn.execute("""
INSERT OR REPLACE INTO bindings (scope, node_type, local_id, remote_id)
VALUES (?, ?, ?, ?)
""", (scope, node_type, local_id, remote_id))
await self.conn.commit()
async def save_bindings_bulk(self, scope: str, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
assert self.conn is not None
if not bindings:
return
rows = [(scope, node_type, local_id, remote_id) for local_id, remote_id in bindings.items()]
await self.conn.executemany(
"INSERT OR REPLACE INTO bindings (scope, node_type, local_id, remote_id) VALUES (?, ?, ?, ?)",
rows,
)
await self.conn.commit()
async def delete_binding(self, scope: str, node_type: str, local_id: str) -> None:
assert self.conn is not None
await self.conn.execute(
"DELETE FROM bindings WHERE scope = ? AND node_type = ? AND local_id = ?",
(scope, node_type, local_id),
)
await self.conn.commit()
async def delete_bindings_bulk(self, scope: str, node_type: str, local_ids: List[str]) -> None:
assert self.conn is not None
if not local_ids:
return
rows = [(scope, node_type, local_id) for local_id in local_ids]
await self.conn.executemany(
"DELETE FROM bindings WHERE scope = ? AND node_type = ? AND local_id = ?",
rows,
)
await self.conn.commit()
async def load_bindings(self, scope: str, node_type: str) -> List[Dict[str, Any]]:
assert self.conn is not None
async with self.conn.execute(
"SELECT local_id, remote_id FROM bindings WHERE scope = ? AND node_type = ?",
(scope, node_type),
) as cursor:
rows = await cursor.fetchall()
return [{"local_id": r["local_id"], "payload": {"remote_id": r["remote_id"]}} for r in rows]
async def load_bind_data_id_overrides(self, node_type: str, scope: str) -> Dict[str, Optional[str]]:
assert self.conn is not None
sql = """
SELECT b.local_id AS src_node_id, rn.data_id AS bind_data_id
FROM bindings b
LEFT JOIN nodes rn
ON rn.node_id = b.remote_id
AND rn.node_type = b.node_type
AND rn.scope = b.scope
WHERE b.node_type = ?
AND b.scope = ?
UNION ALL
SELECT b.remote_id AS src_node_id, ln.data_id AS bind_data_id
FROM bindings b
LEFT JOIN nodes ln
ON ln.node_id = b.local_id
AND ln.node_type = b.node_type
AND ln.scope = b.scope
WHERE b.node_type = ?
AND b.scope = ?
AND b.remote_id IS NOT NULL
"""
async with self.conn.execute(sql, (node_type, scope, node_type, scope)) as cursor:
rows = await cursor.fetchall()
result: Dict[str, Optional[str]] = {}
for row in rows:
src_node_id = row["src_node_id"]
if not src_node_id:
continue
result[str(src_node_id)] = row["bind_data_id"]
return result
async def delete_scope_node_types(self, *, scope: str, node_types: List[str]) -> None:
assert self.conn is not None
filtered_types = [node_type for node_type in node_types if node_type]
if not filtered_types:
return
placeholders = ", ".join("?" for _ in filtered_types)
await self.conn.execute(
f"DELETE FROM nodes WHERE scope = ? AND node_type IN ({placeholders})",
(scope, *filtered_types),
)
await self.conn.execute(
f"DELETE FROM bindings WHERE scope = ? AND node_type IN ({placeholders})",
(scope, *filtered_types),
)
await self.conn.commit()
async def copy_nodes_between_scopes(
self,
*,
collection_id: str,
source_scope: str,
target_scope: str,
node_types: List[str],
) -> None:
assert self.conn is not None
if source_scope == target_scope or not node_types:
return
placeholders = ", ".join("?" for _ in node_types)
sql = f"""
INSERT OR REPLACE INTO nodes (
collection_id, scope, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
)
SELECT
collection_id, ?, node_id, node_type, data_id, bind_data_id,
data, origin_data, action, status, binding_status, error, sync_log, context
FROM nodes
WHERE collection_id = ?
AND scope = ?
AND node_type IN ({placeholders})
"""
await self.conn.execute(sql, (target_scope, collection_id, source_scope, *node_types))
await self.conn.commit()
async def copy_bindings_between_scopes(
self,
*,
source_scope: str,
target_scope: str,
node_types: List[str],
) -> None:
assert self.conn is not None
if source_scope == target_scope or not node_types:
return
placeholders = ", ".join("?" for _ in node_types)
sql = f"""
INSERT OR REPLACE INTO bindings (scope, node_type, local_id, remote_id)
SELECT ?, node_type, local_id, remote_id
FROM bindings
WHERE scope = ?
AND node_type IN ({placeholders})
"""
await self.conn.execute(sql, (target_scope, source_scope, *node_types))
await self.conn.commit()
async def dump_to_runtime(self, filename: str) -> None:
assert self.conn is not None
runtime_path = os.path.join("_runtime", filename)
if os.path.exists(runtime_path):
os.remove(runtime_path)
await self.conn.execute("VACUUM INTO ?", (runtime_path,))
async def close(self) -> None:
if self.conn:
try:
await self.conn.close()
except Exception as exc:
print(f"⚠️ Error closing persistence: {exc}")
finally:
self.conn = None
sync_state_machine/common/registry.py
+2 -13
View File
@@ -17,7 +17,7 @@ Domain 注册中心
node_type="contract", node_type="contract",
schema=ContractResponse, schema=ContractResponse,
node_class=ContractSyncNode, node_class=ContractSyncNode,
jsonl_handler_class=ContractJsonlHandler jsonl_handler_class=ContractJsonlHandler,
) )
# Collection 查询 SyncNode 类 # Collection 查询 SyncNode 类
@@ -47,7 +47,6 @@ class DomainRegistration:
strategy_class: Optional[Type[Any]] = None # 可选，部分 domain 可能只读 strategy_class: Optional[Type[Any]] = None # 可选，部分 domain 可能只读
jsonl_handler_class: Optional[Type[Any]] = None jsonl_handler_class: Optional[Type[Any]] = None
api_handler_class: Optional[Type[Any]] = None api_handler_class: Optional[Type[Any]] = None
db_handler_class: Optional[Type[Any]] = None
handler_classes: Dict[str, Type[Any]] = field(default_factory=dict) handler_classes: Dict[str, Type[Any]] = field(default_factory=dict)
metadata: Dict[str, Any] = field(default_factory=dict) # 额外元数据 metadata: Dict[str, Any] = field(default_factory=dict) # 额外元数据
@@ -69,7 +68,6 @@ class DomainRegistry:
_BUILTIN_HANDLER_ATTRS: Dict[str, str] = { _BUILTIN_HANDLER_ATTRS: Dict[str, str] = {
"jsonl": "jsonl_handler_class", "jsonl": "jsonl_handler_class",
"api": "api_handler_class", "api": "api_handler_class",
"db": "db_handler_class",
} }
@classmethod @classmethod
@@ -81,7 +79,6 @@ class DomainRegistry:
strategy_class: Optional[Type[Any]] = None, strategy_class: Optional[Type[Any]] = None,
jsonl_handler_class: Optional[Type[Any]] = None, jsonl_handler_class: Optional[Type[Any]] = None,
api_handler_class: Optional[Type[Any]] = None, api_handler_class: Optional[Type[Any]] = None,
db_handler_class: Optional[Type[Any]] = None,
**metadata **metadata
) -> None: ) -> None:
""" """
@@ -94,7 +91,6 @@ class DomainRegistry:
strategy_class: Strategy 子类（可选） strategy_class: Strategy 子类（可选）
jsonl_handler_class: JSONL Handler 类（可选） jsonl_handler_class: JSONL Handler 类（可选）
api_handler_class: API Handler 类（可选） api_handler_class: API Handler 类（可选）
db_handler_class: DB Handler 类（可选）
**metadata: 额外的元数据 **metadata: 额外的元数据
Raises: Raises:
@@ -110,7 +106,6 @@ class DomainRegistry:
strategy_class=strategy_class, strategy_class=strategy_class,
jsonl_handler_class=jsonl_handler_class, jsonl_handler_class=jsonl_handler_class,
api_handler_class=api_handler_class, api_handler_class=api_handler_class,
db_handler_class=db_handler_class,
metadata=metadata metadata=metadata
) )
for datasource_type, attr_name in cls._BUILTIN_HANDLER_ATTRS.items(): for datasource_type, attr_name in cls._BUILTIN_HANDLER_ATTRS.items():
@@ -187,10 +182,6 @@ class DomainRegistry:
setattr(reg, attr_name, handler_class) setattr(reg, attr_name, handler_class)
@classmethod @classmethod
def register_db_handler(cls, node_type: str, db_handler_class: Type[Any]) -> None:
"""为已注册 domain 补充 DB Handler。"""
cls.register_handler(node_type, "db", db_handler_class)
@classmethod @classmethod
def get_schema(cls, node_type: str) -> Type[BaseModel]: def get_schema(cls, node_type: str) -> Type[BaseModel]:
"""获取 Schema 类""" """获取 Schema 类"""
@@ -245,7 +236,6 @@ class DomainRegistry:
print(f" Strategy Class: {reg.strategy_class.__name__ if reg.strategy_class else '❌ None'}") print(f" Strategy Class: {reg.strategy_class.__name__ if reg.strategy_class else '❌ None'}")
print(f" JSONL Handler: {reg.jsonl_handler_class.__name__ if reg.jsonl_handler_class else '❌ None'}") print(f" JSONL Handler: {reg.jsonl_handler_class.__name__ if reg.jsonl_handler_class else '❌ None'}")
print(f" API Handler: {reg.api_handler_class.__name__ if reg.api_handler_class else '❌ None'}") print(f" API Handler: {reg.api_handler_class.__name__ if reg.api_handler_class else '❌ None'}")
print(f" DB Handler: {reg.db_handler_class.__name__ if reg.db_handler_class else '❌ None'}")
extra_handler_types = sorted( extra_handler_types = sorted(
handler_type for handler_type in reg.handler_classes if handler_type not in cls._BUILTIN_HANDLER_ATTRS handler_type for handler_type in reg.handler_classes if handler_type not in cls._BUILTIN_HANDLER_ATTRS
) )
@@ -272,7 +262,7 @@ class DomainRegistry:
"has_strategy": True, "has_strategy": True,
"has_jsonl_handler": True, "has_jsonl_handler": True,
"has_api_handler": True, "has_api_handler": True,
"has_db_handler": False "handler_types": []
}, },
... ...
} }
@@ -284,7 +274,6 @@ class DomainRegistry:
"has_strategy": reg.strategy_class is not None, "has_strategy": reg.strategy_class is not None,
"has_jsonl_handler": reg.jsonl_handler_class is not None, "has_jsonl_handler": reg.jsonl_handler_class is not None,
"has_api_handler": reg.api_handler_class is not None, "has_api_handler": reg.api_handler_class is not None,
"has_db_handler": reg.db_handler_class is not None,
"handler_types": sorted(reg.handler_classes.keys()), "handler_types": sorted(reg.handler_classes.keys()),
} }
return summary return summary
sync_state_machine/common/sqlite_backend.py
+3
View File
@@ -0,0 +1,3 @@
from .persistence_backends.sqlite_backend import SQLitePersistenceBackend
__all__ = ["SQLitePersistenceBackend"]
sync_state_machine/config/__init__.py
+10 -2
View File
@@ -24,12 +24,17 @@ from .strategy_config import (
ConfigPresets, ConfigPresets,
) )
from .pipeline_config import PipelineRunConfig, DEFAULT_NODE_TYPES from .pipeline_config import PipelineRunConfig, DEFAULT_NODE_TYPES
from .multi_project_config import (
MultiProjectItemConfig,
MultiProjectRunConfig,
build_multi_project_config_from_file,
is_multi_project_payload,
)
from .builder import ( from .builder import (
build_default_config, build_default_config,
build_config, build_config,
build_config_from_file, build_config_from_file,
apply_config_overrides, apply_config_overrides,
apply_env_overrides,
config_snapshot, config_snapshot,
) )
@@ -59,10 +64,13 @@ __all__ = [
"StrategyRuntimeConfig", "StrategyRuntimeConfig",
"ConfigPresets", "ConfigPresets",
"DEFAULT_NODE_TYPES", "DEFAULT_NODE_TYPES",
"MultiProjectItemConfig",
"MultiProjectRunConfig",
"build_multi_project_config_from_file",
"is_multi_project_payload",
"build_default_config", "build_default_config",
"build_config", "build_config",
"build_config_from_file", "build_config_from_file",
"apply_config_overrides", "apply_config_overrides",
"apply_env_overrides",
"config_snapshot", "config_snapshot",
] ]
sync_state_machine/config/builder.py
+13 -17
View File
@@ -63,13 +63,22 @@ def _apply_strategy_overrides(
runtime_cfg = strategy_map[node_type] runtime_cfg = strategy_map[node_type]
override = dict(raw_override or {}) override = dict(raw_override or {})
deprecated_keys = [key for key in ("force_sync", "load_mode") if key in override]
if deprecated_keys:
joined = ", ".join(deprecated_keys)
raise ValueError(
f"Strategy override for {node_type!r} contains unsupported legacy keys: {joined}. "
"Use skip_sync under strategies.<node_type> or datasource.handler_configs instead."
)
preset = override.pop("preset", None) preset = override.pop("preset", None)
if preset: if preset:
runtime_cfg.config = ConfigPresets.get_preset(str(preset)) runtime_cfg.config = ConfigPresets.get_preset(str(preset))
override.pop("skip_sync", None) skip_sync = override.pop("skip_sync", None)
override.pop("force_sync", None)
override.pop("load_mode", None) if skip_sync is not None:
runtime_cfg.skip_sync = bool(skip_sync)
if "config" in override and isinstance(override["config"], dict): if "config" in override and isinstance(override["config"], dict):
runtime_cfg.config = runtime_cfg.config.model_validate( runtime_cfg.config = runtime_cfg.config.model_validate(
@@ -107,7 +116,6 @@ def build_default_config(project_root: Path) -> PipelineRunConfig:
"read_only": False, "read_only": False,
"handler_configs": {}, "handler_configs": {},
}, },
"target_project_ids": [],
"persist": { "persist": {
"db_path": str(project_root / "test_results" / "sync_state_machine" / "pipeline_default.db"), "db_path": str(project_root / "test_results" / "sync_state_machine" / "pipeline_default.db"),
"enable": True, "enable": True,
@@ -129,29 +137,17 @@ def build_default_config(project_root: Path) -> PipelineRunConfig:
def apply_config_overrides(config: PipelineRunConfig, overrides: Dict[str, Any]) -> PipelineRunConfig: def apply_config_overrides(config: PipelineRunConfig, overrides: Dict[str, Any]) -> PipelineRunConfig:
merged = _deep_merge(config.model_dump(), overrides) merged = _deep_merge(config.model_dump(), overrides)
raw_strategies = merged.pop("strategies", None) raw_strategies = merged.pop("strategies", None)
legacy_strategy_overrides = merged.pop("strategy_overrides", None)
resolved = PipelineRunConfig.model_validate(merged) resolved = PipelineRunConfig.model_validate(merged)
if raw_strategies is not None: if raw_strategies is not None:
resolved.strategies = _apply_strategy_overrides(resolved.node_types, raw_strategies) resolved.strategies = _apply_strategy_overrides(resolved.node_types, raw_strategies)
else: else:
resolved.strategies = _apply_strategy_overrides(resolved.node_types, legacy_strategy_overrides or {}) resolved.strategies = _apply_strategy_overrides(resolved.node_types, {})
return resolved return resolved
def apply_env_overrides(
config: PipelineRunConfig,
project_root: Path,
*,
dotenv_file: Optional[Union[str, Path]] = None,
) -> PipelineRunConfig:
raise NotImplementedError(
"Environment-based overrides are removed. Use config files via overrides_file instead."
)
def build_config( def build_config(
*, *,
project_root: Path, project_root: Path,
sync_state_machine/config/datasource_config.py
-2
View File
@@ -11,7 +11,6 @@ class BaseDataSourceConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True) model_config = ConfigDict(extra="forbid", validate_assignment=True)
type: str type: str
target_project_ids: List[str] = Field(default_factory=list)
handler_configs: Dict[str, Dict[str, Any]] = Field(default_factory=dict) handler_configs: Dict[str, Dict[str, Any]] = Field(default_factory=dict)
@@ -33,7 +32,6 @@ class ApiDataSourceConfig(BaseDataSourceConfig):
poll_interval: float = 0.5 poll_interval: float = 0.5
api_rate_limit_max_requests: int = 10 api_rate_limit_max_requests: int = 10
api_rate_limit_window_seconds: float = 10.0 api_rate_limit_window_seconds: float = 10.0
target_project_ids: List[str] = Field(default_factory=list)
DataSourceConfig = Annotated[ DataSourceConfig = Annotated[
sync_state_machine/config/multi_project_config.py
+39
View File
@@ -0,0 +1,39 @@
from __future__ import annotations
from pathlib import Path
from typing import Any, Dict, List, Optional
from pydantic import BaseModel, ConfigDict
from .run_presets import load_overrides_from_file
class MultiProjectItemConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
local_project_id: str
remote_project_id: str
config_path: Optional[str] = None
@property
def scope(self) -> str:
return f"project_id:{self.local_project_id}"
class MultiProjectRunConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
global_config: str
default_project_config: str
project_bootstrap_node_types: List[str] = []
projects: List[MultiProjectItemConfig]
def is_multi_project_payload(payload: Dict[str, Any]) -> bool:
return all(key in payload for key in ("global_config", "default_project_config", "projects"))
def build_multi_project_config_from_file(*, project_root: Path, file_path: str | Path) -> MultiProjectRunConfig:
path = Path(file_path)
overrides = load_overrides_from_file(path, project_root)
return MultiProjectRunConfig.model_validate(overrides)
sync_state_machine/config/persist_config.py
+2 -2
View File
@@ -1,4 +1,4 @@
from typing import Any, Dict, Literal from typing import Any, Dict
from pydantic import BaseModel, ConfigDict, Field from pydantic import BaseModel, ConfigDict, Field
@@ -6,7 +6,7 @@ from pydantic import BaseModel, ConfigDict, Field
class PersistConfig(BaseModel): class PersistConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True) model_config = ConfigDict(extra="forbid", validate_assignment=True)
backend: Literal["sqlite", "sqlalchemy", "custom"] = "sqlite" backend: str = "sqlite"
db_path: str db_path: str
backend_options: Dict[str, Any] = Field(default_factory=dict) backend_options: Dict[str, Any] = Field(default_factory=dict)
enable: bool = True enable: bool = True
sync_state_machine/config/runtime_config.py
+10 -2
View File
@@ -14,12 +14,17 @@ from .strategy_config import (
ConfigPresets, ConfigPresets,
) )
from .pipeline_config import PipelineRunConfig, DEFAULT_NODE_TYPES from .pipeline_config import PipelineRunConfig, DEFAULT_NODE_TYPES
from .multi_project_config import (
MultiProjectItemConfig,
MultiProjectRunConfig,
build_multi_project_config_from_file,
is_multi_project_payload,
)
from .builder import ( from .builder import (
build_default_config, build_default_config,
build_config, build_config,
build_config_from_file, build_config_from_file,
apply_config_overrides, apply_config_overrides,
apply_env_overrides,
config_snapshot, config_snapshot,
) )
@@ -36,10 +41,13 @@ __all__ = [
"ConfigPresets", "ConfigPresets",
"PipelineRunConfig", "PipelineRunConfig",
"DEFAULT_NODE_TYPES", "DEFAULT_NODE_TYPES",
"MultiProjectItemConfig",
"MultiProjectRunConfig",
"build_multi_project_config_from_file",
"is_multi_project_payload",
"build_default_config", "build_default_config",
"build_config", "build_config",
"build_config_from_file", "build_config_from_file",
"apply_config_overrides", "apply_config_overrides",
"apply_env_overrides",
"config_snapshot", "config_snapshot",
] ]
sync_state_machine/config/strategy_config.py
+7 -4
View File
@@ -47,10 +47,12 @@ class StrategyConfig(BaseModel):
field_direction_key: Optional[str] = None field_direction_key: Optional[str] = None
field_direction_overrides: Dict[str, UpdateDirection] = Field(default_factory=dict) field_direction_overrides: Dict[str, UpdateDirection] = Field(default_factory=dict)
# post-check 一致性比较时，忽略列表型字段内各元素的特定子键。 # post-check 一致性比较时忽略的顶层字段。
# 格式: {字段名: [子键, ...]}，如 {"plan_node": ["is_audit"]}。 # 只影响最终一致性评估，不影响 create/update 的差异检测。
# 用于排除后端硬写死、永远与本地不一致的字段，避免误报。 post_check_ignore_fields: List[str] = Field(default_factory=list)
post_check_ignore_list_item_fields: Dict[str, List[str]] = Field(default_factory=dict)
# schema diff / validate 时忽略的顶层字段。
compare_ignore_fields: List[str] = Field(default_factory=list)
class StrategyRuntimeConfig(BaseModel): class StrategyRuntimeConfig(BaseModel):
@@ -59,6 +61,7 @@ class StrategyRuntimeConfig(BaseModel):
model_config = ConfigDict(validate_assignment=True, extra="forbid") model_config = ConfigDict(validate_assignment=True, extra="forbid")
node_type: str node_type: str
skip_sync: bool = False
config: StrategyConfig = Field(default_factory=StrategyConfig) config: StrategyConfig = Field(default_factory=StrategyConfig)
sync_state_machine/datasource/__init__.py
-7
View File
@@ -25,9 +25,6 @@ from .task_result import TaskResult, TaskStatus
# API DataSource # API DataSource
from .api import ApiClient, ApiDataSource, BaseApiHandler, register as register_api_datasource from .api import ApiClient, ApiDataSource, BaseApiHandler, register as register_api_datasource
# DB base helpers
from .db import DbDataSource, BaseDbHandler
# JSONL DataSource # JSONL DataSource
from .jsonl import JsonlDataSource, BaseJsonlHandler, register as register_jsonl_datasource from .jsonl import JsonlDataSource, BaseJsonlHandler, register as register_jsonl_datasource
@@ -65,10 +62,6 @@ __all__ = [
"ApiDataSource", "ApiDataSource",
"BaseApiHandler", "BaseApiHandler",
# DB base helpers
"DbDataSource",
"BaseDbHandler",
# JSONL DataSource # JSONL DataSource
"JsonlDataSource", "JsonlDataSource",
"BaseJsonlHandler", "BaseJsonlHandler",
sync_state_machine/datasource/api/datasource.py
+8 -9
View File
@@ -12,7 +12,7 @@ import json
from typing import Optional from typing import Optional
from pathlib import Path from pathlib import Path
from ..datasource import BaseDataSource from ..datasource import BaseDataSource
from ..handler import NodeHandler, ensure_handler_compat from ..handler import NodeHandler
from ..task_result import TaskResult from ..task_result import TaskResult
from .client import ApiClient from .client import ApiClient
@@ -65,23 +65,22 @@ class ApiDataSource(BaseDataSource):
"""关闭数据源（关闭 HTTP 客户端）""" """关闭数据源（关闭 HTTP 客户端）"""
await self.client.close() await self.client.close()
def register_handler(self, handler: NodeHandler) -> None: def add_handler(self, handler: NodeHandler) -> None:
""" """
注册业务 Handler 添加业务 Handler
Args: Args:
handler: API Handler 实例 handler: API Handler 实例
""" """
compatible_handler = ensure_handler_compat(handler) handler.set_api_client(self.client)
compatible_handler.set_api_client(self.client) handler.set_poll_mode(self.poll_mode)
compatible_handler.set_poll_mode(self.poll_mode)
# 调用基类注册方法 # 调用基类添加方法
super().register_handler(compatible_handler) super().add_handler(handler)
# 如果 collection 已设置，确保新 handler 拿到引用 # 如果 collection 已设置，确保新 handler 拿到引用
if self._collection is not None: if self._collection is not None:
compatible_handler.set_collection(self._collection) # type: ignore[arg-type] handler.set_collection(self._collection) # type: ignore[arg-type]
@staticmethod @staticmethod
def _fmt_trace_value(value, max_len: int = 600) -> str: def _fmt_trace_value(value, max_len: int = 600) -> str:
sync_state_machine/datasource/api/handler.py
+8 -11
View File
@@ -20,7 +20,7 @@ from typing import ClassVar, Dict, List, Any, Optional, TYPE_CHECKING, TypeVar
from pydantic import BaseModel from pydantic import BaseModel
from ...common.type_safety import get_pydantic_model_fields from ...common.type_safety import get_pydantic_model_fields
from ..handler import NodeHandler from ..handler import BaseNodeHandler
from ..task_result import TaskResult from ..task_result import TaskResult
from ..handler import ValidationResult # Import ValidationResult from ..handler import ValidationResult # Import ValidationResult
@@ -32,7 +32,7 @@ if TYPE_CHECKING:
T = TypeVar("T", bound=BaseModel) T = TypeVar("T", bound=BaseModel)
class BaseApiHandler(NodeHandler[T]): class BaseApiHandler(BaseNodeHandler[T]):
"""API Handler 基类""" """API Handler 基类"""
# 子类需要设置这些属性 # 子类需要设置这些属性
@@ -40,16 +40,20 @@ class BaseApiHandler(NodeHandler[T]):
_node_class: type['SyncNode'] _node_class: type['SyncNode']
_schema: Any _schema: Any
def __init__(self): def __init__(self, **handler_config: Any):
# 子类应该在调用 super().__init__() 之前或之后设置 _node_type, _node_class, _schema # 子类应该在调用 super().__init__() 之前或之后设置 _node_type, _node_class, _schema
super().__init__()
self.api_client: 'ApiClient' = None # type: ignore # 由 DataSource 注入 self.api_client: 'ApiClient' = None # type: ignore # 由 DataSource 注入
self.poll_mode: str = "async" # async | sync self.poll_mode: str = "async" # async | sync
self._collection: 'DataCollection' = None # type: ignore # 由 DataSource 注入 self._collection: 'DataCollection' = None # type: ignore # 由 DataSource 注入
self.handler_config: Dict[str, Any] = {} self.handler_config = dict(handler_config)
# 子类在 __init__ 中设置此列表以声明 create/update 中涉及的 Pydantic schema 类； # 子类在 __init__ 中设置此列表以声明 create/update 中涉及的 Pydantic schema 类；
# post-check 将只比较这些 schema 覆盖的字段，过滤后端只读的派生字段。 # post-check 将只比较这些 schema 覆盖的字段，过滤后端只读的派生字段。
self.update_schemas: List[type] = [] self.update_schemas: List[type] = []
def set_collection(self, collection: 'DataCollection') -> None:
self._collection = collection
def get_update_fields(self, *, exclude: frozenset = frozenset({'id'})) -> List[str]: def get_update_fields(self, *, exclude: frozenset = frozenset({'id'})) -> List[str]:
"""从 update_schemas 中提取所有可写字段名，排除标识键（默认排除 id）。""" """从 update_schemas 中提取所有可写字段名，排除标识键（默认排除 id）。"""
fields: set = set() fields: set = set()
@@ -66,13 +70,6 @@ class BaseApiHandler(NodeHandler[T]):
"""设置 Collection（由 DataSource 调用）""" """设置 Collection（由 DataSource 调用）"""
self._collection = collection self._collection = collection
def set_handler_config(self, config: Dict[str, Any]) -> None:
self.handler_config = dict(config)
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""默认忽略项目过滤列表，具体子类按需覆盖。"""
return
@property @property
def node_type(self) -> str: def node_type(self) -> str:
"""返回节点类型""" """返回节点类型"""
sync_state_machine/datasource/datasource.py
+7 -28
View File
@@ -31,7 +31,6 @@ if TYPE_CHECKING:
from ..common.types import SyncStatus from ..common.types import SyncStatus
from ..common.type_safety import get_pydantic_model_fields from ..common.type_safety import get_pydantic_model_fields
from .handler import ensure_handler_compat
from .task_result import TaskResult from .task_result import TaskResult
from ..engine import ( from ..engine import (
StateMachineConfig, StateMachineConfig,
@@ -58,9 +57,9 @@ class BaseDataSource(ABC):
# 1. 初始化 DataSource # 1. 初始化 DataSource
datasource = MyDataSource() datasource = MyDataSource()
# 2. 注册 Handler # 2. 添加 Handler
datasource.register_handler(project_handler) datasource.add_handler(project_handler)
datasource.register_handler(contract_handler) datasource.add_handler(contract_handler)
# 3. 设置 Collection # 3. 设置 Collection
datasource.set_collection(collection) datasource.set_collection(collection)
@@ -75,7 +74,6 @@ class BaseDataSource(ABC):
def __init__(self, poll_max_retries: int = 1, poll_interval: float = 0.5): def __init__(self, poll_max_retries: int = 1, poll_interval: float = 0.5):
self._handlers: Dict[str, "NodeHandler"] = {} self._handlers: Dict[str, "NodeHandler"] = {}
self._collection: Optional["DataCollection"] = None self._collection: Optional["DataCollection"] = None
self._target_project_ids: List[str] = []
# 按节点类型统计信息 # 按节点类型统计信息
# {"company": {"loaded": 10, "synced": 10, "success": 10, "failed": 0}, ...} # {"company": {"loaded": 10, "synced": 10, "success": 10, "failed": 0}, ...}
@@ -88,13 +86,6 @@ class BaseDataSource(ABC):
self._poll_interval = max(0, poll_interval) self._poll_interval = max(0, poll_interval)
self._sm_runtime: Optional[StateMachineRuntime] = None self._sm_runtime: Optional[StateMachineRuntime] = None
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
self._target_project_ids = [str(item) for item in target_project_ids if str(item)]
@property
def target_project_ids(self) -> List[str]:
return list(self._target_project_ids)
def _ensure_sm_runtime(self) -> StateMachineRuntime: def _ensure_sm_runtime(self) -> StateMachineRuntime:
if self._sm_runtime is None: if self._sm_runtime is None:
cfg_path = Path(__file__).resolve().parents[1] / "config" / "node_state_machine.yaml" cfg_path = Path(__file__).resolve().parents[1] / "config" / "node_state_machine.yaml"
@@ -369,29 +360,17 @@ class BaseDataSource(ABC):
# ========== Handler 管理 ========== # ========== Handler 管理 ==========
def register_handler(self, handler: "NodeHandler") -> None: def add_handler(self, handler: "NodeHandler") -> None:
""" """
注册 Handler 添加 Handler 到当前 datasource
Args: Args:
handler: 节点处理器实例 handler: 节点处理器实例
示例： 示例：
datasource.register_handler(ContractNodeHandler(api_client)) datasource.add_handler(ContractNodeHandler(api_client))
""" """
compatible_handler = ensure_handler_compat(handler) self._handlers[handler.node_type] = handler
self._handlers[compatible_handler.node_type] = compatible_handler
self._apply_target_project_ids_to_handler(compatible_handler)
def set_target_project_ids(self, target_project_ids: Optional[List[str]]) -> None:
"""设置项目过滤列表，并在注册阶段下发给所有 handler。"""
normalized = [str(pid) for pid in (target_project_ids or []) if str(pid)]
self._target_project_ids = normalized
for handler in self._handlers.values():
self._apply_target_project_ids_to_handler(handler)
def _apply_target_project_ids_to_handler(self, handler: "NodeHandler") -> None:
handler.set_target_project_ids(self._target_project_ids)
def get_handler(self, node_type: str) -> "NodeHandler": def get_handler(self, node_type: str) -> "NodeHandler":
"""获取 Handler""" """获取 Handler"""
sync_state_machine/datasource/db/__init__.py
-8
View File
@@ -1,8 +0,0 @@
from __future__ import annotations
"""DB DataSource 基类模块。"""
from .datasource import DbDataSource
from .handler import BaseDbHandler
__all__ = ["DbDataSource", "BaseDbHandler"]
sync_state_machine/datasource/db/datasource.py
-22
View File
@@ -1,22 +0,0 @@
"""DbDataSource - 数据库数据源协调层。"""
from __future__ import annotations
from ..datasource import BaseDataSource
class DbDataSource(BaseDataSource):
"""基于数据库 Handler 的数据源实现。"""
async def close(self) -> None:
return None
def _on_node_loaded(self, handler, node) -> None:
node.append_log(
f"DB链路[load] node_type={node.node_type} data_id={node.data_id or '<empty>'}"
)
def _on_in_progress_result(self, handler, node, result) -> None:
node.append_log(
f"DB链路[submit] action={node.action.value} data_id={node.data_id or '<empty>'}"
)
sync_state_machine/datasource/db/handler.py
-68
View File
@@ -1,68 +0,0 @@
"""BaseDbHandler - DB Handler 基类。"""
from __future__ import annotations
from typing import Any, Dict, List, Optional, Set, TypeVar, TYPE_CHECKING
from pydantic import BaseModel
from ...common.type_safety import get_pydantic_model_fields
from ..handler import BaseNodeHandler
if TYPE_CHECKING:
from ...common.sync_node import SyncNode
from .datasource import DbDataSource
T = TypeVar("T", bound=BaseModel)
class BaseDbHandler(BaseNodeHandler[T]):
"""数据库 Handler 基类。"""
def __init__(
self,
datasource: "DbDataSource",
node_type: str,
node_class: type["SyncNode[T]"],
schema: type[T],
):
super().__init__(node_type, node_class, schema)
self.datasource = datasource
self.update_schemas: List[type] = []
self._target_project_ids: Optional[Set[str]] = None
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
if target_project_ids:
self._target_project_ids = {str(item) for item in target_project_ids if str(item)}
else:
self._target_project_ids = None
def get_update_fields(self, *, exclude: frozenset = frozenset({"id"})) -> List[str]:
fields: set[str] = set()
for schema in self.update_schemas:
for field_name in get_pydantic_model_fields(schema):
if field_name not in exclude:
fields.add(field_name)
return sorted(fields)
def _create_basic_node(self, data: Dict[str, Any], depend_ids: Optional[List[str]] = None) -> "SyncNode[T]":
data_id = self.extract_id(data)
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() before creating nodes.")
if data_id:
existing_node = self._collection.get_by_data_id(self.node_type, data_id)
if existing_node is not None:
existing_node.depend_ids = list(depend_ids or [])
existing_node.set_data(data.copy())
existing_node.set_origin_data(data.copy())
return existing_node
node_id = self._collection.generate_node_id()
return self.node_class(
node_id=node_id,
data_id=data_id,
data=data.copy(),
depend_ids=depend_ids or [],
origin_data=data.copy(),
)
sync_state_machine/datasource/handler.py
+28 -93
View File
@@ -74,10 +74,6 @@ class NodeHandler(Protocol[T]):
"""注入 handler 级别配置。默认可忽略。""" """注入 handler 级别配置。默认可忽略。"""
... ...
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""注入项目过滤列表。默认可忽略。"""
...
def set_api_client(self, client: "ApiClient") -> None: def set_api_client(self, client: "ApiClient") -> None:
"""注入 API 客户端。默认可忽略。""" """注入 API 客户端。默认可忽略。"""
... ...
@@ -341,11 +337,11 @@ class BaseNodeHandler(ABC, Generic[T]):
def __init__( def __init__(
self, self,
node_type: str, node_type: str | None = None,
node_class: Type["SyncNode[T]"], node_class: Type["SyncNode[T]"] | None = None,
schema: Type[T] schema: Type[T] | None = None,
): ):
self._node_type = node_type self._node_type = node_type or ""
self._node_class = node_class self._node_class = node_class
self._schema = schema self._schema = schema
self._collection: Optional["DataCollection"] = None self._collection: Optional["DataCollection"] = None
@@ -360,14 +356,34 @@ class BaseNodeHandler(ABC, Generic[T]):
""" """
self._collection = collection self._collection = collection
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""默认忽略项目过滤能力。"""
return
def set_handler_config(self, config: Dict[str, Any]) -> None: def set_handler_config(self, config: Dict[str, Any]) -> None:
"""默认保存 handler 级别配置，供子类按需读取。""" """默认保存 handler 级别配置，供子类按需读取。"""
self.handler_config = dict(config) self.handler_config = dict(config)
def get_data_id_filter(self) -> List[str]:
if "data_id_filter" in self.handler_config:
value = self.handler_config.get("data_id_filter", [])
if isinstance(value, str):
return [value] if value else []
if isinstance(value, list):
return [str(item) for item in value if str(item)]
return []
if self.node_type != "project" and self._collection is not None:
return [node.data_id for node in self._collection.filter(node_type="project") if node.data_id]
return []
def should_skip_by_data_id_filter(self, item: Dict[str, Any], item_id: str) -> bool:
data_id_filter = set(self.get_data_id_filter())
if not data_id_filter:
return False
if self.node_type == "project":
return item_id not in data_id_filter
project_id = item.get("project_id")
if project_id:
return str(project_id) not in data_id_filter
return False
def set_api_client(self, client: "ApiClient") -> None: def set_api_client(self, client: "ApiClient") -> None:
"""默认忽略 API 客户端注入。""" """默认忽略 API 客户端注入。"""
return return
@@ -479,87 +495,6 @@ class BaseNodeHandler(ABC, Generic[T]):
return self.create_node(data) return self.create_node(data)
class CompatibleNodeHandler(Generic[T]):
"""兼容旧式 Handler，实现扩展后的可选接口并委托给原对象。"""
def __init__(self, handler: NodeHandler[T]):
self._handler = handler
@property
def node_type(self) -> str:
return self._handler.node_type
@property
def node_class(self) -> Type["SyncNode[T]"]:
return self._handler.node_class
@property
def schema(self) -> Type[T]:
return self._handler.schema
def set_collection(self, collection: "DataCollection") -> None:
self._handler.set_collection(collection)
def set_handler_config(self, config: Dict[str, Any]) -> None:
method = getattr(self._handler, "set_handler_config", None)
if callable(method):
method(config)
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
method = getattr(self._handler, "set_target_project_ids", None)
if callable(method):
method(target_project_ids)
def set_api_client(self, client: "ApiClient") -> None:
method = getattr(self._handler, "set_api_client", None)
if callable(method):
method(client)
def set_poll_mode(self, mode: str) -> None:
method = getattr(self._handler, "set_poll_mode", None)
if callable(method):
method(mode)
def get_update_fields(self, *, exclude: frozenset[str] = frozenset({"id"})) -> List[str]:
method = getattr(self._handler, "get_update_fields", None)
if callable(method):
return method(exclude=exclude)
return []
async def load(self) -> List["SyncNode"]:
return await self._handler.load()
async def create_all(self, nodes: List["SyncNode"]) -> List[TaskResult]:
return await self._handler.create_all(nodes)
async def update_all(self, nodes: List["SyncNode"]) -> List[TaskResult]:
return await self._handler.update_all(nodes)
async def delete_all(self, nodes: List["SyncNode"]) -> List[TaskResult]:
return await self._handler.delete_all(nodes)
async def poll_tasks(self, task_ids: List[str]) -> Dict[str, TaskResult]:
return await self._handler.poll_tasks(task_ids)
async def validate(self, data: Dict[str, Any]) -> ValidationResult:
return await self._handler.validate(data)
def extract_id(self, data: Dict[str, Any]) -> str:
return self._handler.extract_id(data)
def _create_node(self, data: Dict[str, Any]) -> "SyncNode[T]":
return self._handler._create_node(data)
def __getattr__(self, item: str) -> Any:
return getattr(self._handler, item)
def ensure_handler_compat(handler: NodeHandler[T]) -> NodeHandler[T]:
if isinstance(handler, BaseNodeHandler) or isinstance(handler, CompatibleNodeHandler):
return handler
return CompatibleNodeHandler(handler)
class NodeHandlerRegistry: class NodeHandlerRegistry:
"""节点处理器注册表""" """节点处理器注册表"""
sync_state_machine/datasource/jsonl/datasource.py
+97 -2
View File
@@ -10,8 +10,10 @@ JsonlDataSource - JSONL 文件数据源
from __future__ import annotations from __future__ import annotations
import json
import re
from pathlib import Path from pathlib import Path
from typing import Dict, Any, Set from typing import Dict, Any, Set, Callable, List, Optional
from ..datasource import BaseDataSource from ..datasource import BaseDataSource
@@ -30,7 +32,7 @@ class JsonlDataSource(BaseDataSource):
示例： 示例：
datasource = JsonlDataSource(Path("filtered_datasource/datasource/ecm-2025-12-02")) datasource = JsonlDataSource(Path("filtered_datasource/datasource/ecm-2025-12-02"))
datasource.register_handler(ContractJsonlHandler(datasource)) datasource.add_handler(ContractJsonlHandler(datasource))
collection = DataCollection() collection = DataCollection()
datasource.set_collection(collection) datasource.set_collection(collection)
@@ -46,6 +48,7 @@ class JsonlDataSource(BaseDataSource):
# 用于 create/update/delete 的内存缓存 # 用于 create/update/delete 的内存缓存
self._data: Dict[str, Dict[str, Dict[str, Any]]] = {} # {node_type: {id: data}} self._data: Dict[str, Dict[str, Dict[str, Any]]] = {} # {node_type: {id: data}}
self._dirty_types: Set[str] = set() self._dirty_types: Set[str] = set()
self._node_items_cache: Dict[str, Dict[str, Any]] = {}
def _on_node_loaded(self, handler, node) -> None: def _on_node_loaded(self, handler, node) -> None:
node.append_log( node.append_log(
@@ -62,6 +65,98 @@ class JsonlDataSource(BaseDataSource):
f"JSONL链路[poll] task_id={task_id} status={result.status.value}" f"JSONL链路[poll] task_id={task_id} status={result.status.value}"
) )
def invalidate_node_cache(self, node_type: str) -> None:
self._node_items_cache.pop(node_type, None)
def _matching_files(self, node_type: str) -> List[Path]:
if not self.dir_path.exists():
return []
pattern = re.compile(rf"^{re.escape(node_type)}_\d+\.jsonl$")
return sorted(
file_path
for file_path in self.dir_path.iterdir()
if pattern.match(file_path.name)
)
def _build_cache_entry(
self,
*,
node_type: str,
extract_id: Callable[[Dict[str, Any]], str],
) -> Dict[str, Any]:
files = self._matching_files(node_type)
signature = tuple(
(file_path.name, file_path.stat().st_mtime_ns, file_path.stat().st_size)
for file_path in files
)
cached = self._node_items_cache.get(node_type)
if cached and cached.get("signature") == signature:
return cached
items: List[Dict[str, Any]] = []
data_id_index: Dict[str, List[Dict[str, Any]]] = {}
project_index: Dict[str, List[Dict[str, Any]]] = {}
no_project_items: List[Dict[str, Any]] = []
for file_path in files:
with open(file_path, "r", encoding="utf-8") as f:
for line_num, line in enumerate(f, 1):
line = line.strip()
if not line:
continue
try:
item = json.loads(line)
except json.JSONDecodeError:
print(f"Warning: Invalid JSON in {file_path.name}:{line_num}")
continue
items.append(item)
item_id = extract_id(item)
if item_id:
data_id_index.setdefault(str(item_id), []).append(item)
project_id = item.get("project_id")
if project_id:
project_index.setdefault(str(project_id), []).append(item)
else:
no_project_items.append(item)
entry = {
"signature": signature,
"items": items,
"data_id_index": data_id_index,
"project_index": project_index,
"no_project_items": no_project_items,
}
self._node_items_cache[node_type] = entry
return entry
def get_items_for_load(
self,
*,
node_type: str,
extract_id: Callable[[Dict[str, Any]], str],
data_id_filter: Optional[List[str]] = None,
filter_on_project_id: bool = False,
) -> List[Dict[str, Any]]:
entry = self._build_cache_entry(node_type=node_type, extract_id=extract_id)
filters = [str(item) for item in (data_id_filter or []) if str(item)]
if not filters:
return list(entry["items"])
if not filter_on_project_id:
matched: List[Dict[str, Any]] = []
for data_id in filters:
matched.extend(entry["data_id_index"].get(data_id, []))
return matched
matched = list(entry["no_project_items"])
for project_id in filters:
matched.extend(entry["project_index"].get(project_id, []))
return matched
# load_all 和 sync_all 继承自 BaseDataSource # load_all 和 sync_all 继承自 BaseDataSource
# 保存逻辑在 BaseJsonlHandler 中实现 # 保存逻辑在 BaseJsonlHandler 中实现
sync_state_machine/datasource/jsonl/handler.py
+10 -41
View File
@@ -12,7 +12,6 @@ BaseJsonlHandler - JSONL Handler 基类
from __future__ import annotations from __future__ import annotations
import json import json
import re
from typing import Dict, List, Any, TYPE_CHECKING, Optional, Set from typing import Dict, List, Any, TYPE_CHECKING, Optional, Set
from uuid import uuid4 from uuid import uuid4
@@ -50,23 +49,6 @@ class BaseJsonlHandler(BaseNodeHandler):
): ):
super().__init__(node_type, node_class, schema) super().__init__(node_type, node_class, schema)
self.datasource = datasource self.datasource = datasource
self._target_project_ids: List[str] = []
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""注入项目过滤列表的本地副本。真正的过滤来源仍然是 datasource。"""
self._target_project_ids = [str(pid) for pid in target_project_ids if str(pid)]
def _should_skip_item_by_project_filter(self, item: Dict[str, Any], item_id: str) -> bool:
target_project_ids = set(self.datasource.target_project_ids)
if not target_project_ids:
return False
if self.node_type == "project":
return item_id not in target_project_ids
project_id = item.get("project_id")
if project_id:
return str(project_id) not in target_project_ids
return False
async def load(self) -> List['SyncNode']: async def load(self) -> List['SyncNode']:
""" """
@@ -82,40 +64,26 @@ class BaseJsonlHandler(BaseNodeHandler):
if not self.datasource.dir_path.exists(): if not self.datasource.dir_path.exists():
return nodes return nodes
# 查找所有匹配的文件：{node_type}_N.jsonl data_id_filter = self.get_data_id_filter()
pattern = re.compile(rf"^{re.escape(self.node_type)}_\d+\.jsonl$") items = self.datasource.get_items_for_load(
node_type=self.node_type,
extract_id=self.extract_id,
data_id_filter=data_id_filter,
filter_on_project_id=self.node_type != "project",
)
for file_path in self.datasource.dir_path.iterdir(): for item in items:
if not pattern.match(file_path.name):
continue
# 读取文件
with open(file_path, "r", encoding="utf-8") as f:
for line_num, line in enumerate(f, 1):
line = line.strip()
if not line:
continue
try:
item = json.loads(line)
item_id = self.extract_id(item) item_id = self.extract_id(item)
if self.should_skip_by_data_id_filter(item, item_id):
if self._should_skip_item_by_project_filter(item, item_id):
continue continue
# 创建节点
node = self.create_node(item) node = self.create_node(item)
nodes.append(node) nodes.append(node)
# 同时加载到缓存
if item_id: if item_id:
bucket = self.datasource._data.setdefault(self.node_type, {}) bucket = self.datasource._data.setdefault(self.node_type, {})
bucket[item_id] = item bucket[item_id] = item
except json.JSONDecodeError as e:
print(f"Warning: Invalid JSON in {file_path.name}:{line_num}: {e}")
# 继续处理下一行
return nodes return nodes
def extract_id(self, data: Dict[str, Any]) -> str: def extract_id(self, data: Dict[str, Any]) -> str:
@@ -293,6 +261,7 @@ class BaseJsonlHandler(BaseNodeHandler):
# 清除该类型的 dirty 标记 # 清除该类型的 dirty 标记
self.datasource._dirty_types.discard(self.node_type) self.datasource._dirty_types.discard(self.node_type)
self.datasource.invalidate_node_cache(self.node_type)
except Exception as e: except Exception as e:
print(f"Error saving {self.node_type}: {str(e)}") print(f"Error saving {self.node_type}: {str(e)}")
sync_state_machine/domain/construction/sync_strategy.py
+10
View File
@@ -18,3 +18,13 @@ class ConstructionTaskSyncStrategy(DefaultSyncStrategy[ConstructionResponse]):
remote_orphan_action=OrphanAction.NONE, remote_orphan_action=OrphanAction.NONE,
update_direction=UpdateDirection.PUSH, update_direction=UpdateDirection.PUSH,
) )
def preprocess_compare_payload(self, payload: dict) -> dict:
plan_nodes = payload.get("plan_node")
if not isinstance(plan_nodes, list):
return payload
payload["plan_node"] = [
{key: value for key, value in item.items() if key != "is_audit"} if isinstance(item, dict) else item
for item in plan_nodes
]
return payload
sync_state_machine/domain/contract_settlement/sync_strategy.py
+10
View File
@@ -18,3 +18,13 @@ class ContractSettlementSyncStrategy(DefaultSyncStrategy[ContractSettlementRespo
remote_orphan_action=OrphanAction.NONE, remote_orphan_action=OrphanAction.NONE,
update_direction=UpdateDirection.PUSH, update_direction=UpdateDirection.PUSH,
) )
def preprocess_compare_payload(self, payload: dict) -> dict:
plan_nodes = payload.get("plan_node")
if not isinstance(plan_nodes, list):
return payload
payload["plan_node"] = [
{key: value for key, value in item.items() if key != "is_audit"} if isinstance(item, dict) else item
for item in plan_nodes
]
return payload
sync_state_machine/domain/material/sync_strategy.py
+10
View File
@@ -18,3 +18,13 @@ class MaterialSyncStrategy(DefaultSyncStrategy[MaterialResponse]):
remote_orphan_action=OrphanAction.NONE, remote_orphan_action=OrphanAction.NONE,
update_direction=UpdateDirection.PUSH, update_direction=UpdateDirection.PUSH,
) )
def preprocess_compare_payload(self, payload: dict) -> dict:
plan_nodes = payload.get("plan_node")
if not isinstance(plan_nodes, list):
return payload
payload["plan_node"] = [
{key: value for key, value in item.items() if key != "is_audit"} if isinstance(item, dict) else item
for item in plan_nodes
]
return payload
sync_state_machine/domain/preparation/api_handler.py
-9
View File
@@ -27,17 +27,8 @@ class PreparationApiHandler(BaseApiHandler):
self._node_type = "preparation" self._node_type = "preparation"
self._node_class = PreparationSyncNode self._node_class = PreparationSyncNode
self._schema = PreparationDetail self._schema = PreparationDetail
self._filter_project_id = None
self.update_schemas = [PostPreparation] self.update_schemas = [PostPreparation]
def set_filter_project_id(self, project_id: str | List[str]):
"""设置要加载的项目ID过滤列表"""
self._filter_project_id = project_id
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""设置目标项目ID列表"""
self.set_filter_project_id(target_project_ids)
async def load(self) -> List[SyncNode]: async def load(self) -> List[SyncNode]:
"""从 API 加载里程碑数据（需要项目上下文）""" """从 API 加载里程碑数据（需要项目上下文）"""
from ..project.api_handler import ProjectApiHandler from ..project.api_handler import ProjectApiHandler
sync_state_machine/domain/project/api_handler.py
+10 -24
View File
@@ -71,26 +71,11 @@ class ProjectApiHandler(BaseApiHandler):
# 类级别缓存：存储 all_info 数据 # 类级别缓存：存储 all_info 数据
_all_info_cache: Dict[str, Dict[str, Any]] = {} _all_info_cache: Dict[str, Dict[str, Any]] = {}
def __init__(self, filter_project_id: str | List[str] | None = None): def __init__(self, **handler_config: Any):
""" super().__init__(**handler_config)
初始化 Project Handler
Args:
filter_project_id: 指定只加载的项目ID列表（可选，用于多项目测试）
"""
super().__init__()
self._node_type = "project" self._node_type = "project"
self._node_class = ProjectSyncNode self._node_class = ProjectSyncNode
self._schema = ProjectResponseBase self._schema = ProjectResponseBase
if filter_project_id is None:
self._filter_project_id = None
elif isinstance(filter_project_id, str):
self._filter_project_id = [filter_project_id]
else:
self._filter_project_id = filter_project_id # 保存过滤条件
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
self._filter_project_id = list(target_project_ids) if target_project_ids else None
@classmethod @classmethod
async def get_all_info(cls, client, project_id: str) -> Dict[str, Any]: async def get_all_info(cls, client, project_id: str) -> Dict[str, Any]:
@@ -127,15 +112,12 @@ class ProjectApiHandler(BaseApiHandler):
nodes = [] nodes = []
# 确定要加载的项目ID列表：参数 > 构造函数配置 project_id_filter = self.get_data_id_filter()
target_project_ids = self._filter_project_id or []
if isinstance(target_project_ids, str):
target_project_ids = [target_project_ids]
if target_project_ids: if project_id_filter:
# 只加载指定项目列表 # 只加载指定项目列表
projects : List[ProjectResponseBase]= [] projects : List[ProjectResponseBase]= []
for project_id in target_project_ids: for project_id in project_id_filter:
project_response = await api_get_project(self.api_client, project_id) project_response = await api_get_project(self.api_client, project_id)
projects.append(project_response) projects.append(project_response)
else: else:
@@ -166,10 +148,14 @@ class ProjectApiHandler(BaseApiHandler):
# 处理所有项目 # 处理所有项目
for project_data in projects: for project_data in projects:
project_item = project_data.model_dump()
if self.should_skip_by_data_id_filter(project_item, project_data.id):
continue
# 预加载并缓存 all_info，同时存储到 node.context # 预加载并缓存 all_info，同时存储到 node.context
all_info = await self.get_all_info(self.api_client, project_data.id) all_info = await self.get_all_info(self.api_client, project_data.id)
normalized_project_data = _merge_project_all_info_fields( normalized_project_data = _merge_project_all_info_fields(
project_data.model_dump(), project_item,
all_info, all_info,
) )
node = self._create_node(normalized_project_data) node = self._create_node(normalized_project_data)
sync_state_machine/domain/registry.py
+4 -4
View File
@@ -4,13 +4,13 @@ Domain Handler Registry - 集中管理所有领域的 API 和 JSONL Handlers
使用方式： 使用方式：
from sync_state_machine.domain.registry import API_HANDLERS, JSONL_HANDLERS from sync_state_machine.domain.registry import API_HANDLERS, JSONL_HANDLERS
# 注册所有 API handlers # 添加所有 API handlers
for handler_class in API_HANDLERS: for handler_class in API_HANDLERS:
api_ds.register_handler(handler_class()) api_ds.add_handler(handler_class())
# 注册所有 JSONL handlers # 添加所有 JSONL handlers
for handler_class in JSONL_HANDLERS: for handler_class in JSONL_HANDLERS:
jsonl_ds.register_handler(handler_class(jsonl_ds)) jsonl_ds.add_handler(handler_class(jsonl_ds))
""" """
# ============================================ # ============================================
sync_state_machine/pipeline/__init__.py
+4 -2
View File
@@ -4,6 +4,7 @@ Pipeline Layer - Pipeline orchestration and workflow management
from .copy_data_pipeline import CopyDataPipeline from .copy_data_pipeline import CopyDataPipeline
from .full_sync_pipeline import FullSyncPipeline from .full_sync_pipeline import FullSyncPipeline
from .multi_project_pipeline import MultiProjectPipeline
from ..config import ( from ..config import (
PipelineRunConfig, PipelineRunConfig,
ApiDataSourceConfig, ApiDataSourceConfig,
@@ -14,7 +15,6 @@ from ..config import (
build_default_config, build_default_config,
build_config, build_config,
apply_config_overrides, apply_config_overrides,
apply_env_overrides,
config_snapshot, config_snapshot,
) )
from .factory import ( from .factory import (
@@ -25,11 +25,13 @@ from .factory import (
create_pipeline_from_config, create_pipeline_from_config,
run_pipeline_from_file, run_pipeline_from_file,
run_pipeline_from_config, run_pipeline_from_config,
run_profile_from_file,
) )
__all__ = [ __all__ = [
"CopyDataPipeline", "CopyDataPipeline",
"FullSyncPipeline", "FullSyncPipeline",
"MultiProjectPipeline",
"create_jsonl_to_api_pipeline", "create_jsonl_to_api_pipeline",
"create_jsonl_to_jsonl_pipeline", "create_jsonl_to_jsonl_pipeline",
"create_api_to_jsonl_pipeline", "create_api_to_jsonl_pipeline",
@@ -37,6 +39,7 @@ __all__ = [
"create_pipeline_from_config", "create_pipeline_from_config",
"run_pipeline_from_file", "run_pipeline_from_file",
"run_pipeline_from_config", "run_pipeline_from_config",
"run_profile_from_file",
"PipelineRunConfig", "PipelineRunConfig",
"ApiDataSourceConfig", "ApiDataSourceConfig",
"JsonlDataSourceConfig", "JsonlDataSourceConfig",
@@ -46,6 +49,5 @@ __all__ = [
"build_default_config", "build_default_config",
"build_config", "build_config",
"apply_config_overrides", "apply_config_overrides",
"apply_env_overrides",
"config_snapshot", "config_snapshot",
] ]
sync_state_machine/pipeline/copy_data_pipeline.py
+3 -3
View File
@@ -67,7 +67,7 @@ class CopyDataPipeline:
supported_types = [] supported_types = []
for handler_class in JSONL_HANDLERS: for handler_class in JSONL_HANDLERS:
handler = handler_class(self.source_datasource) handler = handler_class(self.source_datasource)
self.source_datasource.register_handler(handler) self.source_datasource.add_handler(handler)
supported_types.append(handler.node_type) supported_types.append(handler.node_type)
self._owns_source = True self._owns_source = True
self._supported_types = supported_types self._supported_types = supported_types
@@ -82,7 +82,7 @@ class CopyDataPipeline:
self.target_datasource = JsonlDataSource(dir_path=Path(target_dir), read_only=False) self.target_datasource = JsonlDataSource(dir_path=Path(target_dir), read_only=False)
# 自动注册所有 JSONL handlers # 自动注册所有 JSONL handlers
for handler_class in JSONL_HANDLERS: for handler_class in JSONL_HANDLERS:
self.target_datasource.register_handler(handler_class(self.target_datasource)) self.target_datasource.add_handler(handler_class(self.target_datasource))
self._owns_target = True self._owns_target = True
else: else:
raise ValueError("Must provide either target_datasource or target_dir") raise ValueError("Must provide either target_datasource or target_dir")
@@ -228,7 +228,7 @@ class CopyDataPipeline:
handler_class = DomainRegistry.get_jsonl_handler(node_type) handler_class = DomainRegistry.get_jsonl_handler(node_type)
if handler_class: if handler_class:
handler = handler_class(datasource) handler = handler_class(datasource)
datasource.register_handler(handler) datasource.add_handler(handler)
def _print_summary(self): def _print_summary(self):
"""打印执行摘要""" """打印执行摘要"""
sync_state_machine/pipeline/factory.py
+129 -53
View File
@@ -21,24 +21,29 @@ from typing import Any, Dict, List, Optional
from ..common.binding import BindingManager from ..common.binding import BindingManager
from ..common.collection import DataCollection from ..common.collection import DataCollection
from ..common.persistence import PersistenceBackend from ..common.persistence import PersistenceBackend, create_persistence_backend
from ..common.registry import DomainRegistry from ..common.registry import DomainRegistry
from ..datasource import ensure_builtin_datasource_types_registered from ..datasource import ensure_builtin_datasource_types_registered
from ..datasource.type_registry import get_datasource_factory from ..datasource.type_registry import get_datasource_factory
from ..config import ( from ..config import (
PipelineRunConfig, PipelineRunConfig,
MultiProjectRunConfig,
DataSourceConfig, DataSourceConfig,
ApiDataSourceConfig, ApiDataSourceConfig,
PersistConfig, PersistConfig,
JsonlDataSourceConfig, JsonlDataSourceConfig,
LoggingConfig, LoggingConfig,
build_config,
StrategyRuntimeConfig, StrategyRuntimeConfig,
build_config_from_file, build_config_from_file,
build_multi_project_config_from_file,
apply_config_overrides, apply_config_overrides,
OrphanAction, OrphanAction,
UpdateDirection, UpdateDirection,
resolve_log_file_path, resolve_log_file_path,
config_snapshot, config_snapshot,
is_multi_project_payload,
load_overrides_from_file,
) )
from ..sync_system.strategy import BaseSyncStrategy from ..sync_system.strategy import BaseSyncStrategy
from .run_logger import init_pipeline_logger from .run_logger import init_pipeline_logger
@@ -146,14 +151,14 @@ async def _initialize_datasource(datasource, *, endpoint_name: str) -> None:
ensure_builtin_datasource_types_registered() ensure_builtin_datasource_types_registered()
def _register_handlers_for_endpoint( def _add_handlers_for_endpoint(
*, *,
datasource, datasource,
ds_config: DataSourceConfig, ds_config: DataSourceConfig,
node_types: List[str], node_types: List[str],
) -> None: ) -> None:
for node_type in node_types: for node_type in node_types:
datasource.register_handler( datasource.add_handler(
_create_handler( _create_handler(
node_type=node_type, node_type=node_type,
ds_config=ds_config, ds_config=ds_config,
@@ -162,24 +167,35 @@ def _register_handlers_for_endpoint(
) )
def _register_handlers( def _add_handlers(
config: PipelineRunConfig, config: PipelineRunConfig,
local_ds, local_ds,
remote_ds, remote_ds,
all_types: List[str], all_types: List[str],
) -> None: ) -> None:
_register_handlers_for_endpoint( _add_handlers_for_endpoint(
datasource=local_ds, datasource=local_ds,
ds_config=config.local_datasource, ds_config=config.local_datasource,
node_types=all_types, node_types=all_types,
) )
_register_handlers_for_endpoint( _add_handlers_for_endpoint(
datasource=remote_ds, datasource=remote_ds,
ds_config=config.remote_datasource, ds_config=config.remote_datasource,
node_types=all_types, node_types=all_types,
) )
def _build_collections_and_bindings(
*,
persistence: PersistenceBackend,
scope: str,
) -> tuple[DataCollection, DataCollection, BindingManager]:
local_collection = DataCollection("local", scope=scope, persistence=persistence, auto_persist=False)
remote_collection = DataCollection("remote", scope=scope, persistence=persistence, auto_persist=False)
binding_manager = BindingManager(persistence, auto_persist=False, scope=scope)
return local_collection, remote_collection, binding_manager
def _resolve_strategy_map(config: PipelineRunConfig, node_types: List[str]) -> Dict[str, StrategyRuntimeConfig]: def _resolve_strategy_map(config: PipelineRunConfig, node_types: List[str]) -> Dict[str, StrategyRuntimeConfig]:
if config.strategies: if config.strategies:
return {item.node_type: item for item in config.strategies} return {item.node_type: item for item in config.strategies}
@@ -196,6 +212,28 @@ def _resolve_strategy_map(config: PipelineRunConfig, node_types: List[str]) -> D
return resolved return resolved
def _build_pipeline_run_config(
*,
node_types: List[str],
local_datasource: DataSourceConfig,
remote_datasource: DataSourceConfig,
persist: PersistConfig,
logging_config: LoggingConfig,
strategy_overrides: Optional[Dict[str, Dict[str, Any]]] = None,
) -> PipelineRunConfig:
cfg = PipelineRunConfig(
node_types=node_types,
local_datasource=local_datasource,
remote_datasource=remote_datasource,
strategies=[],
persist=persist,
logging=logging_config,
)
if strategy_overrides:
cfg = apply_config_overrides(cfg, {"strategies": strategy_overrides})
return cfg
def _build_strategies( def _build_strategies(
config: PipelineRunConfig, config: PipelineRunConfig,
node_types: List[str], node_types: List[str],
@@ -225,6 +263,7 @@ def _build_strategies(
runtime_cfg = strategy_map.get(node_type) runtime_cfg = strategy_map.get(node_type)
if runtime_cfg is not None: if runtime_cfg is not None:
strategy.set_config(runtime_cfg.config.model_copy(deep=True)) strategy.set_config(runtime_cfg.config.model_copy(deep=True))
strategy.skip_sync = bool(runtime_cfg.skip_sync)
strategies.append(strategy) strategies.append(strategy)
return strategies return strategies
@@ -233,6 +272,10 @@ def _build_strategies(
async def create_pipeline_from_config( async def create_pipeline_from_config(
config: PipelineRunConfig, config: PipelineRunConfig,
*, *,
scope: str = "global",
pipeline_name: str | None = None,
bootstrap_binding_node_types: Optional[List[str]] = None,
ephemeral_node_types: Optional[List[str]] = None,
local_datasource_override=None, local_datasource_override=None,
remote_datasource_override=None, remote_datasource_override=None,
) -> FullSyncPipeline: ) -> FullSyncPipeline:
@@ -249,9 +292,6 @@ async def create_pipeline_from_config(
all_types = list(config.node_types) all_types = list(config.node_types)
strategy_map = _resolve_strategy_map(config, all_types) strategy_map = _resolve_strategy_map(config, all_types)
if config.persist.backend != "sqlite":
raise NotImplementedError(f"Persistence backend not supported yet: {config.persist.backend}")
db_path = Path(config.persist.db_path) db_path = Path(config.persist.db_path)
if config.persist.wipe_on_start and db_path.exists(): if config.persist.wipe_on_start and db_path.exists():
db_path.unlink() db_path.unlink()
@@ -260,7 +300,7 @@ async def create_pipeline_from_config(
local_ds = None local_ds = None
remote_ds = None remote_ds = None
try: try:
persistence = PersistenceBackend(str(config.persist.db_path), backend=config.persist.backend) persistence = create_persistence_backend(config.persist)
await persistence.initialize() await persistence.initialize()
local_ds = await _create_datasource( local_ds = await _create_datasource(
@@ -277,14 +317,12 @@ async def create_pipeline_from_config(
project_root=_project_root(), project_root=_project_root(),
) )
await _initialize_datasource(remote_ds, endpoint_name="remote") await _initialize_datasource(remote_ds, endpoint_name="remote")
local_ds.set_target_project_ids([str(pid) for pid in config.local_datasource.target_project_ids if str(pid)]) _add_handlers(config, local_ds, remote_ds, all_types)
remote_ds.set_target_project_ids([str(pid) for pid in config.remote_datasource.target_project_ids if str(pid)])
_register_handlers(config, local_ds, remote_ds, all_types) local_collection, remote_collection, binding_manager = _build_collections_and_bindings(
persistence=persistence,
local_collection = DataCollection("local", persistence, auto_persist=False) scope=scope,
remote_collection = DataCollection("remote", persistence, auto_persist=False) )
binding_manager = BindingManager(persistence, auto_persist=False)
strategies = _build_strategies( strategies = _build_strategies(
config, config,
@@ -304,6 +342,10 @@ async def create_pipeline_from_config(
local_collection=local_collection, local_collection=local_collection,
remote_collection=remote_collection, remote_collection=remote_collection,
binding_manager=binding_manager, binding_manager=binding_manager,
scope=scope,
pipeline_name=pipeline_name,
bootstrap_binding_node_types=bootstrap_binding_node_types,
ephemeral_node_types=ephemeral_node_types,
enable_persistence=config.persist.enable, enable_persistence=config.persist.enable,
) )
@@ -328,6 +370,8 @@ async def run_pipeline_from_config(
*, *,
title: str = "sync_state_machine pipeline", title: str = "sync_state_machine pipeline",
print_summary: bool = True, print_summary: bool = True,
scope: str = "global",
bootstrap_binding_node_types: Optional[List[str]] = None,
local_datasource_override=None, local_datasource_override=None,
remote_datasource_override=None, remote_datasource_override=None,
) -> Dict[str, Any]: ) -> Dict[str, Any]:
@@ -344,7 +388,7 @@ async def run_pipeline_from_config(
resolved_cfg = config_snapshot(config) resolved_cfg = config_snapshot(config)
resolved_cfg_text = json.dumps(resolved_cfg, ensure_ascii=False, indent=2) resolved_cfg_text = json.dumps(resolved_cfg, ensure_ascii=False, indent=2)
logger.info( logger.info(
"Resolved Config Summary: node_types=%d local=%s remote=%s", "Resolved Full Config Summary: node_types=%d local=%s remote=%s",
len(config.node_types), len(config.node_types),
config.local_datasource.type, config.local_datasource.type,
config.remote_datasource.type, config.remote_datasource.type,
@@ -352,7 +396,7 @@ async def run_pipeline_from_config(
if config.logging.file_path: if config.logging.file_path:
try: try:
with open(config.logging.file_path, "a", encoding="utf-8") as fp: with open(config.logging.file_path, "a", encoding="utf-8") as fp:
fp.write("Resolved Config:\n") fp.write("Resolved Full Config:\n")
fp.write(resolved_cfg_text) fp.write(resolved_cfg_text)
fp.write("\n") fp.write("\n")
except Exception as exc: except Exception as exc:
@@ -365,15 +409,18 @@ async def run_pipeline_from_config(
print(f"🚀 Creating {title}") print(f"🚀 Creating {title}")
print(f"📝 Log file: {config.logging.file_path or '-'}") print(f"📝 Log file: {config.logging.file_path or '-'}")
print( print(
f"🔧 Resolved Config Summary: node_types={len(config.node_types)} " f"🔧 Resolved Full Config Summary: node_types={len(config.node_types)} "
f"local={config.local_datasource.type} " f"local={config.local_datasource.type} "
f"remote={config.remote_datasource.type}" f"remote={config.remote_datasource.type}"
) )
print(f"🔧 Resolved Config:\n{resolved_cfg_text}") print(f"🔧 Resolved Full Config:\n{resolved_cfg_text}")
print("=" * 80) print("=" * 80)
pipeline = await create_pipeline_from_config( pipeline = await create_pipeline_from_config(
config, config,
scope=scope,
pipeline_name=title,
bootstrap_binding_node_types=bootstrap_binding_node_types,
local_datasource_override=local_datasource_override, local_datasource_override=local_datasource_override,
remote_datasource_override=remote_datasource_override, remote_datasource_override=remote_datasource_override,
) )
@@ -393,12 +440,20 @@ async def create_pipeline_from_file(
*, *,
project_root: Path, project_root: Path,
config_path: str | Path, config_path: str | Path,
scope: str = "global",
pipeline_name: str | None = None,
bootstrap_binding_node_types: Optional[List[str]] = None,
ephemeral_node_types: Optional[List[str]] = None,
local_datasource_override=None, local_datasource_override=None,
remote_datasource_override=None, remote_datasource_override=None,
) -> FullSyncPipeline: ) -> FullSyncPipeline:
config = build_config_from_file(project_root=project_root, file_path=config_path) config = build_config_from_file(project_root=project_root, file_path=config_path)
return await create_pipeline_from_config( return await create_pipeline_from_config(
config, config,
scope=scope,
pipeline_name=pipeline_name,
bootstrap_binding_node_types=bootstrap_binding_node_types,
ephemeral_node_types=ephemeral_node_types,
local_datasource_override=local_datasource_override, local_datasource_override=local_datasource_override,
remote_datasource_override=remote_datasource_override, remote_datasource_override=remote_datasource_override,
) )
@@ -410,6 +465,8 @@ async def run_pipeline_from_file(
config_path: str | Path, config_path: str | Path,
title: str = "sync_state_machine pipeline", title: str = "sync_state_machine pipeline",
print_summary: bool = True, print_summary: bool = True,
scope: str = "global",
bootstrap_binding_node_types: Optional[List[str]] = None,
local_datasource_override=None, local_datasource_override=None,
remote_datasource_override=None, remote_datasource_override=None,
) -> Dict[str, Any]: ) -> Dict[str, Any]:
@@ -418,11 +475,41 @@ async def run_pipeline_from_file(
config, config,
title=title, title=title,
print_summary=print_summary, print_summary=print_summary,
scope=scope,
bootstrap_binding_node_types=bootstrap_binding_node_types,
local_datasource_override=local_datasource_override, local_datasource_override=local_datasource_override,
remote_datasource_override=remote_datasource_override, remote_datasource_override=remote_datasource_override,
) )
async def run_profile_from_file(
*,
project_root: Path,
config_path: str | Path,
print_summary: bool = True,
) -> Dict[str, Any]:
cfg_path = Path(config_path)
raw = load_overrides_from_file(cfg_path, project_root)
if is_multi_project_payload(raw):
from .multi_project_pipeline import MultiProjectPipeline
multi_cfg = MultiProjectRunConfig.model_validate(raw)
pipeline = MultiProjectPipeline(project_root=project_root, config_path=cfg_path, config=multi_cfg)
if print_summary:
print("\n" + "=" * 80)
print(f"🚀 Creating multi-project pipeline from: {cfg_path}")
print("=" * 80)
return await pipeline.run()
config = build_config(project_root=project_root, overrides=raw)
mode = f"{config.local_datasource.type}->{config.remote_datasource.type}"
return await run_pipeline_from_config(
config,
title=f"sync_state_machine pipeline ({mode})",
print_summary=print_summary,
)
async def create_jsonl_to_api_pipeline( async def create_jsonl_to_api_pipeline(
*, *,
# 路径配置 # 路径配置
@@ -434,8 +521,6 @@ async def create_jsonl_to_api_pipeline(
api_secret: str, api_secret: str,
# 节点类型 # 节点类型
node_types: List[str], node_types: List[str],
# 项目过滤
target_project_ids: Optional[List[str]] = None,
strategy_overrides: Optional[Dict[str, Dict[str, Any]]] = None, strategy_overrides: Optional[Dict[str, Dict[str, Any]]] = None,
# Pipeline 行为 # Pipeline 行为
enable_persistence: bool = True, enable_persistence: bool = True,
@@ -446,20 +531,20 @@ async def create_jsonl_to_api_pipeline(
poll_interval: float = 0.5, poll_interval: float = 0.5,
api_rate_limit_max_requests: int = 30, api_rate_limit_max_requests: int = 30,
api_rate_limit_window_seconds: float = 10.0, api_rate_limit_window_seconds: float = 10.0,
# Handler 自定义配置（可选，作用于 API 端） local_handler_configs: Optional[Dict[str, Dict[str, Any]]] = None,
handler_configs: Optional[Dict[str, Dict[str, Any]]] = None, remote_handler_configs: Optional[Dict[str, Dict[str, Any]]] = None,
# Logger 配置 # Logger 配置
initialize_logger: bool = True, initialize_logger: bool = True,
logger_file_path: Optional[str] = None, logger_file_path: Optional[str] = None,
logger_level: int = logging.INFO, logger_level: int = logging.INFO,
) -> FullSyncPipeline: ) -> FullSyncPipeline:
cfg = PipelineRunConfig( cfg = _build_pipeline_run_config(
node_types=node_types, node_types=node_types,
local_datasource=JsonlDataSourceConfig( local_datasource=JsonlDataSourceConfig(
type="jsonl", type="jsonl",
jsonl_dir=local_jsonl_dir, jsonl_dir=local_jsonl_dir,
read_only=False, read_only=False,
target_project_ids=target_project_ids or [], handler_configs=local_handler_configs or {},
), ),
remote_datasource=ApiDataSourceConfig( remote_datasource=ApiDataSourceConfig(
type="api", type="api",
@@ -471,23 +556,20 @@ async def create_jsonl_to_api_pipeline(
poll_interval=poll_interval, poll_interval=poll_interval,
api_rate_limit_max_requests=api_rate_limit_max_requests, api_rate_limit_max_requests=api_rate_limit_max_requests,
api_rate_limit_window_seconds=api_rate_limit_window_seconds, api_rate_limit_window_seconds=api_rate_limit_window_seconds,
target_project_ids=target_project_ids or [], handler_configs=remote_handler_configs or {},
handler_configs=handler_configs or {},
), ),
strategies=[],
persist=PersistConfig( persist=PersistConfig(
db_path=persistence_db, db_path=persistence_db,
enable=enable_persistence, enable=enable_persistence,
wipe_on_start=wipe_persistence_on_start, wipe_on_start=wipe_persistence_on_start,
), ),
logging=LoggingConfig( logging_config=LoggingConfig(
initialize=initialize_logger, initialize=initialize_logger,
file_path=logger_file_path, file_path=logger_file_path,
level=logger_level, level=logger_level,
), ),
strategy_overrides=strategy_overrides,
) )
if strategy_overrides:
cfg = apply_config_overrides(cfg, {"strategies": strategy_overrides})
return await create_pipeline_from_config(cfg) return await create_pipeline_from_config(cfg)
@@ -502,8 +584,6 @@ async def create_api_to_jsonl_pipeline(
persistence_db: str, persistence_db: str,
# 节点类型 # 节点类型
node_types: List[str], node_types: List[str],
# 项目过滤
target_project_ids: Optional[List[str]] = None,
strategy_overrides: Optional[Dict[str, Dict[str, Any]]] = None, strategy_overrides: Optional[Dict[str, Dict[str, Any]]] = None,
# Pipeline 行为 # Pipeline 行为
enable_persistence: bool = True, enable_persistence: bool = True,
@@ -514,14 +594,14 @@ async def create_api_to_jsonl_pipeline(
poll_interval: float = 0.5, poll_interval: float = 0.5,
api_rate_limit_max_requests: int = 30, api_rate_limit_max_requests: int = 30,
api_rate_limit_window_seconds: float = 10.0, api_rate_limit_window_seconds: float = 10.0,
# Handler 自定义配置（可选，作用于 API 端） local_handler_configs: Optional[Dict[str, Dict[str, Any]]] = None,
handler_configs: Optional[Dict[str, Dict[str, Any]]] = None, remote_handler_configs: Optional[Dict[str, Dict[str, Any]]] = None,
# Logger 配置 # Logger 配置
initialize_logger: bool = True, initialize_logger: bool = True,
logger_file_path: Optional[str] = None, logger_file_path: Optional[str] = None,
logger_level: int = logging.INFO, logger_level: int = logging.INFO,
) -> FullSyncPipeline: ) -> FullSyncPipeline:
cfg = PipelineRunConfig( cfg = _build_pipeline_run_config(
node_types=node_types, node_types=node_types,
local_datasource=ApiDataSourceConfig( local_datasource=ApiDataSourceConfig(
type="api", type="api",
@@ -533,29 +613,26 @@ async def create_api_to_jsonl_pipeline(
poll_interval=poll_interval, poll_interval=poll_interval,
api_rate_limit_max_requests=api_rate_limit_max_requests, api_rate_limit_max_requests=api_rate_limit_max_requests,
api_rate_limit_window_seconds=api_rate_limit_window_seconds, api_rate_limit_window_seconds=api_rate_limit_window_seconds,
target_project_ids=target_project_ids or [], handler_configs=local_handler_configs or {},
handler_configs=handler_configs or {},
), ),
remote_datasource=JsonlDataSourceConfig( remote_datasource=JsonlDataSourceConfig(
type="jsonl", type="jsonl",
jsonl_dir=remote_jsonl_dir, jsonl_dir=remote_jsonl_dir,
read_only=False, read_only=False,
target_project_ids=target_project_ids or [], handler_configs=remote_handler_configs or {},
), ),
strategies=[],
persist=PersistConfig( persist=PersistConfig(
db_path=persistence_db, db_path=persistence_db,
enable=enable_persistence, enable=enable_persistence,
wipe_on_start=wipe_persistence_on_start, wipe_on_start=wipe_persistence_on_start,
), ),
logging=LoggingConfig( logging_config=LoggingConfig(
initialize=initialize_logger, initialize=initialize_logger,
file_path=logger_file_path, file_path=logger_file_path,
level=logger_level, level=logger_level,
), ),
strategy_overrides=strategy_overrides,
) )
if strategy_overrides:
cfg = apply_config_overrides(cfg, {"strategies": strategy_overrides})
return await create_pipeline_from_config(cfg) return await create_pipeline_from_config(cfg)
@@ -565,40 +642,39 @@ async def create_jsonl_to_jsonl_pipeline(
remote_jsonl_dir: str, remote_jsonl_dir: str,
persistence_db: str, persistence_db: str,
node_types: List[str], node_types: List[str],
target_project_ids: Optional[List[str]] = None,
enable_persistence: bool = True, enable_persistence: bool = True,
wipe_persistence_on_start: bool = False, wipe_persistence_on_start: bool = False,
strategy_overrides: Optional[Dict[str, Dict[str, Any]]] = None, strategy_overrides: Optional[Dict[str, Dict[str, Any]]] = None,
local_handler_configs: Optional[Dict[str, Dict[str, Any]]] = None,
remote_handler_configs: Optional[Dict[str, Dict[str, Any]]] = None,
initialize_logger: bool = True, initialize_logger: bool = True,
logger_file_path: Optional[str] = None, logger_file_path: Optional[str] = None,
logger_level: int = logging.INFO, logger_level: int = logging.INFO,
) -> FullSyncPipeline: ) -> FullSyncPipeline:
cfg = PipelineRunConfig( cfg = _build_pipeline_run_config(
node_types=node_types, node_types=node_types,
local_datasource=JsonlDataSourceConfig( local_datasource=JsonlDataSourceConfig(
type="jsonl", type="jsonl",
jsonl_dir=local_jsonl_dir, jsonl_dir=local_jsonl_dir,
read_only=False, read_only=False,
target_project_ids=target_project_ids or [], handler_configs=local_handler_configs or {},
), ),
remote_datasource=JsonlDataSourceConfig( remote_datasource=JsonlDataSourceConfig(
type="jsonl", type="jsonl",
jsonl_dir=remote_jsonl_dir, jsonl_dir=remote_jsonl_dir,
read_only=False, read_only=False,
target_project_ids=target_project_ids or [], handler_configs=remote_handler_configs or {},
), ),
strategies=[],
persist=PersistConfig( persist=PersistConfig(
db_path=persistence_db, db_path=persistence_db,
enable=enable_persistence, enable=enable_persistence,
wipe_on_start=wipe_persistence_on_start, wipe_on_start=wipe_persistence_on_start,
), ),
logging=LoggingConfig( logging_config=LoggingConfig(
initialize=initialize_logger, initialize=initialize_logger,
file_path=logger_file_path, file_path=logger_file_path,
level=logger_level, level=logger_level,
), ),
strategy_overrides=strategy_overrides,
) )
if strategy_overrides:
cfg = apply_config_overrides(cfg, {"strategies": strategy_overrides})
return await create_pipeline_from_config(cfg) return await create_pipeline_from_config(cfg)
sync_state_machine/pipeline/full_sync_pipeline.py
+124 -43
View File
@@ -12,6 +12,7 @@ from pathlib import Path
if TYPE_CHECKING: if TYPE_CHECKING:
from ..datasource.datasource import BaseDataSource from ..datasource.datasource import BaseDataSource
from ..datasource.jsonl import JsonlDataSource
from ..common.collection import DataCollection from ..common.collection import DataCollection
from ..common.binding import BindingManager from ..common.binding import BindingManager
from ..common.persistence import PersistenceBackend from ..common.persistence import PersistenceBackend
@@ -47,6 +48,10 @@ class FullSyncPipeline:
local_collection: DataCollection, local_collection: DataCollection,
remote_collection: DataCollection, remote_collection: DataCollection,
binding_manager: BindingManager, binding_manager: BindingManager,
scope: str = "global",
pipeline_name: str | None = None,
bootstrap_binding_node_types: Optional[List[str]] = None,
ephemeral_node_types: Optional[List[str]] = None,
node_types: Optional[List[str]] = None, node_types: Optional[List[str]] = None,
load_order: Optional[List[str]] = None, load_order: Optional[List[str]] = None,
sync_order: Optional[List[str]] = None, sync_order: Optional[List[str]] = None,
@@ -61,7 +66,11 @@ class FullSyncPipeline:
self.local_datasource = local_datasource self.local_datasource = local_datasource
self.remote_datasource = remote_datasource self.remote_datasource = remote_datasource
self.strategies = strategies self.strategies = strategies
self.scope = scope
self.pipeline_name = pipeline_name or f"pipeline[{scope}]"
self.node_types = node_types or [s.node_type for s in self.strategies] self.node_types = node_types or [s.node_type for s in self.strategies]
self.bootstrap_binding_node_types = list(dict.fromkeys(bootstrap_binding_node_types or []))
self.ephemeral_node_types = list(dict.fromkeys(ephemeral_node_types or []))
self.load_order = load_order or self.node_types self.load_order = load_order or self.node_types
self.sync_order = sync_order or self.node_types self.sync_order = sync_order or self.node_types
self.persistence = persistence self.persistence = persistence
@@ -90,6 +99,10 @@ class FullSyncPipeline:
self._consistency_by_type: Dict[str, Dict[str, Any]] = {} self._consistency_by_type: Dict[str, Dict[str, Any]] = {}
self._logger = logger self._logger = logger
self._closed = False self._closed = False
self._loaded_node_types: set[str] = set()
self._preloaded_local_nodes: List[Any] = []
self._preloaded_remote_nodes: List[Any] = []
self._preloaded_bindings: Dict[str, Dict[str, Optional[str]]] = {}
def _resolve_pipeline_runtime(self) -> StateMachineRuntime: def _resolve_pipeline_runtime(self) -> StateMachineRuntime:
if self.strategies: if self.strategies:
@@ -103,22 +116,23 @@ class FullSyncPipeline:
async def run(self) -> Dict[str, Any]: async def run(self) -> Dict[str, Any]:
try: try:
await self.phase_bootstrap() await self.phase_bootstrap()
self._logger.info("✅ Phase bootstrap completed") self._logger.info("✅ Phase bootstrap completed (%s)", self.pipeline_name)
await self.phase_sync_by_node_type() await self.phase_sync_by_node_type()
self._logger.info("✅ Phase sync-by-node-type completed") self._logger.info("✅ Phase sync-by-node-type completed (%s)", self.pipeline_name)
post_ok = await self.phase_post_check() post_ok = await self.phase_post_check()
self.stats["post_check_passed"] = post_ok self.stats["post_check_passed"] = post_ok
self._logger.info("✅ Phase post-check completed") self._logger.info("✅ Phase post-check completed (%s)", self.pipeline_name)
await self.phase_persistence() await self.phase_persistence()
self._logger.info("✅ Phase persistence completed") self._logger.info("✅ Phase persistence completed (%s)", self.pipeline_name)
return self.stats return self.stats
finally: finally:
await self.close() await self.close()
async def phase_sync_by_node_type(self) -> None: async def phase_sync_by_node_type(self) -> None:
self._prepare_datasources()
strategy_map = {s.node_type: s for s in self.strategies} strategy_map = {s.node_type: s for s in self.strategies}
for node_type in self.sync_order: for node_type in self.sync_order:
strategy = strategy_map.get(node_type) strategy = strategy_map.get(node_type)
@@ -130,6 +144,8 @@ class FullSyncPipeline:
self._logger.info(f"🧩 Strategy start: {node_type}") self._logger.info(f"🧩 Strategy start: {node_type}")
self._logger.info("-" * section_width) self._logger.info("-" * section_width)
await self._load_node_type(node_type, reason="pre-strategy refresh")
await strategy.bind() await strategy.bind()
self._logger.info(f"✅ Strategy bind: {node_type}") self._logger.info(f"✅ Strategy bind: {node_type}")
@@ -156,23 +172,20 @@ class FullSyncPipeline:
self._logger.error(f"❌ Strategy failed but pipeline continues: {node_type} | {type(exc).__name__}: {exc}") self._logger.error(f"❌ Strategy failed but pipeline continues: {node_type} | {type(exc).__name__}: {exc}")
continue continue
@staticmethod
def _is_noop_strategy(strategy: BaseSyncStrategy[Any]) -> bool:
cfg = strategy.config
return (
cfg.local_orphan_action == OrphanAction.NONE
and cfg.remote_orphan_action == OrphanAction.NONE
and cfg.update_direction == UpdateDirection.NONE
)
async def phase_bootstrap(self) -> None: async def phase_bootstrap(self) -> None:
await self._load_persistence_state() await self._load_persistence_state()
self._logger.info("✅ Bootstrap: persistence loaded") self._logger.info("✅ Bootstrap: persistence loaded")
await self._load_from_datasource() await self._apply_preloaded_state()
self._logger.info("✅ Bootstrap: datasource fetched") self._logger.info("✅ Bootstrap: shared preload applied")
self._prepare_datasources()
self._logger.info("✅ Bootstrap: datasource prepared")
await self._apply_project_scope_cleanup()
async def phase_bind(self) -> None: async def phase_bind(self) -> None:
self._prepare_datasources()
strategy_map = {s.node_type: s for s in self.strategies} strategy_map = {s.node_type: s for s in self.strategies}
for node_type in self.sync_order: for node_type in self.sync_order:
strategy = strategy_map.get(node_type) strategy = strategy_map.get(node_type)
@@ -186,6 +199,7 @@ class FullSyncPipeline:
continue continue
async def phase_create(self) -> None: async def phase_create(self) -> None:
self._prepare_datasources()
strategy_map = {s.node_type: s for s in self.strategies} strategy_map = {s.node_type: s for s in self.strategies}
for node_type in self.sync_order: for node_type in self.sync_order:
strategy = strategy_map.get(node_type) strategy = strategy_map.get(node_type)
@@ -207,6 +221,7 @@ class FullSyncPipeline:
continue continue
async def phase_update(self) -> None: async def phase_update(self) -> None:
self._prepare_datasources()
strategy_map = {s.node_type: s for s in self.strategies} strategy_map = {s.node_type: s for s in self.strategies}
for node_type in self.sync_order: for node_type in self.sync_order:
strategy = strategy_map.get(node_type) strategy = strategy_map.get(node_type)
@@ -250,10 +265,17 @@ class FullSyncPipeline:
if not self.enable_persistence: if not self.enable_persistence:
return return
excluded_node_types = self.ephemeral_node_types or None
# 加载持久化数据 # 加载持久化数据
await self.local_collection.load_from_persistence() await self.local_collection.load_from_persistence_excluding(excluded_node_types)
await self.remote_collection.load_from_persistence() await self.remote_collection.load_from_persistence_excluding(excluded_node_types)
for node_type in self.node_types: binding_node_types = [
node_type
for node_type in dict.fromkeys([*self.node_types, *self.bootstrap_binding_node_types])
if node_type not in set(self.ephemeral_node_types)
]
for node_type in binding_node_types:
await self.binding_manager.load_from_persistence(node_type) await self.binding_manager.load_from_persistence(node_type)
self._logger.info("🔄 Persistence loaded (state preserved; pending E01 normalization)") self._logger.info("🔄 Persistence loaded (state preserved; pending E01 normalization)")
@@ -271,26 +293,92 @@ class FullSyncPipeline:
self._logger.info(f"🧹 Reset cleanup: {total_cleaned} CREATE-failed zombies cleaned") self._logger.info(f"🧹 Reset cleanup: {total_cleaned} CREATE-failed zombies cleaned")
self._logger.info("🔄 Post-load reset cleanup completed") self._logger.info("🔄 Post-load reset cleanup completed")
async def _load_from_datasource(self) -> None: async def _apply_preloaded_state(self) -> None:
"""从数据源加载数据""" if self._preloaded_local_nodes:
await self.local_collection.import_nodes(self._preloaded_local_nodes)
if self._preloaded_remote_nodes:
await self.remote_collection.import_nodes(self._preloaded_remote_nodes)
for node_type, bindings in self._preloaded_bindings.items():
self.binding_manager.import_bindings(node_type, bindings)
def preload_shared_state(
self,
*,
local_nodes: List[Any],
remote_nodes: List[Any],
bindings_by_type: Dict[str, Dict[str, Optional[str]]],
) -> None:
self._preloaded_local_nodes = list(local_nodes)
self._preloaded_remote_nodes = list(remote_nodes)
self._preloaded_bindings = {
node_type: dict(bindings)
for node_type, bindings in bindings_by_type.items()
}
def _prepare_datasources(self) -> None:
"""为后续按 node_type 加载准备 collection 绑定。"""
self.local_datasource.set_collection(self.local_collection) self.local_datasource.set_collection(self.local_collection)
await self.local_datasource.load_all(order=self.load_order)
self.remote_datasource.set_collection(self.remote_collection) self.remote_datasource.set_collection(self.remote_collection)
await self.remote_datasource.load_all(order=self.load_order)
for node_type in self.node_types: def _get_project_scope_filter_ids(self, datasource: "BaseDataSource") -> List[str]:
try:
handler = datasource.get_handler("project")
get_data_id_filter = getattr(handler, "get_data_id_filter", None)
if callable(get_data_id_filter):
raw_filter = get_data_id_filter()
if isinstance(raw_filter, list):
return list(dict.fromkeys(raw_filter))
except Exception:
pass
return []
async def _apply_project_scope_cleanup(self) -> None:
local_project_ids = self._get_project_scope_filter_ids(self.local_datasource)
remote_project_ids = self._get_project_scope_filter_ids(self.remote_datasource)
if not local_project_ids and not remote_project_ids:
return
deleted_local = await self.local_collection.filter_by_project_ids(local_project_ids)
deleted_remote = await self.remote_collection.filter_by_project_ids(remote_project_ids)
deleted_bindings = await self.binding_manager.prune_missing_nodes(
local_collection=self.local_collection,
remote_collection=self.remote_collection,
node_types=list(dict.fromkeys([*self.node_types, *self.bootstrap_binding_node_types])),
)
if deleted_local or deleted_remote or deleted_bindings:
self._logger.info(
"🧹 Project-scope cleanup: local_deleted=%d remote_deleted=%d bindings_deleted=%d",
deleted_local,
deleted_remote,
deleted_bindings,
)
async def _load_node_type(self, node_type: str, *, reason: str) -> None:
self._logger.info(f"🔄 Load node_type={node_type}, reason={reason}")
await self.local_datasource.load_all(order=[node_type])
await self.remote_datasource.load_all(order=[node_type])
if node_type in self._loaded_node_types:
return
self._loaded_node_types.add(node_type)
self.stats["loaded"] += len(self.local_collection.filter(node_type=node_type)) self.stats["loaded"] += len(self.local_collection.filter(node_type=node_type))
self.stats["loaded"] += len(self.remote_collection.filter(node_type=node_type)) self.stats["loaded"] += len(self.remote_collection.filter(node_type=node_type))
def _get_action_success_count(self, datasource: "BaseDataSource", node_type: str, action_key: str) -> int: def _get_action_success_count(self, datasource: "BaseDataSource", node_type: str, action_key: str) -> int:
summary = datasource.get_action_summary().get(node_type, {}) summary = datasource.get_action_summary().get(node_type, {})
action_summary = summary.get(action_key, {}) if isinstance(summary, dict) else {} action_summary = summary.get(action_key, {}) if isinstance(summary, dict) else {}
return int(action_summary.get("success", 0)) if isinstance(action_summary, dict) else 0 return int(action_summary.get("success", 0)) if isinstance(action_summary, dict) else 0
def _should_skip_reload(self) -> bool:
return isinstance(self.local_datasource, JsonlDataSource) and isinstance(self.remote_datasource, JsonlDataSource)
async def _reload_node_type(self, node_type: str, *, reason: str) -> None: async def _reload_node_type(self, node_type: str, *, reason: str) -> None:
self._logger.info(f"🔄 Reload after write: node_type={node_type}, reason={reason}") if self._should_skip_reload():
await self.local_datasource.load_all(order=[node_type]) self._logger.info("⏭️ Reload skipped for JSONL pipeline: node_type=%s reason=%s", node_type, reason)
await self.remote_datasource.load_all(order=[node_type]) return
await self._load_node_type(node_type, reason=reason)
async def _evaluate_consistency_for_node_type(self, node_type: str) -> Dict[str, Any]: async def _evaluate_consistency_for_node_type(self, node_type: str) -> Dict[str, Any]:
strategy = next((item for item in self.strategies if item.node_type == node_type), None) strategy = next((item for item in self.strategies if item.node_type == node_type), None)
@@ -298,28 +386,18 @@ class FullSyncPipeline:
compare_to_remote = True compare_to_remote = True
if strategy and strategy.config.update_direction == UpdateDirection.PULL: if strategy and strategy.config.update_direction == UpdateDirection.PULL:
compare_to_remote = False compare_to_remote = False
ignore_list_item_fields = dict(strategy.config.post_check_ignore_list_item_fields) if strategy else {} ignore_fields = list(strategy.config.post_check_ignore_fields) if strategy else []
# 从远端 handler 的 update_schemas 提取只读字段过滤白名单
post_check_fields: List[str] | None = None
try:
remote_handler = self.remote_datasource.get_handler(node_type)
fields = remote_handler.get_update_fields()
if fields:
post_check_fields = fields
except Exception:
pass
result = await evaluate_consistency_for_node_type( result = await evaluate_consistency_for_node_type(
node_type=node_type, node_type=node_type,
strategy=strategy,
local_collection=self.local_collection, local_collection=self.local_collection,
remote_collection=self.remote_collection, remote_collection=self.remote_collection,
binding_manager=self.binding_manager, binding_manager=self.binding_manager,
logger=self._logger, logger=self._logger,
depend_fields=depend_fields, depend_fields=depend_fields,
compare_to_remote=compare_to_remote, compare_to_remote=compare_to_remote,
ignore_list_item_fields=ignore_list_item_fields, ignore_fields=ignore_fields,
post_check_fields=post_check_fields,
) )
self._consistency_by_type[node_type] = result self._consistency_by_type[node_type] = result
return result return result
@@ -419,6 +497,9 @@ class FullSyncPipeline:
所有类型均已同步完毕后，统一做一次全量 reload（触发后端派生字段计算）， 所有类型均已同步完毕后，统一做一次全量 reload（触发后端派生字段计算），
再对所有类型做一致性评估，确保 post-check 结果反映最终状态。 再对所有类型做一致性评估，确保 post-check 结果反映最终状态。
""" """
if self._should_skip_reload():
self._logger.info("⏭️ Post-check reload skipped for JSONL pipeline")
else:
self._logger.info("🔄 Post-check: reloading all node types for final consistency evaluation...") self._logger.info("🔄 Post-check: reloading all node types for final consistency evaluation...")
for node_type in self.node_types: for node_type in self.node_types:
await self._reload_node_type(node_type, reason="post-check final reload") await self._reload_node_type(node_type, reason="post-check final reload")
@@ -443,11 +524,11 @@ class FullSyncPipeline:
return return
self._logger.info("🔍 Persisting local collection...") self._logger.info("🔍 Persisting local collection...")
await self.local_collection.persist() await self.local_collection.persist(exclude_node_types=self.ephemeral_node_types)
self._logger.info("🔍 Persisting remote collection...") self._logger.info("🔍 Persisting remote collection...")
await self.remote_collection.persist() await self.remote_collection.persist(exclude_node_types=self.ephemeral_node_types)
self._logger.info("🔍 Persisting binding manager...") self._logger.info("🔍 Persisting binding manager...")
await self.binding_manager.persist() await self.binding_manager.persist(exclude_node_types=self.ephemeral_node_types)
self._logger.info("✅ All data persisted successfully") self._logger.info("✅ All data persisted successfully")
# ========== Summary & Reporting ========== # ========== Summary & Reporting ==========
sync_state_machine/pipeline/multi_project_pipeline.py
+166
View File
@@ -0,0 +1,166 @@
from __future__ import annotations
import logging
import copy
from pathlib import Path
from typing import Dict, Optional, List, TypedDict, Any
from ..config import (
MultiProjectItemConfig,
MultiProjectRunConfig,
PipelineRunConfig,
apply_config_overrides,
build_config_from_file,
)
from .factory import create_pipeline_from_config
logger = logging.getLogger(__name__)
class _SharedStateSnapshot(TypedDict):
local_nodes: List[Any]
remote_nodes: List[Any]
bindings_by_type: Dict[str, Dict[str, Optional[str]]]
def _resolve_embedded_config_path(raw_path: str, *, project_root: Path, profile_path: Path) -> Path:
candidate = Path(raw_path)
if candidate.is_absolute():
return candidate
profile_relative = (profile_path.parent / candidate).resolve()
if profile_relative.exists():
return profile_relative
return (project_root / candidate).resolve()
def _project_scope_overrides(item: MultiProjectItemConfig, *, node_types: list[str]) -> Dict[str, object]:
del node_types
local_handler_configs = {
"project": {"data_id_filter": [item.local_project_id]},
}
remote_handler_configs = {
"project": {"data_id_filter": [item.remote_project_id]},
}
return {
"local_datasource": {
"handler_configs": local_handler_configs,
},
"remote_datasource": {
"handler_configs": remote_handler_configs,
},
}
def _resolve_project_bootstrap_node_types(
*,
config: MultiProjectRunConfig,
shared_node_types: list[str],
) -> list[str]:
configured = [node_type for node_type in config.project_bootstrap_node_types if node_type]
if configured:
return list(dict.fromkeys(configured))
return list(dict.fromkeys(shared_node_types))
class MultiProjectPipeline:
def __init__(self, *, project_root: Path, config_path: Path, config: MultiProjectRunConfig):
self.project_root = project_root
self.config_path = config_path
self.config = config
def _load_pipeline_config(self, raw_path: str) -> PipelineRunConfig:
resolved = _resolve_embedded_config_path(raw_path, project_root=self.project_root, profile_path=self.config_path)
return build_config_from_file(project_root=self.project_root, file_path=resolved)
def _prepare_project_config(
self,
*,
shared_persist: Dict[str, object],
item: MultiProjectItemConfig,
base_path: Optional[str],
) -> PipelineRunConfig:
raw_path = base_path or self.config.default_project_config
cfg = self._load_pipeline_config(raw_path)
cfg = apply_config_overrides(cfg, {"persist": {**shared_persist, "wipe_on_start": False}})
return apply_config_overrides(cfg, _project_scope_overrides(item, node_types=list(cfg.node_types)))
async def _snapshot_shared_state(self, *, pipeline, node_types: List[str]) -> _SharedStateSnapshot:
local_nodes = [
copy.deepcopy(node)
for node_type in node_types
for node in pipeline.local_collection.filter(node_type=node_type)
]
remote_nodes = [
copy.deepcopy(node)
for node_type in node_types
for node in pipeline.remote_collection.filter(node_type=node_type)
]
bindings_by_type: Dict[str, Dict[str, Optional[str]]] = {}
for node_type in node_types:
bindings = await pipeline.binding_manager.get_all_bindings(node_type)
bindings_by_type[node_type] = {local_id: remote_id for local_id, remote_id in bindings}
return {
"local_nodes": local_nodes,
"remote_nodes": remote_nodes,
"bindings_by_type": bindings_by_type,
}
async def run(self) -> Dict[str, Dict[str, object]]:
results: Dict[str, Dict[str, object]] = {}
global_config = self._load_pipeline_config(self.config.global_config)
shared_persist = global_config.persist.model_dump(mode="json")
shared_node_types = list(global_config.node_types)
project_bootstrap_node_types = _resolve_project_bootstrap_node_types(
config=self.config,
shared_node_types=shared_node_types,
)
logger.info("🚀 Multi-project pipeline start: projects=%d", len(self.config.projects))
global_pipeline = await create_pipeline_from_config(
global_config,
scope="global",
pipeline_name="multi-project pipeline [global]",
)
try:
results["global"] = await global_pipeline.run()
shared_state = await self._snapshot_shared_state(
pipeline=global_pipeline,
node_types=project_bootstrap_node_types,
)
finally:
await global_pipeline.close()
for item in self.config.projects:
project_config = self._prepare_project_config(
shared_persist=shared_persist,
item=item,
base_path=item.config_path,
)
scope = item.scope
title = f"multi-project pipeline [{scope}]"
logger.info(
"🚀 Project pipeline start: scope=%s local_project_id=%s remote_project_id=%s",
scope,
item.local_project_id,
item.remote_project_id,
)
project_pipeline = await create_pipeline_from_config(
project_config,
scope=scope,
pipeline_name=title,
bootstrap_binding_node_types=project_bootstrap_node_types,
ephemeral_node_types=project_bootstrap_node_types,
)
try:
project_pipeline.preload_shared_state(
local_nodes=shared_state["local_nodes"],
remote_nodes=shared_state["remote_nodes"],
bindings_by_type=shared_state["bindings_by_type"],
)
results[scope] = await project_pipeline.run()
finally:
await project_pipeline.close()
return results
sync_state_machine/pipeline/summary_report.py
+45 -24
View File
@@ -28,6 +28,43 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
return text[:width] return text[:width]
return text[: width - 1] + "…" return text[: width - 1] + "…"
def print_consistency_summary() -> None:
line_width = 96
def _format_sample(sample: dict) -> str:
local_text = _clip(repr(sample.get("local")), 72)
remote_text = _clip(repr(sample.get("remote")), 72)
return f"{local_text}/{remote_text}"
logger.info(f"\n{'=' * line_width}")
logger.info("🔎 Consistency Summary")
logger.info(f"{'=' * line_width}")
for node_type in node_types:
item = consistency_by_type.get(node_type, {}) or {}
checked_pairs = int(item.get("checked_pairs", 0))
mismatch_count = int(item.get("mismatch_count", 0))
logger.info(f"{node_type} (checked_pairs={checked_pairs}, mismatches={mismatch_count})")
fields = item.get("fields", []) or []
differing_fields = [field for field in fields if int(field.get("mismatch_count", 0)) > 0]
if not differing_fields:
logger.info(" pass")
continue
logger.info(" field mismatch/total samples")
logger.info(" -----------------------------------------------------------")
for field_report in differing_fields:
field_name = str(field_report.get("field_name", ""))
field_mismatch = int(field_report.get("mismatch_count", 0))
field_total = int(field_report.get("total_count", 0))
samples = field_report.get("samples", []) or []
sample_text = _format_sample(samples[0]) if samples else "-"
logger.info(
f" {_clip(field_name, 30):<30} {field_mismatch:>4}/{field_total:<4} "
f"{sample_text}"
)
def print_collection_summary(title: str, datasource, collection) -> None: def print_collection_summary(title: str, datasource, collection) -> None:
action_summary = datasource.get_action_summary() action_summary = datasource.get_action_summary()
line_width = 133 line_width = 133
@@ -110,20 +147,17 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
f"{_clip(total_consistency_sft, consistency_w):<{consistency_w}}" f"{_clip(total_consistency_sft, consistency_w):<{consistency_w}}"
) )
print_collection_summary("Local", local_datasource, local_collection)
print_collection_summary("Remote", remote_datasource, remote_collection)
loaded = int(stats.get("loaded", 0))
synced = int(stats.get("synced", 0))
failed = int(stats.get("failed", 0))
post_check_passed = bool(stats.get("post_check_passed", True))
mismatch_node_types = [ mismatch_node_types = [
node_type node_type
for node_type, item in consistency_by_type.items() for node_type, item in consistency_by_type.items()
if int(item.get("mismatch_count", 0)) > 0 if int(item.get("mismatch_count", 0)) > 0
] ]
loaded = int(stats.get("loaded", 0))
synced = int(stats.get("synced", 0))
failed = int(stats.get("failed", 0))
post_check_passed = bool(stats.get("post_check_passed", True))
logger.info("\n" + "=" * 96) logger.info("\n" + "=" * 96)
logger.info( logger.info(
f"📈 Pipeline Stats: loaded={loaded}, synced={synced}, failed={failed}, post_check_passed={post_check_passed}, mismatch_types={len(mismatch_node_types)}" f"📈 Pipeline Stats: loaded={loaded}, synced={synced}, failed={failed}, post_check_passed={post_check_passed}, mismatch_types={len(mismatch_node_types)}"
@@ -132,19 +166,6 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
logger.info(f"🔎 Mismatch Node Types: {', '.join(mismatch_node_types)}") logger.info(f"🔎 Mismatch Node Types: {', '.join(mismatch_node_types)}")
logger.info("=" * 96) logger.info("=" * 96)
sample_per_type = 3 print_consistency_summary()
has_mismatch_samples = any(int(item.get("mismatch_count", 0)) > 0 for item in consistency_by_type.values()) print_collection_summary("Local", local_datasource, local_collection)
if has_mismatch_samples: print_collection_summary("Remote", remote_datasource, remote_collection)
logger.info("\n" + "=" * 96)
logger.info("🧪 Inconsistency Samples (per type)")
logger.info("=" * 96)
for node_type in node_types:
item = consistency_by_type.get(node_type, {})
mismatch_count = int(item.get("mismatch_count", 0))
if mismatch_count <= 0:
continue
checked_pairs = int(item.get("checked_pairs", 0))
samples = item.get("samples", []) or []
logger.info(f"- {node_type}: mismatches={mismatch_count}, checked_pairs={checked_pairs}")
for sample in samples[:sample_per_type]:
logger.info(f" • {sample}")
sync_state_machine/sync_system/resolve_ids.py
+145 -29
View File
@@ -221,6 +221,22 @@ class IDResolver:
# 判断是否是ID字段 # 判断是否是ID字段
is_id_field = field_name.endswith("_id") or field_name == "id" is_id_field = field_name.endswith("_id") or field_name == "id"
is_primary_id_field = field_name in ("id", "_id") is_primary_id_field = field_name in ("id", "_id")
is_depend_field = field_name in depend_field_names
if isinstance(value, list):
if not is_depend_field:
continue
data_dict[field_name] = await self._resolve_list_id_field_value(
value=value,
hint=id_field_hints.get(field_name),
node=node,
node_type=node_type,
field_name=field_name,
to_remote=to_remote,
logger=logger,
)
continue
if not is_id_field: if not is_id_field:
continue continue
@@ -230,9 +246,6 @@ class IDResolver:
if value == "": if value == "":
continue continue
# 判断是否是依赖字段
is_depend_field = field_name in depend_field_names
# 获取node_type提示 # 获取node_type提示
hint = id_field_hints.get(field_name) hint = id_field_hints.get(field_name)
@@ -240,33 +253,37 @@ class IDResolver:
if is_primary_id_field and not hint: if is_primary_id_field and not hint:
hint = node_type hint = node_type
if isinstance(value, list):
if not is_depend_field:
continue
data_dict[field_name] = await self._resolve_list_id_field_value(
value=value,
hint=hint,
node=node,
node_type=node_type,
field_name=field_name,
to_remote=to_remote,
logger=logger,
)
continue
if not is_id_field and not is_depend_field:
continue
# 解析ID # 解析ID
target_id = await self.resolve_id(str(value), hint, to_remote=to_remote) target_value = await self._resolve_id_field_value(
value=value,
if target_id is None: hint=hint,
# to_remote=True表示从local到remote，所以node是local节点 node=node,
# to_remote=False表示从remote到local，所以node是remote节点 node_type=node_type,
collection_name = "local" if to_remote else "remote" field_name=field_name,
node_data_id = node.data_id or "N/A" is_depend_field=is_depend_field,
direction = "remote" if to_remote else "local" is_primary_id_field=is_primary_id_field,
to_remote=to_remote,
if is_depend_field: logger=logger,
logger.warning(
f"[{node_type}] {collection_name.capitalize()} node {node.node_id} "
f"(data_id={node_data_id}): Dependency field '{field_name}' "
f"with value '{value}' failed to resolve to {direction} ID"
) )
else: data_dict[field_name] = target_value
if not is_primary_id_field:
logger.warning(
f"[{node_type}] {collection_name.capitalize()} node {node.node_id} "
f"(data_id={node_data_id}): Non-dependency ID field '{field_name}' "
f"with value '{value}' failed to resolve to {direction} ID, setting to empty"
)
data_dict[field_name] = ""
else:
# 成功解析，更新字段值
data_dict[field_name] = target_id
# 将修改后的数据写回节点 # 将修改后的数据写回节点
node.set_data(data_dict) node.set_data(data_dict)
@@ -345,9 +362,12 @@ class IDResolver:
original_value = data.get(field_name) original_value = data.get(field_name)
resolved_value = resolved_data.get(field_name) resolved_value = resolved_data.get(field_name)
if isinstance(original_value, list):
continue
# 原本有值（源端 ID），但解析后为空（找不到目标端 ID） # 原本有值（源端 ID），但解析后为空（找不到目标端 ID）
# 说明依赖节点虽然在源端存在，但在目标端还没就绪 # 说明依赖节点虽然在源端存在，但在目标端还没就绪
if original_value and not resolved_value: if self._has_unresolved_dependency_value(original_value, resolved_value):
failed_fields.append(f"{field_name}({dep_type})={original_value}") failed_fields.append(f"{field_name}({dep_type})={original_value}")
if failed_fields: if failed_fields:
@@ -356,3 +376,99 @@ class IDResolver:
return None, error_msg return None, error_msg
return resolved_data, None return resolved_data, None
@staticmethod
def _has_unresolved_dependency_value(original_value: Any, resolved_value: Any) -> bool:
return bool(original_value) and not resolved_value
async def _resolve_id_field_value(
self,
*,
value: Any,
hint: Optional[str],
node: "SyncNode",
node_type: str,
field_name: str,
is_depend_field: bool,
is_primary_id_field: bool,
to_remote: bool,
logger,
) -> Any:
if isinstance(value, list):
return await self._resolve_list_id_field_value(
value=value,
hint=hint,
node=node,
node_type=node_type,
field_name=field_name,
to_remote=to_remote,
logger=logger,
)
target_id = await self.resolve_id(str(value), hint, to_remote=to_remote)
if target_id is None:
self._log_id_resolution_failure(
logger=logger,
node=node,
node_type=node_type,
field_name=field_name,
value=value,
is_depend_field=is_depend_field,
is_primary_id_field=is_primary_id_field,
to_remote=to_remote,
)
return ""
return target_id
async def _resolve_list_id_field_value(
self,
*,
value: list[Any],
hint: Optional[str],
node: "SyncNode",
node_type: str,
field_name: str,
to_remote: bool,
logger,
) -> list[Any]:
resolved_items: list[Any] = []
for item in value:
if item is None or item == "":
continue
target_id = await self.resolve_id(str(item), hint, to_remote=to_remote)
if target_id is not None:
resolved_items.append(target_id)
return resolved_items
@staticmethod
def _log_id_resolution_failure(
*,
logger,
node: "SyncNode",
node_type: str,
field_name: str,
value: Any,
is_depend_field: bool,
is_primary_id_field: bool,
to_remote: bool,
) -> None:
collection_name = "local" if to_remote else "remote"
node_data_id = node.data_id or "N/A"
direction = "remote" if to_remote else "local"
if is_depend_field:
logger.warning(
f"[{node_type}] {collection_name.capitalize()} node {node.node_id} "
f"(data_id={node_data_id}): Dependency field '{field_name}' "
f"with value '{value}' failed to resolve to {direction} ID"
)
elif not is_primary_id_field:
logger.warning(
f"[{node_type}] {collection_name.capitalize()} node {node.node_id} "
f"(data_id={node_data_id}): Non-dependency ID field '{field_name}' "
f"with value '{value}' failed to resolve to {direction} ID, setting to empty"
)
sync_state_machine/sync_system/strategy.py
+41
View File
@@ -19,6 +19,8 @@ from ..engine import (
StateMachineConfig, StateMachineConfig,
StateMachineRuntime, StateMachineRuntime,
) )
from ..validation import SchemaDiffValidator
from .strategy_ops.compare_ops import normalized_data_for_compare
T = TypeVar("T", bound=BaseModel) T = TypeVar("T", bound=BaseModel)
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -70,6 +72,7 @@ class BaseSyncStrategy(Generic[T]):
self.skip_sync = self.default_skip_sync self.skip_sync = self.default_skip_sync
# 统一由 Pipeline 注入 runtime；未注入时在执行阶段显式报错 # 统一由 Pipeline 注入 runtime；未注入时在执行阶段显式报错
self.sm_runtime: Optional[StateMachineRuntime] = None self.sm_runtime: Optional[StateMachineRuntime] = None
self._schema_diff_validator: Optional[SchemaDiffValidator] = None
# 配置逻辑校验 # 配置逻辑校验
self._validate_config() self._validate_config()
@@ -145,6 +148,7 @@ class BaseSyncStrategy(Generic[T]):
def set_config(self, config: StrategyConfig): def set_config(self, config: StrategyConfig):
"""完全替换配置对象""" """完全替换配置对象"""
self.config = config self.config = config
self._schema_diff_validator = None
self._validate_config() self._validate_config()
def update_config(self, **kwargs): def update_config(self, **kwargs):
@@ -163,6 +167,7 @@ class BaseSyncStrategy(Generic[T]):
setattr(self.config, key, value) setattr(self.config, key, value)
else: else:
logger.warning(f"[{self.node_type}] Unknown config key: {key}") logger.warning(f"[{self.node_type}] Unknown config key: {key}")
self._schema_diff_validator = None
# 更新配置后也要校验 # 更新配置后也要校验
self._validate_config() self._validate_config()
@@ -178,9 +183,45 @@ class BaseSyncStrategy(Generic[T]):
""" """
preset_config = ConfigPresets.get_preset(preset_name) preset_config = ConfigPresets.get_preset(preset_name)
self.config = preset_config self.config = preset_config
self._schema_diff_validator = None
self._validate_config() self._validate_config()
logger.info(f"[{self.node_type}] Applied preset: {preset_name}") logger.info(f"[{self.node_type}] Applied preset: {preset_name}")
def preprocess_compare_payload(self, payload: Dict[str, Any]) -> Dict[str, Any]:
return payload
def normalize_compare_payload(
self,
data: Optional[Dict[str, Any]],
data_id_map: Optional[Dict[str, str]] = None,
*,
ignore_fields: Optional[set[str]] = None,
) -> Dict[str, Any]:
prepared = self.preprocess_compare_payload(dict(data or {}))
return normalized_data_for_compare(
prepared,
data_id_map,
ignore_fields=ignore_fields,
)
def get_schema_diff_validator(self) -> SchemaDiffValidator:
if self._schema_diff_validator is None:
self._schema_diff_validator = SchemaDiffValidator(
schema_name=self.schema.__name__,
ignore_fields=self.config.compare_ignore_fields,
)
return self._schema_diff_validator
def reset_schema_diff_validator(self) -> None:
self.get_schema_diff_validator().reset()
def emit_schema_diff_report(self) -> None:
validator = self.get_schema_diff_validator()
if not validator.has_records():
return
for line in validator.format_summary_lines():
logger.info(f"[{self.node_type}] validate {line}")
async def bind(self, **kwargs): async def bind(self, **kwargs):
"""进行绑定逻辑判定，设置 node.binding_status 和 node.action""" """进行绑定逻辑判定，设置 node.binding_status 和 node.action"""
raise NotImplementedError raise NotImplementedError
sync_state_machine/sync_system/strategy_ops/bind_ops.py
+17 -3
View File
@@ -155,6 +155,14 @@ def check_dependency_satisfied(
return True return True
def _normalize_dependency_field_values(value: Any) -> list[Any]:
if isinstance(value, list):
return [item for item in value if item not in (None, "")]
if value in (None, ""):
return []
return [value]
def collect_dependency_errors( def collect_dependency_errors(
dependency_nodes: List[Optional["SyncNode"]], dependency_nodes: List[Optional["SyncNode"]],
field_names: List[str], field_names: List[str],
@@ -211,15 +219,21 @@ async def check_dependencies_for_nodes(strategy, nodes: List["SyncNode"], collec
if has_depend_fields and data_present: if has_depend_fields and data_present:
for field_name, dep_node_type in strategy.config.depend_fields.items(): for field_name, dep_node_type in strategy.config.depend_fields.items():
dep_data_id = node_data.get(field_name) dep_data_values = _normalize_dependency_field_values(node_data.get(field_name))
if not dep_data_id: if not dep_data_values:
continue continue
is_list_field = isinstance(node_data.get(field_name), list)
for dep_data_id in dep_data_values:
dep_node = collection.get_by_data_id(dep_node_type, dep_data_id) dep_node = collection.get_by_data_id(dep_node_type, dep_data_id)
if dep_node is None: if dep_node is None:
maybe_node_id_ref = collection.get_by_node_id(str(dep_data_id)) maybe_node_id_ref = collection.get_by_node_id(str(dep_data_id))
if maybe_node_id_ref is not None and maybe_node_id_ref.node_type == dep_node_type: if maybe_node_id_ref is not None and maybe_node_id_ref.node_type == dep_node_type:
semantic_errors.append(f"依赖字段 {field_name} 使用了 node_id={dep_data_id}，期望 data_id 引用") semantic_errors.append(
f"依赖字段 {field_name} 使用了 node_id={dep_data_id}，期望 data_id 引用"
)
if is_list_field:
continue
depend_nodes.append(dep_node) depend_nodes.append(dep_node)
depend_field_names.append(field_name) depend_field_names.append(field_name)
sync_state_machine/sync_system/strategy_ops/compare_ops.py
+12 -1
View File
@@ -39,7 +39,11 @@ def is_id_like_field(field_name: Optional[str]) -> bool:
return lowered.endswith("_id") or lowered.endswith("_ids") return lowered.endswith("_id") or lowered.endswith("_ids")
def normalize_compare_value(field_name: Optional[str], value: Any, data_id_map: Dict[str, str]) -> Any: def normalize_compare_value(
field_name: Optional[str],
value: Any,
data_id_map: Dict[str, str],
) -> Any:
if isinstance(value, dict): if isinstance(value, dict):
return { return {
key: normalize_compare_value(key, nested, data_id_map) key: normalize_compare_value(key, nested, data_id_map)
@@ -59,14 +63,17 @@ def normalize_compare_value(field_name: Optional[str], value: Any, data_id_map:
def normalized_data_for_compare( def normalized_data_for_compare(
data: Optional[Dict[str, Any]], data: Optional[Dict[str, Any]],
data_id_map: Optional[Dict[str, str]] = None, data_id_map: Optional[Dict[str, str]] = None,
ignore_fields: Optional[set[str]] = None,
) -> Dict[str, Any]: ) -> Dict[str, Any]:
payload = copy.deepcopy(data or {}) payload = copy.deepcopy(data or {})
payload.pop("id", None) payload.pop("id", None)
payload.pop("_id", None) payload.pop("_id", None)
ignored = ignore_fields or set()
mapping = data_id_map or {} mapping = data_id_map or {}
return { return {
key: normalize_compare_value(key, value, mapping) key: normalize_compare_value(key, value, mapping)
for key, value in payload.items() for key, value in payload.items()
if key not in ignored
} }
@@ -75,10 +82,14 @@ def collect_payload_diffs(
target_payload: Dict[str, Any], target_payload: Dict[str, Any],
*, *,
max_items: int = 6, max_items: int = 6,
ignore_fields: Optional[set[str]] = None,
) -> Dict[str, Dict[str, Any]]: ) -> Dict[str, Dict[str, Any]]:
out: Dict[str, Dict[str, Any]] = {} out: Dict[str, Dict[str, Any]] = {}
ignored = ignore_fields or set()
keys = list(dict.fromkeys(list(source_payload.keys()) + list(target_payload.keys()))) keys = list(dict.fromkeys(list(source_payload.keys()) + list(target_payload.keys())))
for key in keys: for key in keys:
if key in ignored:
continue
source_value = source_payload.get(key) source_value = source_payload.get(key)
target_value = target_payload.get(key) target_value = target_payload.get(key)
if source_value == target_value: if source_value == target_value:
sync_state_machine/sync_system/strategy_ops/post_check_ops.py
+39 -64
View File
@@ -3,35 +3,8 @@ from __future__ import annotations
from typing import Any, Dict, List from typing import Any, Dict, List
from ...sync_system.resolve_ids import IDResolver from ...sync_system.resolve_ids import IDResolver
from .compare_ops import ( from ...validation import SchemaDiffValidator
build_data_id_normalization_map, from .compare_ops import build_data_id_normalization_map
normalized_data_for_compare,
collect_payload_diffs,
)
def _strip_list_item_fields(payload: Dict, ignore_map: Dict[str, List[str]]) -> Dict:
"""从 payload 中列表型字段的每个元素里，剔除指定子键。
例如 ignore_map={"plan_node": ["is_audit"]} 会把 payload["plan_node"][*]["is_audit"] 全部去掉。
payload 本身不会被修改（浅拷贝）。
"""
if not ignore_map:
return payload
result = dict(payload)
for field_name, sub_keys in ignore_map.items():
if field_name not in result:
continue
items = result[field_name]
if not isinstance(items, list):
continue
stripped = []
for item in items:
if isinstance(item, dict):
item = {k: v for k, v in item.items() if k not in sub_keys}
stripped.append(item)
result[field_name] = stripped
return result
def _compact_repr(value: Any, *, max_len: int = 80) -> str: def _compact_repr(value: Any, *, max_len: int = 80) -> str:
@@ -54,20 +27,23 @@ def _append_post_check_error(node, message: str) -> None:
async def evaluate_consistency_for_node_type( async def evaluate_consistency_for_node_type(
*, *,
node_type: str, node_type: str,
strategy,
local_collection, local_collection,
remote_collection, remote_collection,
binding_manager, binding_manager,
logger, logger,
depend_fields: Dict[str, str] | None = None, depend_fields: Dict[str, str] | None = None,
compare_to_remote: bool = True, compare_to_remote: bool = True,
ignore_list_item_fields: Dict[str, List[str]] | None = None, ignore_fields: List[str] | None = None,
post_check_fields: List[str] | None = None,
) -> Dict[str, Any]: ) -> Dict[str, Any]:
records = await binding_manager.get_all_records(node_type) records = await binding_manager.get_all_records(node_type)
mismatches: List[str] = []
checked_pairs = 0 checked_pairs = 0
depend_fields = dict(depend_fields or {}) depend_fields = dict(depend_fields or {})
id_resolver = IDResolver(local_collection, remote_collection, binding_manager) id_resolver = IDResolver(local_collection, remote_collection, binding_manager)
validator = SchemaDiffValidator(
schema_name=node_type,
ignore_fields=ignore_fields,
)
data_id_map = await build_data_id_normalization_map( data_id_map = await build_data_id_normalization_map(
node_type=node_type, node_type=node_type,
local_collection=local_collection, local_collection=local_collection,
@@ -117,43 +93,42 @@ async def evaluate_consistency_for_node_type(
if resolved_remote is not None: if resolved_remote is not None:
remote_data = resolved_remote remote_data = resolved_remote
local_payload = normalized_data_for_compare(local_data, data_id_map) local_payload = strategy.normalize_compare_payload(local_data, data_id_map)
remote_payload = normalized_data_for_compare(remote_data, data_id_map) remote_payload = strategy.normalize_compare_payload(remote_data, data_id_map)
if post_check_fields: diff_map = validator.compare_payloads(
local_payload = {k: v for k, v in local_payload.items() if k in post_check_fields} local_payload,
remote_payload = {k: v for k, v in remote_payload.items() if k in post_check_fields} remote_payload,
if ignore_list_item_fields: sample_id=f"{local_id}->{remote_id}",
local_payload = _strip_list_item_fields(local_payload, ignore_list_item_fields) schema_refs=[node_type],
remote_payload = _strip_list_item_fields(remote_payload, ignore_list_item_fields) )
if local_payload != remote_payload: if diff_map:
diff_map = collect_payload_diffs(local_payload, remote_payload) detail_parts = []
diff_parts = []
for field_name, values in diff_map.items(): for field_name, values in diff_map.items():
local_text = _compact_repr(values.get("source")) local_text = _compact_repr(values.get("local"))
remote_text = _compact_repr(values.get("target")) remote_text = _compact_repr(values.get("remote"))
diff_parts.append(f"{field_name}{{local:{local_text},remote:{remote_text}}}") detail_parts.append(f"{field_name}{{local:{local_text},remote:{remote_text}}}")
detail = "; ".join(diff_parts) if diff_parts else "payload differs" detail = "; ".join(detail_parts) if detail_parts else "payload differs"
mismatch_text = f"pair_diff(local_id={local_id}, remote_id={remote_id}, {detail})"
mismatches.append(mismatch_text)
_append_post_check_error(local_node, f"post_check diff: {detail}") _append_post_check_error(local_node, f"post_check diff: {detail}")
_append_post_check_error(remote_node, f"post_check diff: {detail}") _append_post_check_error(remote_node, f"post_check diff: {detail}")
result = { report = validator.build_report() if checked_pairs else {
"ok": len(mismatches) == 0, "schema_name": node_type,
"checked_pairs": checked_pairs, "pairs_compared": 0,
"mismatch_count": len(mismatches), "mismatched_pairs": 0,
"samples": mismatches[:10], "ignore_fields": [],
"fields": [],
} }
if result["ok"]: # 将字段统计补回到结果里，供 summary 输出。
logger.info(f"✅ Consistency check passed: node_type={node_type}, checked_pairs={checked_pairs}") report["fields"] = report.get("fields", [])
else:
logger.error( result = {
f"❌ Consistency check failed: node_type={node_type}, checked_pairs={checked_pairs}, mismatches={len(mismatches)}" "ok": report["mismatched_pairs"] == 0,
) "checked_pairs": checked_pairs,
logger.error(" ↳ consistent语义: 比较绑定对(local_node.data vs remote_node.data)，非 original_data") "mismatch_count": report["mismatched_pairs"],
for item in result["samples"]: "fields": report["fields"],
logger.error(f" - {item}") "schema_name": report["schema_name"],
"ignore_fields": report["ignore_fields"],
}
return result return result
sync_state_machine/sync_system/strategy_ops/update_ops.py
+51 -27
View File
@@ -39,24 +39,6 @@ def _safe_repr(value, *, max_len: int = 120) -> str:
return text[: max_len - 3] + "..." return text[: max_len - 3] + "..."
def _collect_diff_descriptions(source_data, target_data, *, max_items: int = 8) -> List[str]:
if source_data is None or target_data is None:
return []
descriptions: List[str] = []
for key, source_value in source_data.items():
if key in {"id", "_id"}:
continue
target_value = target_data.get(key)
if source_value == target_value:
continue
descriptions.append(f"{key}: {_safe_repr(target_value)} -> {_safe_repr(source_value)}")
if len(descriptions) >= max_items:
break
return descriptions
def default_needs_update( def default_needs_update(
strategy, strategy,
source_node, source_node,
@@ -72,13 +54,27 @@ def default_needs_update(
if source_data is None or target_data is None: if source_data is None or target_data is None:
return False return False
normalized_source = normalized_data_for_compare(source_data, data_id_map) ignore_fields = set(strategy.config.compare_ignore_fields)
normalized_target = normalized_data_for_compare(target_data, data_id_map) normalized_source = strategy.normalize_compare_payload(
diff_map = collect_payload_diffs(normalized_source, normalized_target, max_items=8) source_data,
data_id_map,
ignore_fields=ignore_fields,
)
normalized_target = strategy.normalize_compare_payload(
target_data,
data_id_map,
ignore_fields=ignore_fields,
)
diff_map = strategy.get_schema_diff_validator().compare_payloads(
normalized_source,
normalized_target,
sample_id=str(target_node.data_id or target_node.node_id),
schema_refs=[strategy.schema.__name__],
)
is_different = bool(diff_map) is_different = bool(diff_map)
if is_different and diff_map: if is_different and diff_map:
previews = [f"{key}: {values.get('source')} != {values.get('target')}" for key, values in list(diff_map.items())[:5]] previews = [f"{key}: {values.get('local')} != {values.get('remote')}" for key, values in list(diff_map.items())[:5]]
logger.debug(f"[{strategy.node_type}] Node diff detected: {', '.join(previews)}") logger.debug(f"[{strategy.node_type}] Node diff detected: {', '.join(previews)}")
return is_different return is_different
@@ -88,13 +84,34 @@ def resolve_needs_update_check(strategy) -> Callable:
return strategy._needs_update return strategy._needs_update
def _collect_diff_descriptions(source_data, target_data, *, data_id_map, max_items: int = 8) -> List[str]: def _collect_diff_descriptions(
strategy,
source_data,
target_data,
*,
data_id_map,
ignore_fields: set[str],
max_items: int = 8,
) -> List[str]:
if source_data is None or target_data is None: if source_data is None or target_data is None:
return [] return []
normalized_source = normalized_data_for_compare(source_data, data_id_map) normalized_source = strategy.normalize_compare_payload(
normalized_target = normalized_data_for_compare(target_data, data_id_map) source_data,
diff_map = collect_payload_diffs(normalized_source, normalized_target, max_items=max_items) data_id_map,
ignore_fields=ignore_fields,
)
normalized_target = strategy.normalize_compare_payload(
target_data,
data_id_map,
ignore_fields=ignore_fields,
)
diff_map = collect_payload_diffs(
normalized_source,
normalized_target,
max_items=max_items,
ignore_fields=ignore_fields,
)
descriptions: List[str] = [] descriptions: List[str] = []
for key, values in diff_map.items(): for key, values in diff_map.items():
@@ -184,7 +201,13 @@ async def add_update_action(
should_update = needs_update_check(src_node) should_update = needs_update_check(src_node)
has_diff = bool(should_update) has_diff = bool(should_update)
if has_diff: if has_diff:
diff_descriptions = _collect_diff_descriptions(data, target_node.get_data(), data_id_map=data_id_map or {}) diff_descriptions = _collect_diff_descriptions(
strategy,
data,
target_node.get_data(),
data_id_map=data_id_map or {},
ignore_fields=set(strategy.config.compare_ignore_fields),
)
if steps_changed: if steps_changed:
has_diff = True has_diff = True
@@ -226,6 +249,7 @@ async def add_update_action(
async def run_update(strategy) -> List["SyncNode"]: async def run_update(strategy) -> List["SyncNode"]:
updated_nodes: List["SyncNode"] = [] updated_nodes: List["SyncNode"] = []
strategy.reset_schema_diff_validator()
needs_update_check = resolve_needs_update_check(strategy) needs_update_check = resolve_needs_update_check(strategy)
data_id_map = await build_data_id_normalization_map( data_id_map = await build_data_id_normalization_map(
node_type=strategy.node_type, node_type=strategy.node_type,
sync_state_machine/validation/__init__.py
+3
View File
@@ -0,0 +1,3 @@
from .schema_diff import SchemaDiffValidator
__all__ = ["SchemaDiffValidator"]
sync_state_machine/validation/schema_diff.py
+146
View File
@@ -0,0 +1,146 @@
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Iterable
from ..sync_system.strategy_ops.compare_ops import normalized_data_for_compare
@dataclass
class SchemaDiffSample:
sample_id: str
local: Any
remote: Any
@dataclass
class SchemaFieldDiffStat:
field_name: str
compared_count: int = 0
mismatch_count: int = 0
schema_refs: set[str] = field(default_factory=set)
samples: list[SchemaDiffSample] = field(default_factory=list)
class SchemaDiffValidator:
def __init__(
self,
*,
schema_name: str,
ignore_fields: Iterable[str] | None = None,
sample_limit: int = 3,
) -> None:
self.schema_name = schema_name
self.ignore_fields = {str(field) for field in (ignore_fields or []) if str(field)}
self.sample_limit = max(1, sample_limit)
self.reset()
def reset(self) -> None:
self.pairs_compared = 0
self.mismatched_pairs = 0
self._field_stats: dict[str, SchemaFieldDiffStat] = {}
def has_records(self) -> bool:
return self.pairs_compared > 0
def compare_payloads(
self,
local_payload: Mapping[str, Any] | None,
remote_payload: Mapping[str, Any] | None,
*,
sample_id: str | None = None,
schema_refs: Iterable[str] | None = None,
) -> dict[str, dict[str, Any]]:
local_normalized = normalized_data_for_compare(
dict(local_payload or {}),
ignore_fields=self.ignore_fields,
)
remote_normalized = normalized_data_for_compare(
dict(remote_payload or {}),
ignore_fields=self.ignore_fields,
)
field_names = sorted(set(local_normalized) | set(remote_normalized))
diff_map: dict[str, dict[str, Any]] = {}
refs = {self.schema_name, *(str(item) for item in (schema_refs or []) if str(item))}
self.pairs_compared += 1
current_sample_id = sample_id or f"pair-{self.pairs_compared}"
for field_name in field_names:
stat = self._field_stats.setdefault(field_name, SchemaFieldDiffStat(field_name=field_name))
stat.compared_count += 1
stat.schema_refs.update(refs)
local_value = local_normalized.get(field_name)
remote_value = remote_normalized.get(field_name)
if local_value == remote_value:
continue
stat.mismatch_count += 1
if len(stat.samples) < self.sample_limit:
stat.samples.append(
SchemaDiffSample(
sample_id=current_sample_id,
local=local_value,
remote=remote_value,
)
)
diff_map[field_name] = {"local": local_value, "remote": remote_value}
if diff_map:
self.mismatched_pairs += 1
return diff_map
def build_report(self) -> dict[str, Any]:
fields = []
for stat in sorted(
self._field_stats.values(),
key=lambda item: (-item.mismatch_count, item.field_name),
):
if stat.compared_count <= 0:
continue
fields.append(
{
"field_name": stat.field_name,
"mismatch_count": stat.mismatch_count,
"total_count": stat.compared_count,
"mismatch_rate": stat.mismatch_count / stat.compared_count,
"schema_refs": sorted(stat.schema_refs),
"samples": [
{
"sample_id": sample.sample_id,
"local": sample.local,
"remote": sample.remote,
}
for sample in stat.samples
],
}
)
return {
"schema_name": self.schema_name,
"pairs_compared": self.pairs_compared,
"mismatched_pairs": self.mismatched_pairs,
"ignore_fields": sorted(self.ignore_fields),
"fields": fields,
}
def format_summary_lines(self, *, max_fields: int = 20, max_samples: int = 1) -> list[str]:
report = self.build_report()
lines = [
(
f"schema={report['schema_name']} pairs={report['pairs_compared']} "
f"mismatched_pairs={report['mismatched_pairs']} ignored={report['ignore_fields']}"
)
]
for field_report in report["fields"][:max_fields]:
line = (
f"field={field_report['field_name']} mismatch={field_report['mismatch_count']}/"
f"{field_report['total_count']} refs={field_report['schema_refs']}"
)
samples = field_report["samples"][:max_samples]
if samples:
sample = samples[0]
line += f" sample[{sample['sample_id']}]={sample['local']!r}/{sample['remote']!r}"
lines.append(line)
return lines
tests/README.md
+3 -1
View File
@@ -12,10 +12,12 @@
- 工具链专项：`pytest tests/integration/test_toolchain_config_and_graph.py -q` - 工具链专项：`pytest tests/integration/test_toolchain_config_and_graph.py -q`
- UI 专项：`pytest tests/integration/test_ui_api_smoke.py -q` - UI 专项：`pytest tests/integration/test_ui_api_smoke.py -q`
- 测试总览与远程环境工具：`docs/testing_guide.md` - 测试总览与远程环境工具：`docs/testing_guide.md`
- 最近常用的定向回归：`pytest tests/unit/test_update_ops.py tests/unit/test_post_check_dependency_id_normalization.py -q`
## 设计原则 ## 设计原则
- 不绕过 schema/状态机校验；配置错误必须显式失败。 - 不绕过 schema/状态机校验；配置错误必须显式失败。
- 单元测试聚焦纯逻辑，集成测试覆盖流水线与边界行为。 - 单元测试聚焦纯逻辑，集成测试覆盖流水线与边界行为。
- 优先保留行为测试，避免只验证转发器、setter 或仅供测试存在的空壳包装层。
- 所有测试均从仓库根路径解析配置，避免目录迁移导致路径脆弱。 - 所有测试均从仓库根路径解析配置，避免目录迁移导致路径脆弱。
- 保留高价值回归测试，移除仅重复路径或无断言价值的历史脚本型测试（脚本保留在 `scripts/`，不纳入 pytest）。 - 保留高价值回归测试，移除仅重复路径或无断言价值的历史脚本型测试（脚本保留在 `scripts/`，不纳入 pytest）。
@@ -26,5 +28,5 @@
- 覆盖核心业务链路（bind/create/update 与 full pipeline）。 - 覆盖核心业务链路（bind/create/update 与 full pipeline）。
## 大规模自动回归 ## 大规模自动回归
- 按轮迭代的大规模自动测试方法见 [auto_test_spec/agent_auto_test_method.md](auto_test_spec/agent_auto_test_method.md)。 - 按轮迭代的大规模自动测试方法见 [../docs/auto_test_spec/agent_auto_test_method.md](../docs/auto_test_spec/agent_auto_test_method.md)。
- 执行时优先采用“单 domain 复现 -> 修复/分类 -> 全量聚合回归 -> 记录总计变化”的节奏，避免一次携带过多上下文。 - 执行时优先采用“单 domain 复现 -> 修复/分类 -> 全量聚合回归 -> 记录总计变化”的节奏，避免一次携带过多上下文。
tests/integration/test_full_pipeline_multinode_mock.py
+10 -9
View File
@@ -11,6 +11,7 @@ import pytest_asyncio
from sync_state_machine.common.binding import BindingManager from sync_state_machine.common.binding import BindingManager
from sync_state_machine.common.collection import DataCollection from sync_state_machine.common.collection import DataCollection
from sync_state_machine.common.persistence import PersistenceBackend from sync_state_machine.common.persistence import PersistenceBackend
from sync_state_machine.common.sqlite_backend import SQLitePersistenceBackend
from sync_state_machine.common.types import BindingStatus, SyncAction, SyncStatus from sync_state_machine.common.types import BindingStatus, SyncAction, SyncStatus
from sync_state_machine.pipeline.full_sync_pipeline import FullSyncPipeline from sync_state_machine.pipeline.full_sync_pipeline import FullSyncPipeline
from sync_state_machine.sync_system.strategy_ops.compare_ops import ( from sync_state_machine.sync_system.strategy_ops.compare_ops import (
@@ -39,7 +40,7 @@ async def pipeline_bundle():
register_test_domain() register_test_domain()
db = Path(tempfile.mkdtemp(prefix="sync_state_machine_pipeline_") ) / "state.db" db = Path(tempfile.mkdtemp(prefix="sync_state_machine_pipeline_") ) / "state.db"
persistence = PersistenceBackend(str(db)) persistence = SQLitePersistenceBackend(str(db))
await persistence.initialize() await persistence.initialize()
local_collection = DataCollection("local", persistence=persistence, auto_persist=False) local_collection = DataCollection("local", persistence=persistence, auto_persist=False)
@@ -112,7 +113,7 @@ async def pipeline_bundle():
local_ds = InMemoryDataSource(poll_max_retries=2, poll_interval=0) local_ds = InMemoryDataSource(poll_max_retries=2, poll_interval=0)
remote_ds = InMemoryDataSource(poll_max_retries=2, poll_interval=0) remote_ds = InMemoryDataSource(poll_max_retries=2, poll_interval=0)
local_ds.register_handler( local_ds.add_handler(
InMemoryNodeHandler( InMemoryNodeHandler(
node_type="test_project", node_type="test_project",
node_class=TestProjectNode, node_class=TestProjectNode,
@@ -120,7 +121,7 @@ async def pipeline_bundle():
load_nodes=local_project_nodes, load_nodes=local_project_nodes,
) )
) )
local_ds.register_handler( local_ds.add_handler(
InMemoryNodeHandler( InMemoryNodeHandler(
node_type="test_contract", node_type="test_contract",
node_class=TestContractNode, node_class=TestContractNode,
@@ -129,7 +130,7 @@ async def pipeline_bundle():
) )
) )
remote_ds.register_handler( remote_ds.add_handler(
InMemoryNodeHandler( InMemoryNodeHandler(
node_type="test_project", node_type="test_project",
node_class=TestProjectNode, node_class=TestProjectNode,
@@ -143,7 +144,7 @@ async def pipeline_bundle():
update_mode_by_code={"UPD-1": "success"}, update_mode_by_code={"UPD-1": "success"},
) )
) )
remote_ds.register_handler( remote_ds.add_handler(
InMemoryNodeHandler( InMemoryNodeHandler(
node_type="test_contract", node_type="test_contract",
node_class=TestContractNode, node_class=TestContractNode,
@@ -304,7 +305,7 @@ async def test_load_overrides_bind_data_id_from_bindings(pipeline_bundle):
existing_persistence = local_collection.persistence existing_persistence = local_collection.persistence
assert existing_persistence is not None assert existing_persistence is not None
persistence = PersistenceBackend(existing_persistence.db_path) persistence = SQLitePersistenceBackend(existing_persistence.db_path)
await persistence.initialize() await persistence.initialize()
await persistence.conn.execute( await persistence.conn.execute(
@@ -443,7 +444,7 @@ async def test_pipeline_create_then_update_completes_in_single_run_with_reload()
register_test_domain() register_test_domain()
db = Path(tempfile.mkdtemp(prefix="sync_state_machine_pipeline_reload_") ) / "state.db" db = Path(tempfile.mkdtemp(prefix="sync_state_machine_pipeline_reload_") ) / "state.db"
persistence = PersistenceBackend(str(db)) persistence = SQLitePersistenceBackend(str(db))
await persistence.initialize() await persistence.initialize()
local_collection = DataCollection("local", persistence=persistence, auto_persist=False) local_collection = DataCollection("local", persistence=persistence, auto_persist=False)
@@ -453,7 +454,7 @@ async def test_pipeline_create_then_update_completes_in_single_run_with_reload()
local_ds = InMemoryDataSource(poll_max_retries=2, poll_interval=0) local_ds = InMemoryDataSource(poll_max_retries=2, poll_interval=0)
remote_ds = InMemoryDataSource(poll_max_retries=2, poll_interval=0) remote_ds = InMemoryDataSource(poll_max_retries=2, poll_interval=0)
local_ds.register_handler( local_ds.add_handler(
InMemoryNodeHandler( InMemoryNodeHandler(
node_type="test_project", node_type="test_project",
node_class=TestProjectNode, node_class=TestProjectNode,
@@ -464,7 +465,7 @@ async def test_pipeline_create_then_update_completes_in_single_run_with_reload()
) )
) )
remote_handler = _MutableProjectRemoteHandler() remote_handler = _MutableProjectRemoteHandler()
remote_ds.register_handler(remote_handler) remote_ds.add_handler(remote_handler)
project_strategy = TestProjectStrategy("test_project", local_collection, remote_collection, binding_manager) project_strategy = TestProjectStrategy("test_project", local_collection, remote_collection, binding_manager)
project_strategy.update_config( project_strategy.update_config(
tests/integration/test_full_pipeline_multinode_persisted_db.py
+6 -5
View File
@@ -8,6 +8,7 @@ import pytest
from sync_state_machine.common.binding import BindingManager from sync_state_machine.common.binding import BindingManager
from sync_state_machine.common.collection import DataCollection from sync_state_machine.common.collection import DataCollection
from sync_state_machine.common.persistence import PersistenceBackend from sync_state_machine.common.persistence import PersistenceBackend
from sync_state_machine.common.sqlite_backend import SQLitePersistenceBackend
from sync_state_machine.pipeline.full_sync_pipeline import FullSyncPipeline from sync_state_machine.pipeline.full_sync_pipeline import FullSyncPipeline
from sync_state_machine.sync_system.config import OrphanAction, UpdateDirection from sync_state_machine.sync_system.config import OrphanAction, UpdateDirection
from tests.fixtures.mock_domain import ( from tests.fixtures.mock_domain import (
@@ -36,7 +37,7 @@ async def test_full_pipeline_multinode_mock_persisted_db_snapshot() -> None:
if db_path.exists(): if db_path.exists():
db_path.unlink() db_path.unlink()
persistence = PersistenceBackend(str(db_path)) persistence = SQLitePersistenceBackend(str(db_path))
await persistence.initialize() await persistence.initialize()
local_collection = DataCollection("local", persistence=persistence, auto_persist=False) local_collection = DataCollection("local", persistence=persistence, auto_persist=False)
@@ -73,7 +74,7 @@ async def test_full_pipeline_multinode_mock_persisted_db_snapshot() -> None:
local_ds = InMemoryDataSource(poll_max_retries=2, poll_interval=0) local_ds = InMemoryDataSource(poll_max_retries=2, poll_interval=0)
remote_ds = InMemoryDataSource(poll_max_retries=2, poll_interval=0) remote_ds = InMemoryDataSource(poll_max_retries=2, poll_interval=0)
local_ds.register_handler( local_ds.add_handler(
InMemoryNodeHandler( InMemoryNodeHandler(
node_type="test_project", node_type="test_project",
node_class=TestProjectNode, node_class=TestProjectNode,
@@ -81,7 +82,7 @@ async def test_full_pipeline_multinode_mock_persisted_db_snapshot() -> None:
load_nodes=local_project_nodes, load_nodes=local_project_nodes,
) )
) )
local_ds.register_handler( local_ds.add_handler(
InMemoryNodeHandler( InMemoryNodeHandler(
node_type="test_contract", node_type="test_contract",
node_class=TestContractNode, node_class=TestContractNode,
@@ -90,7 +91,7 @@ async def test_full_pipeline_multinode_mock_persisted_db_snapshot() -> None:
) )
) )
remote_ds.register_handler( remote_ds.add_handler(
InMemoryNodeHandler( InMemoryNodeHandler(
node_type="test_project", node_type="test_project",
node_class=TestProjectNode, node_class=TestProjectNode,
@@ -104,7 +105,7 @@ async def test_full_pipeline_multinode_mock_persisted_db_snapshot() -> None:
update_mode_by_code={"UPD-1": "success"}, update_mode_by_code={"UPD-1": "success"},
) )
) )
remote_ds.register_handler( remote_ds.add_handler(
InMemoryNodeHandler( InMemoryNodeHandler(
node_type="test_contract", node_type="test_contract",
node_class=TestContractNode, node_class=TestContractNode,
tests/integration/test_mock_datasource_state_machine.py
+8 -7
View File
@@ -7,6 +7,7 @@ import pytest
from sync_state_machine.common.collection import DataCollection from sync_state_machine.common.collection import DataCollection
from sync_state_machine.common.persistence import PersistenceBackend from sync_state_machine.common.persistence import PersistenceBackend
from sync_state_machine.common.sqlite_backend import SQLitePersistenceBackend
from sync_state_machine.common.types import BindingStatus, SyncAction, SyncStatus from sync_state_machine.common.types import BindingStatus, SyncAction, SyncStatus
from sync_state_machine.engine.dispatcher import StateMachineRuntime from sync_state_machine.engine.dispatcher import StateMachineRuntime
from sync_state_machine.engine.model import StateMachineConfig from sync_state_machine.engine.model import StateMachineConfig
@@ -21,7 +22,7 @@ from tests.mocks.mock_datasource import (
@pytest.fixture @pytest.fixture
def datasource_bundle(): def datasource_bundle():
db = Path(tempfile.mkdtemp(prefix="sync_state_machine_mock_ds_")) / "state.db" db = Path(tempfile.mkdtemp(prefix="sync_state_machine_mock_ds_")) / "state.db"
persistence = PersistenceBackend(str(db)) persistence = SQLitePersistenceBackend(str(db))
cfg_path = Path(__file__).resolve().parents[2] / "sync_state_machine" / "config" / "node_state_machine.yaml" cfg_path = Path(__file__).resolve().parents[2] / "sync_state_machine" / "config" / "node_state_machine.yaml"
runtime = StateMachineRuntime(StateMachineConfig.load(cfg_path)) runtime = StateMachineRuntime(StateMachineConfig.load(cfg_path))
collection = DataCollection("mock", persistence=None, auto_persist=False) collection = DataCollection("mock", persistence=None, auto_persist=False)
@@ -49,7 +50,7 @@ async def test_create_success_to_s11(datasource_bundle):
node = _new_node("n1", SyncAction.CREATE) node = _new_node("n1", SyncAction.CREATE)
await collection.add(node) await collection.add(node)
datasource.register_handler(ScriptedMockHandler(HandlerScript(create="success"))) datasource.add_handler(ScriptedMockHandler(HandlerScript(create="success")))
datasource.set_collection(collection) datasource.set_collection(collection)
await datasource.sync_all(data_type="mock") await datasource.sync_all(data_type="mock")
@@ -62,7 +63,7 @@ async def test_create_skipped_is_rejected(datasource_bundle):
node = _new_node("n2", SyncAction.CREATE) node = _new_node("n2", SyncAction.CREATE)
await collection.add(node) await collection.add(node)
datasource.register_handler(ScriptedMockHandler(HandlerScript(create="skipped"))) datasource.add_handler(ScriptedMockHandler(HandlerScript(create="skipped")))
datasource.set_collection(collection) datasource.set_collection(collection)
await datasource.sync_all(data_type="mock") await datasource.sync_all(data_type="mock")
@@ -76,7 +77,7 @@ async def test_update_skip_downgrade_to_s14(datasource_bundle):
node = _new_node("n3", SyncAction.UPDATE) node = _new_node("n3", SyncAction.UPDATE)
await collection.add(node) await collection.add(node)
datasource.register_handler( datasource.add_handler(
ScriptedMockHandler(HandlerScript(update="skipped")) ScriptedMockHandler(HandlerScript(update="skipped"))
) )
datasource.set_collection(collection) datasource.set_collection(collection)
@@ -94,7 +95,7 @@ async def test_update_success_writes_back_origin_data(datasource_bundle):
node.set_data({"id": "n3b", "name": "after"}) node.set_data({"id": "n3b", "name": "after"})
await collection.add(node) await collection.add(node)
datasource.register_handler(ScriptedMockHandler(HandlerScript(update="success"))) datasource.add_handler(ScriptedMockHandler(HandlerScript(update="success")))
datasource.set_collection(collection) datasource.set_collection(collection)
await datasource.sync_all(data_type="mock") await datasource.sync_all(data_type="mock")
@@ -109,7 +110,7 @@ async def test_in_progress_poll_success_to_s11(datasource_bundle):
node = _new_node("n4", SyncAction.CREATE) node = _new_node("n4", SyncAction.CREATE)
await collection.add(node) await collection.add(node)
datasource.register_handler( datasource.add_handler(
ScriptedMockHandler(HandlerScript(create="in_progress", poll_sequence=["success"])) ScriptedMockHandler(HandlerScript(create="in_progress", poll_sequence=["success"]))
) )
datasource.set_collection(collection) datasource.set_collection(collection)
@@ -124,7 +125,7 @@ async def test_in_progress_poll_timeout_to_s12(datasource_bundle):
node = _new_node("n5", SyncAction.CREATE) node = _new_node("n5", SyncAction.CREATE)
await collection.add(node) await collection.add(node)
datasource.register_handler( datasource.add_handler(
ScriptedMockHandler(HandlerScript(create="in_progress", poll_sequence=["in_progress", "in_progress"])) ScriptedMockHandler(HandlerScript(create="in_progress", poll_sequence=["in_progress", "in_progress"]))
) )
datasource.set_collection(collection) datasource.set_collection(collection)
tests/integration/test_mock_full_event_state_coverage.py
+4 -3
View File
@@ -10,6 +10,7 @@ import pytest
from sync_state_machine.common.collection import DataCollection from sync_state_machine.common.collection import DataCollection
from sync_state_machine.common.binding import BindingManager from sync_state_machine.common.binding import BindingManager
from sync_state_machine.common.persistence import PersistenceBackend from sync_state_machine.common.persistence import PersistenceBackend
from sync_state_machine.common.sqlite_backend import SQLitePersistenceBackend
from sync_state_machine.common.types import BindingStatus, SyncAction, SyncStatus from sync_state_machine.common.types import BindingStatus, SyncAction, SyncStatus
from sync_state_machine.engine import StateMachineConfig, StateMachineRuntime from sync_state_machine.engine import StateMachineConfig, StateMachineRuntime
from sync_state_machine.engine.events import ( from sync_state_machine.engine.events import (
@@ -464,7 +465,7 @@ async def test_sync_log_cleared_on_initial_load(tmp_path: Path) -> None:
REPORT_DIR.mkdir(parents=True, exist_ok=True) REPORT_DIR.mkdir(parents=True, exist_ok=True)
db = tmp_path / "sync_log_clear_state.db" db = tmp_path / "sync_log_clear_state.db"
persistence = PersistenceBackend(str(db)) persistence = SQLitePersistenceBackend(str(db))
await persistence.initialize() await persistence.initialize()
collection = DataCollection("project-log-clear", persistence=persistence, auto_persist=False) collection = DataCollection("project-log-clear", persistence=persistence, auto_persist=False)
@@ -531,7 +532,7 @@ async def test_load_preserves_state_then_e01_normalizes_to_s00(tmp_path: Path) -
REPORT_DIR.mkdir(parents=True, exist_ok=True) REPORT_DIR.mkdir(parents=True, exist_ok=True)
db = tmp_path / "load_preserve_then_e01_normalize.db" db = tmp_path / "load_preserve_then_e01_normalize.db"
persistence = PersistenceBackend(str(db)) persistence = SQLitePersistenceBackend(str(db))
await persistence.initialize() await persistence.initialize()
collection = DataCollection("project-e01-normalize", persistence=persistence, auto_persist=False) collection = DataCollection("project-e01-normalize", persistence=persistence, auto_persist=False)
@@ -583,7 +584,7 @@ async def test_invalid_persisted_enum_is_bootstrap_blocked_and_skipped_in_reset(
REPORT_DIR.mkdir(parents=True, exist_ok=True) REPORT_DIR.mkdir(parents=True, exist_ok=True)
db = tmp_path / "invalid_enum_bootstrap_blocked.db" db = tmp_path / "invalid_enum_bootstrap_blocked.db"
persistence = PersistenceBackend(str(db)) persistence = SQLitePersistenceBackend(str(db))
await persistence.initialize() await persistence.initialize()
collection_id = "project-invalid-enum" collection_id = "project-invalid-enum"
tests/integration/test_multi_project_pipeline_jsonl_jsonl.py
+157
View File
@@ -0,0 +1,157 @@
from __future__ import annotations
from pathlib import Path
import pytest
from sync_state_machine.common.persistence import PersistenceBackend
from sync_state_machine.pipeline import run_profile_from_file
from sync_state_machine.pipeline.full_sync_pipeline import FullSyncPipeline
def _write_yaml(path: Path, content: str) -> None:
path.write_text(content.strip() + "\n", encoding="utf-8")
@pytest.mark.asyncio
async def test_multi_project_pipeline_runs_jsonl_to_jsonl_with_shared_persistence(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
pytest.importorskip("yaml")
async def _skip_post_check(self) -> bool:
return True
monkeypatch.setattr(FullSyncPipeline, "phase_post_check", _skip_post_check)
project_root = Path(__file__).resolve().parents[2]
local_dir = project_root / "working_local_datasource" / "20260302-020003"
remote_dir = tmp_path / "remote_jsonl"
db_path = tmp_path / "multi_project.db"
logs_dir = tmp_path / "logs"
remote_dir.mkdir(parents=True, exist_ok=True)
logs_dir.mkdir(parents=True, exist_ok=True)
global_cfg = tmp_path / "global.yaml"
project_cfg = tmp_path / "project.yaml"
multi_cfg = tmp_path / "multi.yaml"
_write_yaml(
global_cfg,
f"""
node_types:
- project
local_datasource:
type: jsonl
jsonl_dir: {local_dir.as_posix()}
read_only: false
remote_datasource:
type: jsonl
jsonl_dir: {remote_dir.as_posix()}
read_only: false
persist:
backend: sqlite
db_path: {db_path.as_posix()}
enable: true
wipe_on_start: true
logging:
initialize: true
file_dir: {logs_dir.as_posix()}
file_name_pattern: multi_global_{{timestamp}}.log
level: 20
strategies:
project:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- name
depend_fields: {{}}
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
""",
)
_write_yaml(
project_cfg,
f"""
node_types:
- lar
local_datasource:
type: jsonl
jsonl_dir: {local_dir.as_posix()}
read_only: false
remote_datasource:
type: jsonl
jsonl_dir: {remote_dir.as_posix()}
read_only: false
persist:
backend: sqlite
db_path: {db_path.as_posix()}
enable: true
wipe_on_start: false
logging:
initialize: true
file_dir: {logs_dir.as_posix()}
file_name_pattern: multi_project_{{timestamp}}.log
level: 20
strategies:
lar:
skip_sync: false
config:
auto_bind: true
auto_bind_fields:
- project_id
depend_fields:
project_id: project
local_orphan_action: create_remote
remote_orphan_action: none
update_direction: push
""",
)
_write_yaml(
multi_cfg,
f"""
global_config: {global_cfg.as_posix()}
default_project_config: {project_cfg.as_posix()}
projects:
- local_project_id: f3e02e84-e81c-4aa6-88d4-f5aa108c8227
remote_project_id: f3e02e84-e81c-4aa6-88d4-f5aa108c8227
- local_project_id: 0f9c3e22-f4bb-4803-825a-4f238eeb06d5
remote_project_id: 0f9c3e22-f4bb-4803-825a-4f238eeb06d5
""",
)
results = await run_profile_from_file(project_root=project_root, config_path=multi_cfg, print_summary=False)
assert "global" in results
assert "project_id:f3e02e84-e81c-4aa6-88d4-f5aa108c8227" in results
assert "project_id:0f9c3e22-f4bb-4803-825a-4f238eeb06d5" in results
_, scope_rows = PersistenceBackend.query_records(
backend="sqlite",
db_path=str(db_path),
sql="SELECT DISTINCT scope FROM nodes ORDER BY scope",
)
scopes = [row["scope"] for row in scope_rows]
assert scopes == [
"global",
"project_id:0f9c3e22-f4bb-4803-825a-4f238eeb06d5",
"project_id:f3e02e84-e81c-4aa6-88d4-f5aa108c8227",
]
_, copied_rows = PersistenceBackend.query_records(
backend="sqlite",
db_path=str(db_path),
sql=(
"SELECT scope, collection_id, node_type, COUNT(1) AS c "
"FROM nodes "
"WHERE scope <> 'global' AND node_type = 'project' "
"GROUP BY scope, collection_id, node_type "
"ORDER BY scope, collection_id, node_type"
),
)
assert copied_rows == []
tests/integration/test_strategy_business_points_migrated.py
+2 -1
View File
@@ -9,6 +9,7 @@ import pytest_asyncio
from sync_state_machine.common.binding import BindingManager from sync_state_machine.common.binding import BindingManager
from sync_state_machine.common.collection import DataCollection from sync_state_machine.common.collection import DataCollection
from sync_state_machine.common.persistence import PersistenceBackend from sync_state_machine.common.persistence import PersistenceBackend
from sync_state_machine.common.sqlite_backend import SQLitePersistenceBackend
from sync_state_machine.common.types import BindingStatus, SyncAction, SyncStatus from sync_state_machine.common.types import BindingStatus, SyncAction, SyncStatus
from sync_state_machine.sync_system.resolve_ids import IDResolver from sync_state_machine.sync_system.resolve_ids import IDResolver
from sync_state_machine.sync_system.config import OrphanAction, UpdateDirection from sync_state_machine.sync_system.config import OrphanAction, UpdateDirection
@@ -32,7 +33,7 @@ from tests.fixtures.mock_domain import (
async def setup_env(): async def setup_env():
register_test_domain() register_test_domain()
db = Path(tempfile.mkdtemp(prefix="sync_state_machine_biz_") ) / "state.db" db = Path(tempfile.mkdtemp(prefix="sync_state_machine_biz_") ) / "state.db"
persistence = PersistenceBackend(str(db)) persistence = SQLitePersistenceBackend(str(db))
await persistence.initialize() await persistence.initialize()
local = DataCollection("local", persistence=persistence, auto_persist=False) local = DataCollection("local", persistence=persistence, auto_persist=False)
tests/unit/test_auto_test_project_profile.py
+4 -4
View File
@@ -27,8 +27,8 @@ def test_build_temp_run_profile_uses_explicit_project_prebind(tmp_path: Path) ->
source_profile.write_text( source_profile.write_text(
yaml.safe_dump( yaml.safe_dump(
{ {
"local_datasource": {"target_project_ids": ["old-local"]}, "local_datasource": {"handler_configs": {"project": {"data_id_filter": ["old-local"]}}},
"remote_datasource": {"target_project_ids": ["old-remote"]}, "remote_datasource": {"handler_configs": {"project": {"data_id_filter": ["old-remote"]}}},
"strategies": {"project": {"config": {}}}, "strategies": {"project": {"config": {}}},
}, },
allow_unicode=True, allow_unicode=True,
@@ -55,8 +55,8 @@ def test_build_temp_run_profile_uses_explicit_project_prebind(tmp_path: Path) ->
profile = yaml.safe_load(Path(profile_path).read_text(encoding="utf-8")) profile = yaml.safe_load(Path(profile_path).read_text(encoding="utf-8"))
project_config = profile["strategies"]["project"]["config"] project_config = profile["strategies"]["project"]["config"]
assert profile["local_datasource"]["target_project_ids"] == ["local-project"] assert profile["local_datasource"]["handler_configs"]["project"]["data_id_filter"] == ["local-project"]
assert profile["remote_datasource"]["target_project_ids"] == ["remote-project"] assert profile["remote_datasource"]["handler_configs"]["project"]["data_id_filter"] == ["remote-project"]
assert project_config["auto_bind"] is False assert project_config["auto_bind"] is False
assert project_config["auto_bind_fields"] == [] assert project_config["auto_bind_fields"] == []
assert project_config["pre_bind_data_id"] == [ assert project_config["pre_bind_data_id"] == [
tests/unit/test_datasource_load_continue_on_error.py
+2 -2
View File
@@ -97,8 +97,8 @@ class _DS(BaseDataSource):
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_load_all_continue_when_one_handler_fails(capsys) -> None: async def test_load_all_continue_when_one_handler_fails(capsys) -> None:
ds = _DS() ds = _DS()
ds.register_handler(_FailHandler()) ds.add_handler(_FailHandler())
ds.register_handler(_OkHandler()) ds.add_handler(_OkHandler())
ds.set_collection(DataCollection("local")) ds.set_collection(DataCollection("local"))
await ds.load_all(order=["fail", "ok"]) await ds.load_all(order=["fail", "ok"])
tests/unit/test_datasource_target_project_ids_resolution.py
+11 -11
View File
@@ -21,12 +21,12 @@ def _build_config(
local_datasource=JsonlDataSourceConfig( local_datasource=JsonlDataSourceConfig(
type="jsonl", type="jsonl",
jsonl_dir="./filtered_datasource/datasource/filtered", jsonl_dir="./filtered_datasource/datasource/filtered",
target_project_ids=local_ids or [], handler_configs={"project": {"data_id_filter": local_ids or []}},
), ),
remote_datasource=JsonlDataSourceConfig( remote_datasource=JsonlDataSourceConfig(
type="jsonl", type="jsonl",
jsonl_dir="./remote_datasource/datasource", jsonl_dir="./remote_datasource/datasource",
target_project_ids=remote_ids or [], handler_configs={"project": {"data_id_filter": remote_ids or []}},
), ),
strategies=[], strategies=[],
persist=PersistConfig(db_path="./test_results/sync_state_machine/test.db"), persist=PersistConfig(db_path="./test_results/sync_state_machine/test.db"),
@@ -34,29 +34,29 @@ def _build_config(
) )
def test_pipeline_config_has_no_top_level_target_project_ids() -> None: def test_pipeline_config_uses_handler_level_project_filters() -> None:
cfg = _build_config(local_ids=["p_local"], remote_ids=["p_remote"]) cfg = _build_config(local_ids=["p_local"], remote_ids=["p_remote"])
assert cfg.local_datasource.target_project_ids == ["p_local"] assert cfg.local_datasource.handler_configs["project"]["data_id_filter"] == ["p_local"]
assert cfg.remote_datasource.target_project_ids == ["p_remote"] assert cfg.remote_datasource.handler_configs["project"]["data_id_filter"] == ["p_remote"]
with pytest.raises(AttributeError): legacy_field_name = "_".join(("target", "project", "ids"))
_ = cfg.target_project_ids assert legacy_field_name not in type(cfg).model_fields
def test_api_datasource_allows_empty_target_project_ids() -> None: def test_api_datasource_allows_empty_handler_configs() -> None:
cfg = ApiDataSourceConfig( cfg = ApiDataSourceConfig(
type="api", type="api",
api_base_url="https://example.com", api_base_url="https://example.com",
) )
assert cfg.target_project_ids == [] assert cfg.handler_configs == {}
def test_api_datasource_rate_limit_config_fields() -> None: def test_api_datasource_rate_limit_config_fields() -> None:
cfg = ApiDataSourceConfig( cfg = ApiDataSourceConfig(
type="api", type="api",
api_base_url="https://example.com", api_base_url="https://example.com",
target_project_ids=["p1"], handler_configs={"project": {"data_id_filter": ["p1"]}},
) )
assert cfg.api_rate_limit_max_requests == 10 assert cfg.api_rate_limit_max_requests == 10
assert cfg.api_rate_limit_window_seconds == 10.0 assert cfg.api_rate_limit_window_seconds == 10.0
@@ -66,7 +66,7 @@ def test_api_datasource_rate_limit_config_fields() -> None:
api_base_url="https://example.com", api_base_url="https://example.com",
api_rate_limit_max_requests=5, api_rate_limit_max_requests=5,
api_rate_limit_window_seconds=2.5, api_rate_limit_window_seconds=2.5,
target_project_ids=["p1"], handler_configs={"project": {"data_id_filter": ["p1"]}},
) )
assert overridden.api_rate_limit_max_requests == 5 assert overridden.api_rate_limit_max_requests == 5
assert overridden.api_rate_limit_window_seconds == 2.5 assert overridden.api_rate_limit_window_seconds == 2.5

Some files were not shown because too many files have changed in this diff Show More