Compare commits

...

19 Commits

Author SHA1 Message Date
strepsiades ba73b30a04 增加sqlalchemy后端,优化日志结构 2026-04-08 09:28:14 +08:00
strepsiades c377264610 实现了多project pipeline 2026-04-07 16:52:42 +08:00
strepsiades 4d6f499508 压制了无用日志 2026-04-07 16:14:00 +08:00
strepsiades 966f2e1270 初步实现多project pipeline 2026-04-07 14:45:19 +08:00
strepsiades e3eab5ade1 添加了重构文档 2026-04-07 11:25:19 +08:00
strepsiades aa31243bae 修复一个因为项目双向更新更新导致第二次更新无法进行的问题 2026-04-03 11:44:53 +08:00
strepsiades f5729d5a18 整理代码 2026-04-03 09:18:44 +08:00
strepsiades 543c3ff906 优化加载后删除节点的逻辑;和数据库查询性能 2026-04-02 19:31:04 +08:00
strepsiades 74df53889c 解决了部分detail load效率问题和load失败终止pipeline问题 2026-04-02 14:35:28 +08:00
strepsiades bb8ea931b5 优化大量推送问题 2026-04-02 11:14:44 +08:00
strepsiades f93a6c5c68 优化性能 2026-04-01 18:13:18 +08:00
strepsiades 9a3a34c58b 修改问题 2026-04-01 17:04:26 +08:00
strepsiades 7c49df94fc 修改多个问题,优化代码结构 2026-04-01 16:23:43 +08:00
strepsiades 8f4727a772 优化sync_system代码质量 2026-04-01 11:52:57 +08:00
strepsiades eb02fd32a5 修复了一个project_detail_data create失败报错的问题 2026-03-31 15:54:12 +08:00
strepsiades f0ce8dc37e 修改supplier model,删除了多余字段 2026-03-31 11:33:27 +08:00
strepsiades 14c20a6123 整理update逻辑 2026-03-31 08:48:34 +08:00
strepsiades bdc7bd5069 规范了update data的构建方式,统一使用build_update_request_data方法来处理更新数据的构建和过滤逻辑。这个方法会根据提供的原始数据、更新数据和对应的schema来生成最终的更新请求数据,同时会自动排除掉未变化的字段和空值字段,从而确保只发送必要的更新信息到后端接口。这种方式不仅简化了代码逻辑,还提高了代码的可维护性和一致性。 2026-03-30 14:51:01 +08:00
strepsiades f50dc1aed0 整体同步代码 2026-03-26 14:10:47 +08:00
139 changed files with 8575 additions and 2203 deletions
+2 -1
View File
@@ -108,6 +108,7 @@ ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
- 仓库内直接运行:`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
- 已安装命令行入口:`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
- 多项目 JSONL 示例:`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.multi_project.default.yaml`
- 当前已经支持多 project 运行;标准写法是直接使用多项目编排配置文件,或在支持 implicit multi-project 的 profile 中由配置本身决定运行模式。
- 自定义配置:从 `run_profiles/preset/` 复制一份到 `run_profiles/*.local.yaml`,只改有差异的字段即可。
### API 同步
@@ -144,7 +145,7 @@ ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
- 第一次接手仓库:`docs/getting_started.md`
- 想理解整体分层:`docs/architecture.md`
- 想理解状态与迁移:`docs/node_state_definition.md` + `docs/state_machine.md`
- 想调运行配置:`docs/runtime_config_guide.md` + `run_profiles/README.md`
- 想调运行配置:`docs/runtime_config_guide.md` + `run_profiles/README.md`(包含多项目配置写法)
- 想接新数据源或改 handler`sync_state_machine/datasource/README.md` + `docs/library_integration_guide.md`
- 想排障:`docs/operations.md`
- 想补测试或跑联调:`docs/testing_guide.md` + `tests/README.md`
+6 -6
View File
@@ -7,7 +7,7 @@
- 负责组装并驱动整条流程:加载持久化、执行 bind/create/update、调用 datasource 同步、持久化结果。
- reset 语义:
- 加载阶段仅恢复持久化状态(不做状态重置);
- 加载后通过 `BaseSyncStrategy.run_reset()` 触发 `E01`
- 加载后通过 `run_phase2_cleanup()` 触发 `E01`
- `create_zombie=true` -> `S15`(随后删除);
- `create_zombie=false` -> `S00`(归并为 `UNCHECKED/NONE/PENDING`,并清空 `error`)。
- 若持久化核心枚举非法(`action/status/binding_status`),节点进入加载异常隔离态 `S17`
@@ -26,12 +26,12 @@
### strategy(决策组织层)
- 外部入口:`sync_state_machine/sync_system/strategy.py`
- 具体实现下沉:`sync_state_machine/sync_system/strategy_ops/`
- `bind_ops.py`: `run_bind()` + `phase_core_state()/phase_dependency_check()/phase_auto_binding()`
- `create_ops.py`: `run_create()`
- `update_ops.py`: `run_update()`
- `bind_ops.py`: bind 阶段静态函数与纯工具
- `create_ops.py`: create 阶段静态函数与纯工具
- `update_ops.py`: update 阶段静态函数与纯工具
- `delete_ops.py`: `run_delete()`
- `reset_ops.py`: `get_phase1_reset_defaults()/run_phase2_cleanup()`
- strategy 负责收集上下文并调用状态机事件函数,不直接硬编码状态值。
- `reset_ops.py`: `run_phase2_cleanup()`
- strategy 负责 bind/create/update 的相位内部编排,并调用状态机事件函数,不直接硬编码状态值。
### datasource(执行与回写层)
- 入口:`sync_state_machine/datasource/datasource.py`
+2 -2
View File
@@ -131,8 +131,8 @@ API 数据源中,校验分为三层:
- 例如 `ContractCreate.model_validate(...)``ContractUpdate.model_validate(...)`
- load 响应也会在解析后转为领域 schema
2. 通用更新过滤与差异识别
- `_filter_update_data`仅在 schema 相关字段有变化时提交更新
- `_get_schema_diff`:用于差异字段识别与调试输出
- `build_update_request_data`update 入口,仅在 schema 相关字段有变化时提交更新
- `get_update_schema_diff`update 入口,用于差异字段识别与调试输出
3. 轮询接口输入校验
- `PushIdsSchema` 校验 push_ids 格式后再请求 `/push/result`
+1 -1
View File
@@ -46,7 +46,7 @@ pytest tests/integration/test_toolchain_config_and_graph.py -q
- 想知道系统整体怎么分层:`docs/architecture.md`
- 想知道状态机怎么工作:`docs/node_state_definition.md``docs/state_machine.md`
- 想知道一次同步是怎么跑的:`docs/sync_flow.md`
- 想调运行配置:`docs/runtime_config_guide.md`
- 想调运行配置:`docs/runtime_config_guide.md`(包含单项目、多项目、`persist.url``sqlalchemy`、业务侧注册 datasource 的当前写法)
- 想接新 datasource 或 handler`sync_state_machine/datasource/README.md``docs/library_integration_guide.md`
- 想看测试体系:`docs/testing_guide.md``tests/README.md`
- 想做排障:`docs/operations.md`
+627
View File
@@ -0,0 +1,627 @@
# 多项目同步重构方案
## 1. 背景和目标
### 背景
当前 `multi_project_pipeline` 的主要问题:
1. 顶层概念过重,在单项目同步之上又叠了一层特殊编排。
2. 通过共享快照传递 `collection` / `binding` / 运行态,边界不清晰。
3. 配置入口过多,存在 `global_config` / `default_project_config` / `config_path` 多层模型。
4. 项目完成后哪些对象可以释放,没有明确生命周期。
### 目标
新方案只解决四件事:
1. 顶层仍然是 pipeline,但直接写新的 `ProjectBatchPipeline`
2. `global` 数据只同步一次,不复制到每个项目,不在每个项目中重复校验。
3. 改动尽量限制在 `persistence / collection / binding_manager / datasource / pipeline` 这些基础层。
4. 项目级运行态支持 `load -> sync -> persist -> unload -> close`
### 一个明确结论
这里不建议先做复杂的 session 框架。
第一阶段最合适的方式是:
1.`sync_project(...)` 里创建一个很薄的 `ProjectScopeRuntime`
2. 它只是把“当前项目 scope 下的一组对象”收拢起来。
3. 如果你临时不想单独建类,也可以先写成函数内局部变量。
但从清理顺序和可读性考虑,推荐保留 `ProjectScopeRuntime` 这个薄对象。
它不是复杂 session 系统,本质上就是“单个项目的一次运行上下文”。
## 2. 新的 Batch Project Pipeline 和伪代码
### 结构结论
顶层对象建议是:`ProjectBatchPipeline`
它负责:
1. 跑一次 global phase。
2. 并发调度所有项目。
3. 管理 batch 级共享对象。
4. 管理项目级对象的创建和清理。
它不负责:
1. 业务字段映射。
2. handler 细节。
3. strategy 逻辑。
### 共享对象和项目级对象
#### 整个 batch 共享的对象
1. `BatchConfig`
2. `StateMachineRuntime`
3. `Logger`
4. `PersistenceService`
5. `global` 只读运行态
6. datasource 的共享静态资源,例如 API client、认证信息、限流器、handler registry
7. semaphore 和任务调度器
#### 每个项目内创建的对象
1. `ProjectScopeRuntime`
2. `local_collection`
3. `remote_collection`
4. `binding_manager`
5. 当前项目 scope 对应的 persistence view
6. 当前项目的 local datasource runtime
7. 当前项目的 remote datasource runtime
8. 当前项目统计对象
### Global 数据处理原则
这点直接定死:
1. `global` 数据不复制到每个 project scope。
2. `global` 数据不在每个项目里重复 bootstrap。
3. `global` 数据不在每个项目里重复 post-check。
项目运行时如果需要共享节点或绑定:
1. 先查当前 project scope。
2. 未命中再回退到 `global` 只读运行态。
### 伪代码
```python
class ProjectBatchPipeline:
def __init__(self, config: BatchConfig):
self.config = config
self.runtime = build_state_machine_runtime()
self.logger = build_logger(config.logging)
self.persistence = PersistenceService(config.persist)
self.global_runtime = None
self.shared_local_ds = build_shared_local_datasource_resources(config.local_datasource)
self.shared_remote_ds = build_shared_remote_datasource_resources(config.remote_datasource)
async def run(self) -> None:
self.global_runtime = await self.sync_global()
await self.dispatch_projects()
async def sync_global(self) -> "GlobalReadonlyRuntime":
...
async def dispatch_projects(self) -> None:
...
async def sync_project(self, local_project_id: str, remote_project_id: str) -> None:
scope_runtime = await self.create_project_scope_runtime(local_project_id, remote_project_id)
try:
await scope_runtime.load()
await run_project_sync(scope_runtime, self.config.project_node_types)
await scope_runtime.persist()
finally:
await scope_runtime.unload()
await scope_runtime.close()
```
```python
class ProjectScopeRuntime:
def __init__(self, scope: str, persistence_view, global_runtime, shared_local_ds, shared_remote_ds):
self.scope = scope
self.persistence_view = persistence_view
self.local_collection = DataCollection(...)
self.remote_collection = DataCollection(...)
self.binding_manager = BindingManager(...)
self.local_datasource = build_local_datasource_runtime(shared_local_ds, self.local_collection)
self.remote_datasource = build_remote_datasource_runtime(shared_remote_ds, self.remote_collection)
self.global_runtime = global_runtime
async def load(self) -> None: ...
async def persist(self) -> None: ...
async def unload(self) -> None: ...
async def close(self) -> None: ...
```
## 3. 组件说明
### 3.1 Persistence
#### 角色
`PersistenceService` 是 batch 共享对象。
它共享的是:
1. 后端类型。
2. DB 路径或连接串。
3. 连接池或底层连接能力。
它不共享的是:
1. 当前项目的内存缓存。
2. 当前 scope 的脏状态。
#### 关键语义
建议把 persistence 理解成两层:
1. `PersistenceService`
- 共享后端服务。
2. `ScopedPersistenceView`
- 绑定某个 `scope` 的轻量 view。
所以“scope 属于 persistence 的 scope view”的意思是:
1. 整个 batch 共享一个 persistence service。
2. 每个项目拿一个绑定了当前 `scope` 的轻量操作视图。
不是说每个项目都新建一套 persistence backend。
#### 主要方法伪代码
```python
class PersistenceService:
async def initialize(self) -> None: ...
def view(self, scope: str) -> ScopedPersistenceView: ...
async def close(self) -> None: ...
class ScopedPersistenceView:
async def load_nodes(self, collection_id: str) -> list[dict]: ...
async def save_nodes(self, collection_id: str, nodes: list[SyncNode]) -> None: ...
async def delete_nodes(self, collection_id: str, node_ids: list[str]) -> None: ...
async def load_bindings(self, node_type: str) -> list[dict]: ...
async def save_bindings(self, node_type: str, bindings: dict[str, str | None]) -> None: ...
```
### 3.2 Collection
#### 角色
`Collection` 必须是 scope 级对象,不能做单例。
原因:
1. 它内部天然持有当前 scope 的节点集合。
2. 它持有 data_id 索引。
3. 它持有删除集合和脏状态。
#### 关键语义
project scope 的 `Collection` 只存项目域数据。
对于共享数据,不复制 global,而是支持只读回退:
1. 先查当前 scope。
2. 未命中再查 global 只读 collection。
#### 主要方法伪代码
```python
class DataCollection:
async def load_from_persistence(self) -> None: ...
async def persist(self) -> None: ...
async def unload(self) -> None: ...
def get(self, node_id: str) -> SyncNode | None: ...
def get_by_data_id(self, node_type: str, data_id: str) -> SyncNode | None:
node = self._get_local(node_type, data_id)
if node is not None:
return node
return self._get_global_fallback(node_type, data_id)
```
### 3.3 BindingManager
#### 角色
`BindingManager` 也是 scope 级对象,不应共享。
它负责:
1. 当前 scope 的绑定运行态。
2. 当前 scope 的绑定持久化。
3. 当前 scope 未命中时对 global 只读绑定做回退查询。
#### 主要方法伪代码
```python
class BindingManager:
async def load_from_persistence(self, node_type: str) -> None: ...
async def persist(self) -> None: ...
async def unload(self) -> None: ...
async def get_remote_id(self, node_type: str, local_id: str) -> str | None:
remote_id = self._get_local_remote_id(node_type, local_id)
if remote_id is not None:
return remote_id
return await self._get_global_remote_id(node_type, local_id)
```
### 3.4 DataSource
#### 角色
`local_datasource / remote_datasource / logging` 不应按项目变化,这一点成立。
但按当前实现,`BaseDataSource` 不适合直接做单例,因为它内部持有:
1. `_collection`
2. `_stats`
3. `_action_summary`
4. `_action_by_node_id`
#### 推荐的最小改法
第一阶段不强行拆成两个正式类,而是采用下面的语义:
1. datasource 的静态配置和共享 client 是 batch 级共享的。
2. datasource 的运行态对象仍然是项目级创建的。
也就是说:
1. 配置不按项目变化。
2. runtime 仍然按项目隔离。
这样改动范围最小,也更不容易污染现有 handler 行为。
#### 主要方法伪代码
```python
class ScopedDataSourceRuntime(BaseDataSource):
def set_collection(self, collection: DataCollection) -> None: ...
async def load_all(self, order: list[str]) -> None: ...
async def sync_all(self, order: list[str]) -> None: ...
def reset_runtime_state(self) -> None: ...
async def close(self) -> None: ...
```
## 4. 改进路线图
### 第一阶段:先收敛架构边界
只改基础层,不动业务层:
1. 新建 `ProjectBatchPipeline`
2. 明确 `global``project_id:<local_project_id>` 语义。
3. 引入 `ProjectScopeRuntime` 这个薄的项目运行上下文。
4.`Collection``BindingManager` 增加 global fallback 语义。
5. persistence 侧增加共享 service + scope view 语义。
### 第二阶段:清理生命周期
把项目级清理流程固定下来:
1. `load`
2. `sync`
3. `persist`
4. `unload`
5. `close`
### 第三阶段:再决定 datasource 是否正式拆层
如果第一阶段跑通,再决定是否把 datasource 正式拆成:
1. 共享静态层。
2. scope runtime 层。
这一步不是第一批必须完成。
### 最终落点
最终希望达到的状态:
1. global 数据只同步一次,只读共享。
2. project 数据按 scope 独立运行。
3. 项目跑完即可卸载运行态。
4. 大部分改动都限制在基础设施层。
8. 当前项目统计对象
### 2.3 为什么推荐 `ProjectScopeRuntime`
你问的是:要不要做 session 对象,还是做子 pipeline,还是都放函数里。
建议是:
1. 不做新的复杂 session 框架。
2. 不做 `FullSyncPipeline` 风格的子 pipeline 继承树。
3. 做一个很薄的 `ProjectScopeRuntime`
它的作用只有两个:
1. 把当前项目的一组 scope 对象收拢到一起。
2. 让清理顺序明确,不要把资源散在 `sync_project(...)` 的局部变量里。
如果你坚持更简单,也可以第一版直接写在 `sync_project(...)` 里;但文档层面仍建议用 `ProjectScopeRuntime` 表达这个层次。
### 2.4 伪代码
```python
class ProjectBatchPipeline:
def __init__(self, config: BatchConfig):
self.config = config
self.runtime = build_state_machine_runtime()
self.logger = build_logger(config.logging)
self.persistence = PersistenceService(config.persist)
self.global_runtime = None
self.shared_local_ds = build_shared_local_datasource_resources(config.local_datasource)
self.shared_remote_ds = build_shared_remote_datasource_resources(config.remote_datasource)
async def run(self) -> None:
self.global_runtime = await self.sync_global()
await self.dispatch_projects()
async def sync_global(self) -> "GlobalReadonlyRuntime":
# 只同步 shared_node_types
...
async def dispatch_projects(self) -> None:
# semaphore 控制并发
...
async def sync_project(self, local_project_id: str, remote_project_id: str) -> None:
scope_runtime = await self.create_project_scope_runtime(local_project_id, remote_project_id)
try:
await scope_runtime.load()
await run_project_sync(scope_runtime, self.config.project_node_types)
await scope_runtime.persist()
finally:
await scope_runtime.unload()
await scope_runtime.close()
```
```python
class ProjectScopeRuntime:
def __init__(self, scope: str, persistence_view, global_runtime, shared_local_ds, shared_remote_ds):
self.scope = scope
self.persistence_view = persistence_view
self.local_collection = DataCollection(...)
self.remote_collection = DataCollection(...)
self.binding_manager = BindingManager(...)
self.local_datasource = build_local_datasource_runtime(shared_local_ds, self.local_collection)
self.remote_datasource = build_remote_datasource_runtime(shared_remote_ds, self.remote_collection)
self.global_runtime = global_runtime
async def load(self) -> None: ...
async def persist(self) -> None: ...
async def unload(self) -> None: ...
async def close(self) -> None: ...
```
### 2.5 Global 数据的处理原则
这点现在可以直接定死:
1. `global` 数据不复制到每个 project scope。
2. `global` 数据不在每个项目里重复 bootstrap。
3. `global` 数据不在每个项目里重复 post-check。
项目级运行时如果需要共享节点或绑定,采用:
1. 当前 project scope 先查本地运行态。
2. 未命中时回退到 `global` 只读运行态。
这样可以避免复制大体量 global 数据。
2. `preload_shared_state(...)` 把项目 pipeline 变成了“半冷启动、半热注入”的特殊运行态。
3. `ephemeral_node_types` / `bootstrap_binding_node_types` 这些参数开始承担原本不属于它们的职责。
## 3. 组件说明
### 3.1 Persistence
#### 角色
`PersistenceService` 是 batch 共享对象。
它共享的是:
1. 后端类型。
2. DB 路径或连接串。
3. 连接池或底层连接能力。
它不共享的是:
1. 当前项目的内存缓存。
2. 当前 scope 的脏状态。
#### 关键语义
建议把 `persistence` 理解成两层:
1. `PersistenceService`
- 共享后端服务。
2. `ScopedPersistenceView`
- 绑定某个 `scope` 的轻量 view。
所以“scope 属于 persistence 的 scope view”这句话的意思是:
1. 整个 batch 共享一个 persistence service。
2. 每个项目拿一个绑定了当前 `scope` 的轻量操作视图。
不是说每个项目都新建一套 persistence backend。
#### 主要方法伪代码
```python
class PersistenceService:
async def initialize(self) -> None: ...
def view(self, scope: str) -> ScopedPersistenceView: ...
async def close(self) -> None: ...
class ScopedPersistenceView:
async def load_nodes(self, collection_id: str) -> list[dict]: ...
async def save_nodes(self, collection_id: str, nodes: list[SyncNode]) -> None: ...
async def delete_nodes(self, collection_id: str, node_ids: list[str]) -> None: ...
async def load_bindings(self, node_type: str) -> list[dict]: ...
async def save_bindings(self, node_type: str, bindings: dict[str, str | None]) -> None: ...
```
### 3.2 Collection
#### 角色
`Collection` 必须是 scope 级对象,不能做单例。
原因很简单:
1. 它内部天然持有当前 scope 的节点集合。
2. 它持有 data_id 索引。
3. 它持有删除集合和脏状态。
#### 关键语义
project scope 的 `Collection` 只存项目域数据。
对于共享数据,不复制 global,而是支持只读回退:
1. 先查当前 scope。
2. 未命中再查 global 只读 collection。
#### 主要方法伪代码
```python
class DataCollection:
async def load_from_persistence(self) -> None: ...
async def persist(self) -> None: ...
async def unload(self) -> None: ...
def get(self, node_id: str) -> SyncNode | None: ...
def get_by_data_id(self, node_type: str, data_id: str) -> SyncNode | None:
node = self._get_local(node_type, data_id)
if node is not None:
return node
return self._get_global_fallback(node_type, data_id)
```
### 3.3 BindingManager
#### 角色
`BindingManager` 也是 scope 级对象,不应共享。
它负责:
1. 当前 scope 的绑定运行态。
2. 当前 scope 的绑定持久化。
3. 当前 scope 未命中时对 global 只读绑定做回退查询。
#### 主要方法伪代码
```python
class BindingManager:
async def load_from_persistence(self, node_type: str) -> None: ...
async def persist(self) -> None: ...
async def unload(self) -> None: ...
async def get_remote_id(self, node_type: str, local_id: str) -> str | None:
remote_id = self._get_local_remote_id(node_type, local_id)
if remote_id is not None:
return remote_id
return await self._get_global_remote_id(node_type, local_id)
```
### 3.4 DataSource
#### 角色
`local_datasource / remote_datasource / logging` 不应按项目变化,这一点是对的。
但按当前实现,`BaseDataSource` 也不适合直接做单例,因为它内部持有:
1. `_collection`
2. `_stats`
3. `_action_summary`
4. `_action_by_node_id`
所以第一阶段不建议“直接把当前 `BaseDataSource` 改成单例 + 注入 scope”。
#### 推荐的最小改法
第一阶段先不强行拆成两个正式类,而是采用下面的语义:
1. datasource 的静态配置和共享 client 是 batch 级共享的。
2. datasource 的运行态对象仍然是项目级创建的。
也就是说:
1. 配置不按项目变化。
2. runtime 仍然按项目隔离。
这样改动范围最小,也更不容易污染现有 handler 行为。
#### 主要方法伪代码
```python
class ScopedDataSourceRuntime(BaseDataSource):
def set_collection(self, collection: DataCollection) -> None: ...
async def load_all(self, order: list[str]) -> None: ...
async def sync_all(self, order: list[str]) -> None: ...
def reset_runtime_state(self) -> None: ...
async def close(self) -> None: ...
```
当前多项目配置要求:
1. `global_config`
### 第二阶段:清理生命周期
把项目级清理流程固定下来:
1. `load`
2. `sync`
3. `persist`
4. `unload`
5. `close`
### 第三阶段:再决定 datasource 是否正式拆层
如果第一阶段跑通,可以再决定是否把 datasource 正式拆成:
1. 共享静态层。
2. scope runtime 层。
这一步不是必须第一批完成。
### 最终落点
最终希望达到的状态是:
1. global 数据只同步一次,只读共享。
2. project 数据按 scope 独立运行。
3. 项目跑完即可卸载运行态。
4. 大部分改动都限制在基础设施层。
1. 全局阶段一份配置。
2. 项目阶段一份默认配置。
3. 每个项目还能再覆盖一份配置。
这对于少量项目还能接受,但项目数到了几百个就不可维护了。
如果每个项目都允许自带一份 pipeline 配置,系统就失去“批量同步”的基本前提:
1. 无法判断哪些差异是业务差异,哪些是配置漂移。
2. 无法稳定复用并发调度逻辑。
3. 无法保证结果可复现。
结论是:多项目批量同步不应允许“每项目一套 pipeline 定义”。
+9
View File
@@ -37,6 +37,10 @@
| `ABNORMAL` | 数据损坏或核心绑定异常 | 否 |
| `DEPENDENCY_ERROR` | 依赖不可满足 | 否 |
bootstrap 特殊语义:
- 如果节点在持久化恢复后存在绑定,但 bootstrap datasource refresh 未返回最新数据,则节点进入 `S17` 语义并被 `bootstrap_blocked` 隔离。
- 如果节点仅残留在 persistence 中、没有绑定,且 datasource refresh 也未返回,则节点按 stale zombie 清理,不再进入 bind。
## 3. create / update 语义
### create
@@ -64,6 +68,11 @@
- 状态落地:`Decision.to_state` 对应 YAML `states[Sxx]`
- 不变量:每次落地后执行 `runtime.check_invariants()`
bootstrap 额外输入:
- `is_create_zombie`
- `bootstrap_source_present`
- `has_binding_record`
## 6. 关键一致性约束
- `UNCHECKED` 不允许带动作(`INV-1_UNCHECKED_ACTION_NONE`)。
+68
View File
@@ -0,0 +1,68 @@
# ECM Sync Quality Cleanup Execution Plan
## Goal
This cleanup pass keeps behavior unchanged and prioritizes deleting, merging, and unifying duplicated code on the single-pipeline path.
Acceptance remains unchanged:
- `backend/local_tests/sync_system_tests/depm_to_ecm_bootstrap_pull_then_project_push.pipeline.yaml` must still run successfully.
- Local Summary and Remote Summary must keep the same structure and stable output shape.
## Constraints
- Do not modify `schemas/`.
- Do not modify `filtered_datasource/`.
- Do not bypass schema validation.
- Do not add fallback logic that changes behavior.
- Prefer helper extraction, deletion, and merge over new abstraction layers.
## Execution Order
### Batch 1: Entry + Factory
Status: completed
- Turn `run.py` into a thin wrapper around `sync_state_machine.cli.main`.
- Merge duplicated datasource lifecycle and resolved-config logging in `sync_state_machine/pipeline/factory.py`.
- Verify with focused unit tests and the real DEPM to ECM profile.
### Batch 2: Pipeline Flow + Backend Runner
Status: completed
- Reduce repeated phase-completion, strategy-loop, reload-skip, and close-handling code in `sync_state_machine/pipeline/full_sync_pipeline.py`.
- Reduce repeated orchestration code in `backend/app/thirdparty/ecm_sync/scripts/depm_to_ecm/run_pipeline.py` without changing CLI behavior or fail-fast rules.
- Keep log text stable where practical.
### Batch 3: Datasource + Handler Base Cleanup
Status: completed
- Reduce duplicated handler registration and context injection between datasource base classes and subclasses.
- Reduce duplicated node metadata initialization across handler subclasses by pushing safe defaults into handler base classes.
- Delete domain-level constructor boilerplate where subclasses only repeat node type, node class, schema, or no-op `_create_node` wrappers.
### Batch 4: Domain Repeat Cleanup + Docs
Status: in progress
- Inspect domain handlers for repeated load/update helper logic and merge only stable repeated fragments into shared helpers.
- Update docs so they only describe the real maintained execution path.
- Keep backend runner documentation aligned with current script behavior.
- Remove stale references only after code paths are confirmed stable.
## Validation Plan
1. Run targeted `ecm_sync_system` unit tests for the touched pipeline code.
2. Run the real backend profile with the backend interpreter.
3. Confirm the output still ends with pipeline completion plus Local Summary and Remote Summary tables.
## Current Focus
Current implementation focus is Batch 4.
- repeated load/update helpers under `sync_state_machine/domain/`
- maintained execution-path docs under `docs/`
The intent is code reduction only, not behavior redesign.
+419 -209
View File
@@ -1,298 +1,508 @@
# 运行配置业务说明
# 运行配置写法说明
本文档从业务视角说明配置字段含义、对阶段行为的影响,以及推荐的覆盖方式
这份文档只讲一件事:现在 `ecm_sync_system` 的配置文件应该怎么写
## 0. 配置约定
重点不是把所有默认值抄一遍,而是说明当前真实生效的写法、哪些字段是内置能力、哪些字段来自业务侧扩展,以及单项目和多项目文件各自长什么样。
当前运行配置分成两层:
## 1. 先分清两类配置文件
1. 输入配置
- 你写在 `run_profiles/*.yaml``run_profiles/preset/*.default.yaml` 里的内容。
- 只建议写必要覆盖项,不要把默认值全部展开。
2. Resolved Full Config
- pipeline 启动时打印并写入日志的全量配置。
- 其中已经补齐 datasource 默认值、策略默认值和日志/持久化默认值。
当前有两类 YAML
也就是说:
- 模板应当尽量短。
- 排障时看启动日志里的 **Resolved Full Config**
1. 单 pipeline 配置
- 对应 `PipelineRunConfig`
- 用来描述一条完整同步链路
- 顶层通常包含:`node_types``local_datasource``remote_datasource``persist``logging``strategies`
2. 多项目编排配置
- 对应 `MultiProjectRunConfig`
- 只负责描述“先跑全局,再按项目并发跑哪些项目”
- 顶层包含:`global_config``default_project_config``max_concurrency``project_bootstrap_node_types``projects`
### 0.1 一个最小输入示例
判断规则很简单:
- 如果文件里有 `global_config``default_project_config``projects`,它就是多项目编排配置。
- 否则按单 pipeline 配置解析。
## 2. 配置加载约定
当前约定如下:
- 跟踪在仓库里的模板和示例,只写必要覆盖项,不重复展开默认值。
- pipeline 启动后会打印一份 `Resolved Full Config`,它才是最终生效的完整配置。
- 排障优先看 `Resolved Full Config`,不要只盯着输入 YAML。
## 3. 占位符规则
`ecm_sync_system` 核心库内置只认识一个占位符:
- `${PROJECT_ROOT}`:运行时替换为 `ecm_sync_system` 仓库根目录绝对路径。
例如:
```yaml
persist:
backend: sqlite
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
```
需要特别区分的是:
- `${BACKEND_ROOT}``${ECM_SYNC_LOG_DIR}` 这类占位符不是核心库通用语义。
- 它们通常是 backend 集成脚本在读取 profile 时额外替换的调用方语义。
- 如果你不是通过 backend 的联调脚本运行,而是直接在 `ecm_sync_system` 内跑 profile,就不要假设这些占位符一定可用。
## 4. 单 pipeline 配置的当前写法
### 4.1 最小骨架
```yaml
node_types:
- company
- supplier
- project
local_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
remote_datasource:
type: jsonl
jsonl_dir: ${PROJECT_ROOT}/remote_datasource/datasource
type: api
api_base_url: http://127.0.0.1:8000/api/third/v2
api_uid: your_uid
api_secret: your_secret
persist:
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
backend: sqlite
backend_options: {}
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
logging:
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
file_name_pattern: demo_{timestamp}.log
strategies:
project:
config:
auto_bind: true
auto_bind_fields:
- name
local_orphan_action: none
remote_orphan_action: none
update_direction: push
```
### 0.2 对应的全量 resolved 结果骨架
### 4.2 顶层字段含义
- `node_types`
- 本轮参与同步的节点类型列表。
- 顺序会影响执行顺序,依赖链通常要把上游类型放前面。
- `local_datasource` / `remote_datasource`
- 定义两侧数据源类型和对应参数。
- `persist`
- 定义持久化后端及目标位置。
- `logging`
- 定义日志初始化、落盘位置和级别。
- `strategies`
- 定义每个 `node_type` 的同步策略。
## 5. datasource 字段怎么写
### 5.1 内置 datasource type
核心库内置注册了两种 datasource type
1. `jsonl`
2. `api`
#### `jsonl` 写法
```yaml
node_types:
- company
- supplier
- user
- project
...
local_datasource:
type: jsonl
jsonl_dir: /abs/path/filtered_datasource/datasource/filtered
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
read_only: false
handler_configs: {}
handler_configs:
project:
data_id_filter:
- project-id-1
```
字段说明:
- `jsonl_dir`JSONL 数据目录。
- `read_only`:是否只读。
- `handler_configs`:按 `node_type` 下发给 handler 的扩展参数。
#### `api` 写法
```yaml
remote_datasource:
type: jsonl
jsonl_dir: /abs/path/remote_datasource/datasource
read_only: false
handler_configs: {}
type: api
api_base_url: http://127.0.0.1:8000/api/third/v2
api_uid: your_uid
api_secret: your_secret
api_debug: true
poll_max_retries: 10
poll_interval: 0.5
api_rate_limit_max_requests: 30
api_rate_limit_window_seconds: 10.0
handler_configs:
project:
data_id_filter:
- remote-project-id-1
```
字段说明:
- `api_debug`:开启后输出更详细的请求日志。
- `poll_max_retries` / `poll_interval`:异步任务轮询控制。
- `api_rate_limit_max_requests` / `api_rate_limit_window_seconds`:全局限流窗口。
- `handler_configs`:同样按 `node_type` 下发业务参数。
### 5.2 业务侧注册的 datasource type
核心库允许业务系统在启动阶段注册自己的 datasource type。
因此,配置里不只可以写 `jsonl``api`,还可以写业务侧注册出来的类型,例如 backend 里的 `depm_db`
当前 backend 的 `depm_db` 真实写法是:
```yaml
local_datasource:
type: depm_db
tenant_id: "019976df-0628-7c46-a896-81d3f290a7df"
app:
config_file: ${BACKEND_ROOT}/config.yaml
overrides:
database:
url: mysql+aiomysql://root:123456@127.0.0.1/depm_prod_local_auto_test?charset=utf8mb4
handler_configs:
project:
data_id_filter:
- 019a62a5-2323-7be2-99c8-82534f8befa4
```
这个例子说明两点:
1. datasource 的可写字段由对应注册模型决定,不是所有 type 都共用同一套字段。
2. 如果 `type` 没有先被注册,配置校验会直接报错,不会做降级处理。
`depm_db` 这类业务扩展字段需要以业务侧实现为准。对 backend 当前实现来说:
- `tenant_id`DEPM 本地 datasource 的租户过滤 ID。
- `app.config_file`:backend 配置文件路径,默认可落回 `backend/config.yaml`
- `app.overrides`:直接传给 backend `Settings()` 的覆盖项。
## 6. strategy 写法
### 6.1 当前推荐写法
`strategies` 现在推荐写成“按 `node_type` 为键”的映射,而不是手工写一长串列表。
```yaml
strategies:
- node_type: project
skip_sync: false
supplier:
skip_post_check: true
config:
auto_bind: true
auto_bind_fields:
- name
pre_bind_data_id: []
- credit_code
depend_fields: {}
local_orphan_action: none
remote_orphan_action: create_local
update_direction: none
project:
domain_option:
pull_group_plan_production_date: true
config:
auto_bind: false
auto_bind_fields: []
pre_bind_data_id:
- local_data_id: local-project-id
remote_data_id: remote-project-id
depend_fields:
company_id: company
local_orphan_action: create_remote
local_orphan_action: none
remote_orphan_action: none
update_direction: push
field_direction_key: null
field_direction_overrides: {}
post_check_ignore_fields: []
compare_ignore_fields: []
```
### 6.2 常用字段
- `auto_bind`
- 是否开启自动绑定。
- `auto_bind_fields`
- 自动绑定使用的业务键字段。
- `pre_bind_data_id`
- 显式 local/remote data_id 绑定对。
- `depend_fields`
- 依赖字段到依赖 `node_type` 的映射。
- `local_orphan_action` / `remote_orphan_action`
- 孤儿节点创建策略。
- `update_direction`
- `push` / `pull` / `none`
- `post_check_ignore_fields`
- 只影响最终一致性检查,不影响 create/update 差异检测。
- `compare_ignore_fields`
- schema diff / validate 时忽略的顶层字段。
- `skip_sync`
- 只跳过 create/update,不跳过 load/bind。
- `skip_post_check`
- 跳过该类型的 post-check reload 与一致性校验。
- `domain_option`
- 领域策略自己的扩展参数,由对应 domain 自己解释。
### 6.3 `preset` 仍然可用
如果只是想套一组预设策略,可以这样写:
```yaml
strategies:
project:
preset: pull_only
```
当前内置预设包括:
- `first_sync`
- `daily_sync`
- `repair_sync`
- `pull_only`
- `push_only`
### 6.4 已废弃的旧字段不要再写
以下 legacy key 已明确拒绝:
- `force_sync`
- `load_mode`
如果你在 strategy 里写这两个字段,配置会直接报错。
当前替代方式是:
- 是否跳过同步,用 `skip_sync`
- handler 特有加载参数,写到 `datasource.handler_configs.<node_type>`
## 7. persist 的当前写法
### 7.1 校验规则
当前 `persist` 配置遵循下面的规则:
1. `db_path``url` 至少要有一个。
2.`backend != sqlite` 时,必须显式提供 `url`
3. `wipe_on_start=true` 只会尝试删除可映射到文件路径的持久化文件;对 MySQL 这类远程数据库不会帮你删库。
### 7.2 SQLite 文件持久化
```yaml
persist:
backend: sqlite
db_path: /abs/path/test_results/sync_state_machine/demo.db
backend_options: {}
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/simple_pipeline_sm.db
enable: true
wipe_on_start: false
```
### 7.3 SQLAlchemy 持久化到 SQLite URL
```yaml
persist:
backend: sqlalchemy
url: sqlite+aiosqlite:////abs/path/to/pipeline.db
enable: true
wipe_on_start: false
```
### 7.4 SQLAlchemy 持久化到 MySQL
```yaml
persist:
backend: sqlalchemy
url: mysql+aiomysql://root:123456@127.0.0.1/ecm_sync_system?charset=utf8mb4
enable: true
wipe_on_start: false
```
当前语义:
- `sqlalchemy` backend 会自动创建它自己的 `nodes``bindings` 表及索引。
- 它不会替你创建 MySQL 数据库本身,所以 `ecm_sync_system` 这个库需要提前存在。
- 现有表结构已经兼容 MySQL,不需要再额外给主键字段手工指定前缀长度。
### 7.5 `backend_options` 什么时候用
`backend_options` 仍然保留,但现在优先使用显式字段:
- `db_path`
- `url`
- `enable`
- `wipe_on_start`
除非某个自定义 backend 明确要求,否则不要把主连接信息再塞进 `backend_options`
## 8. logging 的当前写法
```yaml
logging:
initialize: true
file_path: null
file_dir: null
file_name_pattern: null
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
file_name_pattern: simple_pipeline_sm_{timestamp}.log
level: 20
suppress_node_logs: true
```
## 1. 推荐使用方式(先 default,再覆盖)
字段说明:
1. 先使用基线配置运行:
- `run_profiles/preset/jsonl_to_jsonl.default.yaml`
2. 若需项目定制,再复制为:
- `run_profiles/jsonl_to_jsonl.local.yaml`
3. 仅改有业务差异的字段,避免一次改太多导致排障困难。
- `initialize`
- 是否初始化应用日志。
- `file_path`
- 直接指定日志文件绝对路径。
- `file_dir`
- 只指定目录时,实际文件名由 `file_name_pattern` 生成。
- `file_name_pattern`
- 支持 `{timestamp}` 占位。
- `level`
- 默认 `20`,即 `INFO`
- `suppress_node_logs`
- 是否压缩逐节点日志噪音。
占位符:
- `${PROJECT_ROOT}` 会在运行时替换为项目根目录绝对路径。
## 9. 多项目编排文件怎么写
## 2. 顶层字段(你在改什么)
### 9.1 结构
- `node_types`
- 本轮参与同步的业务类型列表。
- 顺序会影响依赖链执行时机(当前按 node_type 串行执行 bind/create/update)。
- `local_datasource` / `remote_datasource`
- 分别定义两侧数据来源与执行端。
- `strategies`
- 每个业务类型的行为策略(依赖、自动绑定、创建方向、更新方向、跳过等)。
- `persist`
- 持久化开关与数据库位置。
- `persist.backend`
- 持久化后端注册表里的名称,例如 `sqlite`
- backend 实现会自行解释 `persist` 的其余字段。
- `persist.backend_options`
- 后端私有配置,后端自己决定如何解释。
### 6. 持久化后端注册
如果你希望在 backend 初始化阶段替换持久化后端,可以先在启动代码里按名称注册,再通过配置选择,不需要改调用方。
#### 6.1 内置 SQLite
多项目文件不是把所有字段都堆在一起,而是引用两份单 pipeline 配置:
```yaml
persist:
backend: sqlite
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
backend_options: {}
global_config: run_profiles/jsonl_to_jsonl.multi_project.global.yaml
default_project_config: run_profiles/jsonl_to_jsonl.multi_project.project.yaml
project_bootstrap_node_types:
- company
- supplier
max_concurrency: 8
projects:
- local_project_id: local-project-1
remote_project_id: remote-project-1
- local_project_id: local-project-2
remote_project_id: remote-project-2
```
#### 6.2 注册自定义后端
字段说明:
```python
from sync_state_machine.common.persistence import register_persistence_backend
- `global_config`
- 全局公共节点要跑的单 pipeline 配置。
- `default_project_config`
- 每个项目默认使用的单 pipeline 配置。
- `project_bootstrap_node_types`
- 需要从 global scope 继承到项目 scope 的公共节点类型,例如 `company``supplier`
- `max_concurrency`
- 项目并发数上限,默认 `20`
- `projects`
- 项目映射清单。
register_persistence_backend("mysql", lambda config: MySQLPersistenceBackend(config))
```
### 9.2 项目项支持单独切配置
然后在配置里选择这个名字
如果某个项目需要特殊策略,可以在项目项里覆盖 `config_path`
```yaml
persist:
backend: mysql
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
backend_options:
url: mysql+aiomysql://user:pass@127.0.0.1/depm_sync?charset=utf8mb4
pool_size: 5
projects:
- local_project_id: local-project-1
remote_project_id: remote-project-1
- local_project_id: local-project-2
remote_project_id: remote-project-2
config_path: run_profiles/special_project.pipeline.yaml
```
后端实现会收到整份 `PersistConfig`,自行决定是读取 `db_path``backend_options`,还是额外扩展别的字段
- `logging`
- 运行日志输出位置与级别。
此时该项目不会再用 `default_project_config`,而是改用自己的单 pipeline 配置文件
## 3. 数据源字段(DataSource
### 9.3 当前多项目 profile 的实际风格
### 3.1 JSONL 数据源
以当前 backend 两项目推送链路为例,常见组合是:
- `type=jsonl`:使用本地 JSONL 文件作为数据源。
- `jsonl_dir`JSONL 目录路径。
- `read_only=false`:允许该侧执行 create/update 回写;若为 true,通常用于只读加载。
- `handler_configs.project.data_id_filter=[]`:项目白名单;为空时不进行项目筛选。
- `handler_configs`:按节点类型下发给各业务 handler 的扩展参数。
- `local_datasource.type: depm_db`
- `remote_datasource.type: api`
- `persist.backend: sqlalchemy`
- `persist.url: mysql+aiomysql://...`
- `strategies.project.config.pre_bind_data_id` 明确写每个项目的 local/remote 映射
### 3.2 API 数据源
这类 profile 本质上仍然是“多项目编排文件 + 两份单 pipeline 文件”的组合,不要把它理解成第三种配置格式。
- `type=api`:对接 HTTP API。
- `api_base_url` / `api_uid` / `api_secret`:接口访问参数。
- `api_debug`:开启后输出更多调试信息。
- `poll_max_retries` / `poll_interval`:异步任务轮询次数与间隔。
- `handler_configs.project.data_id_filter`:建议显式配置的项目白名单;至少需要指定一个项目ID,以避免加载过多远程数据。
- `handler_configs`:业务 handler 扩展参数。
## 10. 当前推荐的写法习惯
## 4. 策略字段(Strategy
### 10.1 只写必要覆盖项
每个 node_type 的 **Resolved Full Config** 都会包含以下字段,但输入配置不需要全部显式写出。
推荐:
### 4.1 `update_direction`
- 只写与你当前环境、项目过滤、持久化目标、策略差异直接相关的字段。
- `push`:本地变更推送到远端。
- `pull`:远端变更拉取到本地。
- `bidirectional`:双向更新(需要明确冲突策略,谨慎使用)。
- `none`:关闭 update 阶段。
不推荐:
### 4.2 `local_orphan_action` / `remote_orphan_action`
- 把默认值全部展开复制一遍。
- `create_remote`:本地孤儿创建到远端。
- `create_local`:远端孤儿创建到本地。
- `none`:不自动创建。
- `delete_local` / `delete_remote`:当前主流程未作为默认路径,谨慎使用。
原因很直接:
### 4.3 `auto_bind` / `auto_bind_fields` / `pre_bind_data_id`
- 默认值一旦调整,你的大段本地 YAML 会悄悄过时。
- 排障时也更难看出你到底改了什么。
- `auto_bind=true`:在 `S02(MISSING)` 上尝试自动匹配。
- `auto_bind=false`:不会做自动匹配,也不会走自动创建识别路径;通常会进入阻断态等待人工处理。
- `auto_bind_fields`:自动匹配使用的业务键字段。
- `pre_bind_data_id`:显式 data_id 预绑定清单(通用能力),格式为:
- `- local_data_id: "..."`
` remote_data_id: "..."`
- 规则:若两侧数据源都存在这对 data_id 对应节点,且当前无绑定记录,则直接绑定。
- 不依赖 `auto_bind/auto_bind_fields`,即使 `auto_bind=false` 也会执行。
- 不做业务键一对一匹配检查,按你给定的映射对优先执行。
### 10.2 handler 特有参数放 datasource 里
### 4.4 `depend_fields`
- 定义依赖字段到依赖 node_type 的映射,例如:`company_id -> company`
- 依赖节点状态非 `NORMAL` 时,会在 bind 的依赖检查阶段进入 `DEPENDENCY_ERROR`
### 4.5 `skip_sync`
- 运行配置中已不建议使用 `skip_sync`
- 推荐使用显式组合:
- `local_orphan_action: none`
- `remote_orphan_action: none`
- `update_direction: none`
- `skip_sync=true` 的当前语义是:
- 仍会执行该 node_type 的 load / bind
- 只跳过 create / update 写入阶段
- 适合“保留绑定与依赖可见性,但临时禁止写入”的场景。
- `none/none/none` 是更推荐的输入风格,因为它表达的是“该类型不做创建、不做补本地、不做更新”。
- 这组配置不会绕过前置 load / bind / 依赖检查;它只会把该类型变成无写入策略。
### 4.6 Handler 级参数(`handler_configs`
- 建议将 handler 特有参数放在 datasource 的 `handler_configs.<node_type>` 下。
- 示例:
- `remote_datasource.handler_configs.supplier.load_mode: persisted_only`
- `persisted_only` 表示只使用已持久化数据,不主动从外部拉新数据(具体以对应 handler 实现为准)。
### 4.7 `field_direction_key` / `field_direction_overrides`
- 这两个字段只在少数 node_type 上需要,典型例子是 `project_detail_data`
- `field_direction_key`
- 指定从节点 data 里取哪个字段作为“方向查表键”。
- 例如 `key`,表示用 `data["key"]` 决定这条记录是 push 还是 pull。
- `field_direction_overrides`
- 指定“字段值 -> 更新方向”的映射。
- 没命中的值继续使用默认 `update_direction`
示例:
比如:
```yaml
project_detail_data:
config:
field_direction_key: key
field_direction_overrides:
ensure_production: pull
climb_production: pull
challenge_production: pull
remote_datasource:
type: api
handler_configs:
contract_settlement_detail:
load_method: aggregate
```
含义:
- `key=ensure_production/climb_production/challenge_production` 时,从远端拉回本地。
- 其余 key 继续按默认 `update_direction=push` 处理。
这类参数应该写在 `datasource.handler_configs.<node_type>`,不要错写到 strategy 里。
## 5. 阶段行为与配置关系
### 10.3 项目过滤显式下沉到 handler
- bind 阶段
- `depend_fields` 决定是否做依赖校验。
- `pre_bind_data_id` 先按显式 local/remote data_id 映射做预绑定。
- `auto_bind/auto_bind_fields` 决定是否继续进行业务键自动绑定。
- create 阶段
- `local_orphan_action` / `remote_orphan_action` 决定孤儿是否创建。
- update 阶段
- `update_direction` 决定是否和向哪侧更新。
- 写入后 reloadPipeline 内置)
- 对 API / 需要重新拉取权威状态的数据源,create 或 update 成功后仍会按 node_type reload。
- 对纯 JSONL -> JSONL pipelinereload 会跳过,直接复用当前内存状态。
- 目的:避免对离线 JSONL 数据做无意义重复扫描,同时保留 API 数据源的权威状态刷新。
- 同步后一致性检查(Pipeline 内置)
- 每个 node_type 在本轮同步后会基于绑定对执行数据一致性校验(比较时忽略 `id/_id` 字段)。
- 若发现差异,会在日志中打印差异样本,并在汇总表新增 `Consistent` 列展示 `Y` / `N(差异数)`
- 全局跳过
- `local_orphan_action=none` + `remote_orphan_action=none` + `update_direction=none`
可将该类型收敛为“只做 load/bind,不做 create/update 写入”的只读同步策略。
例如:
## 6. 覆盖配置建议
```yaml
handler_configs:
project:
data_id_filter:
- project-id-1
- project-id-2
```
- 先在 default 跑通一轮,再做 local 覆盖
- 每次只改一类问题相关字段(例如只改 `project` 的依赖与自动绑定)。
- 覆盖后优先看 `sync_log` 是否出现预期状态迁移。
不要在 pipeline 外层再做一层临时过滤逻辑。项目范围应尽量体现在配置里,并由对应 handler 解释
## 7. 持久化与日志字段
### 10.4 看到报错先修配置,不要做降级
### `persist`
当前系统对配置保持 fail-fast:
- `enable`:是否开启持久化。
- `db_path`SQLite 路径。
- `wipe_on_start`:启动是否清理历史持久化。
- datasource type 未注册会报错
- legacy strategy key 会报错
- `persist` 缺失目标会报错
- profile 结构不符合 schema 会报错
### `logging`
这属于预期行为。优先修正配置,而不是在文档或调用侧加 fallback。
- `file_dir`:日志目录。
- `file_name_pattern`:文件命名模板。
- `level`:日志级别。
## 11. 参考文件
排障建议:运行失败时优先看日志中的 `状态机流转``reason=`,再对照 `docs/state_machine.md``docs/node_state_definition.md` 理解含义。
如果你想直接对照真实样例,优先看这些文件:
- `run_profiles/jsonl_to_api.yaml`
- `run_profiles/jsonl_to_jsonl.multi_project.yaml`
- backend 中的 `local_tests/sync_system_tests/depm_to_ecm_bootstrap_pull_then_project_push.two_projects.sqlalchemy.pipeline.yaml`
如果你想看配置模型定义,直接看:
- `sync_state_machine/config/pipeline_config.py`
- `sync_state_machine/config/datasource_config.py`
- `sync_state_machine/config/persist_config.py`
- `sync_state_machine/config/multi_project_config.py`
- `sync_state_machine/config/strategy_config.py`
排障时优先顺序:
1. 看输入 YAML。
2. 看启动日志里的 `Resolved Full Config`
3. 看对应 datasource type 是否真的已经注册。
4. 再看运行日志和状态机结果。
+7 -4
View File
@@ -21,6 +21,7 @@
- `S12C/S12U/S12D`:按动作拆分的执行失败状态
- `S13`:对侧创建等待态(`PEER_CREATING/NONE/PENDING`
- `S14`UPDATE 运行时降级为 `NONE` 并跳过
- `S17`:bootstrap 加载异常隔离(存在绑定,但 datasource refresh 未返回最新节点)
## 3. 事件到代码映射
@@ -43,14 +44,15 @@
- `E40D_START/E40D`DELETE 执行入口与结果回写
## 4. strategy 调用链(当前)
- bind 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.bind()` -> `sync_state_machine/sync_system/strategy_ops/bind_ops.py::run_bind()`
- create 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.create()` -> `sync_state_machine/sync_system/strategy_ops/create_ops.py::run_create()`
- update 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.update()` -> `sync_state_machine/sync_system/strategy_ops/update_ops.py::run_update()`
- reset 主入口:`sync_state_machine/sync_system/strategy.py::BaseSyncStrategy.run_reset()` -> `sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`
- bind 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.bind()`
- create 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.create()`
- update 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.update()`
- reset 主入口:`sync_state_machine/pipeline/full_sync_pipeline.py::_load_persistence_state()` -> `sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`
## 5. context 字段(以 YAML 为准)
### bind 相关
- `bootstrap_source_present`
- `has_binding_record`
- `core_binding_result` / `peer_core_binding_result` (`NORMAL|WARNING|ABNORMAL`)
- `data_present`
@@ -75,6 +77,7 @@
## 6. 语义边界
- strategy 负责业务语义计算(如依赖是否满足、自动绑定结果、差异检测)。
- 状态机只负责迁移判定与不变量校验,不直接访问数据源。
- bootstrap 阶段的数据源刷新由 pipeline 完成,状态机只消费 `bootstrap_source_present / has_binding_record / is_create_zombie` 这些已计算上下文。
## 7. 不变量
- `INV-1_UNCHECKED_ACTION_NONE`
+35 -12
View File
@@ -11,42 +11,65 @@
## 2. reset 时序
### 2.1 加载时 resetphase1 语义)
### 2.1 加载时恢复
-`sync_state_machine/common/collection.py::load_from_persistence()` 执行。
- 语义:仅恢复持久化状态不在此阶段做状态归并
- 语义:仅恢复持久化状态,并记录本轮 `sync_log``persistence` 恢复来源;不在此阶段做状态归并。
- 恢复后保留 `data/origin_data`,等待 bootstrap 阶段的 datasource 刷新覆盖最新数据。
### 2.2 加载后 cleanupphase2 语义)
- pipeline 调用 `BaseSyncStrategy.run_reset()`
### 2.2 bootstrap datasource refresh
- pipeline 在持久化恢复后、`E01` 之前,按已恢复到内存的 `node_type` 做一次 datasource 刷新
- 实际实现:`sync_state_machine/pipeline/full_sync_pipeline.py::_bootstrap_refresh_latest_data()`
- 语义:
- 如果 datasource 命中节点,则用最新数据覆盖持久化快照,并清除 `_loaded_from_persistence` 标记。
- 如果 datasource 未命中节点,则节点保留持久化快照,供 `E01` 判断它是 stale zombie 还是 bootstrap 异常隔离对象。
### 2.3 加载后 cleanup
- pipeline 在 datasource refresh 完成后调用 `run_phase2_cleanup()`
- 实际实现:`sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`
- 核心事件:`E01``S16 -> S00/S15`)。
- `S16` 表示“持久化加载输入态”:本轮从持久化恢复出的节点集合(可包含上轮遗留节点,也可包含上轮已稳定节点)
- `S16` 表示“bootstrap 输入态”:持久化恢复出的节点,叠加 datasource refresh 后仍待 `E01` 判定的节点集合
- 判定:
- `is_create_zombie=true` -> `S15`,节点删除(并尝试解绑);
- `is_create_zombie=false` -> `S00`节点归并为 `UNCHECKED/NONE/PENDING` 并清空 `error`
- `bootstrap_source_present=false && has_binding_record=false` -> `S15`说明该节点仅残留在 persistence 中且未绑定,按 stale zombie 清理;
- `bootstrap_source_present=false && has_binding_record=true` -> `S17`,说明该节点存在绑定,但 datasource refresh 未返回,进入 bootstrap 加载异常隔离;
- 其余情况 -> `S00`,节点归并为 `UNCHECKED/NONE/PENDING` 并清空 `error`
### 2.2.1 `enable_persistence` 与 bootstrap 关系
- `enable_persistence=true`:先 `load_from_persistence()`,再对加载出的节点执行 `E01``S16` 域内处理)。
- `enable_persistence=true`:先 `load_from_persistence()`,再做 bootstrap datasource refresh,最后对输入节点执行 `E01``S16` 域内处理)。
- `enable_persistence=false`:本轮没有持久化输入,`S16/E01` 不参与;随后 Stage1 新加载节点直接进入本轮 bind/create/update/sync 主流程。
### 2.3 加载异常隔离(S17
### 2.4 加载异常隔离(S17
- 持久化中若出现非法枚举值(`action/status/binding_status`),节点标记为 `bootstrap_blocked``S17` 语义)。
- 持久化恢复后若节点存在绑定,但 bootstrap datasource refresh 未返回该节点,也会进入 `S17` 语义并隔离。
- 这类节点保留 `error`,用于人工排查。
- 这类节点不会进入 `E01`,也不会参与 bind/create/update/sync 自动流程。
## 3. bind 流程(strategy_ops
- 入口:`sync_state_machine/sync_system/strategy_ops/bind_ops.py::run_bind()`
- 入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.bind()`
- 入口状态:`S00`
### bind 编排步骤
- strategy 先通过 `bind_ops.collect_bind_entry_nodes()` 收集两侧 `S00` 节点,作为 bind 入口工作集。
- `phase_core_state()` 对入口工作集触发 `E10`,把“已有绑定记录”和“无绑定记录”的节点分流到 `S01/S03/S04/S02`
- `phase_dependency_check()` 对仍需继续处理的缺绑节点触发 `E15`,把缺数据、依赖不满足、语义错误的节点拦到 `S03/S05/S04`
- strategy 再通过 `bind_ops.collect_auto_bind_ready_nodes()` 收窄工作集,只保留当前 `S02 + binding_status=MISSING` 的节点进入自动绑定判定。
- strategy 在顶层显式构造 orphan create 控制参数,并传给 `bind_ops.plan_auto_bind_updates()` 生成自动绑定/自动创建决策。
- `apply_auto_bind_updates()` 对这些决策逐个触发 `E16`,把节点迁移到 `S01/S04/S05/S13`
- `refresh_bind_data_id()` 最后按绑定关系回填两侧节点的 `context.bind_data_id`
### 核心绑定判定
- `phase_core_state()` 内触发 `E10`
- 有绑定记录:根据本端/对端有效性进入 `S01/S03/S04`
- 无绑定记录:进入 `S02`
注意:如果节点在 bootstrap 阶段已被隔离为 `S17` 语义(`bootstrap_blocked=true`),默认不会进入 bind 工作集。
### 依赖检查
- `phase_dependency_check()` -> `check_dependencies_for_nodes()` 内触发 `E15`
- `data_present=true``dep_satisfied=true` -> 不迁移,节点保持在 `S02`,继续进入自动绑定判定。
- `data_present=false` -> `S03`
- `dep_satisfied=false` -> `S05`
- `depend_fields` 为空 -> `S04`(无法自动处理,阻断后续自动创建)
- `depend_fields` 为空本身不会触发阻断;若节点数据存在,则等价于“当前依赖满足”,继续留在 `S02`
### 自动绑定
- `phase_auto_binding()` 内触发 `E16`
@@ -57,7 +80,7 @@
## 4. create/update 准备
### create
- 入口:`sync_state_machine/sync_system/strategy_ops/create_ops.py::run_create()`
- 入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.create()`
- 运行时事件:`E20``E21/E22` 为配置保留分支,当前 strategy 未接入自动执行)
- 运行时入口状态:`S13`
- 说明:`E16` 负责识别“需自动创建”并将源节点置为 `S13`;业务代码执行对侧 spawn+bind 后,再通过 `E20``S13` 提交到 `S01`(成功);若 spawn 失败,`E20` 不迁移并保留在 `S13` 等待人工/重试。
@@ -66,7 +89,7 @@
- 新创建目标节点在 `S06/S10C` 阶段允许 `data_id` 为空;`CREATE SUCCESS``S11C`)必须携带非空 `result_data_id`,否则事件直接报错。
### update
- 入口:`sync_state_machine/sync_system/strategy_ops/update_ops.py::run_update()`
- 入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.update()`
- 事件:`E30`
- 入口状态(source 节点):`S01`
- `target_has_data_id=false``update_id_resolve_ok=false` -> `S08`
+2
View File
@@ -10,6 +10,8 @@ dependencies = [
"PyYAML>=6.0,<7",
"httpx>=0.27,<1",
"aiosqlite>=0.20,<1",
"SQLAlchemy>=2.0,<3",
"greenlet>=3,<4",
"pytz>=2024.1,<2027",
"fastapi>=0.115,<1",
"uvicorn>=0.30,<1",
+1 -47
View File
@@ -1,8 +1,5 @@
from __future__ import annotations
import argparse
import asyncio
import logging
import sys
from pathlib import Path
@@ -10,50 +7,7 @@ PROJECT_ROOT = Path(__file__).resolve().parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
from sync_state_machine.config import load_overrides_from_file
from sync_state_machine.pipeline import run_profile_from_file
def _resolve_config_path(raw_path: str) -> Path:
path = Path(raw_path)
if not path.is_absolute():
path = PROJECT_ROOT / path
return path.resolve()
async def _main() -> int:
parser = argparse.ArgumentParser(description="Run sync pipeline from config override file")
parser.add_argument(
"--config_path",
required=True,
help="Path to run profile file (JSON/YAML, supports relative path, e.g. run_profiles/jsonl_to_jsonl.local.json)",
)
args = parser.parse_args()
cfg_path = _resolve_config_path(args.config_path)
if not cfg_path.exists() or not cfg_path.is_file():
raise FileNotFoundError(f"Config file not found: {cfg_path}")
print(f"🧩 Loaded config file: {cfg_path}")
await run_profile_from_file(
project_root=PROJECT_ROOT,
config_path=cfg_path,
print_summary=True,
)
return 0
def main() -> int:
try:
return asyncio.run(_main())
except KeyboardInterrupt:
print("\n⚠️ Interrupted by user")
return 130
except Exception as exc:
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
return 1
finally:
logging.shutdown()
from sync_state_machine.cli import main
if __name__ == "__main__":
+29 -5
View File
@@ -5,7 +5,7 @@
from typing import List, Optional
from fastapi import HTTPException
from pydantic import Field, BaseModel, field_validator
from pydantic import Field, BaseModel, field_validator, model_validator
from ..common.push_system import BaseUpdateRequestSchema
from ..common.base import BaseResponseWithID
from ..common.datetime import get_current_date
@@ -73,12 +73,12 @@ class ProjectResponseBase(BaseResponseWithID):
progress_type: ProgressType | None = Field(None, description="推进分类", examples=[ProgressType.A])
key_constraints: KeyConstraints = Field(KeyConstraints.NONE, description="关键制约", examples=[KeyConstraints.NONE])
key_constraints_remark: Optional[str] = Field("", description="关键制约备注", examples=[""])
status: Optional[str] = Field(None, description="项目状态")
# status: Optional[str] = Field(None, description="项目状态")
apply_file_ids: List[str] = Field(default_factory=list, description="申请表附件ID列表", examples=[[]])
# 拓展信息
dbtc_score_list: Optional[list] = Field(default_factory=list, description='达标投产评分列表') # 这里项目用的是list而不是List
approval_status: Optional[str] = Field(None, description="审核状态", examples=["pending"])
# approval_status: Optional[str] = Field(None, description="审核状态", examples=["pending"])
@@ -266,11 +266,35 @@ class ProjectProductionCapacity(BaseModel):
"""
plan_production_date: DateStr | None = Field(None, description="集团公司计划投产日期", examples=["2024-08-01"])
plan_full_production_date: DateStr | None = Field(None, description="集团公司计划全部投产日期", examples=["2024-09-01"])
ic_plan_first_production_date: DateStr | None = Field(None, description="区域公司计划首批投产日期(只有新能源需要)",examples=["2024-08-01"])
ic_plan_full_production_date: DateStr | None = Field(None, description="区域公司计划全投日期(只有新能源需要)",examples=["2024-09-01"])
actual_production_date: DateStr | None = Field(None, description="实际投产日期(只有新能源需要)", examples=["2024-08-15"])
actual_full_production_date: DateStr | None = Field(None, description="实际全投日期(只有新能源需要)", examples=["2024-09-10"])
actual_production_list : list[_ActualProductionNode] | None = Field(None, description="实际投产节点列表(每次必须填写全部)(只有新能源需要)")
@model_validator(mode="before")
@classmethod
def reject_ic_plan_dates(cls, data):
if not isinstance(data, dict):
return data
blocked_fields = [
field for field in ("ic_plan_first_production_date", "ic_plan_full_production_date")
if field in data
]
if blocked_fields:
raise HTTPException(
status_code=400,
detail=f"/production_capacity 接口不支持更新字段: {', '.join(blocked_fields)}"
)
return data
@property
def ic_plan_first_production_date(self) -> str:
return ""
@property
def ic_plan_full_production_date(self) -> str:
return ""
@field_validator("actual_production_date")
def validate_actual_production_date(cls, v):
if v is None:
+1 -1
View File
@@ -36,4 +36,4 @@ class PreparationDetail(BaseResponseWithID):
# approval_status: Optional[str] = Field(None, description="审批状态")
name: Optional[str] = Field(None, description="里程碑名称")
group: Optional[str] = Field(None, description="分组条件", examples=['KGTJLS'])
is_acquired: Optional[int] = Field(0, description="是否取得")
# is_acquired: Optional[int] = Field(0, description="是否取得")
+1 -1
View File
@@ -36,4 +36,4 @@ class ProductionDetail(BaseResponseWithID):
is_exist: Optional[bool] = Field(None, description="是否取得")
# approval_status: Optional[str] = Field(None, description="审批状态")
name: Optional[str] = Field(None, description="投产节点名称")
is_acquired: Optional[int] = Field(0, description="是否取得")
# is_acquired: Optional[int] = Field(0, description="是否取得")
+1 -7
View File
@@ -48,15 +48,9 @@ class SupplierDetailResponse(BaseResponseWithID):
status: str = Field(..., description="供应商状态", examples=["active"])
sort: int = Field(..., description="排序", examples=[1])
credit_code: Optional[str] = Field(None, description="信用代码", examples=["914403007109310121"])
created_by: str = Field(..., description="创建者ID", examples=["user:001"])
created_by_name: str = Field(..., description="创建者名称", examples=["张三"])
created_at: str = Field(..., description="创建时间", examples=["2023-10-01T12:00:00Z"])
updated_by: str = Field(..., description="更新者ID", examples=["user:001"])
updated_by_name: str = Field(..., description="更新者名称", examples=["张三"])
updated_at: str = Field(..., description="更新时间", examples=["2023-10-01T12:00:00Z"])
province: Optional[str] = Field(None, description="省份编码", examples=[None])
city: Optional[str] = Field(None, description="城市编码", examples=[None])
company_nature: List[Optional[str]] = Field(None, description="供应商性质数组", examples=[['supplier']])
company_nature: List[Optional[str]] = Field([], description="供应商性质数组", examples=[['supplier']])
city_name: Optional[str] = Field(None, description="城市名称", examples=[None])
province_name: Optional[str] = Field(None, description="省份名称", examples=[None])
region_path: Optional[list] = Field(default_factory=list, description="供应商国家城市地址编码列表")
+5 -5
View File
@@ -309,11 +309,11 @@ def build_temp_run_profile_for_project_auto_bind(
project_strategy["update_direction"] = project_update_direction or "none"
if project_detail_push_keys:
project_detail_config = profile.setdefault("strategies", {}).setdefault("project_detail_data", {}).setdefault("config", {})
overrides = dict(project_detail_config.get("field_direction_overrides", {}))
for key in project_detail_push_keys:
overrides.pop(key, None)
project_detail_config["field_direction_overrides"] = overrides
fixed_pull_keys = {"ensure_production", "climb_production", "challenge_production"}
if fixed_pull_keys.intersection(project_detail_push_keys):
project_detail_runtime = profile.setdefault("strategies", {}).setdefault("project_detail_data", {})
domain_option = project_detail_runtime.setdefault("domain_option", {})
domain_option["pull_group_production_plans"] = False
runtime_dir = Path(report_root) / "runtime_configs"
runtime_dir.mkdir(parents=True, exist_ok=True)
+56 -11
View File
@@ -15,6 +15,8 @@ from urllib import error, request
import yaml
from sync_state_machine.common.persistence import PersistenceBackend
PROJECT_ROOT = Path(__file__).resolve().parents[1]
@@ -312,26 +314,69 @@ def sqlite_rows(db_path: str | Path, sql: str, params: tuple[Any, ...] = ()) ->
return [{key: row[key] for key in row.keys()} for row in rows]
def persistence_summary(db_path: str | Path, node_type: str | None = None) -> dict[str, Any]:
def persistence_rows(
*,
backend: str,
sql: str,
db_path: str | Path | None = None,
url: str | None = None,
limit: int = 200,
) -> list[dict[str, Any]]:
_, rows = PersistenceBackend.query_records(
backend=backend,
db_path=str(db_path) if db_path is not None else None,
url=url,
sql=sql,
limit=limit,
)
return rows
def persistence_summary(
db_path: str | Path | None,
node_type: str | None = None,
*,
backend: str = "sqlite",
url: str | None = None,
) -> dict[str, Any]:
records = PersistenceBackend.records_summary(
backend=backend,
db_path=str(db_path) if db_path is not None else None,
url=url,
)
summary: dict[str, Any] = {
"backend": backend,
"db_path": str(db_path),
"db_exists": Path(db_path).exists(),
"url": url,
"db_exists": records.get("db_exists", False),
"bindings": [],
"node_status": [],
}
if not Path(db_path).exists():
if not summary["db_exists"]:
return summary
target_db_path = str(db_path) if db_path is not None else None
if node_type:
summary["bindings"] = sqlite_rows(
db_path,
"SELECT node_type, local_id, remote_id, last_updated FROM bindings WHERE node_type = ? ORDER BY last_updated DESC LIMIT 20",
(node_type,),
escaped_node_type = node_type.replace("'", "''")
summary["bindings"] = persistence_rows(
backend=backend,
db_path=target_db_path,
url=url,
sql=(
"SELECT node_type, local_id, remote_id, last_updated "
f"FROM bindings WHERE node_type = '{escaped_node_type}' ORDER BY last_updated DESC LIMIT 20"
),
)
summary["node_status"] = sqlite_rows(
db_path,
"SELECT collection_id, node_type, action, status, binding_status, COUNT(1) AS count FROM nodes WHERE node_type = ? GROUP BY collection_id, node_type, action, status, binding_status ORDER BY collection_id, action, status",
(node_type,),
summary["node_status"] = persistence_rows(
backend=backend,
db_path=target_db_path,
url=url,
sql=(
"SELECT collection_id, node_type, action, status, binding_status, COUNT(1) AS count "
f"FROM nodes WHERE node_type = '{escaped_node_type}' "
"GROUP BY collection_id, node_type, action, status, binding_status "
"ORDER BY collection_id, action, status"
),
)
return summary
+6 -5
View File
@@ -15,6 +15,7 @@ if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
import sync_state_machine.domain # noqa: F401, E402
from sync_state_machine.logging import configure_app_logging, get_logger # noqa: E402
from sync_state_machine.pipeline import ( # noqa: E402
build_config,
run_pipeline_from_config,
@@ -23,15 +24,17 @@ from sync_state_machine.config import load_named_preset_overrides # noqa: E402
PRESET_NAME = "jsonl_sm"
logger = get_logger(__name__)
async def main() -> int:
try:
overrides, preset_file, used_default = load_named_preset_overrides(PROJECT_ROOT, PRESET_NAME)
source_label = "default" if used_default else "custom"
print(f"🧩 Loaded preset ({source_label}): {preset_file}")
config = build_config(project_root=PROJECT_ROOT, overrides=overrides)
configure_app_logging(config.logging, replace_handlers=True)
logger.info("🧩 Loaded preset (%s): %s", source_label, preset_file)
await run_pipeline_from_config(
config,
title="sync_state_machine JSONL -> JSONL pipeline",
@@ -40,12 +43,10 @@ async def main() -> int:
return 0
except KeyboardInterrupt:
print("\n⚠️ Interrupted by user")
logger.warning("⚠️ Interrupted by user")
return 130
except Exception as exc:
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
import traceback
traceback.print_exc()
logger.exception("❌ Pipeline failed: %s: %s", type(exc).__name__, exc)
return 1
+6 -5
View File
@@ -15,6 +15,7 @@ if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
import sync_state_machine.domain # noqa: F401, E402
from sync_state_machine.logging import configure_app_logging, get_logger # noqa: E402
from sync_state_machine.pipeline import ( # noqa: E402
build_config,
run_pipeline_from_config,
@@ -23,15 +24,17 @@ from sync_state_machine.config import load_named_preset_overrides # noqa: E402
PRESET_NAME = "simple_sm"
logger = get_logger(__name__)
async def main() -> int:
try:
overrides, preset_file, used_default = load_named_preset_overrides(PROJECT_ROOT, PRESET_NAME)
source_label = "default" if used_default else "custom"
print(f"🧩 Loaded preset ({source_label}): {preset_file}")
config = build_config(project_root=PROJECT_ROOT, overrides=overrides)
configure_app_logging(config.logging, replace_handlers=True)
logger.info("🧩 Loaded preset (%s): %s", source_label, preset_file)
await run_pipeline_from_config(
config,
title="sync_state_machine JSONL -> API pipeline",
@@ -40,12 +43,10 @@ async def main() -> int:
return 0
except KeyboardInterrupt:
print("\n⚠️ Interrupted by user")
logger.warning("⚠️ Interrupted by user")
return 130
except Exception as exc:
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
import traceback
traceback.print_exc()
logger.exception("❌ Pipeline failed: %s: %s", type(exc).__name__, exc)
return 1
+8 -3
View File
@@ -5,9 +5,13 @@ import asyncio
import logging
from pathlib import Path
from .logging import get_logger, init_app_logger
from .pipeline import run_profile_from_file
logger = get_logger(__name__)
def package_project_root() -> Path:
return Path(__file__).resolve().parents[1]
@@ -20,6 +24,7 @@ def resolve_config_path(raw_path: str, *, project_root: Path) -> Path:
async def _main() -> int:
init_app_logger(replace_handlers=True)
project_root = package_project_root()
parser = argparse.ArgumentParser(description="Run sync pipeline from config override file")
@@ -34,7 +39,7 @@ async def _main() -> int:
if not cfg_path.exists() or not cfg_path.is_file():
raise FileNotFoundError(f"Config file not found: {cfg_path}")
print(f"🧩 Loaded config file: {cfg_path}")
logger.info("🧩 Loaded config file: %s", cfg_path)
await run_profile_from_file(
project_root=project_root,
config_path=cfg_path,
@@ -47,10 +52,10 @@ def main() -> int:
try:
return asyncio.run(_main())
except KeyboardInterrupt:
print("\n⚠️ Interrupted by user")
logger.warning("⚠️ Interrupted by user")
return 130
except Exception as exc:
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
logger.exception("❌ Pipeline failed: %s: %s", type(exc).__name__, exc)
return 1
finally:
logging.shutdown()
+2 -1
View File
@@ -6,7 +6,7 @@ from .persistence import (
get_registered_persistence_backends,
register_persistence_backend,
)
from .persistence_backends import SQLitePersistenceBackend
from .persistence_backends import SQLAlchemyPersistenceBackend, SQLitePersistenceBackend
from .collection import DataCollection
from .binding import BindingManager
@@ -17,6 +17,7 @@ __all__ = [
"SyncNode",
"PersistenceBackend",
"SQLitePersistenceBackend",
"SQLAlchemyPersistenceBackend",
"create_persistence_backend",
"register_persistence_backend",
"get_registered_persistence_backends",
+67 -12
View File
@@ -1,10 +1,13 @@
import json
import logging
from typing import Optional, Dict, Any, List
from ..logging import get_logger, get_scope_display_name
from .persistence import PersistenceBackend
from .persistence_service import ScopedPersistenceView
from .types import BindingStatus, BindingRecord
logger = logging.getLogger(__name__)
logger = get_logger(__name__)
class BindingManager:
@@ -12,14 +15,53 @@ class BindingManager:
BindingManager 负责维护不同 DataCollection 间节点的绑定关系。
所有查询均在内存中进行,持久化仅作为备份。
"""
def __init__(self, persistence: PersistenceBackend, auto_persist: bool = True, scope: str = "global"):
def __init__(
self,
persistence: PersistenceBackend | ScopedPersistenceView,
auto_persist: bool = False,
scope: str = "global",
global_fallback_manager: Optional["BindingManager"] = None,
):
self.persistence = persistence
self.auto_persist = auto_persist
self.scope = scope
self._global_fallback_manager = global_fallback_manager
# { node_type: { local_id: remote_id } }
self._bindings: Dict[str, Dict[str, Optional[str]]] = {}
self._deleted: Dict[str, set[str]] = {}
def set_global_fallback_manager(self, manager: Optional["BindingManager"]) -> None:
self._global_fallback_manager = manager
async def _persistence_save_binding(self, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.save_binding(node_type, local_id, payload_dict)
return
await self.persistence.save_binding(self.scope, node_type, local_id, payload_dict)
async def _persistence_delete_binding(self, node_type: str, local_id: str) -> None:
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.delete_binding(node_type, local_id)
return
await self.persistence.delete_binding(self.scope, node_type, local_id)
async def _persistence_delete_bindings_bulk(self, node_type: str, local_ids: List[str]) -> None:
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.delete_bindings_bulk(node_type, local_ids)
return
await self.persistence.delete_bindings_bulk(self.scope, node_type, local_ids)
async def _persistence_save_bindings_bulk(self, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.save_bindings_bulk(node_type, bindings)
return
await self.persistence.save_bindings_bulk(self.scope, node_type, bindings)
async def _persistence_load_bindings(self, node_type: str) -> List[Dict[str, Any]]:
if isinstance(self.persistence, ScopedPersistenceView):
return await self.persistence.load_bindings(node_type)
return await self.persistence.load_bindings(self.scope, node_type)
def _get_type_bindings(self, node_type: str) -> Dict[str, Optional[str]]:
if node_type not in self._bindings:
self._bindings[node_type] = {}
@@ -56,7 +98,7 @@ class BindingManager:
logger.debug(log_line)
if self.auto_persist:
await self.persistence.save_binding(self.scope, node_type, local_id, {"remote_id": remote_id})
await self._persistence_save_binding(node_type, local_id, {"remote_id": remote_id})
async def unbind(self, node_type: str, local_id: str, manual: bool = False):
"""
@@ -81,14 +123,19 @@ class BindingManager:
)
if self.auto_persist:
await self.persistence.delete_binding(self.scope, node_type, local_id)
await self._persistence_delete_binding(node_type, local_id)
else:
self._get_deleted(node_type).add(local_id)
async def get_remote_id(self, node_type: str, local_id: str) -> Optional[str]:
"""查询本地节点对应的远程 ID"""
bindings = self._get_type_bindings(node_type)
return bindings.get(local_id)
remote_id = bindings.get(local_id)
if remote_id is not None or local_id in bindings:
return remote_id
if self._global_fallback_manager is None:
return None
return await self._global_fallback_manager.get_remote_id(node_type, local_id)
async def get_local_id(self, node_type: str, remote_id: str) -> Optional[str]:
"""查询远程节点对应的本地 ID (内存遍历)"""
@@ -96,11 +143,13 @@ class BindingManager:
for lid, rid in bindings.items():
if rid == remote_id:
return lid
return None
if self._global_fallback_manager is None:
return None
return await self._global_fallback_manager.get_local_id(node_type, remote_id)
async def get_record(self, node_type: str, local_id: str) -> Optional[BindingRecord]:
"""获取结构化绑定记录"""
remote_id = self._get_type_bindings(node_type).get(local_id)
remote_id = await self.get_remote_id(node_type, local_id)
if remote_id is None:
return None
return BindingRecord(
@@ -120,7 +169,7 @@ class BindingManager:
async def load_from_persistence(self, node_type: str):
"""从持久化层加载特定类型的绑定关系到内存"""
raw_bindings = await self.persistence.load_bindings(self.scope, node_type)
raw_bindings = await self._persistence_load_bindings(node_type)
bindings = self._get_type_bindings(node_type)
bindings.clear()
self._get_deleted(node_type).clear()
@@ -163,7 +212,7 @@ class BindingManager:
del bindings[local_id]
deleted_count += 1
if self.auto_persist:
await self.persistence.delete_binding(self.scope, node_type, local_id)
await self._persistence_delete_binding(node_type, local_id)
else:
self._get_deleted(node_type).add(local_id)
@@ -196,13 +245,19 @@ class BindingManager:
touched_types.append(node_type)
if deleted:
await self.persistence.delete_bindings_bulk(self.scope, node_type, deleted)
await self._persistence_delete_bindings_bulk(node_type, deleted)
self._get_deleted(node_type).clear()
if bindings:
await self.persistence.save_bindings_bulk(self.scope, node_type, bindings)
await self._persistence_save_bindings_bulk(node_type, bindings)
touched_text = ",".join(touched_types) if touched_types else "none"
logger.info(
f"🔍 [BindingManager.persist] 完成: scope={self.scope}, types={type_count}, saved={saved_total}, deleted={deleted_total}, touched=[{touched_text}]"
f"🔍 [BindingManager.persist] 完成: scope={get_scope_display_name(self.scope)}, types={type_count}, saved={saved_total}, deleted={deleted_total}, touched=[{touched_text}]"
)
async def unload(self, *, node_types: Optional[List[str]] = None) -> None:
target_types = list(node_types) if node_types is not None else list(self._bindings.keys())
for node_type in target_types:
self._bindings.pop(node_type, None)
self._deleted.pop(node_type, None)
+133 -31
View File
@@ -5,6 +5,7 @@ from typing import Dict, List, Optional, Any, Set, Callable, Literal
from .sync_node import SyncNode
from .types import SyncStatus
from .persistence import PersistenceBackend
from .persistence_service import ScopedPersistenceView
class DataCollection:
@@ -24,18 +25,22 @@ class DataCollection:
self,
collection_id: str,
scope: str = "global",
persistence: Optional[PersistenceBackend] = None,
persistence: Optional[PersistenceBackend | ScopedPersistenceView] = None,
auto_persist: bool = False,
sm_runtime: Optional[Any] = None,
global_fallback_collection: Optional["DataCollection"] = None,
):
self.collection_id = collection_id
self.scope = scope
self.persistence = persistence
self.auto_persist = auto_persist
self._sm_runtime = sm_runtime
self._global_fallback_collection = global_fallback_collection
self._nodes: Dict[str, SyncNode] = {} # 内存缓存
# data_id 全局唯一(按 collection 维度),用于 O(1) 反查 node_id
self._data_id_to_node_id: Dict[str, str] = {}
# node_id -> data_id 反向索引,用于 O(1) 维护 data_id 更新
self._node_id_to_data_id: Dict[str, str] = {}
# auto_persist=False 时,delete() 不会立即落库;这里记录删除集合,persist() 时统一写回
self._deleted_node_ids: set[str] = set()
@@ -51,6 +56,83 @@ class DataCollection:
"""设置 Collection 默认状态机 runtime。"""
self._sm_runtime = runtime
def set_global_fallback_collection(self, collection: Optional["DataCollection"]) -> None:
self._global_fallback_collection = collection
def _get_global_fallback(self) -> Optional["DataCollection"]:
return self._global_fallback_collection
async def _persistence_save_node(self, node: SyncNode) -> None:
if not self.persistence:
return
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.save_node(self.collection_id, node)
return
await self.persistence.save_node(self.collection_id, self.scope, node)
async def _persistence_delete_node(self, node_id: str) -> None:
if not self.persistence:
return
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.delete_node(self.collection_id, node_id)
return
await self.persistence.delete_node(self.collection_id, self.scope, node_id)
async def _persistence_delete_nodes_bulk(self, node_ids: List[str]) -> None:
if not self.persistence:
return
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.delete_nodes_bulk(self.collection_id, node_ids)
return
await self.persistence.delete_nodes_bulk(self.collection_id, self.scope, node_ids)
async def _persistence_save_nodes_bulk(self, nodes: List[SyncNode]) -> None:
if not self.persistence:
return
if isinstance(self.persistence, ScopedPersistenceView):
await self.persistence.save_nodes_bulk(self.collection_id, nodes)
return
await self.persistence.save_nodes_bulk(self.collection_id, self.scope, nodes)
async def _persistence_load_nodes(
self,
*,
exclude_node_types: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
if not self.persistence:
return []
if isinstance(self.persistence, ScopedPersistenceView):
return await self.persistence.load_nodes(
self.collection_id,
exclude_node_types=exclude_node_types,
)
return await self.persistence.load_nodes(
self.collection_id,
self.scope,
exclude_node_types=exclude_node_types,
)
async def _persistence_load_bind_data_id_overrides(self, node_type: str) -> Dict[str, Optional[str]]:
if not self.persistence:
return {}
if isinstance(self.persistence, ScopedPersistenceView):
return await self.persistence.load_bind_data_id_overrides(node_type)
return await self.persistence.load_bind_data_id_overrides(node_type, self.scope)
def _remove_data_id_index(self, node_id: str, data_id: Optional[str] = None) -> None:
target_data_id = data_id
if target_data_id is None:
target_data_id = self._node_id_to_data_id.pop(node_id, None)
else:
self._node_id_to_data_id.pop(node_id, None)
if target_data_id and self._data_id_to_node_id.get(target_data_id) == node_id:
self._data_id_to_node_id.pop(target_data_id, None)
def _set_data_id_index(self, node_id: str, data_id: str) -> None:
self._data_id_to_node_id[data_id] = node_id
self._node_id_to_data_id[node_id] = data_id
async def add(self, node: SyncNode):
# 检查 node_id 唯一性
if node.node_id in self._nodes:
@@ -65,18 +147,17 @@ class DataCollection:
raise ValueError(
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
)
self._data_id_to_node_id[node.data_id] = node.node_id
self._set_data_id_index(node.node_id, node.data_id)
self._nodes[node.node_id] = node
self._deleted_node_ids.discard(node.node_id)
if self.persistence and self.auto_persist:
await self.persistence.save_node(self.collection_id, self.scope, node)
await self._persistence_save_node(node)
async def update(self, node: SyncNode):
old_node = self._nodes.get(node.node_id)
if old_node and old_node.data_id and old_node.data_id != "" and old_node.data_id != node.data_id:
# node_id 不变但 data_id 变更,清理旧索引
self._data_id_to_node_id.pop(old_node.data_id, None)
old_data_id = self._node_id_to_data_id.get(node.node_id)
if old_data_id and old_data_id != node.data_id:
self._remove_data_id_index(node.node_id, old_data_id)
if node.data_id and node.data_id != "":
existing_node_id = self._data_id_to_node_id.get(node.data_id)
@@ -84,27 +165,33 @@ class DataCollection:
raise ValueError(
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
)
self._data_id_to_node_id[node.data_id] = node.node_id
self._set_data_id_index(node.node_id, node.data_id)
else:
self._node_id_to_data_id.pop(node.node_id, None)
self._nodes[node.node_id] = node
self._deleted_node_ids.discard(node.node_id)
if self.persistence and self.auto_persist:
await self.persistence.save_node(self.collection_id, self.scope, node)
await self._persistence_save_node(node)
async def delete(self, node_id: str):
old_node = self._nodes.get(node_id)
if old_node and old_node.data_id and old_node.data_id != "":
self._data_id_to_node_id.pop(old_node.data_id, None)
self._remove_data_id_index(node_id)
if node_id in self._nodes:
del self._nodes[node_id]
if self.persistence:
if self.auto_persist:
await self.persistence.delete_node(self.collection_id, self.scope, node_id)
await self._persistence_delete_node(node_id)
else:
self._deleted_node_ids.add(node_id)
def get(self, node_id: str) -> Optional[SyncNode]:
return self._nodes.get(node_id)
node = self._nodes.get(node_id)
if node is not None:
return node
fallback = self._get_global_fallback()
if fallback is None:
return None
return fallback.get(node_id)
def get_by_node_id(self, node_id: str) -> Optional[SyncNode]:
"""别名方法,与 get() 功能相同"""
@@ -116,14 +203,23 @@ class DataCollection:
def get_node_id_by_data_id(self, data_id: str) -> Optional[str]:
"""按 data_id 反查 node_idO(1))。"""
return self._data_id_to_node_id.get(data_id)
node_id = self._data_id_to_node_id.get(data_id)
if node_id is not None:
return node_id
fallback = self._get_global_fallback()
if fallback is None:
return None
return fallback.get_node_id_by_data_id(data_id)
def get_by_data_id_any(self, data_id: str) -> Optional[SyncNode]:
"""按 data_id 查找节点(不要求 node_type)(O(1))。"""
node_id = self._data_id_to_node_id.get(data_id)
if not node_id:
if node_id:
return self._nodes.get(node_id)
fallback = self._get_global_fallback()
if fallback is None:
return None
return self._nodes.get(node_id)
return fallback.get_by_data_id_any(data_id)
def get_by_data_id(self, node_type: str, data_id: str) -> Optional[SyncNode]:
"""按 node_type 和 data_id 查找节点"""
@@ -362,7 +458,7 @@ class DataCollection:
- data/origin_data 保留(由 DataSource 覆盖)
**初始化归并(E01)**:
- 由 Pipeline 在加载后调用 run_reset() 触发 E01
- 由 Pipeline 在加载后调用 `run_phase2_cleanup()` 触发 E01
- CREATE 僵尸节点进入 S15 并删除
- 其余节点进入 S00UNCHECKED/NONE/PENDING
"""
@@ -371,9 +467,10 @@ class DataCollection:
self._nodes = {}
self._data_id_to_node_id = {}
self._node_id_to_data_id = {}
self._deleted_node_ids.clear()
node_dicts = await self.persistence.load_nodes(self.collection_id, self.scope)
node_dicts = await self._persistence_load_nodes()
from .types import SyncAction, SyncStatus, BindingStatus
from .registry import DomainRegistry
@@ -472,6 +569,7 @@ class DataCollection:
context=context_value,
)
node.context["_loaded_from_persistence"] = True
node.append_log("加载来源: persistence 恢复节点")
except Exception:
import logging
logger = logging.getLogger(__name__)
@@ -490,11 +588,11 @@ class DataCollection:
raise ValueError(
f"Duplicate data_id detected while loading: {node.data_id} already bound to node_id={existing_node_id}"
)
self._data_id_to_node_id[node.data_id] = node.node_id
self._set_data_id_index(node.node_id, node.data_id)
node_types = {node.node_type for node in self._nodes.values()}
for node_type in node_types:
override_map = await self.persistence.load_bind_data_id_overrides(node_type, self.scope)
override_map = await self._persistence_load_bind_data_id_overrides(node_type)
for node in self.filter(node_type=node_type):
bind_data_id = override_map.get(node.node_id)
if bind_data_id:
@@ -509,13 +607,10 @@ class DataCollection:
self._nodes = {}
self._data_id_to_node_id = {}
self._node_id_to_data_id = {}
self._deleted_node_ids.clear()
node_dicts = await self.persistence.load_nodes(
self.collection_id,
self.scope,
exclude_node_types=node_types,
)
node_dicts = await self._persistence_load_nodes(exclude_node_types=node_types)
from .types import SyncAction, SyncStatus, BindingStatus
from .registry import DomainRegistry
@@ -606,6 +701,7 @@ class DataCollection:
context=context_value,
)
node.context["_loaded_from_persistence"] = True
node.append_log("加载来源: persistence 恢复节点")
self._nodes[node.node_id] = node
if node.data_id and node.data_id != "":
@@ -614,11 +710,11 @@ class DataCollection:
raise ValueError(
f"Duplicate data_id detected while loading: {node.data_id} already bound to node_id={existing_node_id}"
)
self._data_id_to_node_id[node.data_id] = node.node_id
self._set_data_id_index(node.node_id, node.data_id)
loaded_node_types = {node.node_type for node in self._nodes.values()}
for node_type in loaded_node_types:
override_map = await self.persistence.load_bind_data_id_overrides(node_type, self.scope)
override_map = await self._persistence_load_bind_data_id_overrides(node_type)
for node in self.filter(node_type=node_type):
bind_data_id = override_map.get(node.node_id)
if bind_data_id:
@@ -639,7 +735,7 @@ class DataCollection:
raise ValueError(
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
)
self._data_id_to_node_id[node.data_id] = node.node_id
self._set_data_id_index(node.node_id, node.data_id)
self._nodes[node.node_id] = node
self._deleted_node_ids.discard(node.node_id)
@@ -651,7 +747,7 @@ class DataCollection:
# 先落库删除(用于 auto_persist=False 的场景)
if self._deleted_node_ids:
await self.persistence.delete_nodes_bulk(self.collection_id, self.scope, list(self._deleted_node_ids))
await self._persistence_delete_nodes_bulk(list(self._deleted_node_ids))
self._deleted_node_ids.clear()
if not self._nodes:
@@ -664,7 +760,13 @@ class DataCollection:
if not nodes_to_persist:
return
await self.persistence.save_nodes_bulk(self.collection_id, self.scope, nodes_to_persist)
await self._persistence_save_nodes_bulk(nodes_to_persist)
async def unload(self) -> None:
self._nodes.clear()
self._data_id_to_node_id.clear()
self._node_id_to_data_id.clear()
self._deleted_node_ids.clear()
async def filter_by_project_ids(self, data_id_filter: List[str]) -> int:
"""按目标项目 data_id 集合过滤当前 collection,返回删除数量。"""
+127 -41
View File
@@ -5,6 +5,9 @@ import sqlite3
from abc import ABC, abstractmethod
from typing import Any, Callable, ClassVar, Dict, List, Optional, TYPE_CHECKING
from sqlalchemy import create_engine, inspect, text
from sqlalchemy.engine import URL, make_url
from ..config.persist_config import PersistConfig
if TYPE_CHECKING:
@@ -26,72 +29,153 @@ class PersistenceBackend(ABC):
cls,
*,
backend: str,
db_path: str,
db_path: str | None = None,
url: str | None = None,
sql: str,
limit: int = 200,
) -> tuple[List[str], List[Dict[str, Any]]]:
if backend != "sqlite":
raise NotImplementedError(f"records query backend not supported yet: {backend}")
if not cls._is_read_only_sql(sql):
raise ValueError("Only read-only SQL is allowed")
path = db_path
if not os.path.exists(path):
raise FileNotFoundError(f"DB not found: {path}")
if backend == "sqlite":
path = db_path
if not path:
raise ValueError("db_path is required for backend=sqlite")
if not os.path.exists(path):
raise FileNotFoundError(f"DB not found: {path}")
with sqlite3.connect(path) as conn:
conn.row_factory = sqlite3.Row
cursor = conn.execute(sql)
rows = cursor.fetchmany(max(1, min(limit, 2000)))
columns = [item[0] for item in (cursor.description or [])]
data = [{k: row[k] for k in columns} for row in rows]
return columns, data
with sqlite3.connect(path) as conn:
conn.row_factory = sqlite3.Row
cursor = conn.execute(sql)
rows = cursor.fetchmany(max(1, min(limit, 2000)))
columns = [item[0] for item in (cursor.description or [])]
data = [{k: row[k] for k in columns} for row in rows]
return columns, data
if backend == "sqlalchemy":
if not url:
raise ValueError("url is required for backend=sqlalchemy")
engine = create_engine(cls._to_sync_sqlalchemy_url(url))
try:
with engine.connect() as conn:
result = conn.execute(text(sql))
rows = result.fetchmany(max(1, min(limit, 2000)))
columns = list(result.keys())
data = [{column: row[index] for index, column in enumerate(columns)} for row in rows]
return columns, data
finally:
engine.dispose()
raise NotImplementedError(f"records query backend not supported yet: {backend}")
@classmethod
def records_summary(
cls,
*,
backend: str,
db_path: str,
db_path: str | None = None,
url: str | None = None,
) -> Dict[str, Any]:
db_exists = cls._target_exists(backend=backend, db_path=db_path, url=url)
summary: Dict[str, Any] = {
"backend": backend,
"db_path": db_path,
"db_exists": os.path.exists(db_path),
"url": url,
"db_exists": db_exists,
"tables": [],
"sample_bindings": [],
}
if not os.path.exists(db_path):
return summary
if backend != "sqlite":
summary["error"] = f"records summary backend not supported yet: {backend}"
if not db_exists:
return summary
with sqlite3.connect(db_path) as conn:
conn.row_factory = sqlite3.Row
tables = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name"
).fetchall()
table_names = [str(r["name"]) for r in tables]
for name in table_names:
count = conn.execute(f"SELECT COUNT(1) AS c FROM {name}").fetchone()["c"]
summary["tables"].append({"name": name, "count": int(count)})
if "bindings" in table_names:
sample = conn.execute(
"SELECT node_type, local_id, remote_id FROM bindings ORDER BY last_updated DESC LIMIT 50"
if backend == "sqlite":
assert db_path is not None
with sqlite3.connect(db_path) as conn:
conn.row_factory = sqlite3.Row
tables = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name"
).fetchall()
summary["sample_bindings"] = [
{
"node_type": row["node_type"],
"local_id": row["local_id"],
"remote_id": row["remote_id"],
}
for row in sample
]
table_names = [str(r["name"]) for r in tables]
for name in table_names:
count = conn.execute(f"SELECT COUNT(1) AS c FROM {name}").fetchone()["c"]
summary["tables"].append({"name": name, "count": int(count)})
if "bindings" in table_names:
sample = conn.execute(
"SELECT node_type, local_id, remote_id FROM bindings ORDER BY last_updated DESC LIMIT 50"
).fetchall()
summary["sample_bindings"] = [
{
"node_type": row["node_type"],
"local_id": row["local_id"],
"remote_id": row["remote_id"],
}
for row in sample
]
return summary
if backend == "sqlalchemy":
if not url:
summary["error"] = "url is required for backend=sqlalchemy"
return summary
engine = create_engine(cls._to_sync_sqlalchemy_url(url))
try:
with engine.connect() as conn:
inspector = inspect(conn)
table_names = sorted(inspector.get_table_names())
for name in table_names:
count = conn.execute(text(f"SELECT COUNT(1) AS c FROM {name}")).scalar_one()
summary["tables"].append({"name": name, "count": int(count)})
if "bindings" in table_names:
sample = conn.execute(
text("SELECT node_type, local_id, remote_id FROM bindings ORDER BY last_updated DESC LIMIT 50")
).fetchall()
summary["sample_bindings"] = [
{
"node_type": row[0],
"local_id": row[1],
"remote_id": row[2],
}
for row in sample
]
finally:
engine.dispose()
return summary
summary["error"] = f"records summary backend not supported yet: {backend}"
return summary
@staticmethod
def _target_exists(*, backend: str, db_path: str | None, url: str | None) -> bool:
if backend == "sqlite":
return bool(db_path) and os.path.exists(db_path)
if backend == "sqlalchemy":
if not url:
return False
parsed = make_url(url)
if parsed.get_backend_name() == "sqlite":
database = parsed.database
if not database or database == ":memory:":
return True
return os.path.exists(database)
return True
return False
@staticmethod
def _to_sync_sqlalchemy_url(url: str) -> str:
parsed = make_url(url)
async_to_sync = {
"sqlite+aiosqlite": "sqlite+pysqlite",
"postgresql+asyncpg": "postgresql+psycopg",
"mysql+aiomysql": "mysql+pymysql",
}
drivername = async_to_sync.get(parsed.drivername, parsed.drivername)
sync_url: URL = parsed.set(drivername=drivername)
return str(sync_url)
@abstractmethod
async def initialize(self) -> None:
raise NotImplementedError
@@ -218,9 +302,11 @@ def create_persistence_backend(
def _register_builtin_persistence_backends() -> None:
from .persistence_backends.sqlalchemy_backend import SQLAlchemyPersistenceBackend
from .persistence_backends.sqlite_backend import SQLitePersistenceBackend
register_persistence_backend("sqlite", lambda config: SQLitePersistenceBackend(config), override=True)
register_persistence_backend("sqlalchemy", lambda config: SQLAlchemyPersistenceBackend(config), override=True)
_register_builtin_persistence_backends()
@@ -1,3 +1,4 @@
from .sqlalchemy_backend import SQLAlchemyPersistenceBackend
from .sqlite_backend import SQLitePersistenceBackend
__all__ = ["SQLitePersistenceBackend"]
__all__ = ["SQLitePersistenceBackend", "SQLAlchemyPersistenceBackend"]
@@ -0,0 +1,428 @@
from __future__ import annotations
import json
import os
from typing import Any, Dict, List, Optional, TYPE_CHECKING
from sqlalchemy import Column, Index, MetaData, PrimaryKeyConstraint, String, Table, Text, delete, insert, select
from sqlalchemy.engine import make_url
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncConnection, create_async_engine
from ...config.persist_config import PersistConfig
from ..persistence import PersistenceBackend
if TYPE_CHECKING:
from ..sync_node import SyncNode
class SQLAlchemyPersistenceBackend(PersistenceBackend):
"""Generic SQLAlchemy persistence backend with explicit schema bootstrap."""
backend_name = "sqlalchemy"
bulk_delete_batch_size = 1000
collection_id_length = 255
scope_length = 255
node_id_length = 255
node_type_length = 128
data_id_length = 255
bind_data_id_length = 255
action_length = 32
status_length = 32
binding_status_length = 32
remote_id_length = 255
last_updated_length = 64
def __init__(self, persist_config: PersistConfig | str | None = None, **_: Any) -> None:
if isinstance(persist_config, PersistConfig):
self.persist_config = persist_config
self.database_url = persist_config.resolved_url()
self.db_path = persist_config.resolved_db_path()
self.backend_options = dict(persist_config.backend_options)
elif isinstance(persist_config, str):
self.persist_config = None
self.database_url = persist_config
parsed = make_url(persist_config)
self.db_path = str(parsed.database) if parsed.get_backend_name() == "sqlite" and parsed.database else None
self.backend_options = {}
else:
raise ValueError("SQLAlchemyPersistenceBackend requires PersistConfig or database URL")
self.engine: Optional[AsyncEngine] = None
self.metadata = MetaData()
self.nodes = Table(
"nodes",
self.metadata,
Column("collection_id", String(self.collection_id_length), nullable=False),
Column("scope", String(self.scope_length), nullable=False, default="global"),
Column("node_id", String(self.node_id_length), nullable=False),
Column("node_type", String(self.node_type_length), nullable=False),
Column("data_id", String(self.data_id_length), nullable=True),
Column("bind_data_id", String(self.bind_data_id_length), nullable=True),
Column("data", Text, nullable=True),
Column("origin_data", Text, nullable=True),
Column("action", String(self.action_length), nullable=True),
Column("status", String(self.status_length), nullable=True),
Column("binding_status", String(self.binding_status_length), nullable=True),
Column("error", Text, nullable=True),
Column("sync_log", Text, nullable=True),
Column("context", Text, nullable=True),
PrimaryKeyConstraint("collection_id", "scope", "node_id", name="pk_nodes"),
Index("idx_nodes_collection_scope_type", "collection_id", "scope", "node_type"),
Index("idx_nodes_collection_scope_data_id", "collection_id", "scope", "data_id"),
Index("idx_nodes_scope_type_node_id", "scope", "node_type", "node_id"),
)
self.bindings = Table(
"bindings",
self.metadata,
Column("scope", String(self.scope_length), nullable=False, default="global"),
Column("node_type", String(self.node_type_length), nullable=False),
Column("local_id", String(self.node_id_length), nullable=False),
Column("remote_id", String(self.remote_id_length), nullable=True),
Column("last_updated", String(self.last_updated_length), nullable=True),
PrimaryKeyConstraint("scope", "node_type", "local_id", name="pk_bindings"),
Index("idx_bindings_scope_type_remote", "scope", "node_type", "remote_id"),
)
async def initialize(self) -> None:
if self.db_path and self.db_path != ":memory:":
dir_path = os.path.dirname(self.db_path)
if dir_path:
os.makedirs(dir_path, exist_ok=True)
engine_options = {"future": True}
engine_options.update(self.backend_options)
self.engine = create_async_engine(self.database_url, **engine_options)
async with self.engine.begin() as conn:
await conn.run_sync(self.metadata.create_all)
async def save_node(self, collection_id: str, scope: str, node: "SyncNode") -> None:
row = self._node_row(collection_id=collection_id, scope=scope, node=node)
async with self._begin() as conn:
await self._replace_node_rows(conn, collection_id=collection_id, scope=scope, node_ids=[node.node_id])
await conn.execute(insert(self.nodes), [row])
async def save_nodes_bulk(self, collection_id: str, scope: str, nodes: List["SyncNode"]) -> None:
if not nodes:
return
rows = [self._node_row(collection_id=collection_id, scope=scope, node=node) for node in nodes]
node_ids = [node.node_id for node in nodes]
async with self._begin() as conn:
await self._replace_node_rows(conn, collection_id=collection_id, scope=scope, node_ids=node_ids)
await conn.execute(insert(self.nodes), rows)
async def delete_node(self, collection_id: str, scope: str, node_id: str) -> None:
async with self._begin() as conn:
await conn.execute(
delete(self.nodes).where(
self.nodes.c.collection_id == collection_id,
self.nodes.c.scope == scope,
self.nodes.c.node_id == node_id,
)
)
async def delete_nodes_bulk(self, collection_id: str, scope: str, node_ids: List[str]) -> None:
if not node_ids:
return
async with self._begin() as conn:
for batch in self._chunked(node_ids):
await conn.execute(
delete(self.nodes).where(
self.nodes.c.collection_id == collection_id,
self.nodes.c.scope == scope,
self.nodes.c.node_id.in_(batch),
)
)
async def load_nodes(
self,
collection_id: str,
scope: str,
*,
exclude_node_types: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
stmt = select(self.nodes).where(
self.nodes.c.collection_id == collection_id,
self.nodes.c.scope == scope,
)
filtered_types = [node_type for node_type in (exclude_node_types or []) if node_type]
if filtered_types:
stmt = stmt.where(self.nodes.c.node_type.not_in(filtered_types))
async with self._connect() as conn:
rows = (await conn.execute(stmt)).mappings().all()
return [self._deserialize_node_row(row) for row in rows]
async def save_binding(self, scope: str, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
row = self._binding_row(scope=scope, node_type=node_type, local_id=local_id, remote_id=payload_dict.get("remote_id") if payload_dict else None)
async with self._begin() as conn:
await self._replace_binding_rows(conn, scope=scope, node_type=node_type, local_ids=[local_id])
await conn.execute(insert(self.bindings), [row])
async def save_bindings_bulk(self, scope: str, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
if not bindings:
return
rows = [
self._binding_row(scope=scope, node_type=node_type, local_id=local_id, remote_id=remote_id)
for local_id, remote_id in bindings.items()
]
async with self._begin() as conn:
await self._replace_binding_rows(conn, scope=scope, node_type=node_type, local_ids=list(bindings.keys()))
await conn.execute(insert(self.bindings), rows)
async def delete_binding(self, scope: str, node_type: str, local_id: str) -> None:
async with self._begin() as conn:
await conn.execute(
delete(self.bindings).where(
self.bindings.c.scope == scope,
self.bindings.c.node_type == node_type,
self.bindings.c.local_id == local_id,
)
)
async def delete_bindings_bulk(self, scope: str, node_type: str, local_ids: List[str]) -> None:
if not local_ids:
return
async with self._begin() as conn:
for batch in self._chunked(local_ids):
await conn.execute(
delete(self.bindings).where(
self.bindings.c.scope == scope,
self.bindings.c.node_type == node_type,
self.bindings.c.local_id.in_(batch),
)
)
async def load_bindings(self, scope: str, node_type: str) -> List[Dict[str, Any]]:
stmt = select(self.bindings.c.local_id, self.bindings.c.remote_id).where(
self.bindings.c.scope == scope,
self.bindings.c.node_type == node_type,
)
async with self._connect() as conn:
rows = (await conn.execute(stmt)).mappings().all()
return [{"local_id": row["local_id"], "payload": {"remote_id": row["remote_id"]}} for row in rows]
async def load_bind_data_id_overrides(self, node_type: str, scope: str) -> Dict[str, Optional[str]]:
local_join = (
select(
self.bindings.c.local_id.label("src_node_id"),
self.nodes.c.data_id.label("bind_data_id"),
)
.select_from(
self.bindings.outerjoin(
self.nodes,
(self.nodes.c.node_id == self.bindings.c.remote_id)
& (self.nodes.c.node_type == self.bindings.c.node_type)
& (self.nodes.c.scope == self.bindings.c.scope),
)
)
.where(
self.bindings.c.node_type == node_type,
self.bindings.c.scope == scope,
)
)
remote_join = (
select(
self.bindings.c.remote_id.label("src_node_id"),
self.nodes.c.data_id.label("bind_data_id"),
)
.select_from(
self.bindings.outerjoin(
self.nodes,
(self.nodes.c.node_id == self.bindings.c.local_id)
& (self.nodes.c.node_type == self.bindings.c.node_type)
& (self.nodes.c.scope == self.bindings.c.scope),
)
)
.where(
self.bindings.c.node_type == node_type,
self.bindings.c.scope == scope,
self.bindings.c.remote_id.is_not(None),
)
)
async with self._connect() as conn:
rows = (await conn.execute(local_join.union_all(remote_join))).mappings().all()
result: Dict[str, Optional[str]] = {}
for row in rows:
src_node_id = row["src_node_id"]
if src_node_id:
result[str(src_node_id)] = row["bind_data_id"]
return result
async def delete_scope_node_types(self, *, scope: str, node_types: List[str]) -> None:
filtered_types = [node_type for node_type in node_types if node_type]
if not filtered_types:
return
async with self._begin() as conn:
await conn.execute(delete(self.nodes).where(self.nodes.c.scope == scope, self.nodes.c.node_type.in_(filtered_types)))
await conn.execute(delete(self.bindings).where(self.bindings.c.scope == scope, self.bindings.c.node_type.in_(filtered_types)))
async def copy_nodes_between_scopes(
self,
*,
collection_id: str,
source_scope: str,
target_scope: str,
node_types: List[str],
) -> None:
if source_scope == target_scope or not node_types:
return
stmt = select(self.nodes).where(
self.nodes.c.collection_id == collection_id,
self.nodes.c.scope == source_scope,
self.nodes.c.node_type.in_(node_types),
)
async with self._begin() as conn:
rows = (await conn.execute(stmt)).mappings().all()
target_rows = [dict(row, scope=target_scope) for row in rows]
if not target_rows:
return
await self._replace_node_rows(
conn,
collection_id=collection_id,
scope=target_scope,
node_ids=[str(row["node_id"]) for row in target_rows],
)
await conn.execute(insert(self.nodes), target_rows)
async def copy_bindings_between_scopes(
self,
*,
source_scope: str,
target_scope: str,
node_types: List[str],
) -> None:
if source_scope == target_scope or not node_types:
return
stmt = select(self.bindings).where(
self.bindings.c.scope == source_scope,
self.bindings.c.node_type.in_(node_types),
)
async with self._begin() as conn:
rows = (await conn.execute(stmt)).mappings().all()
target_rows = [dict(row, scope=target_scope) for row in rows]
if not target_rows:
return
by_type: Dict[str, List[str]] = {}
for row in target_rows:
by_type.setdefault(str(row["node_type"]), []).append(str(row["local_id"]))
for node_type, local_ids in by_type.items():
await self._replace_binding_rows(conn, scope=target_scope, node_type=node_type, local_ids=local_ids)
await conn.execute(insert(self.bindings), target_rows)
async def dump_to_runtime(self, filename: str) -> None:
raise NotImplementedError("dump_to_runtime is only supported by the sqlite backend")
async def close(self) -> None:
if self.engine is None:
return
await self.engine.dispose()
self.engine = None
async def _replace_node_rows(
self,
conn: AsyncConnection,
*,
collection_id: str,
scope: str,
node_ids: List[str],
) -> None:
if not node_ids:
return
for batch in self._chunked(node_ids):
await conn.execute(
delete(self.nodes).where(
self.nodes.c.collection_id == collection_id,
self.nodes.c.scope == scope,
self.nodes.c.node_id.in_(batch),
)
)
async def _replace_binding_rows(
self,
conn: AsyncConnection,
*,
scope: str,
node_type: str,
local_ids: List[str],
) -> None:
if not local_ids:
return
for batch in self._chunked(local_ids):
await conn.execute(
delete(self.bindings).where(
self.bindings.c.scope == scope,
self.bindings.c.node_type == node_type,
self.bindings.c.local_id.in_(batch),
)
)
def _chunked(self, values: List[str]) -> List[List[str]]:
if not values:
return []
batch_size = max(1, int(self.bulk_delete_batch_size))
return [values[index : index + batch_size] for index in range(0, len(values), batch_size)]
def _node_row(self, *, collection_id: str, scope: str, node: "SyncNode") -> Dict[str, Any]:
bind_data_id = None
if isinstance(node.context, dict):
value = node.context.get("bind_data_id")
if value is not None and value != "":
bind_data_id = str(value)
return {
"collection_id": collection_id,
"scope": scope,
"node_id": node.node_id,
"node_type": node.node_type,
"data_id": node.data_id,
"bind_data_id": bind_data_id,
"data": json.dumps(node.data) if node.data else None,
"origin_data": json.dumps(node.origin_data) if node.origin_data else None,
"action": node.action.value,
"status": node.status.value,
"binding_status": node.binding_status.value,
"error": node.error,
"sync_log": node.sync_log,
"context": json.dumps(node.context) if node.context else None,
}
@staticmethod
def _binding_row(*, scope: str, node_type: str, local_id: str, remote_id: Optional[str]) -> Dict[str, Any]:
return {
"scope": scope,
"node_type": node_type,
"local_id": local_id,
"remote_id": remote_id,
"last_updated": None,
}
@staticmethod
def _deserialize_node_row(row: Dict[str, Any]) -> Dict[str, Any]:
item = {
"node_id": row["node_id"],
"node_type": row["node_type"],
"data_id": row["data_id"],
"bind_data_id": row["bind_data_id"],
"depend_ids": [],
"data": json.loads(row["data"]) if row["data"] else None,
"origin_data": json.loads(row["origin_data"]) if row["origin_data"] else None,
"action": row["action"],
"status": row["status"],
"binding_status": row["binding_status"],
"error": row["error"],
"sync_log": row["sync_log"],
"context": json.loads(row["context"]) if row["context"] else {},
}
if item["bind_data_id"] and isinstance(item["context"], dict):
item["context"]["bind_data_id"] = item["bind_data_id"]
return item
def _require_engine(self) -> AsyncEngine:
if self.engine is None:
raise RuntimeError("SQLAlchemyPersistenceBackend is not initialized")
return self.engine
def _connect(self):
return self._require_engine().connect()
def _begin(self):
return self._require_engine().begin()
@@ -20,7 +20,10 @@ class SQLitePersistenceBackend(PersistenceBackend):
def __init__(self, persist_config: PersistConfig | str | None = None, **_: Any) -> None:
if isinstance(persist_config, PersistConfig):
self.persist_config = persist_config
self.db_path = persist_config.db_path
resolved_db_path = persist_config.resolved_db_path()
if not resolved_db_path:
raise ValueError("sqlite persistence requires a resolvable db_path")
self.db_path = resolved_db_path
elif isinstance(persist_config, str):
self.persist_config = None
self.db_path = persist_config
@@ -112,6 +115,9 @@ class SQLitePersistenceBackend(PersistenceBackend):
await self.conn.execute(
"CREATE INDEX IF NOT EXISTS idx_nodes_collection_scope_data_id ON nodes (collection_id, scope, data_id)"
)
await self.conn.execute(
"CREATE INDEX IF NOT EXISTS idx_nodes_scope_type_node_id ON nodes (scope, node_type, node_id)"
)
async def _ensure_bindings_schema(self) -> None:
assert self.conn is not None
@@ -0,0 +1,85 @@
from __future__ import annotations
from typing import Any, Dict, List, Optional
from .persistence import PersistenceBackend
class PersistenceService:
"""Shared persistence service that owns the backend lifecycle."""
def __init__(self, backend: PersistenceBackend):
self._backend = backend
@property
def backend(self) -> PersistenceBackend:
return self._backend
async def initialize(self) -> None:
await self._backend.initialize()
def view(self, scope: str) -> "ScopedPersistenceView":
return ScopedPersistenceView(self._backend, scope=scope)
async def close(self) -> None:
await self._backend.close()
class ScopedPersistenceView:
"""Thin scope-bound persistence view used by project runtimes."""
def __init__(self, backend: PersistenceBackend, *, scope: str):
self._backend = backend
self.scope = scope
@property
def backend(self) -> PersistenceBackend:
return self._backend
async def load_nodes(
self,
collection_id: str,
*,
exclude_node_types: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
return await self._backend.load_nodes(
collection_id,
self.scope,
exclude_node_types=exclude_node_types,
)
async def save_node(self, collection_id: str, node) -> None:
await self._backend.save_node(collection_id, self.scope, node)
async def save_nodes_bulk(self, collection_id: str, nodes) -> None:
await self._backend.save_nodes_bulk(collection_id, self.scope, nodes)
async def delete_node(self, collection_id: str, node_id: str) -> None:
await self._backend.delete_node(collection_id, self.scope, node_id)
async def delete_nodes_bulk(self, collection_id: str, node_ids: List[str]) -> None:
await self._backend.delete_nodes_bulk(collection_id, self.scope, node_ids)
async def load_bindings(self, node_type: str) -> List[Dict[str, Any]]:
return await self._backend.load_bindings(self.scope, node_type)
async def save_binding(self, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
await self._backend.save_binding(self.scope, node_type, local_id, payload_dict)
async def save_bindings_bulk(self, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
await self._backend.save_bindings_bulk(self.scope, node_type, bindings)
async def delete_binding(self, node_type: str, local_id: str) -> None:
await self._backend.delete_binding(self.scope, node_type, local_id)
async def delete_bindings_bulk(self, node_type: str, local_ids: List[str]) -> None:
await self._backend.delete_bindings_bulk(self.scope, node_type, local_ids)
async def load_bind_data_id_overrides(self, node_type: str) -> Dict[str, Optional[str]]:
return await self._backend.load_bind_data_id_overrides(node_type, self.scope)
async def delete_scope_node_types(self, *, node_types: List[str]) -> None:
await self._backend.delete_scope_node_types(scope=self.scope, node_types=node_types)
async def close(self) -> None:
return None
@@ -0,0 +1,3 @@
from .persistence_backends.sqlalchemy_backend import SQLAlchemyPersistenceBackend
__all__ = ["SQLAlchemyPersistenceBackend"]
+76 -38
View File
@@ -9,7 +9,7 @@ from ..common.registry import DomainRegistry
from .datasource_config import DataSourceConfig
from .pipeline_config import PipelineRunConfig
from .run_presets import load_overrides_from_file
from .strategy_config import ConfigPresets, StrategyRuntimeConfig
from .strategy_config import ConfigPresets, StrategyRuntimeConfig, resolve_domain_option_config
def _deep_merge(base: Mapping[str, Any], override: Mapping[str, Any]) -> Dict[str, Any]:
@@ -28,19 +28,76 @@ def _deep_merge(base: Mapping[str, Any], override: Mapping[str, Any]) -> Dict[st
return merged
def _build_default_strategy_runtime_config(node_type: str) -> StrategyRuntimeConfig | None:
strategy_class = DomainRegistry.get_strategy_class(node_type)
if strategy_class is None:
return None
config = strategy_class.default_config.model_copy(deep=True)
config.domain_option = resolve_domain_option_config(
getattr(strategy_class, "domain_option_model", None),
config.domain_option,
)
return StrategyRuntimeConfig(
node_type=node_type,
config=config,
)
def _default_strategy_map(node_types: list[str]) -> Dict[str, StrategyRuntimeConfig]:
strategy_map: Dict[str, StrategyRuntimeConfig] = {}
for node_type in node_types:
strategy_class = DomainRegistry.get_strategy_class(node_type)
if strategy_class is None:
runtime_cfg = _build_default_strategy_runtime_config(node_type)
if runtime_cfg is None:
continue
strategy_map[node_type] = StrategyRuntimeConfig(
node_type=node_type,
config=strategy_class.default_config.model_copy(deep=True),
)
strategy_map[node_type] = runtime_cfg
return strategy_map
def _merge_strategy_runtime_config(
runtime_cfg: StrategyRuntimeConfig,
raw_override: Any,
) -> StrategyRuntimeConfig:
override = dict(raw_override or {})
override.pop("node_type", None)
deprecated_keys = [key for key in ("force_sync", "load_mode") if key in override]
if deprecated_keys:
joined = ", ".join(deprecated_keys)
raise ValueError(
f"Strategy override for {runtime_cfg.node_type!r} contains unsupported legacy keys: {joined}. "
"Use skip_sync under strategies.<node_type> or datasource.handler_configs instead."
)
preset = override.pop("preset", None)
if preset:
runtime_cfg.config = runtime_cfg.config.model_validate(
_deep_merge(
runtime_cfg.config.model_dump(),
ConfigPresets.get_preset(str(preset)).model_dump(exclude_defaults=True),
)
)
if "config" in override and isinstance(override["config"], dict):
runtime_cfg.config = runtime_cfg.config.model_validate(
_deep_merge(runtime_cfg.config.model_dump(), override.pop("config"))
)
if override:
runtime_cfg.config = runtime_cfg.config.model_validate(
_deep_merge(runtime_cfg.config.model_dump(), override)
)
strategy_class = DomainRegistry.get_strategy_class(runtime_cfg.node_type)
runtime_cfg.config.domain_option = resolve_domain_option_config(
getattr(strategy_class, "domain_option_model", None) if strategy_class is not None else None,
runtime_cfg.config.domain_option,
)
return runtime_cfg
def _apply_strategy_overrides(
node_types: list[str],
strategy_overrides: Any,
@@ -49,8 +106,16 @@ def _apply_strategy_overrides(
if isinstance(strategy_overrides, list):
for item in strategy_overrides:
runtime_cfg = StrategyRuntimeConfig.model_validate(item)
strategy_map[runtime_cfg.node_type] = runtime_cfg
raw_item = dict(item or {})
node_type = str(raw_item.get("node_type") or "")
if not node_type:
raise ValueError("Strategy override list item missing required node_type")
runtime_cfg = strategy_map.get(node_type)
if runtime_cfg is None:
runtime_cfg = _build_default_strategy_runtime_config(node_type) or StrategyRuntimeConfig(node_type=node_type)
strategy_map[node_type] = _merge_strategy_runtime_config(runtime_cfg, raw_item)
return [strategy_map[n] for n in node_types if n in strategy_map]
if not isinstance(strategy_overrides, dict):
@@ -58,37 +123,10 @@ def _apply_strategy_overrides(
for node_type, raw_override in strategy_overrides.items():
if node_type not in strategy_map:
strategy_map[node_type] = StrategyRuntimeConfig(node_type=node_type)
strategy_map[node_type] = _build_default_strategy_runtime_config(node_type) or StrategyRuntimeConfig(node_type=node_type)
runtime_cfg = strategy_map[node_type]
override = dict(raw_override or {})
deprecated_keys = [key for key in ("force_sync", "load_mode") if key in override]
if deprecated_keys:
joined = ", ".join(deprecated_keys)
raise ValueError(
f"Strategy override for {node_type!r} contains unsupported legacy keys: {joined}. "
"Use skip_sync under strategies.<node_type> or datasource.handler_configs instead."
)
preset = override.pop("preset", None)
if preset:
runtime_cfg.config = ConfigPresets.get_preset(str(preset))
skip_sync = override.pop("skip_sync", None)
if skip_sync is not None:
runtime_cfg.skip_sync = bool(skip_sync)
if "config" in override and isinstance(override["config"], dict):
runtime_cfg.config = runtime_cfg.config.model_validate(
_deep_merge(runtime_cfg.config.model_dump(), override.pop("config"))
)
if override:
runtime_cfg.config = runtime_cfg.config.model_validate(
_deep_merge(runtime_cfg.config.model_dump(), override)
)
strategy_map[node_type] = _merge_strategy_runtime_config(runtime_cfg, raw_override)
return [strategy_map[n] for n in node_types if n in strategy_map]
@@ -25,6 +25,7 @@ class MultiProjectRunConfig(BaseModel):
global_config: str
default_project_config: str
max_concurrency: int = 20
project_bootstrap_node_types: List[str] = []
projects: List[MultiProjectItemConfig]
@@ -17,6 +17,7 @@ meta:
# represented as separate invocations with different inputs.
context:
is_create_zombie: { type: bool, desc: "初始化时判定为 CREATE 僵尸:action=CREATE 且 (status=FAILED 或 data_id 为空)" }
bootstrap_source_present: { type: bool, desc: "bootstrap 阶段 datasource 刷新后,当前节点是否在最新数据源中命中" }
has_binding_record: { type: bool, desc: "binding_manager 里存在绑定记录 (remote_id/local_id 非空)" }
core_binding_result: { type: enum, values: [NORMAL, WARNING, ABNORMAL], desc: "SyncDecisionEngine.determine_core_status() 给本节点的结果(仅在 has_binding_record=true 时有效)" }
@@ -304,19 +305,30 @@ states:
status: SKIPPED
data_id_exist: true
S17:
binding_status: ABNORMAL
action: NONE
status: FAILED
data_id_exist: [false, true]
desc: |
bootstrap 加载异常隔离态
持久化恢复后存在绑定,但 datasource 刷新未命中最新数据
binding_status: ABNORMAL
action: NONE
status: FAILED
data_id_exist: false | true
pseudo_states:
S15:
desc: "节点被清理删除(zombie cleanup"
S16:
desc: "持久化加载输入态:本轮 E01 的输入节点集合(可包含上轮遗留与已稳定节点)"
S17:
desc: "加载异常隔离态:持久化枚举非法,保留错误供人工处理,不进入自动流程"
stages:
bootstrap:
desc: "启动/初始化(仅针对持久化加载出的节点):僵尸清理与归并到 S00"
desc: "启动/初始化:持久化恢复后先做 datasource 刷新,再按 E01 清理僵尸或归并/隔离"
entry_states: [S16]
exit_states: [S00, S15]
exit_states: [S00, S15, S17]
bind:
desc: "绑定判定:core matrix +(可选)依赖检查 +(可选)自动绑定;stage 仅约束边界,不要求内部顺序"
@@ -346,13 +358,23 @@ stages:
transitions:
E01:
stage: bootstrap
desc: "初始化:删CREATE僵尸;其余重置后进入起点"
desc: "初始化:先依据 datasource 刷新结果判定 source 是否存在,再做僵尸清理/隔离/归并"
from: [S16]
outcomes:
- when: { is_create_zombie: true }
to: S15
- when: { is_create_zombie: false }
- when: { is_create_zombie: false, bootstrap_source_present: false, has_binding_record: false }
to: S15
emit:
note: "持久化孤儿节点未命中最新数据源,按僵尸清理"
- when: { is_create_zombie: false, bootstrap_source_present: false, has_binding_record: true }
to: S17
emit:
note: "存在绑定但最新数据源未返回节点,进入加载异常隔离"
- when: { is_create_zombie: false, bootstrap_source_present: true }
to: S00
emit:
note: "bootstrap 已刷新到最新数据,归并到起点"
# --- Bind phase1 core ---
E10:
+60 -3
View File
@@ -1,13 +1,70 @@
from typing import Any, Dict
from typing import Any, Dict, Optional
from pydantic import BaseModel, ConfigDict, Field
from pydantic import BaseModel, ConfigDict, Field, model_validator
class PersistConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
backend: str = "sqlite"
db_path: str
db_path: Optional[str] = None
url: Optional[str] = None
backend_options: Dict[str, Any] = Field(default_factory=dict)
enable: bool = True
wipe_on_start: bool = False
@model_validator(mode="before")
@classmethod
def _validate_target(cls, data: Any) -> Any:
if not isinstance(data, dict):
return data
normalized = dict(data)
normalized["backend"] = (str(normalized.get("backend") or "sqlite").strip().lower() or "sqlite")
normalized["db_path"] = cls._normalize_optional_text(normalized.get("db_path"))
normalized["url"] = cls._normalize_optional_text(normalized.get("url"))
if not normalized["db_path"] and not normalized["url"]:
raise ValueError("persist requires either db_path or url")
if normalized["backend"] != "sqlite" and not normalized["url"]:
raise ValueError(f"persist.url is required for backend={normalized['backend']}")
return normalized
@staticmethod
def _normalize_optional_text(value: Optional[str]) -> Optional[str]:
if value is None:
return None
normalized = str(value).strip()
return normalized or None
def resolved_url(self) -> str:
if self.url:
return self.url
if self.backend == "sqlite" and self.db_path:
return f"sqlite+aiosqlite:///{self.db_path}"
raise ValueError(f"persist.url is required for backend={self.backend}")
def resolved_db_path(self) -> Optional[str]:
if self.db_path:
return self.db_path
if not self.url:
return None
from sqlalchemy.engine import make_url
parsed = make_url(self.url)
if parsed.get_backend_name() != "sqlite":
return None
database = parsed.database
if not database:
return None
return str(database)
def wipe_file_path(self) -> Optional[str]:
db_path = self.resolved_db_path()
if not db_path or db_path == ":memory:":
return None
return db_path
def display_target(self) -> str:
return self.url or self.db_path or "<unset>"
+74 -10
View File
@@ -1,11 +1,21 @@
from __future__ import annotations
import logging
from enum import Enum
from typing import Dict, List, Optional
from typing import Any, Dict, List, Mapping, Type
from pydantic import BaseModel, ConfigDict, Field
def resolve_domain_option_config(
domain_option_model: Type[BaseModel] | None,
domain_option: Mapping[str, Any] | None,
) -> Dict[str, Any]:
if domain_option_model is None:
return dict(domain_option or {})
return domain_option_model.model_validate(domain_option or {}).model_dump()
class OrphanAction(str, Enum):
CREATE_REMOTE = "create_remote"
CREATE_LOCAL = "create_local"
@@ -17,7 +27,6 @@ class OrphanAction(str, Enum):
class UpdateDirection(str, Enum):
PUSH = "push"
PULL = "pull"
BIDIRECTIONAL = "bidirectional"
NONE = "none"
@@ -40,20 +49,76 @@ class StrategyConfig(BaseModel):
remote_orphan_action: OrphanAction = OrphanAction.NONE
update_direction: UpdateDirection = UpdateDirection.PUSH
# 字段级同步方向覆盖。
# field_direction_key: 节点 data 中用作查表键的字段名,如 "key"。
# field_direction_overrides: 字段值 → UpdateDirection 的映射。
# 未命中映射的节点沿用 update_direction 作为默认方向。
field_direction_key: Optional[str] = None
field_direction_overrides: Dict[str, UpdateDirection] = Field(default_factory=dict)
# post-check 一致性比较时忽略的顶层字段。
# 只影响最终一致性评估,不影响 create/update 的差异检测。
post_check_ignore_fields: List[str] = Field(default_factory=list)
# post-check 一致性比较时,忽略列表型字段内各元素的特定子键。
# 格式: {字段名: [子键, ...]},如 {"plan_node": ["is_audit"]}。
# 用于排除后端硬写死、永远与本地不一致的字段,避免误报。
post_check_ignore_list_item_fields: Dict[str, List[str]] = Field(default_factory=dict)
# schema diff / validate 时忽略的顶层字段。
compare_ignore_fields: List[str] = Field(default_factory=list)
# 当 auto-bind 命中“多对一”候选时,允许稳定选择一条进行绑定,剩余候选按 orphan/create 流程继续处理。
# 默认关闭,避免在一般场景下引入隐式匹配。
allow_many_to_one_auto_bind: bool = False
# 是否跳过该类型的 create/update 阶段。
skip_sync: bool = False
# 是否跳过该类型的 post-check reload 与一致性校验。
skip_post_check: bool = False
# 领域策略运行参数,由各 strategy 的 domain_option_model 负责解释与校验。
domain_option: Dict[str, Any] = Field(default_factory=dict)
def log_logic_warnings(self, *, node_type: str, logger: logging.Logger) -> None:
direction = self.update_direction
local_action = self.local_orphan_action
remote_action = self.remote_orphan_action
if direction == UpdateDirection.PUSH:
if local_action == OrphanAction.CREATE_LOCAL:
logger.warning(
f"[{node_type}] Config inconsistency: "
f"update_direction=PUSH but local_orphan_action=CREATE_LOCAL (本地创建)。"
f"应使用 CREATE_REMOTE (推送到远程)"
)
if remote_action == OrphanAction.CREATE_LOCAL:
logger.warning(
f"[{node_type}] Config inconsistency: "
f"update_direction=PUSH but remote_orphan_action=CREATE_LOCAL (本地创建)。"
f"PUSH 模式应设为 NONE 或 DELETE_REMOTE"
)
elif direction == UpdateDirection.PULL:
if local_action == OrphanAction.CREATE_REMOTE:
logger.warning(
f"[{node_type}] Config inconsistency: "
f"update_direction=PULL but local_orphan_action=CREATE_REMOTE (推送到远程)。"
f"PULL 模式应设为 NONE 或 DELETE_LOCAL"
)
if remote_action in {OrphanAction.CREATE_REMOTE, OrphanAction.DELETE_LOCAL}:
logger.warning(
f"[{node_type}] Config inconsistency: "
f"update_direction=PULL but remote_orphan_action={remote_action}"
f"PULL 模式建议使用 CREATE_LOCAL 或 NONE"
)
elif direction == UpdateDirection.NONE:
if local_action != OrphanAction.NONE:
logger.warning(
f"[{node_type}] Config inconsistency: "
f"update_direction=NONE but local_orphan_action={local_action}"
f"应设为 NONE"
)
if remote_action != OrphanAction.NONE:
logger.warning(
f"[{node_type}] Config inconsistency: "
f"update_direction=NONE but remote_orphan_action={remote_action}"
f"应设为 NONE"
)
class StrategyRuntimeConfig(BaseModel):
"""Pipeline 层策略运行配置(按 node_type)。"""
@@ -61,7 +126,6 @@ class StrategyRuntimeConfig(BaseModel):
model_config = ConfigDict(validate_assignment=True, extra="forbid")
node_type: str
skip_sync: bool = False
config: StrategyConfig = Field(default_factory=StrategyConfig)
+110 -3
View File
@@ -23,6 +23,7 @@ from datetime import datetime
import httpx
from schemas.common.push_system import PushIdsSchema
from ...logging import get_logger, get_scope_display_name
def _build_default_push_steps() -> List[Dict[str, str]]:
@@ -33,6 +34,7 @@ def _build_default_push_steps() -> List[Dict[str, str]]:
class ApiClient:
"""API HTTP 客户端"""
_TRACE_TEXT_MAX_LEN = 1000
_REQUEST_STATS_LOG_PREFIX = "📊 ApiClient request stats"
def __init__(
self,
@@ -41,6 +43,7 @@ class ApiClient:
secret: str,
debug: bool = False,
log_file: Optional[Path] = None,
session_label: Optional[str] = None,
rate_limit_max_requests: int = 30,
rate_limit_window_seconds: float = 10.0,
):
@@ -62,16 +65,58 @@ class ApiClient:
self._client: Optional[httpx.AsyncClient] = None
self._debug = debug
self._log_file = log_file
self._scope = ""
self._session_label = (session_label or "").strip()
self._trace_seq = 0
self._trace_by_push_id: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
self._trace_by_biz_id: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
self._trace_by_op: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
self._load_trace_by_item: Dict[tuple[str, str], deque[Dict[str, Any]]] = defaultdict(deque)
self._logger = logging.getLogger(__name__)
self._logger = get_logger(__name__)
self._rate_limit_max_requests = int(rate_limit_max_requests)
self._rate_limit_window_seconds = float(rate_limit_window_seconds)
self._rate_limit_lock = asyncio.Lock()
self._rate_limit_timestamps: deque[float] = deque()
self._request_total = 0
self._request_success = 0
self._request_failure = 0
def set_scope(self, scope: Optional[str]) -> None:
self._scope = str(scope or "").strip()
def set_session_label(self, label: Optional[str]) -> None:
self._session_label = str(label or "").strip()
def get_log_session_label(self) -> str:
if self._session_label:
return self._session_label
scope_label = get_scope_display_name(self._scope)
return scope_label or self._scope
def _project_name_from_request(self, payload: Optional[Dict[str, Any]]) -> str:
if not isinstance(payload, dict):
return ""
project_id = str(payload.get("project_id") or "").strip()
if not project_id:
return ""
display = get_scope_display_name(f"project_id:{project_id}")
if not display or display == f"project_id:{project_id}":
return ""
return display
def _resolve_request_session_label(
self,
*,
request_params: Optional[Dict[str, Any]] = None,
request_payload: Optional[Dict[str, Any]] = None,
) -> str:
payload_label = self._project_name_from_request(request_payload)
if payload_label:
return payload_label
params_label = self._project_name_from_request(request_params)
if params_label:
return params_label
return self.get_log_session_label()
def _is_rate_limit_enabled(self) -> bool:
return self._rate_limit_max_requests > 0 and self._rate_limit_window_seconds > 0
@@ -105,9 +150,54 @@ class ApiClient:
async def close(self):
"""关闭 HTTP 客户端"""
self.log_request_stats(level="INFO")
if self._client:
await self._client.aclose()
self._client = None
def get_request_stats(self) -> Dict[str, int]:
"""返回当前客户端生命周期内的请求统计。"""
return {
"total": self._request_total,
"success": self._request_success,
"failure": self._request_failure,
}
def format_request_stats(self) -> str:
"""格式化当前客户端生命周期内的请求统计。"""
stats = self.get_request_stats()
session_text = self.get_log_session_label()
session_prefix = f"session={session_text} " if session_text else ""
return (
f"{self._REQUEST_STATS_LOG_PREFIX}: "
f"{session_prefix}"
f"total={stats['total']}, success={stats['success']}, failure={stats['failure']}"
)
def log_request_stats(self, level: str = "DEBUG") -> None:
"""输出当前客户端生命周期内的请求统计。"""
self._log(self.format_request_stats(), level=level)
def _log_request_summary(
self,
*,
method: str,
endpoint: str,
elapsed_seconds: float,
request_params: Optional[Dict[str, Any]] = None,
request_payload: Optional[Dict[str, Any]] = None,
) -> None:
if not self._debug:
return
session_text = self._resolve_request_session_label(
request_params=request_params,
request_payload=request_payload,
)
session_prefix = f"session={session_text} " if session_text else ""
self._log(
f"🔎 ApiClient request: {session_prefix}method={method.upper()} endpoint={endpoint} elapsed={elapsed_seconds:.3f}s",
level="INFO",
)
def _log(self, message: str, level: str = "DEBUG"):
"""
@@ -382,6 +472,7 @@ class ApiClient:
client = self._get_client()
url = f"{self._base_url}{endpoint}"
await self._acquire_rate_limit_slot()
self._request_total += 1
# 记录请求信息
self._log(f"\n{'='*60}", level="DEBUG")
@@ -400,10 +491,10 @@ class ApiClient:
if params is None:
params = {}
params['uid'] = self._uid
# 转换所有参数值为字符串(除了 list/dict)
params = {k: str(v) if not isinstance(v, (list, dict)) else v for k, v in params.items()}
sign_params = params.copy()
# 生成签名并附加到 params
signature = self._generate_signature(sign_params)
@@ -433,6 +524,13 @@ class ApiClient:
# 记录响应时间
elapsed = time.time() - start_time
self._log_request_summary(
method=method,
endpoint=endpoint,
elapsed_seconds=elapsed,
request_params=params,
request_payload=data if method.upper() in ['POST', 'PUT', 'DELETE'] else None,
)
self._log(f"Response Time: {elapsed:.3f}s", level="DEBUG")
# 检查 HTTP 状态
@@ -453,14 +551,23 @@ class ApiClient:
response=result,
elapsed_ms=int(elapsed * 1000),
)
self._request_success += 1
return result
except Exception as e:
import traceback
self._request_failure += 1
# 记录错误
elapsed = time.time() - start_time
self._log_request_summary(
method=method,
endpoint=endpoint,
elapsed_seconds=elapsed,
request_params=params,
request_payload=data if method.upper() in ['POST', 'PUT', 'DELETE'] else None,
)
error_msg = "\n========== [ApiClient HTTP Error] ==========\n"
error_msg += f"Request: {method} {url}\n"
error_msg += f"Params: {params}\n"
+27 -14
View File
@@ -15,6 +15,7 @@ from ..datasource import BaseDataSource
from ..handler import NodeHandler
from ..task_result import TaskResult
from .client import ApiClient
from ...logging import set_scope_display_name
class ApiDataSource(BaseDataSource):
@@ -56,6 +57,29 @@ class ApiDataSource(BaseDataSource):
rate_limit_window_seconds=rate_limit_window_seconds,
)
self.poll_mode = poll_mode
def set_collection(self, collection) -> None:
super().set_collection(collection)
self.client.set_scope(getattr(collection, "scope", ""))
def _update_client_session_label_from_project_node(self, node) -> None:
if self._collection is None or getattr(self._collection, "scope", "") == "global":
return
if getattr(node, "node_type", None) != "project":
return
if not hasattr(node, "get_data"):
return
data = node.get_data()
if not isinstance(data, dict):
return
project_name = str(data.get("name") or "").strip()
if not project_name:
return
set_scope_display_name(self._collection.scope, f"project:{project_name}")
self.client.set_session_label(f"project:{project_name}")
async def initialize(self):
"""初始化数据源(初始化 HTTP 客户端)"""
@@ -65,22 +89,10 @@ class ApiDataSource(BaseDataSource):
"""关闭数据源(关闭 HTTP 客户端)"""
await self.client.close()
def add_handler(self, handler: NodeHandler) -> None:
"""
添加业务 Handler
Args:
handler: API Handler 实例
"""
def _prepare_handler_for_registration(self, handler: NodeHandler) -> None:
handler.set_api_client(self.client)
handler.set_poll_mode(self.poll_mode)
# 调用基类添加方法
super().add_handler(handler)
# 如果 collection 已设置,确保新 handler 拿到引用
if self._collection is not None:
handler.set_collection(self._collection) # type: ignore[arg-type]
super()._prepare_handler_for_registration(handler)
@staticmethod
def _fmt_trace_value(value, max_len: int = 600) -> str:
@@ -187,6 +199,7 @@ class ApiDataSource(BaseDataSource):
return None
def _on_node_loaded(self, handler: NodeHandler, node) -> None:
self._update_client_session_label_from_project_node(node)
data_id = node.data_id or ""
if not data_id:
return
+106 -80
View File
@@ -17,9 +17,10 @@ import asyncio
from abc import abstractmethod
from copy import deepcopy
from typing import ClassVar, Dict, List, Any, Optional, TYPE_CHECKING, TypeVar
from pydantic import BaseModel
from pydantic import BaseModel, ValidationError
from ...common.type_safety import get_pydantic_model_fields
from ...logging import get_logger
from ..handler import BaseNodeHandler
from ..task_result import TaskResult
from ..handler import ValidationResult # Import ValidationResult
@@ -30,6 +31,7 @@ if TYPE_CHECKING:
from .client import ApiClient
T = TypeVar("T", bound=BaseModel)
logger = get_logger(__name__)
class BaseApiHandler(BaseNodeHandler[T]):
@@ -41,8 +43,17 @@ class BaseApiHandler(BaseNodeHandler[T]):
_schema: Any
def __init__(self, **handler_config: Any):
# 子类应该在调用 super().__init__() 之前或之后设置 _node_type, _node_class, _schema
super().__init__()
# 子类可直接传入 node_class;少数特殊场景可显式覆盖 schema/node_type。
node_class = handler_config.pop("node_class", None)
node_type = handler_config.pop("node_type", None)
schema = handler_config.pop("schema", None)
if node_class is not None and node_type is None:
node_type = getattr(node_class, "node_type", None)
if node_class is not None and schema is None:
schema = getattr(node_class, "schema", None)
super().__init__(node_type=node_type, node_class=node_class, schema=schema)
self.api_client: 'ApiClient' = None # type: ignore # 由 DataSource 注入
self.poll_mode: str = "async" # async | sync
self._collection: 'DataCollection' = None # type: ignore # 由 DataSource 注入
@@ -51,9 +62,6 @@ class BaseApiHandler(BaseNodeHandler[T]):
# post-check 将只比较这些 schema 覆盖的字段,过滤后端只读的派生字段。
self.update_schemas: List[type] = []
def set_collection(self, collection: 'DataCollection') -> None:
self._collection = collection
def get_update_fields(self, *, exclude: frozenset = frozenset({'id'})) -> List[str]:
"""从 update_schemas 中提取所有可写字段名,排除标识键(默认排除 id)。"""
fields: set = set()
@@ -68,7 +76,17 @@ class BaseApiHandler(BaseNodeHandler[T]):
def set_collection(self, collection: 'DataCollection') -> None:
"""设置 Collection(由 DataSource 调用)"""
self._collection = collection
super().set_collection(collection)
def ensure_api_client(self) -> 'ApiClient':
if self.api_client is None:
raise RuntimeError("API client not set. Call set_api_client() before invoking API handler methods.")
return self.api_client
def ensure_collection(self) -> 'DataCollection':
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() before invoking handler methods.")
return self._collection
@property
def node_type(self) -> str:
@@ -112,8 +130,7 @@ class BaseApiHandler(BaseNodeHandler[T]):
def _create_basic_node(self, data: Dict[str, Any], depend_ids: Optional[List[str]] = None) -> "SyncNode":
"""创建或复用基础节点(默认使用 data['id'] 或 data['_id']"""
data_id = str(data.get("id") or data.get("_id") or "")
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() before creating nodes.")
self.ensure_collection()
if data_id:
existing_node = self._collection.get_by_data_id(self.node_type, data_id)
@@ -167,10 +184,6 @@ class BaseApiHandler(BaseNodeHandler[T]):
payload["steps"] = self._get_request_steps(node) if node is not None else []
return payload
def _build_full_update_data(self, new_data: Dict[str, Any], schema: Any) -> Dict[str, Any]:
validated = schema.model_validate(new_data)
return validated.model_dump(exclude_none=True)
def _mark_approval_snapshot_synced(self, node: 'SyncNode') -> None:
snapshot = self._get_active_approval_snapshot(node)
if snapshot:
@@ -255,8 +268,53 @@ class BaseApiHandler(BaseNodeHandler[T]):
return None
except Exception as e:
return str(e)
def _get_update_schema_payload(
self,
schema: Any,
data: Dict[str, Any],
*,
require_valid: bool,
) -> Dict[str, Any] | None:
"""Update 内部实现:提取指定 schema 下的 payload。
这里明确只服务“update schema”,不负责 response model 一致性校验。
这个方法只给 BaseApiHandler 的 update 入口方法复用,不作为 handler 入口。
处理顺序:
1. 先尝试用 update schema 校验,成功则直接使用校验后的 model_dump。
2. 如果校验失败且 require_valid=False,则退化为只抽取 schema 定义的字段。
之所以允许退化,是因为 update 比较时两边快照常常不完整:
某些字段在旧快照缺失,但新数据已经有值。此时应该继续比较 schema
相关字段,而不是把它误判成“无差异”。
空字符串和 None 会被统一过滤掉,因为当前 update handler 并不把它们
当成“清空字段”的显式语义,而是当作“不参与本次更新”。
"""
try:
validated = schema.model_validate(data)
payload = validated.model_dump(exclude_unset=True)
except ValidationError:
if require_valid:
return None
schema_fields = get_pydantic_model_fields(schema)
if not schema_fields:
return {}
payload = {
field_name: data.get(field_name)
for field_name in schema_fields.keys()
if field_name in data
}
return {
field_name: value
for field_name, value in payload.items()
if value not in (None, "")
}
def _get_schema_diff(
def get_update_schema_diff(
self,
schema,
data: Dict[str, Any],
@@ -264,8 +322,7 @@ class BaseApiHandler(BaseNodeHandler[T]):
node_id: Optional[str] = None,
force_full_schema: bool = False,
) -> Dict[str, Any] | None:
"""
获取数据在指定 schema 下的差异字段
"""Update 入口:获取指定 update schema 下的差异字段。
Args:
schema: Pydantic schema 类
@@ -276,46 +333,37 @@ class BaseApiHandler(BaseNodeHandler[T]):
Returns:
Dict[str, Any] | None: 有差异返回差异字段字典,无差异或字段不全返回 None
"""
from pydantic import ValidationError
try:
new_model = schema.model_validate(data)
origin_model = schema.model_validate(origin_data)
new_dict = new_model.model_dump(exclude_unset=True)
origin_dict = origin_model.model_dump(exclude_unset=True)
if new_dict == origin_dict:
if force_full_schema:
return new_dict
return None
# 返回有差异的字段
diff_fields = {k: v for k, v in new_dict.items() if origin_dict.get(k) != v}
if diff_fields:
# 打印差异详情
node_info = f"[{node_id}]" if node_id else ""
print(f" [DIFF] {self.node_type}{node_info} schema={schema.__name__}")
for k, v in diff_fields.items():
orig_v = origin_dict.get(k)
print(f"{k}: {orig_v!r}{v!r}")
return diff_fields if diff_fields else None
except ValidationError:
# schema 字段不全,视为无差异
# 当前待发送的数据必须能构成合法 update payload;允许旧快照不完整。
new_dict = self._get_update_schema_payload(schema, data, require_valid=True)
origin_dict = self._get_update_schema_payload(schema, origin_data, require_valid=False)
if new_dict is None or origin_dict is None:
return None
if new_dict == origin_dict:
if force_full_schema:
return new_dict
return None
diff_fields = {k: v for k, v in new_dict.items() if origin_dict.get(k) != v}
if diff_fields:
node_info = f"[{node_id}]" if node_id else ""
logger.debug(" [DIFF] %s%s schema=%s", self.node_type, node_info, schema.__name__)
for k, v in diff_fields.items():
orig_v = origin_dict.get(k)
logger.debug("%s: %r%r", k, orig_v, v)
return diff_fields if diff_fields else None
def _filter_update_data(
def build_update_request_data(
self,
new_data: Dict[str, Any],
original_data: Dict[str, Any],
schema: Optional[Any] = None,
force_full_schema: bool = False,
) -> Dict[str, Any]:
"""
过滤出需要更新的字段
"""Update 入口:构造最终要发送给更新接口的请求数据。
规则:
1. 提取 schema 定义的所有字段(包括必填)
@@ -342,29 +390,10 @@ class BaseApiHandler(BaseNodeHandler[T]):
update_data[key] = new_value
return update_data
# 1. 提取 schema 定义的所有字段
schema_fields = set(schema_fields.keys())
schema_data = {}
has_change = False
for key in schema_fields:
new_value = new_data.get(key)
# 排除空值
if new_value == "" or new_value is None:
continue
schema_data[key] = new_value
# 检查是否变化
original_value = original_data.get(key)
if new_value != original_value:
# 记录具体差异以便排查
# print(f" [DEBUG] Field '{key}' changed: '{original_value}' -> '{new_value}'")
has_change = True
# 2. 只有存在变化时才返回数据
if has_change or force_full_schema:
schema_data = self._get_update_schema_payload(schema, new_data, require_valid=True) or {}
origin_schema_data = self._get_update_schema_payload(schema, original_data, require_valid=False) or {}
if force_full_schema or schema_data != origin_schema_data:
return schema_data
return {}
@@ -374,16 +403,12 @@ class BaseApiHandler(BaseNodeHandler[T]):
new_data: Dict[str, Any],
origin_data: Dict[str, Any],
) -> List[str]:
schema_fields = get_pydantic_model_fields(schema)
if not schema_fields:
return []
changed: List[str] = []
for field_name in schema_fields.keys():
if new_data.get(field_name) != origin_data.get(field_name):
changed.append(field_name)
return changed
new_payload = self._get_update_schema_payload(schema, new_data, require_valid=True) or {}
origin_payload = self._get_update_schema_payload(schema, origin_data, require_valid=False) or {}
field_names = sorted(set(new_payload) | set(origin_payload))
return [field_name for field_name in field_names if new_payload.get(field_name) != origin_payload.get(field_name)]
def _build_update_schema_status(
def build_update_schema_status(
self,
*,
schema_name: str,
@@ -393,6 +418,7 @@ class BaseApiHandler(BaseNodeHandler[T]):
will_update: bool,
update_error: Optional[str] = None,
) -> str:
"""Update 入口:生成 update schema 的状态说明文本。"""
new_error = self._validate_schema(new_data, schema)
origin_error = self._validate_schema(origin_data, schema)
changed_fields = self._schema_changed_fields(schema, new_data, origin_data)
+39 -15
View File
@@ -37,9 +37,10 @@ from ..engine import (
StateMachineRuntime,
e40_sync_execute,
)
from ..logging import get_logger, get_scope_display_name
logger = logging.getLogger(__name__)
logger = get_logger(__name__)
class BaseDataSource(ABC):
@@ -74,6 +75,8 @@ class BaseDataSource(ABC):
def __init__(self, poll_max_retries: int = 1, poll_interval: float = 0.5):
self._handlers: Dict[str, "NodeHandler"] = {}
self._collection: Optional["DataCollection"] = None
self._endpoint_name: str = "unknown"
self._datasource_type: str = self.__class__.__name__.lower()
# 按节点类型统计信息
# {"company": {"loaded": 10, "synced": 10, "success": 10, "failed": 0}, ...}
@@ -86,6 +89,17 @@ class BaseDataSource(ABC):
self._poll_interval = max(0, poll_interval)
self._sm_runtime: Optional[StateMachineRuntime] = None
def set_runtime_logging_context(self, *, endpoint_name: str, datasource_type: str) -> None:
self._endpoint_name = endpoint_name
self._datasource_type = datasource_type
def _datasource_log_prefix(self) -> str:
scope = self._collection.scope if self._collection is not None else "unbound"
return f"[{get_scope_display_name(scope)}][{self._endpoint_name}:{self._datasource_type}]"
def _node_log_prefix(self, node_type: str) -> str:
return f"{self._datasource_log_prefix()}[{node_type}]"
def _ensure_sm_runtime(self) -> StateMachineRuntime:
if self._sm_runtime is None:
cfg_path = Path(__file__).resolve().parents[1] / "config" / "node_state_machine.yaml"
@@ -162,6 +176,11 @@ class BaseDataSource(ABC):
"""Datasource-specific hook after a FAILED result is applied to node."""
return
def _prepare_handler_for_registration(self, handler: "NodeHandler") -> None:
"""Allow subclasses to inject datasource-specific dependencies before registration."""
if self._collection is not None:
handler.set_collection(self._collection)
async def _handle_success_result(self, node: "SyncNode", result: "TaskResult") -> None:
previous_data_id = node.data_id
ok = self._apply_sync_execute_state(
@@ -370,6 +389,7 @@ class BaseDataSource(ABC):
示例:
datasource.add_handler(ContractNodeHandler(api_client))
"""
self._prepare_handler_for_registration(handler)
self._handlers[handler.node_type] = handler
def get_handler(self, node_type: str) -> "NodeHandler":
@@ -418,7 +438,9 @@ class BaseDataSource(ABC):
async def load_all(
self,
order: Optional[List[str]] = None,
data_type: Optional[str] = None
data_type: Optional[str] = None,
*,
raise_on_error: bool = False,
) -> None:
"""
按依赖顺序加载所有数据
@@ -460,7 +482,7 @@ class BaseDataSource(ABC):
# Handler 加载数据并创建节点
nodes = await handler.load()
print(f"Loaded nodes for {node_type}: {len(nodes)}")
logger.debug("%s Loaded nodes for %s: %d", self._datasource_log_prefix(), node_type, len(nodes))
# 将节点写入 Collection:优先按 data_id 命中已有节点并更新
await self._upsert_loaded_nodes(handler, node_type, nodes)
@@ -468,7 +490,9 @@ class BaseDataSource(ABC):
# 统计加载的节点数
self._stats[node_type]["loaded"] += len(nodes)
except Exception as exc:
print(f"❌ [{node_type}] 加载失败: {exc}")
logger.error("%s ❌ 加载失败: %s", self._node_log_prefix(node_type), exc)
if raise_on_error:
raise RuntimeError(f"[{node_type}] 加载失败: {exc}") from exc
continue
# ========== 同步执行 ==========
@@ -617,7 +641,7 @@ class BaseDataSource(ABC):
# 批量创建(传入节点列表)
if create_nodes:
logger.info(f"[{handler.node_type}] create提交开始: nodes={len(create_nodes)}")
logger.debug(f"{self._node_log_prefix(handler.node_type)} create提交开始: nodes={len(create_nodes)}")
try:
create_results = await handler.create_all(create_nodes)
create_success = 0
@@ -634,12 +658,12 @@ class BaseDataSource(ABC):
elif result.status == _TaskStatus.SKIPPED:
create_skipped += 1
await self._handle_task_result(handler, result, async_tasks)
logger.info(
f"[{handler.node_type}] create提交完成: success={create_success}, "
logger.debug(
f"{self._node_log_prefix(handler.node_type)} create提交完成: success={create_success}, "
f"in_progress={create_in_progress}, failed={create_failed}, skipped={create_skipped}"
)
except Exception as e:
print(f"❌ [{handler.node_type}] create_all 异常: {e}")
logger.error("❌ [%s] create_all 异常: %s", handler.node_type, e)
for node in create_nodes:
ok = self._apply_sync_execute_state(
node,
@@ -656,7 +680,7 @@ class BaseDataSource(ABC):
for result in update_results:
await self._handle_task_result(handler, result, async_tasks)
except Exception as e:
print(f"❌ [{handler.node_type}] update_all 异常: {e}")
logger.error("❌ [%s] update_all 异常: %s", handler.node_type, e)
for node in update_nodes:
ok = self._apply_sync_execute_state(
node,
@@ -670,8 +694,8 @@ class BaseDataSource(ABC):
# 轮询异步任务
if async_tasks and poll_async_tasks:
logger.info(
f"[{handler.node_type}] create已提交,push_id检查即将开始: pending={len(async_tasks)}, "
logger.debug(
f"{self._node_log_prefix(handler.node_type)} create已提交,push_id检查即将开始: pending={len(async_tasks)}, "
f"first_wait={self._poll_interval:.3f}s"
)
await self._poll_async_tasks(handler, async_tasks)
@@ -821,14 +845,14 @@ class BaseDataSource(ABC):
return
if async_tasks and poll_interval > 0:
logger.info(
f"[{handler.node_type}] push_id首次检查前等待 {poll_interval:.3f}s, pending={len(async_tasks)}"
logger.debug(
f"{self._node_log_prefix(handler.node_type)} push_id首次检查前等待 {poll_interval:.3f}s, pending={len(async_tasks)}"
)
await asyncio.sleep(poll_interval)
while async_tasks and retry_count < max_retries:
logger.info(
f"[{handler.node_type}] push_id检查 {retry_count + 1}/{max_retries}, pending={len(async_tasks)}"
logger.debug(
f"{self._node_log_prefix(handler.node_type)} push_id检查 {retry_count + 1}/{max_retries}, pending={len(async_tasks)}"
)
# 批量查询
task_ids = list(async_tasks.values())
+14 -4
View File
@@ -43,11 +43,21 @@ class BaseJsonlHandler(BaseNodeHandler):
def __init__(
self,
datasource: JsonlDataSource,
node_type: str,
node_class: type,
schema: type
node_class_or_node_type,
node_class: type | None = None,
schema: type | None = None,
node_type: str | None = None,
):
super().__init__(node_type, node_class, schema)
if isinstance(node_class_or_node_type, str):
resolved_node_type = node_class_or_node_type
resolved_node_class = node_class
resolved_schema = schema
else:
resolved_node_class = node_class_or_node_type
resolved_node_type = node_type or getattr(resolved_node_class, "node_type", None)
resolved_schema = schema or getattr(resolved_node_class, "schema", None)
super().__init__(resolved_node_type, resolved_node_class, resolved_schema)
self.datasource = datasource
async def load(self) -> List['SyncNode']:
@@ -3,8 +3,10 @@ from __future__ import annotations
import logging
from pathlib import Path
from ...logging import get_logger
logger = logging.getLogger(__name__)
logger = get_logger(__name__)
def prepare_remote_jsonl_dir(remote_dir: Path, *, project_root: Path) -> Path:
@@ -17,10 +17,7 @@ class AttachmentApiHandler(BaseApiHandler):
"""附件 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "attachment"
self._node_class = AttachmentSyncNode
self._schema = AttachmentSchema
super().__init__(node_class=AttachmentSyncNode, schema=AttachmentSchema)
async def load(self) -> List[SyncNode]:
"""加载附件列表(依赖 Project"""
@@ -9,4 +9,4 @@ from .sync_node import AttachmentSyncNode
class AttachmentJsonlHandler(BaseJsonlHandler):
"""附件 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "attachment", AttachmentSyncNode, Dict[str, Any])
super().__init__(datasource, AttachmentSyncNode, schema=Dict[str, Any])
@@ -6,7 +6,7 @@ Company API Handler (只读)
- 不支持创建/更新/删除
"""
from typing import List, Dict, Any
from typing import List
from ...datasource.api.handler import BaseApiHandler
from ...datasource.task_result import TaskResult
from ...common.sync_node import SyncNode
@@ -18,10 +18,7 @@ class CompanyApiHandler(BaseApiHandler):
"""公司 API Handler (只读)"""
def __init__(self):
super().__init__()
self._node_type = "company"
self._node_class = CompanySyncNode
self._schema = CompanyResponse
super().__init__(node_class=CompanySyncNode, schema=CompanyResponse)
async def load(self) -> List[SyncNode]:
"""加载公司列表(支持分页)"""
@@ -85,10 +82,6 @@ class CompanyApiHandler(BaseApiHandler):
"""轮询异步任务状态(占位)"""
return await super().poll_tasks(task_ids)
def _create_node(self, data: Dict[str, Any]) -> SyncNode:
return self._create_basic_node(data, depend_ids=[])
# ========== 静态 API 函数 ==========
async def api_get_company_list(client, page: int = 1, page_size: int = 100) -> tuple[List[CompanyResponse], int]:
@@ -7,5 +7,5 @@ from schemas.company.company import CompanyResponse
class CompanyJsonlHandler(BaseJsonlHandler):
"""公司 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "company", CompanySyncNode, CompanyResponse)
super().__init__(datasource, CompanySyncNode, schema=CompanyResponse)
@@ -28,10 +28,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
"""施工任务 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "construction_task"
self._node_class = ConstructionTaskSyncNode
self._schema = ConstructionResponse
super().__init__(node_class=ConstructionTaskSyncNode, schema=ConstructionResponse)
self.update_schemas = [ConstructionCreate, ConstructionUpdate, ConstructionPlanUpdate]
async def load(self) -> List[SyncNode]:
@@ -101,7 +98,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
else:
results.append(TaskResult.in_progress(node_id=node.node_id, task_id=push_id))
except Exception as e:
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
results.append(TaskResult.failed(node_id=node.node_id, error=str(e), push_id=push_id))
return results
@@ -135,6 +132,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
node_updated = False
error = None
push_failure_id = None
schema_status_lines: List[str] = []
# 1. 基础信息更新 (ConstructionUpdate)
@@ -186,11 +184,12 @@ class ConstructionTaskApiHandler(BaseApiHandler):
except Exception as e:
error = str(e)
base_update_error = str(e)
push_failure_id = push_id
else:
base_update_error = "show_name changed but task_type != QT"
schema_status_lines.append(
self._build_update_schema_status(
self.build_update_schema_status(
schema_name="ConstructionUpdate",
schema=ConstructionUpdate,
new_data=data,
@@ -236,9 +235,10 @@ class ConstructionTaskApiHandler(BaseApiHandler):
except Exception as e:
error = str(e)
plan_update_error = str(e)
push_failure_id = push_id
schema_status_lines.append(
self._build_update_schema_status(
self.build_update_schema_status(
schema_name="ConstructionPlanUpdate",
schema=ConstructionPlanUpdate,
new_data=data,
@@ -253,7 +253,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
# 汇总结果
if error:
results.append(TaskResult.failed(node_id=node.node_id, error=error))
results.append(TaskResult.failed(node_id=node.node_id, error=error, push_id=push_failure_id))
elif node_updated:
results.append(TaskResult.success(node_id=node.node_id))
else:
@@ -279,7 +279,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
await api_delete_construction_task(self.api_client, biz_id, push_id)
results.append(TaskResult.success(node_id=node.node_id))
except Exception as e:
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
results.append(TaskResult.failed(node_id=node.node_id, error=str(e), push_id=push_id))
return results
@@ -7,4 +7,4 @@ from schemas.construction.construction import ConstructionResponse
class ConstructionTaskJsonlHandler(BaseJsonlHandler):
"""施工任务 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "construction_task", ConstructionTaskSyncNode, ConstructionResponse)
super().__init__(datasource, ConstructionTaskSyncNode, schema=ConstructionResponse)
@@ -19,7 +19,8 @@ class ConstructionTaskSyncStrategy(DefaultSyncStrategy[ConstructionResponse]):
update_direction=UpdateDirection.PUSH,
)
def preprocess_compare_payload(self, payload: dict) -> dict:
def normalize_compare_payload(self, data: dict | None, data_id_map=None) -> dict:
payload = super().normalize_compare_payload(data, data_id_map)
plan_nodes = payload.get("plan_node")
if not isinstance(plan_nodes, list):
return payload
@@ -17,39 +17,68 @@ from schemas.construction.construction_detail import (
ConstructionDetailCreate,
ConstructionDetailUpdate
)
from ..project.api_handler import ProjectApiHandler
class ConstructionTaskDetailApiHandler(BaseApiHandler):
"""施工明细 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "construction_task_detail"
self._node_class = ConstructionTaskDetailSyncNode
self._schema = ConstructionDetailResponse
def __init__(self, **handler_config: Any):
super().__init__(node_class=ConstructionTaskDetailSyncNode, schema=ConstructionDetailResponse, **handler_config)
self.update_schemas = [ConstructionDetailCreate, ConstructionDetailUpdate]
async def load(self) -> List[SyncNode]:
"""加载施工明细列表(依赖 ConstructionTask"""
def _use_aggregate_load(self) -> bool:
return str(self.handler_config.get("load_method", "legacy")).lower() == "aggregate"
async def _load_via_project_aggregate(self) -> List[SyncNode]:
projects = self._collection.filter(node_type="project")
project_ids = [project.data_id for project in projects if project.data_id]
if not project_ids:
return []
all_details: List[SyncNode] = []
aggregate_limit = int(self.handler_config.get("aggregate_limit", 10000))
for project_id in project_ids:
aggregate = await ProjectApiHandler.get_aggregate(
self.api_client,
project_id,
domains=["construction_details"],
limit=aggregate_limit,
)
truncated = set(aggregate.get("meta", {}).get("truncated_domains", []))
if "construction_details" in truncated:
raise RuntimeError(
f"project aggregate truncated domain construction_details for project {project_id}"
)
for detail_data in aggregate.get("data", {}).get("construction_details", []):
all_details.append(self._create_node(dict(detail_data)))
return all_details
async def _load_via_legacy_list(self) -> List[SyncNode]:
tasks = self._collection.filter(node_type="construction_task")
task_ids = [t.data_id for t in tasks if t.data_id]
if not task_ids:
return []
all_details = []
for task_id in task_ids:
try:
details = await api_get_construction_task_detail_list(self.api_client, task_id)
for detail_model in details:
# 已验证的 Pydantic model,转换为 dict
detail_data = detail_model.model_dump()
all_details.append(self._create_node(detail_data))
except Exception as e:
print(f"❌ Failed to load construction details for task {task_id}: {e}")
return all_details
async def load(self) -> List[SyncNode]:
"""加载施工明细列表(依赖 ConstructionTask"""
if self._use_aggregate_load():
return await self._load_via_project_aggregate()
return await self._load_via_legacy_list()
async def create_all(self, nodes: List[SyncNode]) -> List[TaskResult]:
"""批量创建施工明细"""
from pydantic import ValidationError
@@ -98,7 +127,7 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
else:
results.append(TaskResult.in_progress(node_id=node.node_id, task_id=push_id))
except Exception as e:
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
results.append(TaskResult.failed(node_id=node.node_id, error=str(e), push_id=push_id))
return results
@@ -124,9 +153,9 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
origin_data = node.get_origin_data() or {}
# 使用 _get_schema_diff 检查差异并打印日志
# 使用 get_update_schema_diff 作为 update 入口检查差异并打印日志
force_steps_update = self._steps_changed(node)
diff_fields = self._get_schema_diff(
diff_fields = self.get_update_schema_diff(
ConstructionDetailUpdate,
data,
origin_data,
@@ -145,7 +174,7 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
continue
# 需要完整的 schema 数据(包括必填字段)
update_data = self._filter_update_data(
update_data = self.build_update_request_data(
data,
origin_data,
ConstructionDetailUpdate,
@@ -190,7 +219,7 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
# 整组失败
error_msg = str(e)
for node, _ in group_items:
results.append(TaskResult.failed(node_id=node.node_id, error=error_msg))
results.append(TaskResult.failed(node_id=node.node_id, error=error_msg, push_id=push_id))
return results
@@ -208,7 +237,7 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
await api_delete_construction_task_detail(self.api_client, biz_id, push_id)
results.append(TaskResult.success(node_id=node.node_id))
except Exception as e:
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
results.append(TaskResult.failed(node_id=node.node_id, error=str(e), push_id=push_id))
return results
@@ -9,4 +9,4 @@ from schemas.construction.construction_detail import ConstructionDetailResponse
class ConstructionTaskDetailJsonlHandler(BaseJsonlHandler):
"""施工任务明细 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "construction_task_detail", ConstructionTaskDetailSyncNode, ConstructionDetailResponse)
super().__init__(datasource, ConstructionTaskDetailSyncNode, schema=ConstructionDetailResponse)
@@ -101,10 +101,7 @@ class ContractApiHandler(BaseApiHandler):
"""合同 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "contract"
self._node_class = ContractSyncNode
self._schema = ContractResponse
super().__init__(node_class=ContractSyncNode, schema=ContractResponse)
self.update_schemas = [ContractCreate, ContractUpdate]
# ========== Handler 接口实现 ==========
@@ -256,7 +253,7 @@ class ContractApiHandler(BaseApiHandler):
# 过滤出需要更新的字段(排除空值、未变化的字段)
origin_data = node.get_origin_data() or {}
force_steps_update = self._steps_changed(node)
update_data = self._filter_update_data(
update_data = self.build_update_request_data(
new_data=data,
original_data=origin_data,
schema=ContractUpdate,
@@ -9,6 +9,6 @@ from schemas.contract.contract import ContractResponse
class ContractJsonlHandler(BaseJsonlHandler):
"""合同 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "contract", ContractSyncNode, ContractResponse)
super().__init__(datasource, ContractSyncNode, schema=ContractResponse)
@@ -24,11 +24,7 @@ class ContractChangeApiHandler(BaseApiHandler):
"""合同变更 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "contract_change"
self._node_class = ContractChangeSyncNode
# TODO: ContractChangeResponse schema in schemas/project_extensions/contract_change.py is missing 'project_id'
self._schema = ContractChangeResponse
super().__init__(node_class=ContractChangeSyncNode, schema=ContractChangeResponse)
async def load(self) -> List[SyncNode]:
"""加载合同变更数据"""
@@ -93,7 +89,7 @@ class ContractChangeApiHandler(BaseApiHandler):
origin_data = node.get_origin_data() or {}
force_steps_update = self._steps_changed(node)
update_data = self._filter_update_data(
update_data = self.build_update_request_data(
node_data,
origin_data,
PostContractChange,
@@ -7,4 +7,4 @@ from schemas.project_extensions.contract_change import ContractChangeResponse
class ContractChangeJsonlHandler(BaseJsonlHandler):
"""合同变更 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "contract_change", ContractChangeSyncNode, ContractChangeResponse)
super().__init__(datasource, ContractChangeSyncNode, schema=ContractChangeResponse)
@@ -28,10 +28,7 @@ class ContractSettlementApiHandler(BaseApiHandler):
"""合同结算 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "contract_settlement"
self._node_class = ContractSettlementSyncNode
self._schema = ContractSettlementResponse
super().__init__(node_class=ContractSettlementSyncNode, schema=ContractSettlementResponse)
self.update_schemas = [ContractSettlementCreate, ContractSettlementUpdate, ContractSettlementPlanUpdate]
async def load(self) -> List[SyncNode]:
@@ -140,7 +137,7 @@ class ContractSettlementApiHandler(BaseApiHandler):
# 1. 基础信息更新 (ContractSettlementUpdate)
force_steps_update = self._steps_changed(node)
base_diff = self._get_schema_diff(
base_diff = self.get_update_schema_diff(
ContractSettlementUpdate,
data,
origin_data,
@@ -167,7 +164,7 @@ class ContractSettlementApiHandler(BaseApiHandler):
error = f"API error: {e}"
# 2. 计划信息更新 (ContractSettlementPlanUpdate)
plan_diff = self._get_schema_diff(ContractSettlementPlanUpdate, data, origin_data, node.node_id)
plan_diff = self.get_update_schema_diff(ContractSettlementPlanUpdate, data, origin_data, node.node_id)
if not error and plan_diff:
try:
plan_model = ContractSettlementPlanUpdate.model_validate(data)
@@ -7,4 +7,4 @@ from schemas.contract_settlement.contract_settlement import ContractSettlementRe
class ContractSettlementJsonlHandler(BaseJsonlHandler):
"""合同结算 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "contract_settlement", ContractSettlementSyncNode, ContractSettlementResponse)
super().__init__(datasource, ContractSettlementSyncNode, schema=ContractSettlementResponse)
@@ -19,7 +19,8 @@ class ContractSettlementSyncStrategy(DefaultSyncStrategy[ContractSettlementRespo
update_direction=UpdateDirection.PUSH,
)
def preprocess_compare_payload(self, payload: dict) -> dict:
def normalize_compare_payload(self, data: dict | None, data_id_map=None) -> dict:
payload = super().normalize_compare_payload(data, data_id_map)
plan_nodes = payload.get("plan_node")
if not isinstance(plan_nodes, list):
return payload
@@ -17,26 +17,50 @@ from schemas.contract_settlement.contract_settlement_detail import (
ContractSettlementDetailCreate,
ContractSettlementDetailUpdate
)
from ..project.api_handler import ProjectApiHandler
class ContractSettlementDetailApiHandler(BaseApiHandler):
"""合同结算明细 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "contract_settlement_detail"
self._node_class = ContractSettlementDetailSyncNode
self._schema = ContractSettlementDetailResponse
def __init__(self, **handler_config: Any):
super().__init__(node_class=ContractSettlementDetailSyncNode, schema=ContractSettlementDetailResponse, **handler_config)
self.update_schemas = [ContractSettlementDetailCreate, ContractSettlementDetailUpdate]
async def load(self) -> List[SyncNode]:
"""加载合同结算明细列表(依赖 ContractSettlement"""
def _use_aggregate_load(self) -> bool:
return str(self.handler_config.get("load_method", "legacy")).lower() == "aggregate"
async def _load_via_project_aggregate(self) -> List[SyncNode]:
projects = self._collection.filter(node_type="project")
project_ids = [project.data_id for project in projects if project.data_id]
if not project_ids:
return []
all_details: List[SyncNode] = []
aggregate_limit = int(self.handler_config.get("aggregate_limit", 10000))
for project_id in project_ids:
aggregate = await ProjectApiHandler.get_aggregate(
self.api_client,
project_id,
domains=["contract_settlement_details"],
limit=aggregate_limit,
)
truncated = set(aggregate.get("meta", {}).get("truncated_domains", []))
if "contract_settlement_details" in truncated:
raise RuntimeError(
f"project aggregate truncated domain contract_settlement_details for project {project_id}"
)
for detail_data in aggregate.get("data", {}).get("contract_settlement_details", []):
all_details.append(self._create_node(dict(detail_data)))
return all_details
async def _load_via_legacy_list(self) -> List[SyncNode]:
settlements = self._collection.filter(node_type="contract_settlement")
settlement_ids = [s.data_id for s in settlements if s.data_id]
if not settlement_ids:
return []
all_details = []
for settlement_id in settlement_ids:
try:
@@ -46,9 +70,15 @@ class ContractSettlementDetailApiHandler(BaseApiHandler):
all_details.append(self._create_node(detail_data))
except Exception as e:
print(f"❌ Failed to load settlement details for {settlement_id}: {e}")
return all_details
async def load(self) -> List[SyncNode]:
"""加载合同结算明细列表(依赖 ContractSettlement"""
if self._use_aggregate_load():
return await self._load_via_project_aggregate()
return await self._load_via_legacy_list()
async def create_all(self, nodes: List[SyncNode]) -> List[TaskResult]:
"""批量创建合同结算明细"""
from pydantic import ValidationError
@@ -122,9 +152,9 @@ class ContractSettlementDetailApiHandler(BaseApiHandler):
origin_data = node.get_origin_data() or {}
# 使用 _get_schema_diff 检查差异并打印日志
# 使用 get_update_schema_diff 作为 update 入口检查差异并打印日志
force_steps_update = self._steps_changed(node)
diff_fields = self._get_schema_diff(
diff_fields = self.get_update_schema_diff(
ContractSettlementDetailUpdate,
data,
origin_data,
@@ -143,7 +173,7 @@ class ContractSettlementDetailApiHandler(BaseApiHandler):
continue
# 需要完整的 schema 数据(包括必填字段)
update_data = self._filter_update_data(
update_data = self.build_update_request_data(
data,
origin_data,
ContractSettlementDetailUpdate,
@@ -9,4 +9,4 @@ from schemas.contract_settlement.contract_settlement_detail import ContractSettl
class ContractSettlementDetailJsonlHandler(BaseJsonlHandler):
"""合同结算明细 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "contract_settlement_detail", ContractSettlementDetailSyncNode, ContractSettlementDetailResponse)
super().__init__(datasource, ContractSettlementDetailSyncNode, schema=ContractSettlementDetailResponse)
+2 -5
View File
@@ -31,10 +31,7 @@ class LarApiHandler(BaseApiHandler):
"""移民征地 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "lar"
self._node_class = LarSyncNode
self._schema = LarDetailSchema
super().__init__(node_class=LarSyncNode, schema=LarDetailSchema)
async def load(self) -> List[SyncNode]:
"""
@@ -118,7 +115,7 @@ class LarApiHandler(BaseApiHandler):
for schema, api_func in update_configs:
# 获取过滤后的更新数据
update_data = self._filter_update_data(
update_data = self.build_update_request_data(
node_data,
origin_data,
schema,
@@ -7,4 +7,4 @@ from schemas.project_extensions.lar import LarDetailSchema
class LarJsonlHandler(BaseJsonlHandler):
"""移民征地 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "lar", LarSyncNode, LarDetailSchema)
super().__init__(datasource, LarSyncNode, schema=LarDetailSchema)
@@ -28,10 +28,7 @@ class LarChangeApiHandler(BaseApiHandler):
"""移民变更 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "lar_change"
self._node_class = LarChangeSyncNode
self._schema = LarChangeDetailSchema
super().__init__(node_class=LarChangeSyncNode, schema=LarChangeDetailSchema)
async def load(self) -> List[SyncNode]:
"""
@@ -104,7 +101,7 @@ class LarChangeApiHandler(BaseApiHandler):
continue
origin_data = node.get_origin_data() or {}
force_steps_update = self._steps_changed(node)
update_data = self._filter_update_data(
update_data = self.build_update_request_data(
node_data,
origin_data,
LarChangeUpdateSchema,
@@ -9,4 +9,4 @@ class LarChangeJsonlHandler(BaseJsonlHandler):
"""LAR Change (移民变更) JSONL 数据处理器"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "lar_change", LarChangeSyncNode, LarChangeDetailSchema)
super().__init__(datasource, LarChangeSyncNode, schema=LarChangeDetailSchema)
@@ -23,10 +23,7 @@ class MaterialApiHandler(BaseApiHandler):
"""物资 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "material"
self._node_class = MaterialSyncNode
self._schema = MaterialResponse
super().__init__(node_class=MaterialSyncNode, schema=MaterialResponse)
self.update_schemas = [MaterialCreate, MaterialUpdate, MaterialPlanUpdate]
async def load(self) -> List[SyncNode]:
@@ -106,7 +103,7 @@ class MaterialApiHandler(BaseApiHandler):
# 1. 基础信息更新 (MaterialUpdate)
force_steps_update = self._steps_changed(node)
base_diff = self._get_schema_diff(
base_diff = self.get_update_schema_diff(
MaterialUpdate,
data,
origin_data,
@@ -131,7 +128,7 @@ class MaterialApiHandler(BaseApiHandler):
error = f"{ERROR_VALIDATION_FAILED}: {e}"
# 2. 计划信息更新 (MaterialPlanUpdate)
plan_diff = self._get_schema_diff(MaterialPlanUpdate, data, origin_data, node.node_id)
plan_diff = self.get_update_schema_diff(MaterialPlanUpdate, data, origin_data, node.node_id)
if not error and plan_diff:
try:
plan_model = MaterialPlanUpdate.model_validate(data)
@@ -7,4 +7,4 @@ from schemas.material.material import MaterialResponse
class MaterialJsonlHandler(BaseJsonlHandler):
"""物资 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "material", MaterialSyncNode, MaterialResponse)
super().__init__(datasource, MaterialSyncNode, schema=MaterialResponse)
@@ -19,7 +19,8 @@ class MaterialSyncStrategy(DefaultSyncStrategy[MaterialResponse]):
update_direction=UpdateDirection.PUSH,
)
def preprocess_compare_payload(self, payload: dict) -> dict:
def normalize_compare_payload(self, data: dict | None, data_id_map=None) -> dict:
payload = super().normalize_compare_payload(data, data_id_map)
plan_nodes = payload.get("plan_node")
if not isinstance(plan_nodes, list):
return payload
@@ -32,49 +32,75 @@ from schemas.material.material_detail import (
MaterialDetailCreate,
MaterialDetailUpdate
)
from ..project.api_handler import ProjectApiHandler
class MaterialDetailApiHandler(BaseApiHandler):
"""物资明细 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "material_detail"
self._node_class = MaterialDetailSyncNode
self._schema = MaterialDetailResponse
def __init__(self, **handler_config: Any):
super().__init__(node_class=MaterialDetailSyncNode, schema=MaterialDetailResponse, **handler_config)
self.update_schemas = [MaterialDetailCreate, MaterialDetailUpdate]
def _use_aggregate_load(self) -> bool:
return str(self.handler_config.get("load_method", "legacy")).lower() == "aggregate"
async def _load_via_project_aggregate(self) -> List[SyncNode]:
projects = self._collection.filter(node_type="project")
project_ids = [project.data_id for project in projects if project.data_id]
if not project_ids:
return []
all_nodes: List[SyncNode] = []
aggregate_limit = int(self.handler_config.get("aggregate_limit", 10000))
for project_id in project_ids:
aggregate = await ProjectApiHandler.get_aggregate(
self.api_client,
project_id,
domains=["material_details"],
limit=aggregate_limit,
)
truncated = set(aggregate.get("meta", {}).get("truncated_domains", []))
if "material_details" in truncated:
raise RuntimeError(
f"project aggregate truncated domain material_details for project {project_id}"
)
for detail_data in aggregate.get("data", {}).get("material_details", []):
node = self._create_node(dict(detail_data))
all_nodes.append(node)
return all_nodes
async def _load_via_legacy_list(self) -> List[SyncNode]:
materials = self._collection.filter(node_type="material")
material_ids = [m.data_id for m in materials if m.data_id]
if not material_ids:
return []
all_nodes = []
for material_id in material_ids:
try:
details = await api_get_material_detail_list(self.api_client, material_id)
for detail_model in details:
detail_data = detail_model.model_dump()
node = self._create_node(detail_data)
all_nodes.append(node)
except Exception as e:
print(f"❌ Failed to load material details for {material_id}: {e}")
return all_nodes
async def load(self) -> List[SyncNode]:
"""加载物资明细列表(依赖 Material)"""
try:
materials = self._collection.filter(node_type="material")
material_ids = [m.data_id for m in materials if m.data_id]
if not material_ids:
return []
all_nodes = []
for material_id in material_ids:
try:
details = await api_get_material_detail_list(self.api_client, material_id)
for detail_model in details:
# 已验证的 Pydantic model,转换为 dict
detail_data = detail_model.model_dump()
node = self._create_node(detail_data)
all_nodes.append(node)
except Exception as e:
print(f"❌ Failed to load material details for {material_id}: {e}")
return all_nodes
except Exception as e:
print(f"❌ Failed to load material details: {e}")
return []
if self._use_aggregate_load():
return await self._load_via_project_aggregate()
return await self._load_via_legacy_list()
async def create_all(self, nodes: List[SyncNode]) -> List[TaskResult]:
"""批量创建物资明细"""
results = []
for node in nodes:
push_id: str | None = None
try:
data = node.get_data()
if not data:
@@ -118,7 +144,13 @@ class MaterialDetailApiHandler(BaseApiHandler):
else:
results.append(TaskResult.in_progress(node_id=node.node_id, task_id=push_id))
except Exception as e:
results.append(TaskResult.failed(node_id=node.node_id, error=f"Create failed: {str(e)}"))
results.append(
TaskResult.failed(
node_id=node.node_id,
error=f"Create failed: {str(e)}",
push_id=push_id,
)
)
return results
@@ -144,9 +176,9 @@ class MaterialDetailApiHandler(BaseApiHandler):
origin_data = node.get_origin_data() or {}
# 使用 _get_schema_diff 检查差异并打印日志
# 使用 get_update_schema_diff 作为 update 入口检查差异并打印日志
force_steps_update = self._steps_changed(node)
diff_fields = self._get_schema_diff(
diff_fields = self.get_update_schema_diff(
MaterialDetailUpdate,
data,
origin_data,
@@ -165,7 +197,7 @@ class MaterialDetailApiHandler(BaseApiHandler):
continue
# 需要完整的 schema 数据(包括必填字段)
update_data = self._filter_update_data(
update_data = self.build_update_request_data(
data,
origin_data,
MaterialDetailUpdate,
@@ -209,7 +241,7 @@ class MaterialDetailApiHandler(BaseApiHandler):
# 整组失败
error_msg = str(e)
for node, _ in group_items:
results.append(TaskResult.failed(node_id=node.node_id, error=error_msg))
results.append(TaskResult.failed(node_id=node.node_id, error=error_msg, push_id=push_id))
return results
@@ -9,4 +9,4 @@ from schemas.material.material_detail import MaterialDetailResponse
class MaterialDetailJsonlHandler(BaseJsonlHandler):
"""物资明细 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "material_detail", MaterialDetailSyncNode, MaterialDetailResponse)
super().__init__(datasource, MaterialDetailSyncNode, schema=MaterialDetailResponse)
@@ -19,10 +19,7 @@ class PersonnelApiHandler(BaseApiHandler):
"""人员 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "personnel"
self._node_class = PersonnelSyncNode
self._schema = PowerResponse
super().__init__(node_class=PersonnelSyncNode, schema=PowerResponse)
self.update_schemas = [PowerCreate, PowerUpdate, PowerPlanUpdate]
async def load(self) -> List[SyncNode]:
@@ -121,7 +118,7 @@ class PersonnelApiHandler(BaseApiHandler):
# 1. 基础信息更新 (PowerUpdate)
force_steps_update = self._steps_changed(node)
base_diff = self._get_schema_diff(
base_diff = self.get_update_schema_diff(
PowerUpdate,
data,
origin_data,
@@ -7,4 +7,4 @@ from schemas.power.power import PowerResponse
class PersonnelJsonlHandler(BaseJsonlHandler):
"""人员 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "personnel", PersonnelSyncNode, PowerResponse)
super().__init__(datasource, PersonnelSyncNode, schema=PowerResponse)
@@ -13,26 +13,51 @@ from ...common.sync_node import SyncNode
from ...datasource.api.errors import ERROR_NODE_DATA_NONE, ERROR_ID_NOT_FOUND
from .sync_node import PersonnelDetailSyncNode
from schemas.power.power_detail import PowerDetail, PowerDetailCreate, PowerDetailUpdate
from ..project.api_handler import ProjectApiHandler
class PersonnelDetailApiHandler(BaseApiHandler):
"""人员明细 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "personnel_detail"
self._node_class = PersonnelDetailSyncNode
self._schema = PowerDetail
def __init__(self, **handler_config: Any):
super().__init__(node_class=PersonnelDetailSyncNode, schema=PowerDetail, **handler_config)
self.update_schemas = [PowerDetailCreate, PowerDetailUpdate]
async def load(self) -> List[SyncNode]:
"""加载人员明细列表(依赖 Personnel)"""
def _use_aggregate_load(self) -> bool:
return str(self.handler_config.get("load_method", "legacy")).lower() == "aggregate"
async def _load_via_project_aggregate(self) -> List[SyncNode]:
projects = self._collection.filter(node_type="project")
project_ids = [project.data_id for project in projects if project.data_id]
if not project_ids:
return []
all_details: List[SyncNode] = []
aggregate_limit = int(self.handler_config.get("aggregate_limit", 10000))
for project_id in project_ids:
aggregate = await ProjectApiHandler.get_aggregate(
self.api_client,
project_id,
domains=["power_details"],
limit=aggregate_limit,
)
truncated = set(aggregate.get("meta", {}).get("truncated_domains", []))
if "power_details" in truncated:
raise RuntimeError(
f"project aggregate truncated domain power_details for project {project_id}"
)
for detail_data in aggregate.get("data", {}).get("power_details", []):
validated = PowerDetail.model_validate(detail_data)
all_details.append(self._create_node(validated.model_dump()))
return all_details
async def _load_via_legacy_list(self) -> List[SyncNode]:
personnel_list = self._collection.filter(node_type="personnel")
personnel_ids = [p.data_id for p in personnel_list if p.data_id]
if not personnel_ids:
return []
all_details = []
for personnel_id in personnel_ids:
try:
@@ -42,9 +67,15 @@ class PersonnelDetailApiHandler(BaseApiHandler):
all_details.append(self._create_node(validated.model_dump()))
except Exception as e:
print(f"❌ Failed to load personnel details for {personnel_id}: {e}")
return all_details
async def load(self) -> List[SyncNode]:
"""加载人员明细列表(依赖 Personnel)"""
if self._use_aggregate_load():
return await self._load_via_project_aggregate()
return await self._load_via_legacy_list()
async def create_all(self, nodes: List[SyncNode]) -> List[TaskResult]:
"""批量创建人员明细"""
results = []
@@ -111,9 +142,9 @@ class PersonnelDetailApiHandler(BaseApiHandler):
origin_data = node.get_origin_data() or {}
# 使用 _get_schema_diff 检查差异并打印日志
# 使用 get_update_schema_diff 作为 update 入口检查差异并打印日志
force_steps_update = self._steps_changed(node)
diff_fields = self._get_schema_diff(
diff_fields = self.get_update_schema_diff(
PowerDetailUpdate,
data,
origin_data,
@@ -132,7 +163,7 @@ class PersonnelDetailApiHandler(BaseApiHandler):
continue
# 需要完整的 schema 数据(包括必填字段)
update_data = self._filter_update_data(
update_data = self.build_update_request_data(
data,
origin_data,
PowerDetailUpdate,
@@ -9,4 +9,4 @@ from schemas.power.power_detail import PowerDetail
class PersonnelDetailJsonlHandler(BaseJsonlHandler):
"""人员明细 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "personnel_detail", PersonnelDetailSyncNode, PowerDetail)
super().__init__(datasource, PersonnelDetailSyncNode, schema=PowerDetail)
@@ -10,7 +10,7 @@ PUT:
- /preparation - 更新里程碑数据
"""
from typing import List, Dict, Any
from typing import List, Dict
from ...datasource.api.handler import BaseApiHandler
from ...datasource.task_result import TaskResult
from ...common.sync_node import SyncNode
@@ -23,10 +23,7 @@ class PreparationApiHandler(BaseApiHandler):
"""里程碑 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "preparation"
self._node_class = PreparationSyncNode
self._schema = PreparationDetail
super().__init__(node_class=PreparationSyncNode, schema=PreparationDetail)
self.update_schemas = [PostPreparation]
async def load(self) -> List[SyncNode]:
@@ -94,7 +91,7 @@ class PreparationApiHandler(BaseApiHandler):
origin_data = node.get_origin_data() or {}
force_steps_update = self._steps_changed(node)
update_data = self._filter_update_data(
update_data = self.build_update_request_data(
node_data,
origin_data,
PostPreparation,
@@ -141,10 +138,6 @@ class PreparationApiHandler(BaseApiHandler):
for node in nodes
]
def _create_node(self, data: Dict[str, Any]) -> SyncNode:
return self._create_basic_node(data, depend_ids=[])
# ========== 静态 API 函数 ==========
async def api_update_preparation(
@@ -8,4 +8,4 @@ from schemas.project_extensions.preparation import PreparationDetail
class PreparationJsonlHandler(BaseJsonlHandler):
"""里程碑 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "preparation", PreparationSyncNode, PreparationDetail)
super().__init__(datasource, PreparationSyncNode, schema=PreparationDetail)
@@ -20,16 +20,13 @@ class PreparationSyncStrategy(DefaultSyncStrategy[PreparationDetail]):
update_direction=UpdateDirection.PUSH,
)
def _needs_update(self, source_node, target_node, resolved_data=None, data_id_map=None) -> bool:
if resolved_data is None:
raise ValueError(f"[{self.node_type}] resolved_data is required for differential sync.")
target_data = target_node.get_data()
def should_update_pair(self, source_node, target_node, *, source_data, target_data, data_id_map=None) -> bool:
if target_data is None:
return False
# 这里只按 PostPreparation 可写字段判定是否需要 update,避免 response-only 字段差异触发空更新。
update_fields = set(PostPreparation.model_fields.keys())
source_payload = {key: value for key, value in resolved_data.items() if key in update_fields}
source_payload = {key: value for key, value in source_data.items() if key in update_fields}
target_payload = {key: value for key, value in target_data.items() if key in update_fields}
normalized_source = normalized_data_for_compare(source_payload, data_id_map or {})
@@ -23,10 +23,7 @@ class ProductionApiHandler(BaseApiHandler):
"""投产节点 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "production"
self._node_class = ProductionSyncNode
self._schema = ProductionDetail
super().__init__(node_class=ProductionSyncNode, schema=ProductionDetail)
self.update_schemas = [PostProduction]
async def load(self) -> List[SyncNode]:
@@ -99,7 +96,7 @@ class ProductionApiHandler(BaseApiHandler):
# 获取过滤后的更新数据
force_steps_update = self._steps_changed(node)
update_data = self._filter_update_data(
update_data = self.build_update_request_data(
node_data,
origin_data,
PostProduction,
@@ -7,4 +7,4 @@ from schemas.project_extensions.production import ProductionDetail
class ProductionJsonlHandler(BaseJsonlHandler):
"""投产节点 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "production", ProductionSyncNode, ProductionDetail)
super().__init__(datasource, ProductionSyncNode, schema=ProductionDetail)
@@ -20,16 +20,13 @@ class ProductionSyncStrategy(DefaultSyncStrategy[ProductionDetail]):
update_direction=UpdateDirection.PUSH,
)
def _needs_update(self, source_node, target_node, resolved_data=None, data_id_map=None) -> bool:
if resolved_data is None:
raise ValueError(f"[{self.node_type}] resolved_data is required for differential sync.")
target_data = target_node.get_data()
def should_update_pair(self, source_node, target_node, *, source_data, target_data, data_id_map=None) -> bool:
if target_data is None:
return False
# 这里只按 PostProduction 可写字段判定是否需要 update,避免 response-only 字段差异触发空更新。
update_fields = set(PostProduction.model_fields.keys())
source_payload = {key: value for key, value in resolved_data.items() if key in update_fields}
source_payload = {key: value for key, value in source_data.items() if key in update_fields}
target_payload = {key: value for key, value in target_data.items() if key in update_fields}
normalized_source = normalized_data_for_compare(source_payload, data_id_map or {})
@@ -34,6 +34,7 @@ from schemas.project.project_base import (
ProjectSettlementInvestment
)
from schemas.project.project import ProjectInfoResponse
from schemas.project.project_aggregate import ProjectAggregateResponse
_PROJECT_EXTENSION_FIELD_NAMES = {
@@ -72,10 +73,7 @@ class ProjectApiHandler(BaseApiHandler):
_all_info_cache: Dict[str, Dict[str, Any]] = {}
def __init__(self, **handler_config: Any):
super().__init__(**handler_config)
self._node_type = "project"
self._node_class = ProjectSyncNode
self._schema = ProjectResponseBase
super().__init__(node_class=ProjectSyncNode, schema=ProjectResponseBase, **handler_config)
@classmethod
async def get_all_info(cls, client, project_id: str) -> Dict[str, Any]:
@@ -94,6 +92,23 @@ class ProjectApiHandler(BaseApiHandler):
all_info = await api_get_project_all_info(client, project_id)
cls._all_info_cache[project_id] = all_info
return all_info
@classmethod
async def get_aggregate(
cls,
client,
project_id: str,
*,
domains: List[str],
limit: int = 10000,
) -> Dict[str, Any]:
normalized_domains = tuple(dict.fromkeys(domains))
return await api_get_project_aggregate(
client,
project_id,
domains=list(normalized_domains),
limit=limit,
)
@classmethod
def clear_cache(cls):
@@ -107,6 +122,8 @@ class ProjectApiHandler(BaseApiHandler):
project_id: 可选指定项目ID列表则只加载这些项目优先级高于构造函数的filter_project_id
"""
try:
self.ensure_api_client()
self.ensure_collection()
# 清空旧缓存
self.clear_cache()
@@ -234,7 +251,7 @@ class ProjectApiHandler(BaseApiHandler):
# 遍历所有更新 schema,检查差异并更新
for schema, api_func in update_configs:
update_data = self._get_schema_diff(
update_data = self.get_update_schema_diff(
schema,
data,
original_data,
@@ -244,9 +261,9 @@ class ProjectApiHandler(BaseApiHandler):
if not update_data:
continue
# 执行更新(差异已由 _get_schema_diff 打印)
# 执行更新(差异已由 get_update_schema_diff 打印)
try:
full_update_data = self._filter_update_data(
full_update_data = self.build_update_request_data(
data,
original_data,
schema,
@@ -405,6 +422,27 @@ async def api_get_project_all_info(client, project_id: str) -> Dict[str, Any]:
return validated.model_dump()
async def api_get_project_aggregate(
client,
project_id: str,
*,
domains: List[str],
limit: int = 10000,
) -> Dict[str, Any]:
"""POST /project/aggregate - 获取项目聚合数据(校验为 ProjectAggregateResponse"""
response = await client.request(
method="POST",
endpoint="/project/aggregate",
data={"project_id": project_id, "domains": domains, "limit": limit},
)
code = response.get("code")
if code != 200:
raise Exception(f"API error {code}: {response.get('message', 'Unknown error')}")
validated = ProjectAggregateResponse.model_validate(response.get("data", {}))
return validated.model_dump()
async def api_update_project_base_info(client, data: Dict[str, Any], push_id: str, biz_id: str, *, steps: List[Dict[str, Any]] | None = None) -> None:
"""PUT /project - 更新项目基本信息"""
ProjectBaseInfoUpdate.model_validate(data)
@@ -9,4 +9,4 @@ from schemas.project.project_base import ProjectResponseBase
class ProjectJsonlHandler(BaseJsonlHandler):
"""项目 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "project", ProjectSyncNode, ProjectResponseBase)
super().__init__(datasource, ProjectSyncNode, schema=ProjectResponseBase)
@@ -21,6 +21,8 @@ class ProjectSyncSchema(ProjectResponseBase):
completed_investment: float | None = Field(None, description="已完成投资")
total_contract_quantity: float | None = Field(None, description="合同总量")
completed_settlement_quantity: float | None = Field(None, description="已结算量")
ic_plan_first_production_date: str | None = Field(None, description="集团计划首批投产日期")
ic_plan_full_production_date: str | None = Field(None, description="集团计划全容量投产日期")
class ProjectSyncNode(SyncNode[ProjectSyncSchema]):
@@ -1,8 +1,16 @@
import copy
from pydantic import BaseModel
from ...sync_system.strategy import DefaultSyncStrategy
from ...sync_system.config import StrategyConfig, OrphanAction, UpdateDirection
from schemas.project.project_base import ProjectResponseBase
class ProjectDomainOption(BaseModel):
pull_group_plan_production_date: bool = False
class ProjectSyncStrategy(DefaultSyncStrategy[ProjectResponseBase]):
"""
Project 同步策略
@@ -17,6 +25,12 @@ class ProjectSyncStrategy(DefaultSyncStrategy[ProjectResponseBase]):
# 类变量:schema 固定为 ProjectResponseBase
schema = ProjectResponseBase
domain_option_model = ProjectDomainOption
_PULL_ONLY_FIELDS = [
"ic_plan_first_production_date",
"ic_plan_full_production_date",
]
# 类变量:默认配置
default_config = StrategyConfig(
@@ -26,4 +40,89 @@ class ProjectSyncStrategy(DefaultSyncStrategy[ProjectResponseBase]):
local_orphan_action=OrphanAction.CREATE_REMOTE,
remote_orphan_action=OrphanAction.NONE,
update_direction=UpdateDirection.PUSH,
)
)
@property
def domain_option(self) -> ProjectDomainOption:
return ProjectDomainOption.model_validate(self.config.domain_option)
def get_node_update_payload(self, node):
payload = dict(node.get_data() or {})
context = getattr(node, "context", None)
if not isinstance(context, dict):
return payload
all_info = context.get("all_info")
if not isinstance(all_info, dict):
return payload
for section_name, section_data in all_info.items():
if not section_name.endswith("_extension") or not isinstance(section_data, dict):
continue
for field_name in self._PULL_ONLY_FIELDS:
if payload.get(field_name) not in (None, ""):
continue
if field_name in section_data:
payload[field_name] = section_data.get(field_name)
return payload
@staticmethod
def _snapshot_node(node):
# project 是一个很特殊的双向更新场景:
# 1. 先做一次 PULL,把远端 ic_plan_* 回填到本地;
# 2. 再做一次常规 PUSH,把本地其余字段推到远端。
#
# 但 prepare_update_for_direction -> e30_update_prepare 会校验
# source_node 当前仍然处于可进入更新准备的稳定态(S01)。
# 如果直接复用同一个真实节点作为两次 update 的 source,第一次
# 准备完成后真实节点的 action/status 已经变化,第二次就不再满足
# “source 仍在 S01”的前置条件,导致第二次更新被状态机拒绝。
#
# 这里复制一个只用于“提供源数据/源状态视图”的快照,让两次 update
# 都从各自的原始稳定态出发做判断;真正被推进到 S07 / 写入 payload
# 的仍然是目标侧真实节点,而不是这个 snapshot。
snapshot = copy.deepcopy(node)
if hasattr(snapshot, "sync_log"):
snapshot.sync_log = None
return snapshot
async def update_pair(self, local_node, remote_node, data_id_map=None):
updated_by_id = {}
# 注意:这两个 snapshot 只会在各自方向上充当 source_node。
# - PULL: source=remote_snapshot, target=local_node
# - PUSH: source=local_snapshot, target=remote_node
#
# 因此真实 project 节点的状态变更仍只发生在 target 节点上,
# 不会因为传入 snapshot 而把本体节点替换掉,也不会把本体节点的
# action/status/error/sync_log 写到 snapshot 上后丢失掉。
local_snapshot = self._snapshot_node(local_node)
remote_snapshot = self._snapshot_node(remote_node)
if self.domain_option.pull_group_plan_production_date:
pull_update = await self.prepare_update_for_direction(
local_node,
remote_snapshot,
UpdateDirection.PULL,
data_id_map,
include_fields=self._PULL_ONLY_FIELDS,
)
if pull_update is not None:
updated_by_id[pull_update.node_id] = pull_update
generic_exclude_fields = self._PULL_ONLY_FIELDS
else:
generic_exclude_fields = None
if self.config.update_direction != UpdateDirection.NONE:
generic_update = await self.prepare_update_for_direction(
local_snapshot,
remote_node,
self.config.update_direction,
data_id_map,
exclude_fields=generic_exclude_fields,
)
if generic_update is not None:
updated_by_id[generic_update.node_id] = generic_update
return list(updated_by_id.values())
@@ -40,8 +40,6 @@ from schemas.project.project_base import (
_PRODUCTION_PROJECT_FIELDS = [
"plan_production_date",
"plan_full_production_date",
"ic_plan_first_production_date",
"ic_plan_full_production_date",
"actual_production_date",
"actual_full_production_date",
]
@@ -51,10 +49,7 @@ class ProjectDetailApiHandler(BaseApiHandler):
"""项目容量信息 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "project_detail_data"
self._node_class = ProjectDetailSyncNode
self._schema = ProjectDetailProject
super().__init__(node_class=ProjectDetailSyncNode, schema=ProjectDetailProject)
self._pending_push_context: Dict[str, Dict[str, str]] = {}
async def load(self) -> List[SyncNode]:
@@ -185,6 +180,9 @@ class ProjectDetailApiHandler(BaseApiHandler):
except ValidationError as e:
results.append(TaskResult.failed(node_id=node.node_id, error=f"{ERROR_VALIDATION_FAILED}: {e}"))
continue
except Exception as e:
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
continue
push_id = self._generate_push_id()
biz_id = project_id
@@ -207,7 +205,16 @@ class ProjectDetailApiHandler(BaseApiHandler):
self._mark_approval_snapshot_synced(node)
except Exception as e:
self._pending_push_context.pop(push_id, None)
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
results.append(
TaskResult.failed(
node_id=node.node_id,
error=str(e),
push_id=push_id,
biz_id=biz_id,
key=str(key),
project_id=str(project_id),
)
)
return results
@@ -299,46 +306,13 @@ class ProjectDetailApiHandler(BaseApiHandler):
def _create_node(self, data: Dict[str, Any]) -> SyncNode:
project_id_raw = data.get("project_id")
key_raw = self._normalize_key(data.get("key"))
project_id = str(project_id_raw) if project_id_raw else ""
key = str(key_raw) if key_raw else ""
loaded_data_id = str(data.get("id") or data.get("_id") or "")
existing_by_pair = self._find_existing_by_project_key(project_id=project_id, key=key)
if existing_by_pair is not None:
if loaded_data_id and existing_by_pair.data_id != loaded_data_id:
with existing_by_pair.allow_core_state_write("project_detail_pair_rebind"):
existing_by_pair.data_id = loaded_data_id
existing_by_pair.set_data(data)
existing_by_pair.set_origin_data(data)
if project_id:
existing_by_pair.context["project_id"] = project_id
return existing_by_pair
node = self._create_basic_node(data, depend_ids=[])
if project_id:
node.context["project_id"] = project_id
return node
def _find_existing_by_project_key(self, *, project_id: str, key: str) -> Optional[SyncNode]:
if not project_id or not key or self._collection is None:
return None
candidates = self._collection.filter(
node_type=self.node_type,
node_filter=lambda n: (
((n.get_data() or {}).get("project_id") == project_id)
and (self._normalize_key((n.get_data() or {}).get("key")) == key)
),
)
if not candidates:
return None
if len(candidates) > 1:
raise RuntimeError(
f"Duplicate project_detail_data detected: project_id={project_id}, key={key}, count={len(candidates)}"
)
return candidates[0]
async def _get_project_all_info(self, project_id: str, *, force_refresh: bool = False) -> Dict[str, Any]:
from ..project.api_handler import ProjectApiHandler
@@ -7,4 +7,4 @@ from schemas.project_extensions.project_detail import ProjectDetailProject
class ProjectDetailJsonlHandler(BaseJsonlHandler):
"""项目容量信息 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "project_detail_data", ProjectDetailSyncNode, ProjectDetailProject)
super().__init__(datasource, ProjectDetailSyncNode, schema=ProjectDetailProject)
@@ -1,6 +1,25 @@
from typing import Any
from pydantic import BaseModel, Field
from ...common.sync_node import SyncNode
from ...sync_system.strategy import DefaultSyncStrategy
from ...sync_system.config import StrategyConfig, OrphanAction, UpdateDirection
from schemas.project_extensions.project_detail import ProjectDetailProject
from ...sync_system.strategy_ops.bind_ops import (
apply_auto_bind_updates,
collect_auto_bind_ready_nodes,
collect_bind_entry_nodes,
plan_auto_bind_updates,
phase_core_state,
phase_dependency_check,
refresh_bind_data_id,
)
from ...sync_system.strategy_ops import BoundNodePair
from schemas.project_extensions.project_detail import ProjectDetailKey, ProjectDetailProject
class ProjectDetailDomainOption(BaseModel):
pull_group_production_plans: bool = Field(True, description="固定将集团保供/爬坡/挑战产能计划按 pull 处理")
class ProjectDetailSyncStrategy(DefaultSyncStrategy[ProjectDetailProject]):
@@ -13,6 +32,18 @@ class ProjectDetailSyncStrategy(DefaultSyncStrategy[ProjectDetailProject]):
"""
schema = ProjectDetailProject
domain_option_model = ProjectDetailDomainOption
_GROUP_PRODUCTION_PLAN_KEYS = {
"ensure_production",
"climb_production",
"challenge_production",
}
_PRODUCTION_PROJECT_FIELDS = (
"plan_production_date",
"plan_full_production_date",
"actual_production_date",
"actual_full_production_date",
)
default_config = StrategyConfig(
auto_bind=True,
@@ -21,10 +52,232 @@ class ProjectDetailSyncStrategy(DefaultSyncStrategy[ProjectDetailProject]):
local_orphan_action=OrphanAction.NONE,
remote_orphan_action=OrphanAction.NONE,
update_direction=UpdateDirection.PUSH,
field_direction_key="key",
field_direction_overrides={
"ensure_production": UpdateDirection.PULL,
"climb_production": UpdateDirection.PULL,
"challenge_production": UpdateDirection.PULL,
},
)
async def bind(self) -> None:
runtime = self.ensure_runtime()
local_nodes, remote_nodes = collect_bind_entry_nodes(
node_type=self.node_type,
local_collection=self.local_collection,
remote_collection=self.remote_collection,
)
await phase_core_state(
node_type=self.node_type,
runtime=runtime,
binding_manager=self.binding_manager,
local_nodes=local_nodes,
remote_nodes=remote_nodes,
)
await phase_dependency_check(
node_type=self.node_type,
runtime=runtime,
depend_fields=dict(self.config.depend_fields),
local_nodes=local_nodes,
remote_nodes=remote_nodes,
local_collection=self.local_collection,
remote_collection=self.remote_collection,
)
local_nodes, remote_nodes = collect_auto_bind_ready_nodes(
node_type=self.node_type,
local_collection=self.local_collection,
remote_collection=self.remote_collection,
)
local_create_enabled = self.config.local_orphan_action == OrphanAction.CREATE_REMOTE
remote_create_enabled = self.config.remote_orphan_action == OrphanAction.CREATE_LOCAL
local_orphan_create_enabled_by_node = {
node.node_id: local_create_enabled for node in local_nodes
}
remote_orphan_create_enabled_by_node = {
node.node_id: remote_create_enabled for node in remote_nodes
}
if self.domain_option.pull_group_production_plans:
for node in local_nodes:
if self._extract_node_key(node) in self._GROUP_PRODUCTION_PLAN_KEYS:
local_orphan_create_enabled_by_node[node.node_id] = False
for node in remote_nodes:
if self._extract_node_key(node) in self._GROUP_PRODUCTION_PLAN_KEYS:
remote_orphan_create_enabled_by_node[node.node_id] = True
pending_updates = await plan_auto_bind_updates(
node_type=self.node_type,
config=self.config,
local_collection=self.local_collection,
remote_collection=self.remote_collection,
binding_manager=self.binding_manager,
local_nodes=local_nodes,
remote_nodes=remote_nodes,
id_field_hints=dict(self.config.depend_fields),
local_orphan_create_enabled_by_node=local_orphan_create_enabled_by_node,
remote_orphan_create_enabled_by_node=remote_orphan_create_enabled_by_node,
)
await apply_auto_bind_updates(
node_type=self.node_type,
runtime=runtime,
binding_manager=self.binding_manager,
pending_auto_bind_updates=pending_updates,
)
await refresh_bind_data_id(
node_type=self.node_type,
local_collection=self.local_collection,
remote_collection=self.remote_collection,
binding_manager=self.binding_manager,
)
async def process_update_pairs(
self,
pairs: list[BoundNodePair],
data_id_map: dict[str, str] | None = None,
) -> dict[str, SyncNode]:
pull_keys = self._GROUP_PRODUCTION_PLAN_KEYS if self.domain_option.pull_group_production_plans else set()
pull_pairs = []
generic_pairs = []
for pair in pairs:
pair_key = self._extract_pair_key(pair.local_node, pair.remote_node)
if pair_key in pull_keys:
pull_pairs.append(pair)
else:
generic_pairs.append(pair)
updated_by_id: dict[str, SyncNode] = {}
for pair in pull_pairs:
updated_node = await self.prepare_update_for_direction(
pair.local_node,
pair.remote_node,
UpdateDirection.PULL,
data_id_map,
)
if updated_node is not None:
updated_by_id[updated_node.node_id] = updated_node
for pair in generic_pairs:
for updated_node in await self.update_pair(pair.local_node, pair.remote_node, data_id_map):
updated_by_id[updated_node.node_id] = updated_node
return updated_by_id
@property
def domain_option(self) -> ProjectDetailDomainOption:
return ProjectDetailDomainOption.model_validate(self.config.domain_option)
async def create_for_direction(self, *, source_is_local: bool) -> list[SyncNode]:
created_nodes = await super().create_for_direction(source_is_local=source_is_local)
await self._enrich_created_production_nodes(created_nodes, source_is_local=source_is_local)
return created_nodes
def normalize_compare_payload(self, data: dict | None, data_id_map=None) -> dict:
normalized = super().normalize_compare_payload(data, data_id_map)
key = normalized.get("key")
if isinstance(key, ProjectDetailKey):
key = key.value
normalized["key"] = key
if isinstance(key, str):
normalized["value"] = self._normalize_detail_value_for_compare(key, normalized.get("value"))
return normalized
@staticmethod
def _normalize_detail_value_for_compare(key: str, value: object) -> list[dict[str, object]]:
if not isinstance(value, list):
return []
items: list[dict[str, object]] = []
for item in value:
if not isinstance(item, dict):
continue
normalized_item: dict[str, object] = {}
if "capacity" in item:
normalized_item["capacity"] = item.get("capacity")
if key in {"plan_construction", "actual_construction", "production"}:
date_value = item.get("date")
if date_value in (None, ""):
continue
normalized_item["date"] = date_value
elif key in {"ensure_production", "climb_production", "challenge_production"}:
year_value = item.get("year")
if year_value in (None, ""):
date_value = item.get("date")
if isinstance(date_value, str) and len(date_value) >= 4 and date_value[:4].isdigit():
year_value = int(date_value[:4])
if year_value in (None, ""):
continue
normalized_item["year"] = year_value
else:
if item.get("year") not in (None, ""):
normalized_item["year"] = item.get("year")
if item.get("date") not in (None, ""):
normalized_item["date"] = item.get("date")
items.append(normalized_item)
return items
@staticmethod
def _extract_node_key(node: SyncNode) -> str | None:
data = node.get_data() or {}
key = data.get("key")
if isinstance(key, ProjectDetailKey):
return key.value
if isinstance(key, str):
return key
return None
@staticmethod
def _extract_pair_key(local_node, remote_node) -> str | None:
local_data = local_node.get_data() or {}
remote_data = remote_node.get_data() or {}
key = local_data.get("key", remote_data.get("key"))
if isinstance(key, ProjectDetailKey):
return key.value
if isinstance(key, str):
return key
return None
async def _enrich_created_production_nodes(self, created_nodes: list[SyncNode], *, source_is_local: bool) -> None:
if not created_nodes:
return
source_collection = self.local_collection if source_is_local else self.remote_collection
for created_node in created_nodes:
created_data = created_node.get_data() or {}
if self._extract_node_key(created_node) != "production":
continue
if all(created_data.get(field) not in (None, "") for field in self._PRODUCTION_PROJECT_FIELDS):
continue
if source_is_local:
source_node_id = await self.binding_manager.get_local_id(self.node_type, created_node.node_id)
else:
source_node_id = await self.binding_manager.get_remote_id(self.node_type, created_node.node_id)
if not source_node_id:
continue
source_node = source_collection.get(source_node_id)
if source_node is None:
continue
source_data = source_node.get_data() or {}
project_id = source_data.get("project_id")
if not project_id:
continue
project_node = source_collection.get_by_data_id("project", str(project_id))
if project_node is None:
continue
project_data = project_node.get_data() or {}
updated_data: dict[str, Any] = dict(created_data)
changed = False
for field in self._PRODUCTION_PROJECT_FIELDS:
if updated_data.get(field) in (None, "") and project_data.get(field) not in (None, ""):
updated_data[field] = project_data.get(field)
changed = True
if changed:
created_node.set_data(updated_data)
@@ -6,7 +6,7 @@ Supplier API Handler (只读)
- 不支持创建/更新/删除
"""
from typing import List, Dict, Any
from typing import List, Dict
from ...datasource.api.handler import BaseApiHandler
from ...datasource.task_result import TaskResult
from ...common.sync_node import SyncNode
@@ -18,10 +18,7 @@ class SupplierApiHandler(BaseApiHandler):
"""供应商 API Handler (只读)"""
def __init__(self, load_mode: str = "api", max_total: int | None = None):
super().__init__()
self._node_type = "supplier"
self._node_class = SupplierSyncNode
self._schema = SupplierDetailResponse
super().__init__(node_class=SupplierSyncNode, schema=SupplierDetailResponse)
self._load_mode = load_mode
self._max_total = max_total
@@ -99,10 +96,6 @@ class SupplierApiHandler(BaseApiHandler):
"""轮询异步任务状态(占位)"""
return await super().poll_tasks(task_ids)
def _create_node(self, data: Dict[str, Any]) -> SyncNode:
return self._create_basic_node(data, depend_ids=[])
# ========== 静态 API 函数 ==========
async def api_get_supplier_list(client, page: int = 1, page_size: int = 100) -> tuple[List[SupplierDetailResponse], int]:
@@ -9,4 +9,4 @@ from schemas.supplier.supplier import SupplierDetailResponse
class SupplierJsonlHandler(BaseJsonlHandler):
"""供应商 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "supplier", SupplierSyncNode, SupplierDetailResponse)
super().__init__(datasource, SupplierSyncNode, schema=SupplierDetailResponse)
@@ -24,19 +24,16 @@ class SupplierSyncStrategy(DefaultSyncStrategy[SupplierDetailResponse]):
update_direction=UpdateDirection.PULL,
)
def _needs_update(
def should_update_pair(
self,
source_node,
target_node,
resolved_data=None,
*,
source_data,
target_data,
data_id_map=None,
) -> bool:
"""Supplier 更新忽略 id 字段差异。"""
if resolved_data is None:
raise ValueError(f"[{self.node_type}] resolved_data is required for differential sync.")
source_data = resolved_data
target_data = target_node.get_data()
if source_data is None or target_data is None:
return False
@@ -23,10 +23,7 @@ class UnitsApiHandler(BaseApiHandler):
"""机组 API Handler"""
def __init__(self):
super().__init__()
self._node_type = "units"
self._node_class = UnitsSyncNode
self._schema = GeneratorUnitProject
super().__init__(node_class=UnitsSyncNode, schema=GeneratorUnitProject)
async def load(self) -> List[SyncNode]:
"""加载机组数据(从 project all_info 中提取)"""
@@ -94,7 +91,7 @@ class UnitsApiHandler(BaseApiHandler):
origin_data = node.get_origin_data() or {}
force_steps_update = self._steps_changed(node)
update_data = self._filter_update_data(
update_data = self.build_update_request_data(
node_data,
origin_data,
PostGeneratorUnits,
@@ -11,7 +11,7 @@ from schemas.project_extensions.generator_unit import GeneratorUnitProject
class UnitsJsonlHandler(BaseJsonlHandler):
"""机组 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "units", UnitsSyncNode, GeneratorUnitProject)
super().__init__(datasource, UnitsSyncNode, schema=GeneratorUnitProject)
async def load(self) -> List['SyncNode']:
"""加载机组数据(支持 generator_unit_N.jsonl 与 units_N.jsonl"""
@@ -10,7 +10,7 @@ GET:
- /user/list - 获取用户列表
"""
from typing import List, Dict, Any
from typing import List, Dict
from ...datasource.api.handler import BaseApiHandler
from ...datasource.task_result import TaskResult
from ...common.sync_node import SyncNode
@@ -22,10 +22,7 @@ class UserApiHandler(BaseApiHandler):
"""用户 API Handler (只读)"""
def __init__(self):
super().__init__()
self._node_type = "user"
self._node_class = UserSyncNode
self._schema = UserListDetailItem
super().__init__(node_class=UserSyncNode, schema=UserListDetailItem)
async def load(self) -> List[SyncNode]:
"""加载用户列表(支持分页)"""
@@ -87,10 +84,6 @@ class UserApiHandler(BaseApiHandler):
"""轮询异步任务状态(占位)"""
return await super().poll_tasks(task_ids)
def _create_node(self, data: Dict[str, Any]) -> SyncNode:
return self._create_basic_node(data, depend_ids=[])
# ========== 静态 API 函数 ==========
async def api_get_user_list(client, page: int = 1, page_size: int = 100) -> tuple[List[UserListDetailItem], int]:
@@ -7,4 +7,4 @@ from schemas.user.user import UserListDetailItem
class UserJsonlHandler(BaseJsonlHandler):
"""用户 JSONL Handler"""
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "user", UserSyncNode, UserListDetailItem)
super().__init__(datasource, UserSyncNode, schema=UserListDetailItem)
+16 -1
View File
@@ -276,8 +276,14 @@ def e01_bootstrap(
*,
node: "SyncNode",
is_create_zombie: bool,
bootstrap_source_present: bool = True,
has_binding_record: bool = False,
) -> Decision:
context = {"is_create_zombie": is_create_zombie}
context = {
"is_create_zombie": is_create_zombie,
"bootstrap_source_present": bootstrap_source_present,
"has_binding_record": has_binding_record,
}
decision = _dispatch_required(
runtime,
node=node,
@@ -289,6 +295,15 @@ def e01_bootstrap(
_apply_decision(runtime, node=node, decision=decision, fields={"binding_status", "action", "status"})
node.error = None
return decision
if decision.to_state == "S17":
_apply_decision(runtime, node=node, decision=decision, fields={"binding_status", "action", "status"})
reason_text = decision_note(decision) or "<no-reason>"
node.context["bootstrap_blocked"] = True
node.context["bootstrap_block_reason"] = "source_missing_after_persistence_refresh"
node.context["bootstrap_block_errors"] = [reason_text]
node.context["bootstrap_source_present"] = False
node.error = f"BOOTSTRAP_SOURCE_MISSING: {reason_text}"
return decision
if decision.to_state == "S15":
reason_text = decision_note(decision) or "<no-reason>"
node.append_log(
+146
View File
@@ -0,0 +1,146 @@
from __future__ import annotations
import logging
from pathlib import Path
from typing import Optional
APP_LOGGER_NAME = "sync_state_machine"
_SCOPE_DISPLAY_NAMES: dict[str, str] = {}
def set_scope_display_name(scope: str, display_name: str) -> None:
normalized_scope = str(scope).strip()
normalized_name = str(display_name).strip()
if not normalized_scope:
return
if not normalized_name:
_SCOPE_DISPLAY_NAMES.pop(normalized_scope, None)
return
_SCOPE_DISPLAY_NAMES[normalized_scope] = normalized_name
def get_scope_display_name(scope: Optional[str]) -> str:
normalized_scope = str(scope or "").strip()
if not normalized_scope:
return ""
return _SCOPE_DISPLAY_NAMES.get(normalized_scope, normalized_scope)
def clear_scope_display_names() -> None:
_SCOPE_DISPLAY_NAMES.clear()
class ScopeLoggerAdapter(logging.LoggerAdapter):
def process(self, msg, kwargs):
scope = get_scope_display_name(self.extra.get("scope"))
if scope and not str(msg).startswith("["):
msg = f"[{scope}] {msg}"
return msg, kwargs
class _NodeLogFilter(logging.Filter):
def filter(self, record: logging.LogRecord) -> bool:
message = record.getMessage()
if "node=" in message:
return False
if "节点 " in message and "状态" in message:
return False
return True
def get_app_logger() -> logging.Logger:
return logging.getLogger(APP_LOGGER_NAME)
def get_logger(name: Optional[str] = None) -> logging.Logger:
if not name or name == "__main__":
return get_app_logger()
if name == APP_LOGGER_NAME or name.startswith(f"{APP_LOGGER_NAME}."):
return logging.getLogger(name)
return logging.getLogger(f"{APP_LOGGER_NAME}.{name}")
def get_scope_logger(scope: str, name: Optional[str] = None) -> logging.LoggerAdapter:
return ScopeLoggerAdapter(get_logger(name), {"scope": scope})
def clear_app_logger_handlers(logger: Optional[logging.Logger] = None) -> None:
target = logger or get_app_logger()
handlers = list(target.handlers)
for handler in handlers:
target.removeHandler(handler)
try:
handler.close()
except Exception:
pass
target.propagate = True
def init_app_logger(
*,
log_file_path: Optional[str] = None,
level: int = logging.INFO,
mirror_console: bool = True,
replace_handlers: bool = True,
suppress_node_logs: bool = True,
) -> logging.Logger:
logger = get_app_logger()
logger.setLevel(level)
logger.propagate = False
if logger.handlers and not replace_handlers:
return logger
if replace_handlers:
clear_app_logger_handlers(logger)
formatter = logging.Formatter("%(message)s")
node_filter = _NodeLogFilter() if suppress_node_logs else None
if mirror_console:
console_handler = logging.StreamHandler()
console_handler.setLevel(level)
console_handler.setFormatter(formatter)
if node_filter is not None:
console_handler.addFilter(node_filter)
logger.addHandler(console_handler)
if log_file_path:
path = Path(log_file_path)
path.parent.mkdir(parents=True, exist_ok=True)
file_handler = logging.FileHandler(path, mode="w", encoding="utf-8")
file_handler.setLevel(level)
file_handler.setFormatter(formatter)
if node_filter is not None:
file_handler.addFilter(node_filter)
logger.addHandler(file_handler)
return logger
def configure_app_logging(config, *, replace_handlers: bool = False, mirror_console: bool = True) -> logging.Logger:
if not getattr(config, "initialize", True):
return get_app_logger()
if not getattr(config, "file_path", None):
from .config.logging_config import resolve_log_file_path
config.file_path = resolve_log_file_path(config)
return init_app_logger(
log_file_path=config.file_path,
level=config.level,
mirror_console=mirror_console,
replace_handlers=replace_handlers,
suppress_node_logs=config.suppress_node_logs,
)
def ensure_app_logging(config, *, mirror_console: bool = True) -> logging.Logger:
logger = get_app_logger()
if not getattr(config, "initialize", True):
return logger
if logger.handlers:
return logger
return configure_app_logging(config, replace_handlers=False, mirror_console=mirror_console)
+260 -139
View File
@@ -17,11 +17,12 @@ import inspect
import json
import logging
from pathlib import Path
from typing import Any, Dict, List, Optional
from typing import Any, Dict, List, Literal, Optional
from ..common.binding import BindingManager
from ..common.collection import DataCollection
from ..common.persistence import PersistenceBackend, create_persistence_backend
from ..common.persistence_service import PersistenceService
from ..common.registry import DomainRegistry
from ..datasource import ensure_builtin_datasource_types_registered
from ..datasource.type_registry import get_datasource_factory
@@ -36,21 +37,23 @@ from ..config import (
build_config,
StrategyRuntimeConfig,
build_config_from_file,
build_multi_project_config_from_file,
apply_config_overrides,
OrphanAction,
UpdateDirection,
resolve_log_file_path,
config_snapshot,
is_multi_project_payload,
load_overrides_from_file,
)
from ..config.strategy_config import resolve_domain_option_config
from ..sync_system.strategy import BaseSyncStrategy
from .run_logger import init_pipeline_logger
from ..logging import configure_app_logging, ensure_app_logging, get_logger
from .full_sync_pipeline import FullSyncPipeline
from .project_scope_runtime import ProjectScopeRuntime
logger = logging.getLogger(__name__)
logger = get_logger(__name__)
PipelineType = Literal["auto", "full", "multi-project"]
def _project_root() -> Path:
@@ -63,29 +66,6 @@ def _prepare_remote_jsonl_dir(remote_dir: Path, *, project_root: Optional[Path]
return prepare_remote_jsonl_dir(remote_dir, project_root=(project_root or _project_root()))
def _ensure_pipeline_logger(
*,
initialize_logger: bool,
logger_file_path: Optional[str],
logger_level: int,
suppress_node_logs: bool,
) -> None:
if not initialize_logger:
return
app_logger = logging.getLogger("sync_state_machine")
if app_logger.handlers:
return
init_pipeline_logger(
log_file_path=logger_file_path,
level=logger_level,
mirror_console=True,
replace_handlers=False,
suppress_node_logs=suppress_node_logs,
)
def _build_handler_config(
node_type: str,
ds_config: DataSourceConfig,
@@ -139,6 +119,11 @@ async def _create_datasource(
datasource = datasource_factory(ds_config, endpoint_name, **factory_kwargs)
if inspect.isawaitable(datasource):
datasource = await datasource
if hasattr(datasource, "set_runtime_logging_context"):
datasource.set_runtime_logging_context(
endpoint_name=endpoint_name,
datasource_type=ds_config.type,
)
return datasource
@@ -148,6 +133,72 @@ async def _initialize_datasource(datasource, *, endpoint_name: str) -> None:
except Exception as exc:
raise RuntimeError(f"Failed to initialize {endpoint_name} datasource: {type(exc).__name__}: {exc}") from exc
async def _create_and_initialize_datasource(
ds_config: DataSourceConfig,
*,
datasource_override=None,
endpoint_name: str,
project_root: Path | None = None,
):
datasource = await _create_datasource(
ds_config,
datasource_override=datasource_override,
endpoint_name=endpoint_name,
project_root=project_root,
)
await _initialize_datasource(datasource, endpoint_name=endpoint_name)
return datasource
async def _safe_close_resource(name: str, resource) -> None:
if resource is None:
return
try:
await asyncio.wait_for(resource.close(), timeout=3.0)
except Exception as close_exc:
logger.warning("Failed to close %s during create cleanup: %s", name, close_exc)
def _resolved_config_summary(config: PipelineRunConfig) -> tuple[str, str]:
summary = (
"Resolved Full Config Summary: node_types=%d local=%s remote=%s",
len(config.node_types),
config.local_datasource.type,
config.remote_datasource.type,
)
return summary[0], json.dumps(config_snapshot(config), ensure_ascii=False, indent=2)
def _log_resolved_config(config: PipelineRunConfig) -> tuple[str, str]:
summary_template, resolved_cfg_text = _resolved_config_summary(config)
logger.info(
summary_template,
len(config.node_types),
config.local_datasource.type,
config.remote_datasource.type,
)
logger.info("Resolved Full Config:\n%s", resolved_cfg_text)
return summary_template, resolved_cfg_text
def _log_pipeline_start(config: PipelineRunConfig, *, title: str) -> None:
logger.info("\n%s", "=" * 80)
logger.info("🚀 Creating %s", title)
logger.info("📝 Log file: %s", config.logging.file_path or "-")
logger.info(
"🔧 Resolved Full Config Summary: node_types=%d local=%s remote=%s",
len(config.node_types),
config.local_datasource.type,
config.remote_datasource.type,
)
logger.info("=" * 80)
def _log_pipeline_end(config: PipelineRunConfig, *, stats: Dict[str, Any]) -> None:
logger.info("✅ Pipeline completed: stats=%s", stats)
logger.info("🗃️ Persistence Target: %s", config.persist.display_target())
ensure_builtin_datasource_types_registered()
@@ -187,27 +238,59 @@ def _add_handlers(
def _build_collections_and_bindings(
*,
persistence: PersistenceBackend,
persistence,
scope: str,
local_fallback_collection: Optional[DataCollection] = None,
remote_fallback_collection: Optional[DataCollection] = None,
binding_fallback_manager: Optional[BindingManager] = None,
) -> tuple[DataCollection, DataCollection, BindingManager]:
local_collection = DataCollection("local", scope=scope, persistence=persistence, auto_persist=False)
remote_collection = DataCollection("remote", scope=scope, persistence=persistence, auto_persist=False)
binding_manager = BindingManager(persistence, auto_persist=False, scope=scope)
local_collection = DataCollection(
"local",
scope=scope,
persistence=persistence,
auto_persist=False,
global_fallback_collection=local_fallback_collection,
)
remote_collection = DataCollection(
"remote",
scope=scope,
persistence=persistence,
auto_persist=False,
global_fallback_collection=remote_fallback_collection,
)
binding_manager = BindingManager(
persistence,
auto_persist=False,
scope=scope,
global_fallback_manager=binding_fallback_manager,
)
return local_collection, remote_collection, binding_manager
def _resolve_strategy_map(config: PipelineRunConfig, node_types: List[str]) -> Dict[str, StrategyRuntimeConfig]:
if config.strategies:
return {item.node_type: item for item in config.strategies}
resolved_from_config = {item.node_type: item for item in config.strategies}
for node_type, runtime_cfg in resolved_from_config.items():
strategy_class = DomainRegistry.get_strategy_class(node_type)
runtime_cfg.config.domain_option = resolve_domain_option_config(
getattr(strategy_class, "domain_option_model", None) if strategy_class is not None else None,
runtime_cfg.config.domain_option,
)
return resolved_from_config
resolved: Dict[str, StrategyRuntimeConfig] = {}
for node_type in node_types:
strategy_class = DomainRegistry.get_strategy_class(node_type)
if strategy_class is None:
continue
strategy_config = strategy_class.default_config.model_copy(deep=True)
strategy_config.domain_option = resolve_domain_option_config(
getattr(strategy_class, "domain_option_model", None),
strategy_config.domain_option,
)
resolved[node_type] = StrategyRuntimeConfig(
node_type=node_type,
config=strategy_class.default_config.model_copy(deep=True),
config=strategy_config,
)
return resolved
@@ -251,19 +334,18 @@ def _build_strategies(
raise RuntimeError(f"Missing strategy for node_type: {node_type}")
strategy = strategy_class(node_type, local_collection, remote_collection, binding_manager)
if both_jsonl:
strategy.update_config(
local_orphan_action=OrphanAction.CREATE_REMOTE,
remote_orphan_action=OrphanAction.NONE,
update_direction=UpdateDirection.PUSH,
)
if node_type == "contract":
strategy.update_config(depend_fields={"project_id": "project"})
runtime_cfg = strategy_map.get(node_type)
if runtime_cfg is not None:
strategy.set_config(runtime_cfg.config.model_copy(deep=True))
strategy.skip_sync = bool(runtime_cfg.skip_sync)
strategy.config = runtime_cfg.config.model_copy(deep=True)
strategy.config.log_logic_warnings(node_type=strategy.node_type, logger=logger)
if both_jsonl:
strategy.config.local_orphan_action = OrphanAction.CREATE_REMOTE
strategy.config.remote_orphan_action = OrphanAction.NONE
strategy.config.update_direction = UpdateDirection.PUSH
if node_type == "contract":
strategy.config.depend_fields = {"project_id": "project"}
strategy.config.log_logic_warnings(node_type=strategy.node_type, logger=logger)
strategies.append(strategy)
return strategies
@@ -278,90 +360,129 @@ async def create_pipeline_from_config(
ephemeral_node_types: Optional[List[str]] = None,
local_datasource_override=None,
remote_datasource_override=None,
local_fallback_collection: Optional[DataCollection] = None,
remote_fallback_collection: Optional[DataCollection] = None,
binding_fallback_manager: Optional[BindingManager] = None,
persistence_service: Optional[PersistenceService] = None,
scope_runtime: Optional[ProjectScopeRuntime] = None,
) -> FullSyncPipeline:
if config.logging.initialize and not config.logging.file_path:
config.logging.file_path = resolve_log_file_path(config.logging)
_ensure_pipeline_logger(
initialize_logger=config.logging.initialize,
logger_file_path=config.logging.file_path,
logger_level=config.logging.level,
suppress_node_logs=config.logging.suppress_node_logs,
)
ensure_app_logging(config.logging)
all_types = list(config.node_types)
strategy_map = _resolve_strategy_map(config, all_types)
db_path = Path(config.persist.db_path)
if config.persist.wipe_on_start and db_path.exists():
db_path.unlink()
runtime = scope_runtime
close_resources = runtime is None
persistence: Optional[PersistenceBackend] = None
if runtime is None:
runtime = await create_scope_runtime_from_config(
config,
scope=scope,
local_datasource_override=local_datasource_override,
remote_datasource_override=remote_datasource_override,
local_fallback_collection=local_fallback_collection,
remote_fallback_collection=remote_fallback_collection,
binding_fallback_manager=binding_fallback_manager,
persistence_service=persistence_service,
)
strategies = _build_strategies(
config,
all_types,
runtime.local_collection,
runtime.remote_collection,
runtime.binding_manager,
strategy_map,
)
return FullSyncPipeline(
local_datasource=runtime.local_datasource,
remote_datasource=runtime.remote_datasource,
strategies=strategies,
node_types=all_types,
persistence=runtime.persistence_backend,
local_collection=runtime.local_collection,
remote_collection=runtime.remote_collection,
binding_manager=runtime.binding_manager,
scope=scope,
pipeline_name=pipeline_name,
bootstrap_binding_node_types=bootstrap_binding_node_types,
ephemeral_node_types=ephemeral_node_types,
enable_persistence=config.persist.enable,
scope_runtime=runtime,
close_resources=close_resources,
)
async def create_scope_runtime_from_config(
config: PipelineRunConfig,
*,
scope: str = "global",
local_datasource_override=None,
remote_datasource_override=None,
local_fallback_collection: Optional[DataCollection] = None,
remote_fallback_collection: Optional[DataCollection] = None,
binding_fallback_manager: Optional[BindingManager] = None,
persistence_service: Optional[PersistenceService] = None,
global_runtime: Optional[ProjectScopeRuntime] = None,
) -> ProjectScopeRuntime:
ensure_app_logging(config.logging)
owns_persistence_service = persistence_service is None
created_backend: Optional[PersistenceBackend] = None
local_ds = None
remote_ds = None
try:
persistence = create_persistence_backend(config.persist)
await persistence.initialize()
if persistence_service is None:
wipe_file_path = config.persist.wipe_file_path()
if config.persist.wipe_on_start and wipe_file_path:
db_path = Path(wipe_file_path)
if db_path.exists():
db_path.unlink()
created_backend = create_persistence_backend(config.persist)
persistence_service = PersistenceService(created_backend)
await persistence_service.initialize()
local_ds = await _create_datasource(
local_ds = await _create_and_initialize_datasource(
config.local_datasource,
datasource_override=local_datasource_override,
endpoint_name="local",
project_root=_project_root(),
)
await _initialize_datasource(local_ds, endpoint_name="local")
remote_ds = await _create_datasource(
remote_ds = await _create_and_initialize_datasource(
config.remote_datasource,
datasource_override=remote_datasource_override,
endpoint_name="remote",
project_root=_project_root(),
)
await _initialize_datasource(remote_ds, endpoint_name="remote")
_add_handlers(config, local_ds, remote_ds, all_types)
_add_handlers(config, local_ds, remote_ds, list(config.node_types))
persistence_view = persistence_service.view(scope)
local_collection, remote_collection, binding_manager = _build_collections_and_bindings(
persistence=persistence,
persistence=persistence_view,
scope=scope,
local_fallback_collection=local_fallback_collection,
remote_fallback_collection=remote_fallback_collection,
binding_fallback_manager=binding_fallback_manager,
)
strategies = _build_strategies(
config,
all_types,
local_collection,
remote_collection,
binding_manager,
strategy_map,
)
pipeline = FullSyncPipeline(
local_datasource=local_ds,
remote_datasource=remote_ds,
strategies=strategies,
node_types=all_types,
persistence=persistence,
return ProjectScopeRuntime(
scope=scope,
persistence_service=persistence_service,
persistence_view=persistence_view,
local_collection=local_collection,
remote_collection=remote_collection,
binding_manager=binding_manager,
scope=scope,
pipeline_name=pipeline_name,
bootstrap_binding_node_types=bootstrap_binding_node_types,
ephemeral_node_types=ephemeral_node_types,
enable_persistence=config.persist.enable,
local_datasource=local_ds,
remote_datasource=remote_ds,
global_runtime=global_runtime,
owns_persistence_service=owns_persistence_service,
)
return pipeline
except Exception:
async def _safe_close(name: str, resource) -> None:
if resource is None:
return
try:
await asyncio.wait_for(resource.close(), timeout=3.0)
except Exception as close_exc:
logger.warning("Failed to close %s during create cleanup: %s", name, close_exc)
await _safe_close("remote_datasource", remote_ds)
await _safe_close("local_datasource", local_ds)
await _safe_close("persistence", persistence)
await _safe_close_resource("remote_datasource", remote_ds)
await _safe_close_resource("local_datasource", local_ds)
if owns_persistence_service:
await _safe_close_resource("persistence", persistence_service or created_backend)
raise
@@ -374,47 +495,40 @@ async def run_pipeline_from_config(
bootstrap_binding_node_types: Optional[List[str]] = None,
local_datasource_override=None,
remote_datasource_override=None,
pipeline_type: PipelineType = "auto",
) -> Dict[str, Any]:
if config.logging.initialize and not config.logging.file_path:
config.logging.file_path = resolve_log_file_path(config.logging)
ensure_app_logging(config.logging)
_ensure_pipeline_logger(
initialize_logger=config.logging.initialize,
logger_file_path=config.logging.file_path,
logger_level=config.logging.level,
suppress_node_logs=config.logging.suppress_node_logs,
)
from .multi_project_pipeline import is_implicit_project_batch_config, run_implicit_project_batch_from_config
resolved_cfg = config_snapshot(config)
resolved_cfg_text = json.dumps(resolved_cfg, ensure_ascii=False, indent=2)
logger.info(
"Resolved Full Config Summary: node_types=%d local=%s remote=%s",
len(config.node_types),
config.local_datasource.type,
config.remote_datasource.type,
)
if config.logging.file_path:
try:
with open(config.logging.file_path, "a", encoding="utf-8") as fp:
fp.write("Resolved Full Config:\n")
fp.write(resolved_cfg_text)
fp.write("\n")
except Exception as exc:
logger.warning("Failed to write resolved config into log file: %s", exc)
implicit_multi_project = is_implicit_project_batch_config(config)
if pipeline_type == "full" and implicit_multi_project:
logger.info("pipeline_type=full specified, skipping implicit multi-project routing")
if pipeline_type == "multi-project":
_log_resolved_config(config)
return await run_implicit_project_batch_from_config(
config,
title=title,
print_summary=print_summary,
max_concurrency=getattr(config, "max_concurrency", None),
)
if pipeline_type != "full" and implicit_multi_project:
_log_resolved_config(config)
return await run_implicit_project_batch_from_config(
config,
title=title,
print_summary=print_summary,
max_concurrency=getattr(config, "max_concurrency", None),
)
_log_resolved_config(config)
pipeline: Optional[FullSyncPipeline] = None
try:
if print_summary:
print("\n" + "=" * 80)
print(f"🚀 Creating {title}")
print(f"📝 Log file: {config.logging.file_path or '-'}")
print(
f"🔧 Resolved Full Config Summary: node_types={len(config.node_types)} "
f"local={config.local_datasource.type} "
f"remote={config.remote_datasource.type}"
)
print(f"🔧 Resolved Full Config:\n{resolved_cfg_text}")
print("=" * 80)
_log_pipeline_start(config, title=title)
pipeline = await create_pipeline_from_config(
config,
@@ -427,8 +541,7 @@ async def run_pipeline_from_config(
stats = await pipeline.run()
if print_summary:
print(f"\n✅ Pipeline completed: stats={stats}")
print(f"🗃️ Persistence DB: {config.persist.db_path}")
_log_pipeline_end(config, stats=stats)
return stats
finally:
@@ -491,17 +604,25 @@ async def run_profile_from_file(
cfg_path = Path(config_path)
raw = load_overrides_from_file(cfg_path, project_root)
if is_multi_project_payload(raw):
from .multi_project_pipeline import MultiProjectPipeline
from .multi_project_pipeline import MultiProjectPipeline, _resolve_embedded_config_path
multi_cfg = MultiProjectRunConfig.model_validate(raw)
global_cfg_path = _resolve_embedded_config_path(
multi_cfg.global_config,
project_root=project_root,
profile_path=cfg_path,
)
global_config = build_config_from_file(project_root=project_root, file_path=global_cfg_path)
configure_app_logging(global_config.logging, replace_handlers=True)
pipeline = MultiProjectPipeline(project_root=project_root, config_path=cfg_path, config=multi_cfg)
if print_summary:
print("\n" + "=" * 80)
print(f"🚀 Creating multi-project pipeline from: {cfg_path}")
print("=" * 80)
logger.info("\n%s", "=" * 80)
logger.info("🚀 Creating multi-project pipeline from: %s", cfg_path)
logger.info("=" * 80)
return await pipeline.run()
config = build_config(project_root=project_root, overrides=raw)
configure_app_logging(config.logging, replace_handlers=True)
mode = f"{config.local_datasource.type}->{config.remote_datasource.type}"
return await run_pipeline_from_config(
config,
+373 -118
View File
@@ -5,9 +5,9 @@ Full Sync Pipeline - Implements document-specified end-to-end workflow.
from __future__ import annotations
import asyncio
import logging
from typing import Any, Dict, List, Optional, TYPE_CHECKING
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING, Any, Dict, List, Optional
if TYPE_CHECKING:
from ..datasource.datasource import BaseDataSource
@@ -15,17 +15,31 @@ if TYPE_CHECKING:
from ..datasource.jsonl import JsonlDataSource
from ..common.collection import DataCollection
from ..common.binding import BindingManager
from ..logging import get_scope_logger
from ..common.persistence import PersistenceBackend
from ..sync_system.config import UpdateDirection
from ..sync_system.strategy import BaseSyncStrategy
from ..sync_system.config import OrphanAction, UpdateDirection
from ..engine import StateMachineConfig, StateMachineRuntime
from ..engine import e41_post_create_ready
from ..sync_system.strategy_ops import finalize_peer_creating_sources
from ..sync_system.strategy_ops import finalize_peer_creating_sources, run_phase2_cleanup
from ..sync_system.strategy_ops.bind_ops import refresh_bind_data_id
from ..sync_system.strategy_ops.post_check_ops import evaluate_consistency_for_node_type
from .summary_report import print_pipeline_summary
from .project_scope_runtime import ProjectScopeRuntime
logger = logging.getLogger(__name__)
class NodeTypeLoadError(RuntimeError):
"""Raised when a datasource load for a node type fails and the pipeline must stop."""
@dataclass(frozen=True)
class WriteOutcome:
local_written: bool = False
remote_written: bool = False
@property
def any_written(self) -> bool:
return self.local_written or self.remote_written
class FullSyncPipeline:
@@ -56,6 +70,8 @@ class FullSyncPipeline:
load_order: Optional[List[str]] = None,
sync_order: Optional[List[str]] = None,
enable_persistence: bool = True,
scope_runtime: Optional[ProjectScopeRuntime] = None,
close_resources: bool = True,
):
# 验证必需参数(由工厂函数创建)
if strategies is None:
@@ -75,6 +91,8 @@ class FullSyncPipeline:
self.sync_order = sync_order or self.node_types
self.persistence = persistence
self.enable_persistence = enable_persistence
self.scope_runtime = scope_runtime
self.close_resources = close_resources
self.local_collection = local_collection
self.remote_collection = remote_collection
@@ -97,12 +115,9 @@ class FullSyncPipeline:
"post_check_passed": True,
}
self._consistency_by_type: Dict[str, Dict[str, Any]] = {}
self._logger = logger
self._logger = get_scope_logger(scope, __name__)
self._closed = False
self._loaded_node_types: set[str] = set()
self._preloaded_local_nodes: List[Any] = []
self._preloaded_remote_nodes: List[Any] = []
self._preloaded_bindings: Dict[str, Dict[str, Optional[str]]] = {}
def _resolve_pipeline_runtime(self) -> StateMachineRuntime:
if self.strategies:
@@ -113,133 +128,257 @@ class FullSyncPipeline:
cfg_path = Path(__file__).resolve().parents[1] / "config" / "node_state_machine.yaml"
return StateMachineRuntime(StateMachineConfig.load(cfg_path))
def _strategy_map(self) -> Dict[str, BaseSyncStrategy]:
return {strategy.node_type: strategy for strategy in self.strategies}
def _iter_sync_strategies(self):
strategy_map = self._strategy_map()
for node_type in self.sync_order:
strategy = strategy_map.get(node_type)
if strategy is not None:
yield node_type, strategy
def _get_strategy(self, node_type: str) -> BaseSyncStrategy | None:
return next((item for item in self.strategies if item.node_type == node_type), None)
def _log_phase_completed(self, phase_name: str) -> None:
self._logger.info("✅ Phase %s completed (%s)", phase_name, self.pipeline_name)
def _log_strategy_start(self, node_type: str) -> None:
section_width = 96
self._logger.info("\n" + "-" * section_width)
self._logger.info("🧩 Strategy start: %s", node_type)
self._logger.info("-" * section_width)
def _log_strategy_result(self, action: str, node_type: str, count: int | None = None) -> None:
if count is None:
self._logger.info("✅ Strategy %s: %s", action, node_type)
return
self._logger.info("✅ Strategy %s: %s (%s=%d)", action, node_type, action, count)
def _log_strategy_done(self, node_type: str) -> None:
self._logger.info("✅ Strategy done: %s", node_type)
def _log_strategy_failure(self, stage: str, node_type: str, exc: Exception) -> None:
self._logger.error(
"❌ Strategy %s failed but pipeline continues: %s | %s: %s",
stage,
node_type,
type(exc).__name__,
exc,
)
def _log_reload_skipped(self, node_type: str, reason: str) -> None:
self._logger.info("⏭️ Reload skipped for JSONL pipeline: node_type=%s reason=%s", node_type, reason)
async def _safe_close(self, name: str, close_coro, timeout: float = 3.0) -> None:
try:
await asyncio.wait_for(close_coro, timeout=timeout)
except asyncio.TimeoutError:
self._logger.warning("⚠️ close timeout: %s (>%ss)", name, timeout)
except Exception as exc:
self._logger.warning("⚠️ close failed: %s (%s)", name, exc)
async def run(self) -> Dict[str, Any]:
persistence_started = False
try:
await self.phase_bootstrap()
self._logger.info("✅ Phase bootstrap completed (%s)", self.pipeline_name)
self._log_phase_completed("bootstrap")
await self.phase_sync_by_node_type()
self._logger.info("✅ Phase sync-by-node-type completed (%s)", self.pipeline_name)
self._log_phase_completed("sync-by-node-type")
post_ok = await self.phase_post_check()
self.stats["post_check_passed"] = post_ok
self._logger.info("✅ Phase post-check completed (%s)", self.pipeline_name)
self._log_phase_completed("post-check")
persistence_started = True
await self.phase_persistence()
self._logger.info("✅ Phase persistence completed (%s)", self.pipeline_name)
self._log_phase_completed("persistence")
return self.stats
except Exception as exc:
if not persistence_started:
await self._persist_after_failure(exc)
raise
finally:
await self.close()
async def _persist_after_failure(self, cause: Exception) -> None:
if not self.enable_persistence:
self._logger.info(
"⏭️ Persistence disabled after pipeline failure: %s (%s)",
type(cause).__name__,
self.pipeline_name,
)
return
self._logger.info(
"🔄 Pipeline failed before persistence, persisting current state: %s | %s",
type(cause).__name__,
cause,
)
try:
await self.phase_persistence()
self._logger.info("✅ Phase persistence completed after failure (%s)", self.pipeline_name)
except Exception as persist_exc:
self._logger.error(
"❌ Phase persistence failed after pipeline failure: %s | %s",
type(persist_exc).__name__,
persist_exc,
)
async def phase_sync_by_node_type(self) -> None:
self._prepare_datasources()
strategy_map = {s.node_type: s for s in self.strategies}
for node_type in self.sync_order:
strategy = strategy_map.get(node_type)
if strategy is None:
continue
for node_type, strategy in self._iter_sync_strategies():
try:
section_width = 96
self._logger.info("\n" + "-" * section_width)
self._logger.info(f"🧩 Strategy start: {node_type}")
self._logger.info("-" * section_width)
self._log_strategy_start(node_type)
await self._load_node_type(node_type, reason="pre-strategy refresh")
# Load failure is a hard stop: create/update cannot safely treat missing data as empty data.
await strategy.bind()
self._logger.info(f"✅ Strategy bind: {node_type}")
self._log_strategy_result("bind", node_type)
if strategy.skip_sync:
self._logger.info(f"⏭️ Skipping create/update for {node_type} (skip_sync=True)")
self._logger.info("⏭️ Skipping create/update for %s (skip_sync=True)", node_type)
continue
created = await strategy.create()
self._logger.info(f"✅ Strategy create: {node_type} (created={len(created)})")
create_written = await self.commit_creates(node_type)
self._log_strategy_result("create", node_type, len(created))
created_local_node_ids = {
node.node_id for node in created if self.local_collection.get_by_node_id(node.node_id) is not None
}
created_remote_node_ids = {
node.node_id for node in created if self.remote_collection.get_by_node_id(node.node_id) is not None
}
create_outcome = await self.commit_creates(node_type)
await self.normalize_create_success_to_update_entry(node_type)
if create_written:
await self._reload_node_type(node_type, reason="create wrote data")
if create_outcome.any_written:
await self._reload_node_type_after_write(
node_type,
reason="create wrote data",
reload_local=create_outcome.local_written,
reload_remote=create_outcome.remote_written,
)
await refresh_bind_data_id(
node_type=node_type,
local_collection=self.local_collection,
remote_collection=self.remote_collection,
binding_manager=self.binding_manager,
local_node_ids=created_local_node_ids,
remote_node_ids=created_remote_node_ids,
)
updated = await strategy.update()
self._logger.info(f"✅ Strategy update: {node_type} (updated={len(updated)})")
update_written = await self.commit_updates(node_type)
if update_written:
await self._reload_node_type(node_type, reason="update wrote data")
self._log_strategy_result("update", node_type, len(updated))
update_outcome = await self.commit_updates(node_type)
if update_outcome.any_written:
await self._reload_node_type_after_write(
node_type,
reason="update wrote data",
reload_local=update_outcome.local_written,
reload_remote=update_outcome.remote_written,
)
self._logger.info(f"✅ Strategy done: {node_type}")
self._log_strategy_done(node_type)
except NodeTypeLoadError:
raise
except Exception as exc:
self.stats["failed"] += 1
self._logger.error(f"❌ Strategy failed but pipeline continues: {node_type} | {type(exc).__name__}: {exc}")
self._log_strategy_failure("sync", node_type, exc)
continue
async def phase_bootstrap(self) -> None:
await self._load_persistence_state()
if self.scope_runtime is not None:
await self.scope_runtime.load(
node_types=self.node_types,
bootstrap_binding_node_types=self.bootstrap_binding_node_types,
enable_persistence=self.enable_persistence,
ephemeral_node_types=self.ephemeral_node_types,
)
else:
await self._load_persistence_state()
self._logger.info("✅ Bootstrap: persistence loaded")
await self._apply_preloaded_state()
self._logger.info("✅ Bootstrap: shared preload applied")
self._prepare_datasources()
self._logger.info("✅ Bootstrap: datasource prepared")
await self._bootstrap_refresh_latest_data()
self._logger.info("✅ Bootstrap: datasource refresh completed")
await self._run_bootstrap_reset_cleanup()
self._logger.info("✅ Bootstrap: reset cleanup completed")
await self._apply_project_scope_cleanup()
async def phase_bind(self) -> None:
self._prepare_datasources()
strategy_map = {s.node_type: s for s in self.strategies}
for node_type in self.sync_order:
strategy = strategy_map.get(node_type)
if strategy is None:
continue
for node_type, strategy in self._iter_sync_strategies():
try:
await strategy.bind()
except Exception as exc:
self.stats["failed"] += 1
self._logger.error(f"❌ Strategy bind failed but pipeline continues: {node_type} | {type(exc).__name__}: {exc}")
self._log_strategy_failure("bind", node_type, exc)
continue
async def phase_create(self) -> None:
self._prepare_datasources()
strategy_map = {s.node_type: s for s in self.strategies}
for node_type in self.sync_order:
strategy = strategy_map.get(node_type)
if strategy is None:
continue
for node_type, strategy in self._iter_sync_strategies():
if strategy.skip_sync:
self._logger.info(f"⏭️ Skipping create for {node_type} (skip_sync=True)")
self._logger.info("⏭️ Skipping create for %s (skip_sync=True)", node_type)
continue
try:
created = await strategy.create()
self._logger.info(f"✅ Strategy create: {node_type} (created={len(created)})")
create_written = await self.commit_creates(node_type)
self._log_strategy_result("create", node_type, len(created))
created_local_node_ids = {
node.node_id for node in created if self.local_collection.get_by_node_id(node.node_id) is not None
}
created_remote_node_ids = {
node.node_id for node in created if self.remote_collection.get_by_node_id(node.node_id) is not None
}
create_outcome = await self.commit_creates(node_type)
await self.normalize_create_success_to_update_entry(node_type)
if create_written:
await self._reload_node_type(node_type, reason="create wrote data")
if create_outcome.any_written:
await self._reload_node_type_after_write(
node_type,
reason="create wrote data",
reload_local=create_outcome.local_written,
reload_remote=create_outcome.remote_written,
)
await refresh_bind_data_id(
node_type=node_type,
local_collection=self.local_collection,
remote_collection=self.remote_collection,
binding_manager=self.binding_manager,
local_node_ids=created_local_node_ids,
remote_node_ids=created_remote_node_ids,
)
except Exception as exc:
self.stats["failed"] += 1
self._logger.error(f"❌ Strategy create failed but pipeline continues: {node_type} | {type(exc).__name__}: {exc}")
self._log_strategy_failure("create", node_type, exc)
continue
async def phase_update(self) -> None:
self._prepare_datasources()
strategy_map = {s.node_type: s for s in self.strategies}
for node_type in self.sync_order:
strategy = strategy_map.get(node_type)
if strategy is None:
continue
for node_type, strategy in self._iter_sync_strategies():
if strategy.skip_sync:
self._logger.info(f"⏭️ Skipping update for {node_type} (skip_sync=True)")
self._logger.info("⏭️ Skipping update for %s (skip_sync=True)", node_type)
continue
try:
updated = await strategy.update()
self._logger.info(f"✅ Strategy update: {node_type} (updated={len(updated)})")
update_written = await self.commit_updates(node_type)
if update_written:
await self._reload_node_type(node_type, reason="update wrote data")
self._log_strategy_result("update", node_type, len(updated))
update_outcome = await self.commit_updates(node_type)
if update_outcome.any_written:
await self._reload_node_type_after_write(
node_type,
reason="update wrote data",
reload_local=update_outcome.local_written,
reload_remote=update_outcome.remote_written,
)
await self._evaluate_consistency_for_node_type(node_type)
except Exception as exc:
self.stats["failed"] += 1
self._logger.error(f"❌ Strategy update failed but pipeline continues: {node_type} | {type(exc).__name__}: {exc}")
self._log_strategy_failure("update", node_type, exc)
continue
async def close(self) -> None:
@@ -248,20 +387,20 @@ class FullSyncPipeline:
return # 已经关闭过,避免重复关闭
self._closed = True
async def _safe_close(name: str, close_coro, timeout: float = 3.0) -> None:
try:
await asyncio.wait_for(close_coro, timeout=timeout)
except asyncio.TimeoutError:
self._logger.warning("⚠️ close timeout: %s (>%ss)", name, timeout)
except Exception as exc:
self._logger.warning("⚠️ close failed: %s (%s)", name, exc)
if not self.close_resources:
return
await _safe_close("remote_datasource", self.remote_datasource.close())
await _safe_close("local_datasource", self.local_datasource.close())
await _safe_close("persistence", self.persistence.close())
if self.scope_runtime is not None:
await self.scope_runtime.unload()
await self.scope_runtime.close()
return
await self._safe_close("remote_datasource", self.remote_datasource.close())
await self._safe_close("local_datasource", self.local_datasource.close())
await self._safe_close("persistence", self.persistence.close())
async def _load_persistence_state(self) -> None:
"""加载持久化数据并清理僵尸节点"""
"""加载持久化数据到内存,等待 bootstrap datasource refresh 与 E01 cleanup。"""
if not self.enable_persistence:
return
@@ -278,11 +417,51 @@ class FullSyncPipeline:
for node_type in binding_node_types:
await self.binding_manager.load_from_persistence(node_type)
self._logger.info("🔄 Persistence loaded (state preserved; pending E01 normalization)")
# 加载后 reset 清理:清理 CREATE 失败的僵尸绑定
def _resolve_bootstrap_refresh_node_types(self) -> List[str]:
local_types = {node.node_type for node in self.local_collection.filter()}
remote_types = {node.node_type for node in self.remote_collection.filter()}
candidate_types = local_types | remote_types
if not candidate_types:
return []
ordered: List[str] = []
for node_type in [*self.load_order, *self.bootstrap_binding_node_types, *sorted(candidate_types)]:
if node_type in candidate_types and node_type not in ordered:
ordered.append(node_type)
return ordered
async def _bootstrap_refresh_latest_data(self) -> None:
if not self.enable_persistence:
return
node_types = self._resolve_bootstrap_refresh_node_types()
if not node_types:
self._logger.info("⏭️ Bootstrap datasource refresh skipped: no persisted node types")
return
for node_type in node_types:
self._logger.info("🔄 Bootstrap refresh node_type=%s from datasource", node_type)
await self._load_node_type_from_datasource(
self.local_datasource,
datasource_role="local",
node_type=node_type,
reason="bootstrap persistence refresh",
)
await self._load_node_type_from_datasource(
self.remote_datasource,
datasource_role="remote",
node_type=node_type,
reason="bootstrap persistence refresh",
)
async def _run_bootstrap_reset_cleanup(self) -> None:
if not self.enable_persistence:
return
if not self.strategies:
raise RuntimeError("no strategies configured: cannot run bootstrap event E01")
total_cleaned = await BaseSyncStrategy.run_reset(
total_cleaned = await run_phase2_cleanup(
node_types=self.node_types,
local_collection=self.local_collection,
remote_collection=self.remote_collection,
@@ -290,30 +469,7 @@ class FullSyncPipeline:
runtime=self.sm_runtime,
)
if total_cleaned > 0:
self._logger.info(f"🧹 Reset cleanup: {total_cleaned} CREATE-failed zombies cleaned")
self._logger.info("🔄 Post-load reset cleanup completed")
async def _apply_preloaded_state(self) -> None:
if self._preloaded_local_nodes:
await self.local_collection.import_nodes(self._preloaded_local_nodes)
if self._preloaded_remote_nodes:
await self.remote_collection.import_nodes(self._preloaded_remote_nodes)
for node_type, bindings in self._preloaded_bindings.items():
self.binding_manager.import_bindings(node_type, bindings)
def preload_shared_state(
self,
*,
local_nodes: List[Any],
remote_nodes: List[Any],
bindings_by_type: Dict[str, Dict[str, Optional[str]]],
) -> None:
self._preloaded_local_nodes = list(local_nodes)
self._preloaded_remote_nodes = list(remote_nodes)
self._preloaded_bindings = {
node_type: dict(bindings)
for node_type, bindings in bindings_by_type.items()
}
self._logger.info(f"🧹 Reset cleanup: {total_cleaned} bootstrap zombies cleaned")
def _prepare_datasources(self) -> None:
"""为后续按 node_type 加载准备 collection 绑定。"""
@@ -355,9 +511,19 @@ class FullSyncPipeline:
)
async def _load_node_type(self, node_type: str, *, reason: str) -> None:
self._logger.info(f"🔄 Load node_type={node_type}, reason={reason}")
await self.local_datasource.load_all(order=[node_type])
await self.remote_datasource.load_all(order=[node_type])
self._logger.debug(f"🔄 Load node_type={node_type}, reason={reason}")
await self._load_node_type_from_datasource(
self.local_datasource,
datasource_role="local",
node_type=node_type,
reason=reason,
)
await self._load_node_type_from_datasource(
self.remote_datasource,
datasource_role="remote",
node_type=node_type,
reason=reason,
)
if node_type in self._loaded_node_types:
return
@@ -366,6 +532,21 @@ class FullSyncPipeline:
self.stats["loaded"] += len(self.local_collection.filter(node_type=node_type))
self.stats["loaded"] += len(self.remote_collection.filter(node_type=node_type))
async def _load_node_type_from_datasource(
self,
datasource: "BaseDataSource",
*,
datasource_role: str,
node_type: str,
reason: str,
) -> None:
try:
await datasource.load_all(order=[node_type], raise_on_error=True)
except Exception as exc:
raise NodeTypeLoadError(
f"Load failed for node_type={node_type}, datasource={datasource_role}, reason={reason}: {exc}"
) from exc
def _get_action_success_count(self, datasource: "BaseDataSource", node_type: str, action_key: str) -> int:
summary = datasource.get_action_summary().get(node_type, {})
action_summary = summary.get(action_key, {}) if isinstance(summary, dict) else {}
@@ -376,25 +557,69 @@ class FullSyncPipeline:
async def _reload_node_type(self, node_type: str, *, reason: str) -> None:
if self._should_skip_reload():
self._logger.info("⏭️ Reload skipped for JSONL pipeline: node_type=%s reason=%s", node_type, reason)
self._log_reload_skipped(node_type, reason)
return
await self._load_node_type(node_type, reason=reason)
async def _reload_node_type_after_write(
self,
node_type: str,
*,
reason: str,
reload_local: bool,
reload_remote: bool,
) -> None:
if not reload_local and not reload_remote:
return
if self._should_skip_reload():
self._log_reload_skipped(node_type, reason)
return
self._logger.debug(
"🔄 Reload node_type=%s after write, reason=%s, local=%s, remote=%s",
node_type,
reason,
reload_local,
reload_remote,
)
if reload_local:
await self._load_node_type_from_datasource(
self.local_datasource,
datasource_role="local",
node_type=node_type,
reason=reason,
)
if reload_remote:
await self._load_node_type_from_datasource(
self.remote_datasource,
datasource_role="remote",
node_type=node_type,
reason=reason,
)
async def _evaluate_consistency_for_node_type(self, node_type: str) -> Dict[str, Any]:
strategy = next((item for item in self.strategies if item.node_type == node_type), None)
if strategy is None:
raise RuntimeError(f"Missing strategy for node_type={node_type}")
depend_fields = dict(strategy.config.depend_fields) if strategy and strategy.config.depend_fields else {}
compare_to_remote = True
if strategy and strategy.config.update_direction == UpdateDirection.PULL:
compare_to_remote = False
ignore_fields = list(strategy.config.post_check_ignore_fields) if strategy else []
ignore_fields = list(
dict.fromkeys([
*strategy.config.compare_ignore_fields,
*strategy.config.post_check_ignore_fields,
])
)
result = await evaluate_consistency_for_node_type(
node_type=node_type,
strategy=strategy,
local_collection=self.local_collection,
remote_collection=self.remote_collection,
binding_manager=self.binding_manager,
logger=self._logger,
normalize_compare_payload=strategy.normalize_compare_payload,
depend_fields=depend_fields,
compare_to_remote=compare_to_remote,
ignore_fields=ignore_fields,
@@ -402,7 +627,7 @@ class FullSyncPipeline:
self._consistency_by_type[node_type] = result
return result
async def commit_creates(self, node_type: str) -> bool:
async def commit_creates(self, node_type: str) -> WriteOutcome:
"""提交 CREATE 操作(包含异步轮询)"""
# 执行同步(sync_all 内部会根据 action 筛选节点)
await self.local_datasource.sync_all(
@@ -427,8 +652,10 @@ class FullSyncPipeline:
local_success = self._get_action_success_count(self.local_datasource, node_type, "create")
remote_success = self._get_action_success_count(self.remote_datasource, node_type, "create")
wrote = (local_success + remote_success) > 0
return wrote
return WriteOutcome(
local_written=local_success > 0,
remote_written=remote_success > 0,
)
async def _finalize_peer_creating_sources(self, node_type: str) -> None:
finalized_success, finalized_failed = await finalize_peer_creating_sources(
@@ -440,7 +667,7 @@ class FullSyncPipeline:
)
if finalized_success or finalized_failed:
self._logger.info(
self._logger.debug(
f"🔁 Peer-creating finalize: node_type={node_type}, success={finalized_success}, failed={finalized_failed}"
)
@@ -463,9 +690,9 @@ class FullSyncPipeline:
normalized += 1
if normalized > 0:
self._logger.info(f"🔁 Post-create normalization: {normalized} nodes moved S11C->S01 for {node_type}")
self._logger.debug(f"🔁 Post-create normalization: {normalized} nodes moved S11C->S01 for {node_type}")
async def commit_updates(self, node_type: str) -> bool:
async def commit_updates(self, node_type: str) -> WriteOutcome:
"""提交 UPDATE 操作(包含异步轮询)"""
# 执行同步(sync_all 内部会根据 action 筛选节点)
await self.local_datasource.sync_all(
@@ -488,8 +715,10 @@ class FullSyncPipeline:
local_success = self._get_action_success_count(self.local_datasource, node_type, "update")
remote_success = self._get_action_success_count(self.remote_datasource, node_type, "update")
wrote = (local_success + remote_success) > 0
return wrote
return WriteOutcome(
local_written=local_success > 0,
remote_written=remote_success > 0,
)
async def phase_post_check(self) -> bool:
"""同步结果报告。
@@ -502,8 +731,22 @@ class FullSyncPipeline:
else:
self._logger.info("🔄 Post-check: reloading all node types for final consistency evaluation...")
for node_type in self.node_types:
strategy = self._get_strategy(node_type)
if strategy is not None and strategy.skip_post_check:
self._logger.info("⏭️ Post-check reload skipped: node_type=%s", node_type)
continue
await self._reload_node_type(node_type, reason="post-check final reload")
for node_type in self.node_types:
strategy = self._get_strategy(node_type)
if strategy is not None and strategy.skip_post_check:
self._logger.info("⏭️ Post-check skipped: node_type=%s", node_type)
self._consistency_by_type[node_type] = {
"ok": True,
"checked_pairs": 0,
"mismatch_count": 0,
"skipped": True,
}
continue
await self._evaluate_consistency_for_node_type(node_type)
self.stats["consistency"] = {
@@ -519,6 +762,18 @@ class FullSyncPipeline:
async def phase_persistence(self) -> None:
"""持久化所有数据。"""
if self.scope_runtime is not None:
if not self.enable_persistence:
self._logger.info("⏭️ Persistence disabled, skipping...")
return
self._logger.info("🔍 Persisting scope runtime...")
await self.scope_runtime.persist(
enable_persistence=self.enable_persistence,
ephemeral_node_types=self.ephemeral_node_types,
)
self._logger.info("✅ All data persisted successfully")
return
if not self.enable_persistence:
self._logger.info("⏭️ Persistence disabled, skipping...")
return
@@ -1,27 +1,34 @@
from __future__ import annotations
import asyncio
import logging
import copy
from dataclasses import dataclass
from pathlib import Path
from typing import Dict, Optional, List, TypedDict, Any
from typing import Any, Dict, Optional, List
from ..config import (
MultiProjectItemConfig,
MultiProjectRunConfig,
PipelineRunConfig,
StrategyRuntimeConfig,
apply_config_overrides,
build_config_from_file,
)
from .factory import create_pipeline_from_config
from ..logging import get_scope_logger, get_scope_display_name
from .factory import create_pipeline_from_config, create_scope_runtime_from_config
logger = logging.getLogger(__name__)
logger = get_scope_logger("multi-project", __name__)
class _SharedStateSnapshot(TypedDict):
local_nodes: List[Any]
remote_nodes: List[Any]
bindings_by_type: Dict[str, Dict[str, Optional[str]]]
@dataclass(frozen=True)
class ImplicitProjectItem:
local_project_id: str
remote_project_id: str
@property
def scope(self) -> str:
return f"project_id:{self.local_project_id}"
def _resolve_embedded_config_path(raw_path: str, *, project_root: Path, profile_path: Path) -> Path:
@@ -64,7 +71,215 @@ def _resolve_project_bootstrap_node_types(
return list(dict.fromkeys(shared_node_types))
class MultiProjectPipeline:
def _normalize_project_id_items(raw_items: Any) -> list[str]:
items = raw_items if isinstance(raw_items, list) else []
normalized: list[str] = []
for item in items:
value = str(item).strip()
if value and value not in normalized:
normalized.append(value)
return normalized
def _get_project_handler_filters(config: PipelineRunConfig) -> tuple[list[str], list[str]]:
local_filters = _normalize_project_id_items(
config.local_datasource.handler_configs.get("project", {}).get("data_id_filter", [])
)
remote_filters = _normalize_project_id_items(
config.remote_datasource.handler_configs.get("project", {}).get("data_id_filter", [])
)
return local_filters, remote_filters
def _get_project_strategy_runtime(config: PipelineRunConfig) -> StrategyRuntimeConfig | None:
for item in config.strategies:
if item.node_type == "project":
return item
return None
def _get_project_prebind_pairs(config: PipelineRunConfig) -> list[tuple[str, str]]:
runtime_cfg = _get_project_strategy_runtime(config)
if runtime_cfg is None:
return []
pairs: list[tuple[str, str]] = []
for item in runtime_cfg.config.pre_bind_data_id:
local_id = str(item.local_data_id).strip()
remote_id = str(item.remote_data_id).strip()
if not local_id or not remote_id:
continue
pairs.append((local_id, remote_id))
return pairs
def is_implicit_project_batch_config(config: PipelineRunConfig) -> bool:
if "project" not in config.node_types:
return False
local_filters, remote_filters = _get_project_handler_filters(config)
prebind_pairs = _get_project_prebind_pairs(config)
return max(len(local_filters), len(remote_filters), len(prebind_pairs)) > 1
def _extract_implicit_project_items(config: PipelineRunConfig) -> list[ImplicitProjectItem]:
local_filters, remote_filters = _get_project_handler_filters(config)
prebind_pairs = _get_project_prebind_pairs(config)
if not prebind_pairs:
raise ValueError("implicit multi-project config requires strategies.project.config.pre_bind_data_id")
pair_local_ids = [local_id for local_id, _ in prebind_pairs]
pair_remote_ids = [remote_id for _, remote_id in prebind_pairs]
effective_local_filters = local_filters or list(dict.fromkeys(pair_local_ids))
effective_remote_filters = remote_filters or list(dict.fromkeys(pair_remote_ids))
if set(effective_local_filters) != set(pair_local_ids):
raise ValueError(
"implicit multi-project config mismatch: local project data_id_filter must match project pre_bind_data_id local_data_id"
)
if set(effective_remote_filters) != set(pair_remote_ids):
raise ValueError(
"implicit multi-project config mismatch: remote project data_id_filter must match project pre_bind_data_id remote_data_id"
)
seen_local_ids: set[str] = set()
seen_remote_ids: set[str] = set()
items: list[ImplicitProjectItem] = []
for local_id, remote_id in prebind_pairs:
if local_id in seen_local_ids:
raise ValueError(f"duplicate local project id in implicit multi-project config: {local_id}")
if remote_id in seen_remote_ids:
raise ValueError(f"duplicate remote project id in implicit multi-project config: {remote_id}")
seen_local_ids.add(local_id)
seen_remote_ids.add(remote_id)
items.append(ImplicitProjectItem(local_project_id=local_id, remote_project_id=remote_id))
return items
def _split_implicit_project_node_types(node_types: list[str]) -> tuple[list[str], list[str]]:
if "project" not in node_types:
raise ValueError("implicit multi-project config requires node_types to include 'project'")
project_index = node_types.index("project")
return node_types[:project_index], node_types[project_index:]
def _build_implicit_project_config(
base_config: PipelineRunConfig,
*,
item: ImplicitProjectItem,
project_node_types: list[str],
) -> PipelineRunConfig:
return apply_config_overrides(
base_config,
{
"node_types": project_node_types,
"persist": {"wipe_on_start": False},
"local_datasource": {
"handler_configs": {
"project": {"data_id_filter": [item.local_project_id]},
}
},
"remote_datasource": {
"handler_configs": {
"project": {"data_id_filter": [item.remote_project_id]},
}
},
"strategies": {
"project": {
"config": {
"pre_bind_data_id": [
{
"local_data_id": item.local_project_id,
"remote_data_id": item.remote_project_id,
}
]
}
}
},
},
)
async def run_implicit_project_batch_from_config(
config: PipelineRunConfig,
*,
title: str,
print_summary: bool = True,
max_concurrency: int | None = None,
) -> Dict[str, Dict[str, object]]:
from .factory import create_pipeline_from_config, create_scope_runtime_from_config
project_items = _extract_implicit_project_items(config)
shared_node_types, project_node_types = _split_implicit_project_node_types(list(config.node_types))
results: Dict[str, Dict[str, object]] = {}
global_runtime = None
global_pipeline = None
global_config = None
if print_summary:
logger.info("\n%s", "=" * 80)
logger.info("🚀 Creating implicit multi-project pipeline from in-memory config")
logger.info("📦 Projects=%d", len(project_items))
logger.info("=" * 80)
try:
if shared_node_types:
global_config = apply_config_overrides(
config,
{
"node_types": shared_node_types,
},
)
global_runtime = await create_scope_runtime_from_config(global_config, scope="global")
global_pipeline = await create_pipeline_from_config(
global_config,
scope="global",
pipeline_name=f"{title} [global]",
scope_runtime=global_runtime,
)
results["global"] = await global_pipeline.run()
semaphore = asyncio.Semaphore(max(1, max_concurrency or len(project_items)))
async def _run_project(item: ImplicitProjectItem) -> None:
project_config = _build_implicit_project_config(
config,
item=item,
project_node_types=project_node_types,
)
async with semaphore:
project_runtime = await create_scope_runtime_from_config(
project_config,
scope=item.scope,
persistence_service=global_runtime.persistence_service if global_runtime is not None else None,
local_fallback_collection=global_runtime.local_collection if global_runtime is not None else None,
remote_fallback_collection=global_runtime.remote_collection if global_runtime is not None else None,
binding_fallback_manager=global_runtime.binding_manager if global_runtime is not None else None,
global_runtime=global_runtime,
)
try:
project_pipeline = await create_pipeline_from_config(
project_config,
scope=item.scope,
pipeline_name=f"{title} [{item.scope}]",
bootstrap_binding_node_types=shared_node_types,
ephemeral_node_types=shared_node_types,
scope_runtime=project_runtime,
)
results[item.scope] = await project_pipeline.run()
finally:
await project_runtime.unload()
await project_runtime.close()
await asyncio.gather(*[_run_project(item) for item in project_items])
return results
finally:
if global_runtime is not None:
await global_runtime.unload()
await global_runtime.close()
class ProjectBatchPipeline:
def __init__(self, *, project_root: Path, config_path: Path, config: MultiProjectRunConfig):
self.project_root = project_root
self.config_path = config_path
@@ -86,27 +301,6 @@ class MultiProjectPipeline:
cfg = apply_config_overrides(cfg, {"persist": {**shared_persist, "wipe_on_start": False}})
return apply_config_overrides(cfg, _project_scope_overrides(item, node_types=list(cfg.node_types)))
async def _snapshot_shared_state(self, *, pipeline, node_types: List[str]) -> _SharedStateSnapshot:
local_nodes = [
copy.deepcopy(node)
for node_type in node_types
for node in pipeline.local_collection.filter(node_type=node_type)
]
remote_nodes = [
copy.deepcopy(node)
for node_type in node_types
for node in pipeline.remote_collection.filter(node_type=node_type)
]
bindings_by_type: Dict[str, Dict[str, Optional[str]]] = {}
for node_type in node_types:
bindings = await pipeline.binding_manager.get_all_bindings(node_type)
bindings_by_type[node_type] = {local_id: remote_id for local_id, remote_id in bindings}
return {
"local_nodes": local_nodes,
"remote_nodes": remote_nodes,
"bindings_by_type": bindings_by_type,
}
async def run(self) -> Dict[str, Dict[str, object]]:
results: Dict[str, Dict[str, object]] = {}
@@ -118,49 +312,67 @@ class MultiProjectPipeline:
shared_node_types=shared_node_types,
)
logger.info("🚀 Multi-project pipeline start: projects=%d", len(self.config.projects))
global_pipeline = await create_pipeline_from_config(
global_runtime = await create_scope_runtime_from_config(
global_config,
scope="global",
pipeline_name="multi-project pipeline [global]",
)
global_pipeline = None
try:
global_pipeline = await create_pipeline_from_config(
global_config,
scope="global",
pipeline_name="multi-project pipeline [global]",
scope_runtime=global_runtime,
)
results["global"] = await global_pipeline.run()
shared_state = await self._snapshot_shared_state(
pipeline=global_pipeline,
node_types=project_bootstrap_node_types,
)
finally:
await global_pipeline.close()
semaphore = asyncio.Semaphore(self.config.max_concurrency)
for item in self.config.projects:
project_config = self._prepare_project_config(
shared_persist=shared_persist,
item=item,
base_path=item.config_path,
)
scope = item.scope
title = f"multi-project pipeline [{scope}]"
logger.info(
"🚀 Project pipeline start: scope=%s local_project_id=%s remote_project_id=%s",
scope,
item.local_project_id,
item.remote_project_id,
)
project_pipeline = await create_pipeline_from_config(
project_config,
scope=scope,
pipeline_name=title,
bootstrap_binding_node_types=project_bootstrap_node_types,
ephemeral_node_types=project_bootstrap_node_types,
)
try:
project_pipeline.preload_shared_state(
local_nodes=shared_state["local_nodes"],
remote_nodes=shared_state["remote_nodes"],
bindings_by_type=shared_state["bindings_by_type"],
async def _run_project(item: MultiProjectItemConfig) -> None:
project_config = self._prepare_project_config(
shared_persist=shared_persist,
item=item,
base_path=item.config_path,
)
results[scope] = await project_pipeline.run()
finally:
await project_pipeline.close()
scope = item.scope
title = f"multi-project pipeline [{scope}]"
logger.info(
"🚀 Project pipeline start: scope=%s local_project_id=%s remote_project_id=%s",
get_scope_display_name(scope),
item.local_project_id,
item.remote_project_id,
)
async with semaphore:
project_runtime = await create_scope_runtime_from_config(
project_config,
scope=scope,
persistence_service=global_runtime.persistence_service,
local_fallback_collection=global_runtime.local_collection,
remote_fallback_collection=global_runtime.remote_collection,
binding_fallback_manager=global_runtime.binding_manager,
global_runtime=global_runtime,
)
project_pipeline = None
try:
project_pipeline = await create_pipeline_from_config(
project_config,
scope=scope,
pipeline_name=title,
bootstrap_binding_node_types=project_bootstrap_node_types,
ephemeral_node_types=project_bootstrap_node_types,
scope_runtime=project_runtime,
)
results[scope] = await project_pipeline.run()
finally:
await project_runtime.unload()
await project_runtime.close()
return results
await asyncio.gather(*[_run_project(item) for item in self.config.projects])
finally:
await global_runtime.unload()
await global_runtime.close()
return results
class MultiProjectPipeline(ProjectBatchPipeline):
pass
@@ -0,0 +1,102 @@
from __future__ import annotations
import asyncio
from typing import List, Optional
from ..common.binding import BindingManager
from ..common.collection import DataCollection
from ..common.persistence_service import PersistenceService, ScopedPersistenceView
class ProjectScopeRuntime:
"""Thin runtime container for one scope execution."""
def __init__(
self,
*,
scope: str,
persistence_service: PersistenceService,
persistence_view: ScopedPersistenceView,
local_collection: DataCollection,
remote_collection: DataCollection,
binding_manager: BindingManager,
local_datasource,
remote_datasource,
global_runtime: Optional["ProjectScopeRuntime"] = None,
owns_persistence_service: bool = False,
) -> None:
self.scope = scope
self.persistence_service = persistence_service
self.persistence_view = persistence_view
self.local_collection = local_collection
self.remote_collection = remote_collection
self.binding_manager = binding_manager
self.local_datasource = local_datasource
self.remote_datasource = remote_datasource
self.global_runtime = global_runtime
self.owns_persistence_service = owns_persistence_service
self._closed = False
@property
def persistence_backend(self):
return self.persistence_service.backend
async def load(
self,
*,
node_types: List[str],
bootstrap_binding_node_types: Optional[List[str]] = None,
enable_persistence: bool = True,
ephemeral_node_types: Optional[List[str]] = None,
) -> None:
if not enable_persistence:
return
excluded_node_types = list(dict.fromkeys(ephemeral_node_types or [])) or None
await self.local_collection.load_from_persistence_excluding(excluded_node_types)
await self.remote_collection.load_from_persistence_excluding(excluded_node_types)
excluded = set(ephemeral_node_types or [])
binding_node_types = [
node_type
for node_type in dict.fromkeys([*node_types, *(bootstrap_binding_node_types or [])])
if node_type not in excluded
]
for node_type in binding_node_types:
await self.binding_manager.load_from_persistence(node_type)
async def persist(
self,
*,
enable_persistence: bool = True,
ephemeral_node_types: Optional[List[str]] = None,
) -> None:
if not enable_persistence:
return
await self.local_collection.persist(exclude_node_types=ephemeral_node_types)
await self.remote_collection.persist(exclude_node_types=ephemeral_node_types)
await self.binding_manager.persist(exclude_node_types=ephemeral_node_types)
async def unload(self) -> None:
await self.local_collection.unload()
await self.remote_collection.unload()
await self.binding_manager.unload()
async def close(self) -> None:
if self._closed:
return
self._closed = True
await self._safe_close("remote_datasource", self.remote_datasource.close())
await self._safe_close("local_datasource", self.local_datasource.close())
if self.owns_persistence_service:
await self._safe_close("persistence_service", self.persistence_service.close())
async def _safe_close(self, name: str, close_coro, timeout: float = 3.0) -> None:
try:
await asyncio.wait_for(close_coro, timeout=timeout)
except asyncio.TimeoutError:
return
except Exception:
return
+9 -47
View File
@@ -1,18 +1,9 @@
from __future__ import annotations
import logging
from pathlib import Path
from typing import Optional
class _NodeLogFilter(logging.Filter):
def filter(self, record: logging.LogRecord) -> bool:
message = record.getMessage()
if "node=" in message:
return False
if "节点 " in message and "状态" in message:
return False
return True
from ..logging import clear_app_logger_handlers, init_app_logger
def init_pipeline_logger(
@@ -23,43 +14,14 @@ def init_pipeline_logger(
replace_handlers: bool = True,
suppress_node_logs: bool = True,
) -> logging.Logger:
logger = logging.getLogger("sync_state_machine")
logger.setLevel(level)
logger.propagate = False
if replace_handlers:
clear_pipeline_logger_handlers(logger)
formatter = logging.Formatter("%(message)s")
node_filter = _NodeLogFilter() if suppress_node_logs else None
if mirror_console:
console_handler = logging.StreamHandler()
console_handler.setLevel(level)
console_handler.setFormatter(formatter)
if node_filter is not None:
console_handler.addFilter(node_filter)
logger.addHandler(console_handler)
if log_file_path:
path = Path(log_file_path)
path.parent.mkdir(parents=True, exist_ok=True)
file_handler = logging.FileHandler(path, mode="w", encoding="utf-8")
file_handler.setLevel(level)
file_handler.setFormatter(formatter)
if node_filter is not None:
file_handler.addFilter(node_filter)
logger.addHandler(file_handler)
return logger
return init_app_logger(
log_file_path=log_file_path,
level=level,
mirror_console=mirror_console,
replace_handlers=replace_handlers,
suppress_node_logs=suppress_node_logs,
)
def clear_pipeline_logger_handlers(logger: Optional[logging.Logger] = None) -> None:
target = logger or logging.getLogger("sync_state_machine")
handlers = list(target.handlers)
for handler in handlers:
target.removeHandler(handler)
try:
handler.close()
except Exception:
pass
clear_app_logger_handlers(logger)
+33 -22
View File
@@ -1,7 +1,18 @@
from __future__ import annotations
import logging
async def print_pipeline_summary(*, logger, node_types, binding_manager, local_datasource, remote_datasource, local_collection, remote_collection, consistency_by_type, stats) -> None:
scoped_logger = logger
plain_logger = logger.logger if isinstance(logger, logging.LoggerAdapter) else logger
def _log_header(message: str) -> None:
scoped_logger.info(message)
def _log_body(message: str) -> None:
plain_logger.info(message)
binding_counts = {}
for node_type in node_types:
records = await binding_manager.get_all_records(node_type)
@@ -36,31 +47,31 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
remote_text = _clip(repr(sample.get("remote")), 72)
return f"{local_text}/{remote_text}"
logger.info(f"\n{'=' * line_width}")
logger.info("🔎 Consistency Summary")
logger.info(f"{'=' * line_width}")
_log_header(f"\n{'=' * line_width}")
_log_header("🔎 Consistency Summary")
_log_header(f"{'=' * line_width}")
for node_type in node_types:
item = consistency_by_type.get(node_type, {}) or {}
checked_pairs = int(item.get("checked_pairs", 0))
mismatch_count = int(item.get("mismatch_count", 0))
logger.info(f"{node_type} (checked_pairs={checked_pairs}, mismatches={mismatch_count})")
_log_body(f"{node_type} (checked_pairs={checked_pairs}, mismatches={mismatch_count})")
fields = item.get("fields", []) or []
differing_fields = [field for field in fields if int(field.get("mismatch_count", 0)) > 0]
if not differing_fields:
logger.info(" pass")
_log_body(" pass")
continue
logger.info(" field mismatch/total samples")
logger.info(" -----------------------------------------------------------")
_log_body(" field mismatch/total samples")
_log_body(" -----------------------------------------------------------")
for field_report in differing_fields:
field_name = str(field_report.get("field_name", ""))
field_mismatch = int(field_report.get("mismatch_count", 0))
field_total = int(field_report.get("total_count", 0))
samples = field_report.get("samples", []) or []
sample_text = _format_sample(samples[0]) if samples else "-"
logger.info(
_log_body(
f" {_clip(field_name, 30):<30} {field_mismatch:>4}/{field_total:<4} "
f"{sample_text}"
)
@@ -74,15 +85,15 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
action_w = 16
consistency_w = 18
logger.info(f"\n{'='*line_width}")
logger.info(f"📊 {title} Summary")
logger.info(f"{'='*line_width}")
logger.info("说明: Create/Update/Delete = 成功/失败/总数 (失败=FAILED+SKIPPED)")
logger.info("说明: Consistent = 一致对数/不一致对数/已检查绑定对数")
logger.info(
_log_header(f"\n{'='*line_width}")
_log_header(f"📊 {title} Summary")
_log_header(f"{'='*line_width}")
_log_body("说明: Create/Update/Delete = 成功/失败/总数 (失败=FAILED+SKIPPED)")
_log_body("说明: Consistent = 一致对数/不一致对数/已检查绑定对数")
_log_body(
f"{'Node Type':<{node_type_w}} {'Bound/Rec/Load':<{bound_w}} {'Create(S/F/T)':<{action_w}} {'Update(S/F/T)':<{action_w}} {'Delete(S/F/T)':<{action_w}} {'Consistent(S/F/T)':<{consistency_w}}"
)
logger.info(f"{'-'*line_width}")
_log_body(f"{'-'*line_width}")
total_bound_normal = 0
total_bindings = 0
@@ -105,7 +116,7 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
consistency_str = format_consistency_sft(node_type)
bound_str = f"{bound_normal_count}/{bindings}/{loaded_count}"
logger.info(
_log_body(
f"{_clip(node_type, node_type_w):<{node_type_w}} "
f"{_clip(bound_str, bound_w):<{bound_w}} "
f"{_clip(create_str, action_w):<{action_w}} "
@@ -131,14 +142,14 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
total_consistency["checked"] += int(consistency.get("checked_pairs", 0))
total_consistency["mismatch"] += int(consistency.get("mismatch_count", 0))
logger.info(f"{'-'*line_width}")
_log_body(f"{'-'*line_width}")
total_consistent = max(0, total_consistency["checked"] - total_consistency["mismatch"])
total_create_sft = f"{total_create['success']}/{total_create['failed_effective']}/{total_create['total']}"
total_update_sft = f"{total_update['success']}/{total_update['failed_effective']}/{total_update['total']}"
total_delete_sft = f"{total_delete['success']}/{total_delete['failed_effective']}/{total_delete['total']}"
total_consistency_sft = f"{total_consistent}/{total_consistency['mismatch']}/{total_consistency['checked']}"
total_bound_str = f"{total_bound_normal}/{total_bindings}/{total_loaded}"
logger.info(
_log_body(
f"{'TOTAL':<{node_type_w}} "
f"{_clip(total_bound_str, bound_w):<{bound_w}} "
f"{_clip(total_create_sft, action_w):<{action_w}} "
@@ -158,13 +169,13 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
failed = int(stats.get("failed", 0))
post_check_passed = bool(stats.get("post_check_passed", True))
logger.info("\n" + "=" * 96)
logger.info(
_log_header("\n" + "=" * 96)
_log_header(
f"📈 Pipeline Stats: loaded={loaded}, synced={synced}, failed={failed}, post_check_passed={post_check_passed}, mismatch_types={len(mismatch_node_types)}"
)
if mismatch_node_types:
logger.info(f"🔎 Mismatch Node Types: {', '.join(mismatch_node_types)}")
logger.info("=" * 96)
_log_body(f"🔎 Mismatch Node Types: {', '.join(mismatch_node_types)}")
_log_header("=" * 96)
print_consistency_summary()
print_collection_summary("Local", local_datasource, local_collection)

Some files were not shown because too many files have changed in this diff Show More