Compare commits
19 Commits
4b81b83fd8
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| ba73b30a04 | |||
| c377264610 | |||
| 4d6f499508 | |||
| 966f2e1270 | |||
| e3eab5ade1 | |||
| aa31243bae | |||
| f5729d5a18 | |||
| 543c3ff906 | |||
| 74df53889c | |||
| bb8ea931b5 | |||
| f93a6c5c68 | |||
| 9a3a34c58b | |||
| 7c49df94fc | |||
| 8f4727a772 | |||
| eb02fd32a5 | |||
| f0ce8dc37e | |||
| 14c20a6123 | |||
| bdc7bd5069 | |||
| f50dc1aed0 |
@@ -108,6 +108,7 @@ ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
|
||||
- 仓库内直接运行:`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
|
||||
- 已安装命令行入口:`ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml`
|
||||
- 多项目 JSONL 示例:`python run.py --config_path=run_profiles/preset/jsonl_to_jsonl.multi_project.default.yaml`
|
||||
- 当前已经支持多 project 运行;标准写法是直接使用多项目编排配置文件,或在支持 implicit multi-project 的 profile 中由配置本身决定运行模式。
|
||||
- 自定义配置:从 `run_profiles/preset/` 复制一份到 `run_profiles/*.local.yaml`,只改有差异的字段即可。
|
||||
|
||||
### API 同步
|
||||
@@ -144,7 +145,7 @@ ecm-sync-run --config_path=run_profiles/preset/jsonl_to_jsonl.default.yaml
|
||||
- 第一次接手仓库:`docs/getting_started.md`
|
||||
- 想理解整体分层:`docs/architecture.md`
|
||||
- 想理解状态与迁移:`docs/node_state_definition.md` + `docs/state_machine.md`
|
||||
- 想调运行配置:`docs/runtime_config_guide.md` + `run_profiles/README.md`
|
||||
- 想调运行配置:`docs/runtime_config_guide.md` + `run_profiles/README.md`(包含多项目配置写法)
|
||||
- 想接新数据源或改 handler:`sync_state_machine/datasource/README.md` + `docs/library_integration_guide.md`
|
||||
- 想排障:`docs/operations.md`
|
||||
- 想补测试或跑联调:`docs/testing_guide.md` + `tests/README.md`
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
- 负责组装并驱动整条流程:加载持久化、执行 bind/create/update、调用 datasource 同步、持久化结果。
|
||||
- reset 语义:
|
||||
- 加载阶段仅恢复持久化状态(不做状态重置);
|
||||
- 加载后通过 `BaseSyncStrategy.run_reset()` 触发 `E01`:
|
||||
- 加载后通过 `run_phase2_cleanup()` 触发 `E01`:
|
||||
- `create_zombie=true` -> `S15`(随后删除);
|
||||
- `create_zombie=false` -> `S00`(归并为 `UNCHECKED/NONE/PENDING`,并清空 `error`)。
|
||||
- 若持久化核心枚举非法(`action/status/binding_status`),节点进入加载异常隔离态 `S17`:
|
||||
@@ -26,12 +26,12 @@
|
||||
### strategy(决策组织层)
|
||||
- 外部入口:`sync_state_machine/sync_system/strategy.py`
|
||||
- 具体实现下沉:`sync_state_machine/sync_system/strategy_ops/`
|
||||
- `bind_ops.py`: `run_bind()` + `phase_core_state()/phase_dependency_check()/phase_auto_binding()`
|
||||
- `create_ops.py`: `run_create()`
|
||||
- `update_ops.py`: `run_update()`
|
||||
- `bind_ops.py`: bind 阶段静态函数与纯工具
|
||||
- `create_ops.py`: create 阶段静态函数与纯工具
|
||||
- `update_ops.py`: update 阶段静态函数与纯工具
|
||||
- `delete_ops.py`: `run_delete()`
|
||||
- `reset_ops.py`: `get_phase1_reset_defaults()/run_phase2_cleanup()`
|
||||
- strategy 负责收集上下文并调用状态机事件函数,不直接硬编码状态值。
|
||||
- `reset_ops.py`: `run_phase2_cleanup()`
|
||||
- strategy 负责 bind/create/update 的相位内部编排,并调用状态机事件函数,不直接硬编码状态值。
|
||||
|
||||
### datasource(执行与回写层)
|
||||
- 入口:`sync_state_machine/datasource/datasource.py`
|
||||
|
||||
@@ -131,8 +131,8 @@ API 数据源中,校验分为三层:
|
||||
- 例如 `ContractCreate.model_validate(...)`、`ContractUpdate.model_validate(...)`
|
||||
- load 响应也会在解析后转为领域 schema
|
||||
2. 通用更新过滤与差异识别
|
||||
- `_filter_update_data`:仅在 schema 相关字段有变化时提交更新
|
||||
- `_get_schema_diff`:用于差异字段识别与调试输出
|
||||
- `build_update_request_data`:update 入口,仅在 schema 相关字段有变化时提交更新
|
||||
- `get_update_schema_diff`:update 入口,用于差异字段识别与调试输出
|
||||
3. 轮询接口输入校验
|
||||
- `PushIdsSchema` 校验 push_ids 格式后再请求 `/push/result`
|
||||
|
||||
|
||||
@@ -46,7 +46,7 @@ pytest tests/integration/test_toolchain_config_and_graph.py -q
|
||||
- 想知道系统整体怎么分层:`docs/architecture.md`
|
||||
- 想知道状态机怎么工作:`docs/node_state_definition.md` 和 `docs/state_machine.md`
|
||||
- 想知道一次同步是怎么跑的:`docs/sync_flow.md`
|
||||
- 想调运行配置:`docs/runtime_config_guide.md`
|
||||
- 想调运行配置:`docs/runtime_config_guide.md`(包含单项目、多项目、`persist.url`、`sqlalchemy`、业务侧注册 datasource 的当前写法)
|
||||
- 想接新 datasource 或 handler:`sync_state_machine/datasource/README.md` 和 `docs/library_integration_guide.md`
|
||||
- 想看测试体系:`docs/testing_guide.md` 和 `tests/README.md`
|
||||
- 想做排障:`docs/operations.md`
|
||||
|
||||
@@ -0,0 +1,627 @@
|
||||
# 多项目同步重构方案
|
||||
|
||||
## 1. 背景和目标
|
||||
|
||||
### 背景
|
||||
|
||||
当前 `multi_project_pipeline` 的主要问题:
|
||||
|
||||
1. 顶层概念过重,在单项目同步之上又叠了一层特殊编排。
|
||||
2. 通过共享快照传递 `collection` / `binding` / 运行态,边界不清晰。
|
||||
3. 配置入口过多,存在 `global_config` / `default_project_config` / `config_path` 多层模型。
|
||||
4. 项目完成后哪些对象可以释放,没有明确生命周期。
|
||||
|
||||
### 目标
|
||||
|
||||
新方案只解决四件事:
|
||||
|
||||
1. 顶层仍然是 pipeline,但直接写新的 `ProjectBatchPipeline`。
|
||||
2. `global` 数据只同步一次,不复制到每个项目,不在每个项目中重复校验。
|
||||
3. 改动尽量限制在 `persistence / collection / binding_manager / datasource / pipeline` 这些基础层。
|
||||
4. 项目级运行态支持 `load -> sync -> persist -> unload -> close`。
|
||||
|
||||
### 一个明确结论
|
||||
|
||||
这里不建议先做复杂的 session 框架。
|
||||
|
||||
第一阶段最合适的方式是:
|
||||
|
||||
1. 在 `sync_project(...)` 里创建一个很薄的 `ProjectScopeRuntime`。
|
||||
2. 它只是把“当前项目 scope 下的一组对象”收拢起来。
|
||||
3. 如果你临时不想单独建类,也可以先写成函数内局部变量。
|
||||
|
||||
但从清理顺序和可读性考虑,推荐保留 `ProjectScopeRuntime` 这个薄对象。
|
||||
|
||||
它不是复杂 session 系统,本质上就是“单个项目的一次运行上下文”。
|
||||
|
||||
## 2. 新的 Batch Project Pipeline 和伪代码
|
||||
|
||||
### 结构结论
|
||||
|
||||
顶层对象建议是:`ProjectBatchPipeline`
|
||||
|
||||
它负责:
|
||||
|
||||
1. 跑一次 global phase。
|
||||
2. 并发调度所有项目。
|
||||
3. 管理 batch 级共享对象。
|
||||
4. 管理项目级对象的创建和清理。
|
||||
|
||||
它不负责:
|
||||
|
||||
1. 业务字段映射。
|
||||
2. handler 细节。
|
||||
3. strategy 逻辑。
|
||||
|
||||
### 共享对象和项目级对象
|
||||
|
||||
#### 整个 batch 共享的对象
|
||||
|
||||
1. `BatchConfig`
|
||||
2. `StateMachineRuntime`
|
||||
3. `Logger`
|
||||
4. `PersistenceService`
|
||||
5. `global` 只读运行态
|
||||
6. datasource 的共享静态资源,例如 API client、认证信息、限流器、handler registry
|
||||
7. semaphore 和任务调度器
|
||||
|
||||
#### 每个项目内创建的对象
|
||||
|
||||
1. `ProjectScopeRuntime`
|
||||
2. `local_collection`
|
||||
3. `remote_collection`
|
||||
4. `binding_manager`
|
||||
5. 当前项目 scope 对应的 persistence view
|
||||
6. 当前项目的 local datasource runtime
|
||||
7. 当前项目的 remote datasource runtime
|
||||
8. 当前项目统计对象
|
||||
|
||||
### Global 数据处理原则
|
||||
|
||||
这点直接定死:
|
||||
|
||||
1. `global` 数据不复制到每个 project scope。
|
||||
2. `global` 数据不在每个项目里重复 bootstrap。
|
||||
3. `global` 数据不在每个项目里重复 post-check。
|
||||
|
||||
项目运行时如果需要共享节点或绑定:
|
||||
|
||||
1. 先查当前 project scope。
|
||||
2. 未命中再回退到 `global` 只读运行态。
|
||||
|
||||
### 伪代码
|
||||
|
||||
```python
|
||||
class ProjectBatchPipeline:
|
||||
def __init__(self, config: BatchConfig):
|
||||
self.config = config
|
||||
self.runtime = build_state_machine_runtime()
|
||||
self.logger = build_logger(config.logging)
|
||||
self.persistence = PersistenceService(config.persist)
|
||||
self.global_runtime = None
|
||||
self.shared_local_ds = build_shared_local_datasource_resources(config.local_datasource)
|
||||
self.shared_remote_ds = build_shared_remote_datasource_resources(config.remote_datasource)
|
||||
|
||||
async def run(self) -> None:
|
||||
self.global_runtime = await self.sync_global()
|
||||
await self.dispatch_projects()
|
||||
|
||||
async def sync_global(self) -> "GlobalReadonlyRuntime":
|
||||
...
|
||||
|
||||
async def dispatch_projects(self) -> None:
|
||||
...
|
||||
|
||||
async def sync_project(self, local_project_id: str, remote_project_id: str) -> None:
|
||||
scope_runtime = await self.create_project_scope_runtime(local_project_id, remote_project_id)
|
||||
try:
|
||||
await scope_runtime.load()
|
||||
await run_project_sync(scope_runtime, self.config.project_node_types)
|
||||
await scope_runtime.persist()
|
||||
finally:
|
||||
await scope_runtime.unload()
|
||||
await scope_runtime.close()
|
||||
```
|
||||
|
||||
```python
|
||||
class ProjectScopeRuntime:
|
||||
def __init__(self, scope: str, persistence_view, global_runtime, shared_local_ds, shared_remote_ds):
|
||||
self.scope = scope
|
||||
self.persistence_view = persistence_view
|
||||
self.local_collection = DataCollection(...)
|
||||
self.remote_collection = DataCollection(...)
|
||||
self.binding_manager = BindingManager(...)
|
||||
self.local_datasource = build_local_datasource_runtime(shared_local_ds, self.local_collection)
|
||||
self.remote_datasource = build_remote_datasource_runtime(shared_remote_ds, self.remote_collection)
|
||||
self.global_runtime = global_runtime
|
||||
|
||||
async def load(self) -> None: ...
|
||||
async def persist(self) -> None: ...
|
||||
async def unload(self) -> None: ...
|
||||
async def close(self) -> None: ...
|
||||
```
|
||||
|
||||
## 3. 组件说明
|
||||
|
||||
### 3.1 Persistence
|
||||
|
||||
#### 角色
|
||||
|
||||
`PersistenceService` 是 batch 共享对象。
|
||||
|
||||
它共享的是:
|
||||
|
||||
1. 后端类型。
|
||||
2. DB 路径或连接串。
|
||||
3. 连接池或底层连接能力。
|
||||
|
||||
它不共享的是:
|
||||
|
||||
1. 当前项目的内存缓存。
|
||||
2. 当前 scope 的脏状态。
|
||||
|
||||
#### 关键语义
|
||||
|
||||
建议把 persistence 理解成两层:
|
||||
|
||||
1. `PersistenceService`
|
||||
- 共享后端服务。
|
||||
2. `ScopedPersistenceView`
|
||||
- 绑定某个 `scope` 的轻量 view。
|
||||
|
||||
所以“scope 属于 persistence 的 scope view”的意思是:
|
||||
|
||||
1. 整个 batch 共享一个 persistence service。
|
||||
2. 每个项目拿一个绑定了当前 `scope` 的轻量操作视图。
|
||||
|
||||
不是说每个项目都新建一套 persistence backend。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class PersistenceService:
|
||||
async def initialize(self) -> None: ...
|
||||
def view(self, scope: str) -> ScopedPersistenceView: ...
|
||||
async def close(self) -> None: ...
|
||||
|
||||
|
||||
class ScopedPersistenceView:
|
||||
async def load_nodes(self, collection_id: str) -> list[dict]: ...
|
||||
async def save_nodes(self, collection_id: str, nodes: list[SyncNode]) -> None: ...
|
||||
async def delete_nodes(self, collection_id: str, node_ids: list[str]) -> None: ...
|
||||
async def load_bindings(self, node_type: str) -> list[dict]: ...
|
||||
async def save_bindings(self, node_type: str, bindings: dict[str, str | None]) -> None: ...
|
||||
```
|
||||
|
||||
### 3.2 Collection
|
||||
|
||||
#### 角色
|
||||
|
||||
`Collection` 必须是 scope 级对象,不能做单例。
|
||||
|
||||
原因:
|
||||
|
||||
1. 它内部天然持有当前 scope 的节点集合。
|
||||
2. 它持有 data_id 索引。
|
||||
3. 它持有删除集合和脏状态。
|
||||
|
||||
#### 关键语义
|
||||
|
||||
project scope 的 `Collection` 只存项目域数据。
|
||||
|
||||
对于共享数据,不复制 global,而是支持只读回退:
|
||||
|
||||
1. 先查当前 scope。
|
||||
2. 未命中再查 global 只读 collection。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class DataCollection:
|
||||
async def load_from_persistence(self) -> None: ...
|
||||
async def persist(self) -> None: ...
|
||||
async def unload(self) -> None: ...
|
||||
|
||||
def get(self, node_id: str) -> SyncNode | None: ...
|
||||
|
||||
def get_by_data_id(self, node_type: str, data_id: str) -> SyncNode | None:
|
||||
node = self._get_local(node_type, data_id)
|
||||
if node is not None:
|
||||
return node
|
||||
return self._get_global_fallback(node_type, data_id)
|
||||
```
|
||||
|
||||
### 3.3 BindingManager
|
||||
|
||||
#### 角色
|
||||
|
||||
`BindingManager` 也是 scope 级对象,不应共享。
|
||||
|
||||
它负责:
|
||||
|
||||
1. 当前 scope 的绑定运行态。
|
||||
2. 当前 scope 的绑定持久化。
|
||||
3. 当前 scope 未命中时对 global 只读绑定做回退查询。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class BindingManager:
|
||||
async def load_from_persistence(self, node_type: str) -> None: ...
|
||||
async def persist(self) -> None: ...
|
||||
async def unload(self) -> None: ...
|
||||
|
||||
async def get_remote_id(self, node_type: str, local_id: str) -> str | None:
|
||||
remote_id = self._get_local_remote_id(node_type, local_id)
|
||||
if remote_id is not None:
|
||||
return remote_id
|
||||
return await self._get_global_remote_id(node_type, local_id)
|
||||
```
|
||||
|
||||
### 3.4 DataSource
|
||||
|
||||
#### 角色
|
||||
|
||||
`local_datasource / remote_datasource / logging` 不应按项目变化,这一点成立。
|
||||
|
||||
但按当前实现,`BaseDataSource` 不适合直接做单例,因为它内部持有:
|
||||
|
||||
1. `_collection`
|
||||
2. `_stats`
|
||||
3. `_action_summary`
|
||||
4. `_action_by_node_id`
|
||||
|
||||
#### 推荐的最小改法
|
||||
|
||||
第一阶段不强行拆成两个正式类,而是采用下面的语义:
|
||||
|
||||
1. datasource 的静态配置和共享 client 是 batch 级共享的。
|
||||
2. datasource 的运行态对象仍然是项目级创建的。
|
||||
|
||||
也就是说:
|
||||
|
||||
1. 配置不按项目变化。
|
||||
2. runtime 仍然按项目隔离。
|
||||
|
||||
这样改动范围最小,也更不容易污染现有 handler 行为。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class ScopedDataSourceRuntime(BaseDataSource):
|
||||
def set_collection(self, collection: DataCollection) -> None: ...
|
||||
async def load_all(self, order: list[str]) -> None: ...
|
||||
async def sync_all(self, order: list[str]) -> None: ...
|
||||
def reset_runtime_state(self) -> None: ...
|
||||
async def close(self) -> None: ...
|
||||
```
|
||||
|
||||
## 4. 改进路线图
|
||||
|
||||
### 第一阶段:先收敛架构边界
|
||||
|
||||
只改基础层,不动业务层:
|
||||
|
||||
1. 新建 `ProjectBatchPipeline`。
|
||||
2. 明确 `global` 和 `project_id:<local_project_id>` 语义。
|
||||
3. 引入 `ProjectScopeRuntime` 这个薄的项目运行上下文。
|
||||
4. 给 `Collection` 和 `BindingManager` 增加 global fallback 语义。
|
||||
5. persistence 侧增加共享 service + scope view 语义。
|
||||
|
||||
### 第二阶段:清理生命周期
|
||||
|
||||
把项目级清理流程固定下来:
|
||||
|
||||
1. `load`
|
||||
2. `sync`
|
||||
3. `persist`
|
||||
4. `unload`
|
||||
5. `close`
|
||||
|
||||
### 第三阶段:再决定 datasource 是否正式拆层
|
||||
|
||||
如果第一阶段跑通,再决定是否把 datasource 正式拆成:
|
||||
|
||||
1. 共享静态层。
|
||||
2. scope runtime 层。
|
||||
|
||||
这一步不是第一批必须完成。
|
||||
|
||||
### 最终落点
|
||||
|
||||
最终希望达到的状态:
|
||||
|
||||
1. global 数据只同步一次,只读共享。
|
||||
2. project 数据按 scope 独立运行。
|
||||
3. 项目跑完即可卸载运行态。
|
||||
4. 大部分改动都限制在基础设施层。
|
||||
8. 当前项目统计对象
|
||||
|
||||
### 2.3 为什么推荐 `ProjectScopeRuntime`
|
||||
|
||||
你问的是:要不要做 session 对象,还是做子 pipeline,还是都放函数里。
|
||||
|
||||
建议是:
|
||||
|
||||
1. 不做新的复杂 session 框架。
|
||||
2. 不做 `FullSyncPipeline` 风格的子 pipeline 继承树。
|
||||
3. 做一个很薄的 `ProjectScopeRuntime`。
|
||||
|
||||
它的作用只有两个:
|
||||
|
||||
1. 把当前项目的一组 scope 对象收拢到一起。
|
||||
2. 让清理顺序明确,不要把资源散在 `sync_project(...)` 的局部变量里。
|
||||
|
||||
如果你坚持更简单,也可以第一版直接写在 `sync_project(...)` 里;但文档层面仍建议用 `ProjectScopeRuntime` 表达这个层次。
|
||||
|
||||
### 2.4 伪代码
|
||||
|
||||
```python
|
||||
class ProjectBatchPipeline:
|
||||
def __init__(self, config: BatchConfig):
|
||||
self.config = config
|
||||
self.runtime = build_state_machine_runtime()
|
||||
self.logger = build_logger(config.logging)
|
||||
self.persistence = PersistenceService(config.persist)
|
||||
self.global_runtime = None
|
||||
self.shared_local_ds = build_shared_local_datasource_resources(config.local_datasource)
|
||||
self.shared_remote_ds = build_shared_remote_datasource_resources(config.remote_datasource)
|
||||
|
||||
async def run(self) -> None:
|
||||
self.global_runtime = await self.sync_global()
|
||||
await self.dispatch_projects()
|
||||
|
||||
async def sync_global(self) -> "GlobalReadonlyRuntime":
|
||||
# 只同步 shared_node_types
|
||||
...
|
||||
|
||||
async def dispatch_projects(self) -> None:
|
||||
# semaphore 控制并发
|
||||
...
|
||||
|
||||
async def sync_project(self, local_project_id: str, remote_project_id: str) -> None:
|
||||
scope_runtime = await self.create_project_scope_runtime(local_project_id, remote_project_id)
|
||||
try:
|
||||
await scope_runtime.load()
|
||||
await run_project_sync(scope_runtime, self.config.project_node_types)
|
||||
await scope_runtime.persist()
|
||||
finally:
|
||||
await scope_runtime.unload()
|
||||
await scope_runtime.close()
|
||||
```
|
||||
|
||||
```python
|
||||
class ProjectScopeRuntime:
|
||||
def __init__(self, scope: str, persistence_view, global_runtime, shared_local_ds, shared_remote_ds):
|
||||
self.scope = scope
|
||||
self.persistence_view = persistence_view
|
||||
self.local_collection = DataCollection(...)
|
||||
self.remote_collection = DataCollection(...)
|
||||
self.binding_manager = BindingManager(...)
|
||||
self.local_datasource = build_local_datasource_runtime(shared_local_ds, self.local_collection)
|
||||
self.remote_datasource = build_remote_datasource_runtime(shared_remote_ds, self.remote_collection)
|
||||
self.global_runtime = global_runtime
|
||||
|
||||
async def load(self) -> None: ...
|
||||
async def persist(self) -> None: ...
|
||||
async def unload(self) -> None: ...
|
||||
async def close(self) -> None: ...
|
||||
```
|
||||
|
||||
### 2.5 Global 数据的处理原则
|
||||
|
||||
这点现在可以直接定死:
|
||||
|
||||
1. `global` 数据不复制到每个 project scope。
|
||||
2. `global` 数据不在每个项目里重复 bootstrap。
|
||||
3. `global` 数据不在每个项目里重复 post-check。
|
||||
|
||||
项目级运行时如果需要共享节点或绑定,采用:
|
||||
|
||||
1. 当前 project scope 先查本地运行态。
|
||||
2. 未命中时回退到 `global` 只读运行态。
|
||||
|
||||
这样可以避免复制大体量 global 数据。
|
||||
|
||||
2. `preload_shared_state(...)` 把项目 pipeline 变成了“半冷启动、半热注入”的特殊运行态。
|
||||
3. `ephemeral_node_types` / `bootstrap_binding_node_types` 这些参数开始承担原本不属于它们的职责。
|
||||
## 3. 组件说明
|
||||
|
||||
### 3.1 Persistence
|
||||
|
||||
#### 角色
|
||||
|
||||
`PersistenceService` 是 batch 共享对象。
|
||||
|
||||
它共享的是:
|
||||
|
||||
1. 后端类型。
|
||||
2. DB 路径或连接串。
|
||||
3. 连接池或底层连接能力。
|
||||
|
||||
它不共享的是:
|
||||
|
||||
1. 当前项目的内存缓存。
|
||||
2. 当前 scope 的脏状态。
|
||||
|
||||
#### 关键语义
|
||||
|
||||
建议把 `persistence` 理解成两层:
|
||||
|
||||
1. `PersistenceService`
|
||||
- 共享后端服务。
|
||||
2. `ScopedPersistenceView`
|
||||
- 绑定某个 `scope` 的轻量 view。
|
||||
|
||||
所以“scope 属于 persistence 的 scope view”这句话的意思是:
|
||||
|
||||
1. 整个 batch 共享一个 persistence service。
|
||||
2. 每个项目拿一个绑定了当前 `scope` 的轻量操作视图。
|
||||
|
||||
不是说每个项目都新建一套 persistence backend。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class PersistenceService:
|
||||
async def initialize(self) -> None: ...
|
||||
def view(self, scope: str) -> ScopedPersistenceView: ...
|
||||
async def close(self) -> None: ...
|
||||
|
||||
|
||||
class ScopedPersistenceView:
|
||||
async def load_nodes(self, collection_id: str) -> list[dict]: ...
|
||||
async def save_nodes(self, collection_id: str, nodes: list[SyncNode]) -> None: ...
|
||||
async def delete_nodes(self, collection_id: str, node_ids: list[str]) -> None: ...
|
||||
async def load_bindings(self, node_type: str) -> list[dict]: ...
|
||||
async def save_bindings(self, node_type: str, bindings: dict[str, str | None]) -> None: ...
|
||||
```
|
||||
|
||||
### 3.2 Collection
|
||||
|
||||
#### 角色
|
||||
|
||||
`Collection` 必须是 scope 级对象,不能做单例。
|
||||
|
||||
原因很简单:
|
||||
|
||||
1. 它内部天然持有当前 scope 的节点集合。
|
||||
2. 它持有 data_id 索引。
|
||||
3. 它持有删除集合和脏状态。
|
||||
|
||||
#### 关键语义
|
||||
|
||||
project scope 的 `Collection` 只存项目域数据。
|
||||
|
||||
对于共享数据,不复制 global,而是支持只读回退:
|
||||
|
||||
1. 先查当前 scope。
|
||||
2. 未命中再查 global 只读 collection。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class DataCollection:
|
||||
async def load_from_persistence(self) -> None: ...
|
||||
async def persist(self) -> None: ...
|
||||
async def unload(self) -> None: ...
|
||||
|
||||
def get(self, node_id: str) -> SyncNode | None: ...
|
||||
def get_by_data_id(self, node_type: str, data_id: str) -> SyncNode | None:
|
||||
node = self._get_local(node_type, data_id)
|
||||
if node is not None:
|
||||
return node
|
||||
return self._get_global_fallback(node_type, data_id)
|
||||
```
|
||||
|
||||
### 3.3 BindingManager
|
||||
|
||||
#### 角色
|
||||
|
||||
`BindingManager` 也是 scope 级对象,不应共享。
|
||||
|
||||
它负责:
|
||||
|
||||
1. 当前 scope 的绑定运行态。
|
||||
2. 当前 scope 的绑定持久化。
|
||||
3. 当前 scope 未命中时对 global 只读绑定做回退查询。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class BindingManager:
|
||||
async def load_from_persistence(self, node_type: str) -> None: ...
|
||||
async def persist(self) -> None: ...
|
||||
async def unload(self) -> None: ...
|
||||
|
||||
async def get_remote_id(self, node_type: str, local_id: str) -> str | None:
|
||||
remote_id = self._get_local_remote_id(node_type, local_id)
|
||||
if remote_id is not None:
|
||||
return remote_id
|
||||
return await self._get_global_remote_id(node_type, local_id)
|
||||
```
|
||||
|
||||
### 3.4 DataSource
|
||||
|
||||
#### 角色
|
||||
|
||||
`local_datasource / remote_datasource / logging` 不应按项目变化,这一点是对的。
|
||||
|
||||
但按当前实现,`BaseDataSource` 也不适合直接做单例,因为它内部持有:
|
||||
|
||||
1. `_collection`
|
||||
2. `_stats`
|
||||
3. `_action_summary`
|
||||
4. `_action_by_node_id`
|
||||
|
||||
所以第一阶段不建议“直接把当前 `BaseDataSource` 改成单例 + 注入 scope”。
|
||||
|
||||
#### 推荐的最小改法
|
||||
|
||||
第一阶段先不强行拆成两个正式类,而是采用下面的语义:
|
||||
|
||||
1. datasource 的静态配置和共享 client 是 batch 级共享的。
|
||||
2. datasource 的运行态对象仍然是项目级创建的。
|
||||
|
||||
也就是说:
|
||||
|
||||
1. 配置不按项目变化。
|
||||
2. runtime 仍然按项目隔离。
|
||||
|
||||
这样改动范围最小,也更不容易污染现有 handler 行为。
|
||||
|
||||
#### 主要方法伪代码
|
||||
|
||||
```python
|
||||
class ScopedDataSourceRuntime(BaseDataSource):
|
||||
def set_collection(self, collection: DataCollection) -> None: ...
|
||||
async def load_all(self, order: list[str]) -> None: ...
|
||||
async def sync_all(self, order: list[str]) -> None: ...
|
||||
def reset_runtime_state(self) -> None: ...
|
||||
async def close(self) -> None: ...
|
||||
```
|
||||
|
||||
当前多项目配置要求:
|
||||
|
||||
1. `global_config`
|
||||
### 第二阶段:清理生命周期
|
||||
|
||||
把项目级清理流程固定下来:
|
||||
|
||||
1. `load`
|
||||
2. `sync`
|
||||
3. `persist`
|
||||
4. `unload`
|
||||
5. `close`
|
||||
|
||||
### 第三阶段:再决定 datasource 是否正式拆层
|
||||
|
||||
如果第一阶段跑通,可以再决定是否把 datasource 正式拆成:
|
||||
|
||||
1. 共享静态层。
|
||||
2. scope runtime 层。
|
||||
|
||||
这一步不是必须第一批完成。
|
||||
|
||||
### 最终落点
|
||||
|
||||
最终希望达到的状态是:
|
||||
|
||||
1. global 数据只同步一次,只读共享。
|
||||
2. project 数据按 scope 独立运行。
|
||||
3. 项目跑完即可卸载运行态。
|
||||
4. 大部分改动都限制在基础设施层。
|
||||
1. 全局阶段一份配置。
|
||||
2. 项目阶段一份默认配置。
|
||||
3. 每个项目还能再覆盖一份配置。
|
||||
|
||||
这对于少量项目还能接受,但项目数到了几百个就不可维护了。
|
||||
|
||||
如果每个项目都允许自带一份 pipeline 配置,系统就失去“批量同步”的基本前提:
|
||||
|
||||
1. 无法判断哪些差异是业务差异,哪些是配置漂移。
|
||||
2. 无法稳定复用并发调度逻辑。
|
||||
3. 无法保证结果可复现。
|
||||
|
||||
结论是:多项目批量同步不应允许“每项目一套 pipeline 定义”。
|
||||
|
||||
@@ -37,6 +37,10 @@
|
||||
| `ABNORMAL` | 数据损坏或核心绑定异常 | 否 |
|
||||
| `DEPENDENCY_ERROR` | 依赖不可满足 | 否 |
|
||||
|
||||
bootstrap 特殊语义:
|
||||
- 如果节点在持久化恢复后存在绑定,但 bootstrap datasource refresh 未返回最新数据,则节点进入 `S17` 语义并被 `bootstrap_blocked` 隔离。
|
||||
- 如果节点仅残留在 persistence 中、没有绑定,且 datasource refresh 也未返回,则节点按 stale zombie 清理,不再进入 bind。
|
||||
|
||||
## 3. create / update 语义
|
||||
|
||||
### create
|
||||
@@ -64,6 +68,11 @@
|
||||
- 状态落地:`Decision.to_state` 对应 YAML `states[Sxx]`。
|
||||
- 不变量:每次落地后执行 `runtime.check_invariants()`。
|
||||
|
||||
bootstrap 额外输入:
|
||||
- `is_create_zombie`
|
||||
- `bootstrap_source_present`
|
||||
- `has_binding_record`
|
||||
|
||||
## 6. 关键一致性约束
|
||||
|
||||
- `UNCHECKED` 不允许带动作(`INV-1_UNCHECKED_ACTION_NONE`)。
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
# ECM Sync Quality Cleanup Execution Plan
|
||||
|
||||
## Goal
|
||||
|
||||
This cleanup pass keeps behavior unchanged and prioritizes deleting, merging, and unifying duplicated code on the single-pipeline path.
|
||||
|
||||
Acceptance remains unchanged:
|
||||
|
||||
- `backend/local_tests/sync_system_tests/depm_to_ecm_bootstrap_pull_then_project_push.pipeline.yaml` must still run successfully.
|
||||
- Local Summary and Remote Summary must keep the same structure and stable output shape.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Do not modify `schemas/`.
|
||||
- Do not modify `filtered_datasource/`.
|
||||
- Do not bypass schema validation.
|
||||
- Do not add fallback logic that changes behavior.
|
||||
- Prefer helper extraction, deletion, and merge over new abstraction layers.
|
||||
|
||||
## Execution Order
|
||||
|
||||
### Batch 1: Entry + Factory
|
||||
|
||||
Status: completed
|
||||
|
||||
- Turn `run.py` into a thin wrapper around `sync_state_machine.cli.main`.
|
||||
- Merge duplicated datasource lifecycle and resolved-config logging in `sync_state_machine/pipeline/factory.py`.
|
||||
- Verify with focused unit tests and the real DEPM to ECM profile.
|
||||
|
||||
### Batch 2: Pipeline Flow + Backend Runner
|
||||
|
||||
Status: completed
|
||||
|
||||
- Reduce repeated phase-completion, strategy-loop, reload-skip, and close-handling code in `sync_state_machine/pipeline/full_sync_pipeline.py`.
|
||||
- Reduce repeated orchestration code in `backend/app/thirdparty/ecm_sync/scripts/depm_to_ecm/run_pipeline.py` without changing CLI behavior or fail-fast rules.
|
||||
- Keep log text stable where practical.
|
||||
|
||||
### Batch 3: Datasource + Handler Base Cleanup
|
||||
|
||||
Status: completed
|
||||
|
||||
- Reduce duplicated handler registration and context injection between datasource base classes and subclasses.
|
||||
- Reduce duplicated node metadata initialization across handler subclasses by pushing safe defaults into handler base classes.
|
||||
- Delete domain-level constructor boilerplate where subclasses only repeat node type, node class, schema, or no-op `_create_node` wrappers.
|
||||
|
||||
### Batch 4: Domain Repeat Cleanup + Docs
|
||||
|
||||
Status: in progress
|
||||
|
||||
- Inspect domain handlers for repeated load/update helper logic and merge only stable repeated fragments into shared helpers.
|
||||
- Update docs so they only describe the real maintained execution path.
|
||||
- Keep backend runner documentation aligned with current script behavior.
|
||||
- Remove stale references only after code paths are confirmed stable.
|
||||
|
||||
## Validation Plan
|
||||
|
||||
1. Run targeted `ecm_sync_system` unit tests for the touched pipeline code.
|
||||
2. Run the real backend profile with the backend interpreter.
|
||||
3. Confirm the output still ends with pipeline completion plus Local Summary and Remote Summary tables.
|
||||
|
||||
## Current Focus
|
||||
|
||||
Current implementation focus is Batch 4.
|
||||
|
||||
- repeated load/update helpers under `sync_state_machine/domain/`
|
||||
- maintained execution-path docs under `docs/`
|
||||
|
||||
The intent is code reduction only, not behavior redesign.
|
||||
+419
-209
@@ -1,298 +1,508 @@
|
||||
# 运行配置业务说明
|
||||
# 运行配置写法说明
|
||||
|
||||
本文档从业务视角说明配置字段含义、对阶段行为的影响,以及推荐的覆盖方式。
|
||||
这份文档只讲一件事:现在 `ecm_sync_system` 的配置文件应该怎么写。
|
||||
|
||||
## 0. 配置约定
|
||||
重点不是把所有默认值抄一遍,而是说明当前真实生效的写法、哪些字段是内置能力、哪些字段来自业务侧扩展,以及单项目和多项目文件各自长什么样。
|
||||
|
||||
当前运行配置分成两层:
|
||||
## 1. 先分清两类配置文件
|
||||
|
||||
1. 输入配置
|
||||
- 你写在 `run_profiles/*.yaml` 或 `run_profiles/preset/*.default.yaml` 里的内容。
|
||||
- 只建议写必要覆盖项,不要把默认值全部展开。
|
||||
2. Resolved Full Config
|
||||
- pipeline 启动时打印并写入日志的全量配置。
|
||||
- 其中已经补齐 datasource 默认值、策略默认值和日志/持久化默认值。
|
||||
当前有两类 YAML:
|
||||
|
||||
也就是说:
|
||||
- 模板应当尽量短。
|
||||
- 排障时看启动日志里的 **Resolved Full Config**。
|
||||
1. 单 pipeline 配置
|
||||
- 对应 `PipelineRunConfig`
|
||||
- 用来描述一条完整同步链路
|
||||
- 顶层通常包含:`node_types`、`local_datasource`、`remote_datasource`、`persist`、`logging`、`strategies`
|
||||
2. 多项目编排配置
|
||||
- 对应 `MultiProjectRunConfig`
|
||||
- 只负责描述“先跑全局,再按项目并发跑哪些项目”
|
||||
- 顶层包含:`global_config`、`default_project_config`、`max_concurrency`、`project_bootstrap_node_types`、`projects`
|
||||
|
||||
### 0.1 一个最小输入示例
|
||||
判断规则很简单:
|
||||
|
||||
- 如果文件里有 `global_config`、`default_project_config`、`projects`,它就是多项目编排配置。
|
||||
- 否则按单 pipeline 配置解析。
|
||||
|
||||
## 2. 配置加载约定
|
||||
|
||||
当前约定如下:
|
||||
|
||||
- 跟踪在仓库里的模板和示例,只写必要覆盖项,不重复展开默认值。
|
||||
- pipeline 启动后会打印一份 `Resolved Full Config`,它才是最终生效的完整配置。
|
||||
- 排障优先看 `Resolved Full Config`,不要只盯着输入 YAML。
|
||||
|
||||
## 3. 占位符规则
|
||||
|
||||
`ecm_sync_system` 核心库内置只认识一个占位符:
|
||||
|
||||
- `${PROJECT_ROOT}`:运行时替换为 `ecm_sync_system` 仓库根目录绝对路径。
|
||||
|
||||
例如:
|
||||
|
||||
```yaml
|
||||
persist:
|
||||
backend: sqlite
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
|
||||
```
|
||||
|
||||
需要特别区分的是:
|
||||
|
||||
- `${BACKEND_ROOT}`、`${ECM_SYNC_LOG_DIR}` 这类占位符不是核心库通用语义。
|
||||
- 它们通常是 backend 集成脚本在读取 profile 时额外替换的调用方语义。
|
||||
- 如果你不是通过 backend 的联调脚本运行,而是直接在 `ecm_sync_system` 内跑 profile,就不要假设这些占位符一定可用。
|
||||
|
||||
## 4. 单 pipeline 配置的当前写法
|
||||
|
||||
### 4.1 最小骨架
|
||||
|
||||
```yaml
|
||||
node_types:
|
||||
- company
|
||||
- supplier
|
||||
- project
|
||||
|
||||
local_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
|
||||
|
||||
remote_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: ${PROJECT_ROOT}/remote_datasource/datasource
|
||||
type: api
|
||||
api_base_url: http://127.0.0.1:8000/api/third/v2
|
||||
api_uid: your_uid
|
||||
api_secret: your_secret
|
||||
|
||||
persist:
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
|
||||
backend: sqlite
|
||||
backend_options: {}
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
|
||||
|
||||
logging:
|
||||
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
|
||||
file_name_pattern: demo_{timestamp}.log
|
||||
|
||||
strategies:
|
||||
project:
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- name
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
```
|
||||
|
||||
### 0.2 对应的全量 resolved 结果骨架
|
||||
### 4.2 顶层字段含义
|
||||
|
||||
- `node_types`
|
||||
- 本轮参与同步的节点类型列表。
|
||||
- 顺序会影响执行顺序,依赖链通常要把上游类型放前面。
|
||||
- `local_datasource` / `remote_datasource`
|
||||
- 定义两侧数据源类型和对应参数。
|
||||
- `persist`
|
||||
- 定义持久化后端及目标位置。
|
||||
- `logging`
|
||||
- 定义日志初始化、落盘位置和级别。
|
||||
- `strategies`
|
||||
- 定义每个 `node_type` 的同步策略。
|
||||
|
||||
## 5. datasource 字段怎么写
|
||||
|
||||
### 5.1 内置 datasource type
|
||||
|
||||
核心库内置注册了两种 datasource type:
|
||||
|
||||
1. `jsonl`
|
||||
2. `api`
|
||||
|
||||
#### `jsonl` 写法
|
||||
|
||||
```yaml
|
||||
node_types:
|
||||
- company
|
||||
- supplier
|
||||
- user
|
||||
- project
|
||||
...
|
||||
local_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: /abs/path/filtered_datasource/datasource/filtered
|
||||
jsonl_dir: ${PROJECT_ROOT}/filtered_datasource/datasource/filtered
|
||||
read_only: false
|
||||
handler_configs: {}
|
||||
handler_configs:
|
||||
project:
|
||||
data_id_filter:
|
||||
- project-id-1
|
||||
```
|
||||
|
||||
字段说明:
|
||||
|
||||
- `jsonl_dir`:JSONL 数据目录。
|
||||
- `read_only`:是否只读。
|
||||
- `handler_configs`:按 `node_type` 下发给 handler 的扩展参数。
|
||||
|
||||
#### `api` 写法
|
||||
|
||||
```yaml
|
||||
remote_datasource:
|
||||
type: jsonl
|
||||
jsonl_dir: /abs/path/remote_datasource/datasource
|
||||
read_only: false
|
||||
handler_configs: {}
|
||||
type: api
|
||||
api_base_url: http://127.0.0.1:8000/api/third/v2
|
||||
api_uid: your_uid
|
||||
api_secret: your_secret
|
||||
api_debug: true
|
||||
poll_max_retries: 10
|
||||
poll_interval: 0.5
|
||||
api_rate_limit_max_requests: 30
|
||||
api_rate_limit_window_seconds: 10.0
|
||||
handler_configs:
|
||||
project:
|
||||
data_id_filter:
|
||||
- remote-project-id-1
|
||||
```
|
||||
|
||||
字段说明:
|
||||
|
||||
- `api_debug`:开启后输出更详细的请求日志。
|
||||
- `poll_max_retries` / `poll_interval`:异步任务轮询控制。
|
||||
- `api_rate_limit_max_requests` / `api_rate_limit_window_seconds`:全局限流窗口。
|
||||
- `handler_configs`:同样按 `node_type` 下发业务参数。
|
||||
|
||||
### 5.2 业务侧注册的 datasource type
|
||||
|
||||
核心库允许业务系统在启动阶段注册自己的 datasource type。
|
||||
|
||||
因此,配置里不只可以写 `jsonl` 和 `api`,还可以写业务侧注册出来的类型,例如 backend 里的 `depm_db`。
|
||||
|
||||
当前 backend 的 `depm_db` 真实写法是:
|
||||
|
||||
```yaml
|
||||
local_datasource:
|
||||
type: depm_db
|
||||
tenant_id: "019976df-0628-7c46-a896-81d3f290a7df"
|
||||
app:
|
||||
config_file: ${BACKEND_ROOT}/config.yaml
|
||||
overrides:
|
||||
database:
|
||||
url: mysql+aiomysql://root:123456@127.0.0.1/depm_prod_local_auto_test?charset=utf8mb4
|
||||
handler_configs:
|
||||
project:
|
||||
data_id_filter:
|
||||
- 019a62a5-2323-7be2-99c8-82534f8befa4
|
||||
```
|
||||
|
||||
这个例子说明两点:
|
||||
|
||||
1. datasource 的可写字段由对应注册模型决定,不是所有 type 都共用同一套字段。
|
||||
2. 如果 `type` 没有先被注册,配置校验会直接报错,不会做降级处理。
|
||||
|
||||
`depm_db` 这类业务扩展字段需要以业务侧实现为准。对 backend 当前实现来说:
|
||||
|
||||
- `tenant_id`:DEPM 本地 datasource 的租户过滤 ID。
|
||||
- `app.config_file`:backend 配置文件路径,默认可落回 `backend/config.yaml`。
|
||||
- `app.overrides`:直接传给 backend `Settings()` 的覆盖项。
|
||||
|
||||
## 6. strategy 写法
|
||||
|
||||
### 6.1 当前推荐写法
|
||||
|
||||
`strategies` 现在推荐写成“按 `node_type` 为键”的映射,而不是手工写一长串列表。
|
||||
|
||||
```yaml
|
||||
strategies:
|
||||
- node_type: project
|
||||
skip_sync: false
|
||||
supplier:
|
||||
skip_post_check: true
|
||||
config:
|
||||
auto_bind: true
|
||||
auto_bind_fields:
|
||||
- name
|
||||
pre_bind_data_id: []
|
||||
- credit_code
|
||||
depend_fields: {}
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: create_local
|
||||
update_direction: none
|
||||
project:
|
||||
domain_option:
|
||||
pull_group_plan_production_date: true
|
||||
config:
|
||||
auto_bind: false
|
||||
auto_bind_fields: []
|
||||
pre_bind_data_id:
|
||||
- local_data_id: local-project-id
|
||||
remote_data_id: remote-project-id
|
||||
depend_fields:
|
||||
company_id: company
|
||||
local_orphan_action: create_remote
|
||||
local_orphan_action: none
|
||||
remote_orphan_action: none
|
||||
update_direction: push
|
||||
field_direction_key: null
|
||||
field_direction_overrides: {}
|
||||
post_check_ignore_fields: []
|
||||
compare_ignore_fields: []
|
||||
```
|
||||
|
||||
### 6.2 常用字段
|
||||
|
||||
- `auto_bind`
|
||||
- 是否开启自动绑定。
|
||||
- `auto_bind_fields`
|
||||
- 自动绑定使用的业务键字段。
|
||||
- `pre_bind_data_id`
|
||||
- 显式 local/remote data_id 绑定对。
|
||||
- `depend_fields`
|
||||
- 依赖字段到依赖 `node_type` 的映射。
|
||||
- `local_orphan_action` / `remote_orphan_action`
|
||||
- 孤儿节点创建策略。
|
||||
- `update_direction`
|
||||
- `push` / `pull` / `none`。
|
||||
- `post_check_ignore_fields`
|
||||
- 只影响最终一致性检查,不影响 create/update 差异检测。
|
||||
- `compare_ignore_fields`
|
||||
- schema diff / validate 时忽略的顶层字段。
|
||||
- `skip_sync`
|
||||
- 只跳过 create/update,不跳过 load/bind。
|
||||
- `skip_post_check`
|
||||
- 跳过该类型的 post-check reload 与一致性校验。
|
||||
- `domain_option`
|
||||
- 领域策略自己的扩展参数,由对应 domain 自己解释。
|
||||
|
||||
### 6.3 `preset` 仍然可用
|
||||
|
||||
如果只是想套一组预设策略,可以这样写:
|
||||
|
||||
```yaml
|
||||
strategies:
|
||||
project:
|
||||
preset: pull_only
|
||||
```
|
||||
|
||||
当前内置预设包括:
|
||||
|
||||
- `first_sync`
|
||||
- `daily_sync`
|
||||
- `repair_sync`
|
||||
- `pull_only`
|
||||
- `push_only`
|
||||
|
||||
### 6.4 已废弃的旧字段不要再写
|
||||
|
||||
以下 legacy key 已明确拒绝:
|
||||
|
||||
- `force_sync`
|
||||
- `load_mode`
|
||||
|
||||
如果你在 strategy 里写这两个字段,配置会直接报错。
|
||||
|
||||
当前替代方式是:
|
||||
|
||||
- 是否跳过同步,用 `skip_sync`
|
||||
- handler 特有加载参数,写到 `datasource.handler_configs.<node_type>`
|
||||
|
||||
## 7. persist 的当前写法
|
||||
|
||||
### 7.1 校验规则
|
||||
|
||||
当前 `persist` 配置遵循下面的规则:
|
||||
|
||||
1. `db_path` 和 `url` 至少要有一个。
|
||||
2. 当 `backend != sqlite` 时,必须显式提供 `url`。
|
||||
3. `wipe_on_start=true` 只会尝试删除可映射到文件路径的持久化文件;对 MySQL 这类远程数据库不会帮你删库。
|
||||
|
||||
### 7.2 SQLite 文件持久化
|
||||
|
||||
```yaml
|
||||
persist:
|
||||
backend: sqlite
|
||||
db_path: /abs/path/test_results/sync_state_machine/demo.db
|
||||
backend_options: {}
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/simple_pipeline_sm.db
|
||||
enable: true
|
||||
wipe_on_start: false
|
||||
```
|
||||
|
||||
### 7.3 SQLAlchemy 持久化到 SQLite URL
|
||||
|
||||
```yaml
|
||||
persist:
|
||||
backend: sqlalchemy
|
||||
url: sqlite+aiosqlite:////abs/path/to/pipeline.db
|
||||
enable: true
|
||||
wipe_on_start: false
|
||||
```
|
||||
|
||||
### 7.4 SQLAlchemy 持久化到 MySQL
|
||||
|
||||
```yaml
|
||||
persist:
|
||||
backend: sqlalchemy
|
||||
url: mysql+aiomysql://root:123456@127.0.0.1/ecm_sync_system?charset=utf8mb4
|
||||
enable: true
|
||||
wipe_on_start: false
|
||||
```
|
||||
|
||||
当前语义:
|
||||
|
||||
- `sqlalchemy` backend 会自动创建它自己的 `nodes` 和 `bindings` 表及索引。
|
||||
- 它不会替你创建 MySQL 数据库本身,所以 `ecm_sync_system` 这个库需要提前存在。
|
||||
- 现有表结构已经兼容 MySQL,不需要再额外给主键字段手工指定前缀长度。
|
||||
|
||||
### 7.5 `backend_options` 什么时候用
|
||||
|
||||
`backend_options` 仍然保留,但现在优先使用显式字段:
|
||||
|
||||
- `db_path`
|
||||
- `url`
|
||||
- `enable`
|
||||
- `wipe_on_start`
|
||||
|
||||
除非某个自定义 backend 明确要求,否则不要把主连接信息再塞进 `backend_options`。
|
||||
|
||||
## 8. logging 的当前写法
|
||||
|
||||
```yaml
|
||||
logging:
|
||||
initialize: true
|
||||
file_path: null
|
||||
file_dir: null
|
||||
file_name_pattern: null
|
||||
file_dir: ${PROJECT_ROOT}/test_results/sync_state_machine
|
||||
file_name_pattern: simple_pipeline_sm_{timestamp}.log
|
||||
level: 20
|
||||
suppress_node_logs: true
|
||||
```
|
||||
|
||||
## 1. 推荐使用方式(先 default,再覆盖)
|
||||
字段说明:
|
||||
|
||||
1. 先使用基线配置运行:
|
||||
- `run_profiles/preset/jsonl_to_jsonl.default.yaml`
|
||||
2. 若需项目定制,再复制为:
|
||||
- `run_profiles/jsonl_to_jsonl.local.yaml`
|
||||
3. 仅改有业务差异的字段,避免一次改太多导致排障困难。
|
||||
- `initialize`
|
||||
- 是否初始化应用日志。
|
||||
- `file_path`
|
||||
- 直接指定日志文件绝对路径。
|
||||
- `file_dir`
|
||||
- 只指定目录时,实际文件名由 `file_name_pattern` 生成。
|
||||
- `file_name_pattern`
|
||||
- 支持 `{timestamp}` 占位。
|
||||
- `level`
|
||||
- 默认 `20`,即 `INFO`。
|
||||
- `suppress_node_logs`
|
||||
- 是否压缩逐节点日志噪音。
|
||||
|
||||
占位符:
|
||||
- `${PROJECT_ROOT}` 会在运行时替换为项目根目录绝对路径。
|
||||
## 9. 多项目编排文件怎么写
|
||||
|
||||
## 2. 顶层字段(你在改什么)
|
||||
### 9.1 结构
|
||||
|
||||
- `node_types`
|
||||
- 本轮参与同步的业务类型列表。
|
||||
- 顺序会影响依赖链执行时机(当前按 node_type 串行执行 bind/create/update)。
|
||||
- `local_datasource` / `remote_datasource`
|
||||
- 分别定义两侧数据来源与执行端。
|
||||
- `strategies`
|
||||
- 每个业务类型的行为策略(依赖、自动绑定、创建方向、更新方向、跳过等)。
|
||||
- `persist`
|
||||
- 持久化开关与数据库位置。
|
||||
- `persist.backend`
|
||||
- 持久化后端注册表里的名称,例如 `sqlite`。
|
||||
- backend 实现会自行解释 `persist` 的其余字段。
|
||||
- `persist.backend_options`
|
||||
- 后端私有配置,后端自己决定如何解释。
|
||||
|
||||
### 6. 持久化后端注册
|
||||
|
||||
如果你希望在 backend 初始化阶段替换持久化后端,可以先在启动代码里按名称注册,再通过配置选择,不需要改调用方。
|
||||
|
||||
#### 6.1 内置 SQLite
|
||||
多项目文件不是把所有字段都堆在一起,而是引用两份单 pipeline 配置:
|
||||
|
||||
```yaml
|
||||
persist:
|
||||
backend: sqlite
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
|
||||
backend_options: {}
|
||||
global_config: run_profiles/jsonl_to_jsonl.multi_project.global.yaml
|
||||
default_project_config: run_profiles/jsonl_to_jsonl.multi_project.project.yaml
|
||||
|
||||
project_bootstrap_node_types:
|
||||
- company
|
||||
- supplier
|
||||
|
||||
max_concurrency: 8
|
||||
|
||||
projects:
|
||||
- local_project_id: local-project-1
|
||||
remote_project_id: remote-project-1
|
||||
- local_project_id: local-project-2
|
||||
remote_project_id: remote-project-2
|
||||
```
|
||||
|
||||
#### 6.2 注册自定义后端
|
||||
字段说明:
|
||||
|
||||
```python
|
||||
from sync_state_machine.common.persistence import register_persistence_backend
|
||||
- `global_config`
|
||||
- 全局公共节点要跑的单 pipeline 配置。
|
||||
- `default_project_config`
|
||||
- 每个项目默认使用的单 pipeline 配置。
|
||||
- `project_bootstrap_node_types`
|
||||
- 需要从 global scope 继承到项目 scope 的公共节点类型,例如 `company`、`supplier`。
|
||||
- `max_concurrency`
|
||||
- 项目并发数上限,默认 `20`。
|
||||
- `projects`
|
||||
- 项目映射清单。
|
||||
|
||||
register_persistence_backend("mysql", lambda config: MySQLPersistenceBackend(config))
|
||||
```
|
||||
### 9.2 项目项支持单独切配置
|
||||
|
||||
然后在配置里选择这个名字:
|
||||
如果某个项目需要特殊策略,可以在项目项里覆盖 `config_path`:
|
||||
|
||||
```yaml
|
||||
persist:
|
||||
backend: mysql
|
||||
db_path: ${PROJECT_ROOT}/test_results/sync_state_machine/demo.db
|
||||
backend_options:
|
||||
url: mysql+aiomysql://user:pass@127.0.0.1/depm_sync?charset=utf8mb4
|
||||
pool_size: 5
|
||||
projects:
|
||||
- local_project_id: local-project-1
|
||||
remote_project_id: remote-project-1
|
||||
- local_project_id: local-project-2
|
||||
remote_project_id: remote-project-2
|
||||
config_path: run_profiles/special_project.pipeline.yaml
|
||||
```
|
||||
|
||||
后端实现会收到整份 `PersistConfig`,自行决定是读取 `db_path`、`backend_options`,还是额外扩展别的字段。
|
||||
- `logging`
|
||||
- 运行日志输出位置与级别。
|
||||
此时该项目不会再用 `default_project_config`,而是改用自己的单 pipeline 配置文件。
|
||||
|
||||
## 3. 数据源字段(DataSource)
|
||||
### 9.3 当前多项目 profile 的实际风格
|
||||
|
||||
### 3.1 JSONL 数据源
|
||||
以当前 backend 两项目推送链路为例,常见组合是:
|
||||
|
||||
- `type=jsonl`:使用本地 JSONL 文件作为数据源。
|
||||
- `jsonl_dir`:JSONL 目录路径。
|
||||
- `read_only=false`:允许该侧执行 create/update 回写;若为 true,通常用于只读加载。
|
||||
- `handler_configs.project.data_id_filter=[]`:项目白名单;为空时不进行项目筛选。
|
||||
- `handler_configs`:按节点类型下发给各业务 handler 的扩展参数。
|
||||
- `local_datasource.type: depm_db`
|
||||
- `remote_datasource.type: api`
|
||||
- `persist.backend: sqlalchemy`
|
||||
- `persist.url: mysql+aiomysql://...`
|
||||
- `strategies.project.config.pre_bind_data_id` 明确写每个项目的 local/remote 映射
|
||||
|
||||
### 3.2 API 数据源
|
||||
这类 profile 本质上仍然是“多项目编排文件 + 两份单 pipeline 文件”的组合,不要把它理解成第三种配置格式。
|
||||
|
||||
- `type=api`:对接 HTTP API。
|
||||
- `api_base_url` / `api_uid` / `api_secret`:接口访问参数。
|
||||
- `api_debug`:开启后输出更多调试信息。
|
||||
- `poll_max_retries` / `poll_interval`:异步任务轮询次数与间隔。
|
||||
- `handler_configs.project.data_id_filter`:建议显式配置的项目白名单;至少需要指定一个项目ID,以避免加载过多远程数据。
|
||||
- `handler_configs`:业务 handler 扩展参数。
|
||||
## 10. 当前推荐的写法习惯
|
||||
|
||||
## 4. 策略字段(Strategy)
|
||||
### 10.1 只写必要覆盖项
|
||||
|
||||
每个 node_type 的 **Resolved Full Config** 都会包含以下字段,但输入配置不需要全部显式写出。
|
||||
推荐:
|
||||
|
||||
### 4.1 `update_direction`
|
||||
- 只写与你当前环境、项目过滤、持久化目标、策略差异直接相关的字段。
|
||||
|
||||
- `push`:本地变更推送到远端。
|
||||
- `pull`:远端变更拉取到本地。
|
||||
- `bidirectional`:双向更新(需要明确冲突策略,谨慎使用)。
|
||||
- `none`:关闭 update 阶段。
|
||||
不推荐:
|
||||
|
||||
### 4.2 `local_orphan_action` / `remote_orphan_action`
|
||||
- 把默认值全部展开复制一遍。
|
||||
|
||||
- `create_remote`:本地孤儿创建到远端。
|
||||
- `create_local`:远端孤儿创建到本地。
|
||||
- `none`:不自动创建。
|
||||
- `delete_local` / `delete_remote`:当前主流程未作为默认路径,谨慎使用。
|
||||
原因很直接:
|
||||
|
||||
### 4.3 `auto_bind` / `auto_bind_fields` / `pre_bind_data_id`
|
||||
- 默认值一旦调整,你的大段本地 YAML 会悄悄过时。
|
||||
- 排障时也更难看出你到底改了什么。
|
||||
|
||||
- `auto_bind=true`:在 `S02(MISSING)` 上尝试自动匹配。
|
||||
- `auto_bind=false`:不会做自动匹配,也不会走自动创建识别路径;通常会进入阻断态等待人工处理。
|
||||
- `auto_bind_fields`:自动匹配使用的业务键字段。
|
||||
- `pre_bind_data_id`:显式 data_id 预绑定清单(通用能力),格式为:
|
||||
- `- local_data_id: "..."`
|
||||
` remote_data_id: "..."`
|
||||
- 规则:若两侧数据源都存在这对 data_id 对应节点,且当前无绑定记录,则直接绑定。
|
||||
- 不依赖 `auto_bind/auto_bind_fields`,即使 `auto_bind=false` 也会执行。
|
||||
- 不做业务键一对一匹配检查,按你给定的映射对优先执行。
|
||||
### 10.2 handler 特有参数放 datasource 里
|
||||
|
||||
### 4.4 `depend_fields`
|
||||
|
||||
- 定义依赖字段到依赖 node_type 的映射,例如:`company_id -> company`。
|
||||
- 依赖节点状态非 `NORMAL` 时,会在 bind 的依赖检查阶段进入 `DEPENDENCY_ERROR`。
|
||||
|
||||
### 4.5 `skip_sync`
|
||||
|
||||
- 运行配置中已不建议使用 `skip_sync`。
|
||||
- 推荐使用显式组合:
|
||||
- `local_orphan_action: none`
|
||||
- `remote_orphan_action: none`
|
||||
- `update_direction: none`
|
||||
- `skip_sync=true` 的当前语义是:
|
||||
- 仍会执行该 node_type 的 load / bind
|
||||
- 只跳过 create / update 写入阶段
|
||||
- 适合“保留绑定与依赖可见性,但临时禁止写入”的场景。
|
||||
- `none/none/none` 是更推荐的输入风格,因为它表达的是“该类型不做创建、不做补本地、不做更新”。
|
||||
- 这组配置不会绕过前置 load / bind / 依赖检查;它只会把该类型变成无写入策略。
|
||||
|
||||
### 4.6 Handler 级参数(`handler_configs`)
|
||||
|
||||
- 建议将 handler 特有参数放在 datasource 的 `handler_configs.<node_type>` 下。
|
||||
- 示例:
|
||||
- `remote_datasource.handler_configs.supplier.load_mode: persisted_only`
|
||||
- `persisted_only` 表示只使用已持久化数据,不主动从外部拉新数据(具体以对应 handler 实现为准)。
|
||||
|
||||
### 4.7 `field_direction_key` / `field_direction_overrides`
|
||||
|
||||
- 这两个字段只在少数 node_type 上需要,典型例子是 `project_detail_data`。
|
||||
- `field_direction_key`
|
||||
- 指定从节点 data 里取哪个字段作为“方向查表键”。
|
||||
- 例如 `key`,表示用 `data["key"]` 决定这条记录是 push 还是 pull。
|
||||
- `field_direction_overrides`
|
||||
- 指定“字段值 -> 更新方向”的映射。
|
||||
- 没命中的值继续使用默认 `update_direction`。
|
||||
|
||||
示例:
|
||||
比如:
|
||||
|
||||
```yaml
|
||||
project_detail_data:
|
||||
config:
|
||||
field_direction_key: key
|
||||
field_direction_overrides:
|
||||
ensure_production: pull
|
||||
climb_production: pull
|
||||
challenge_production: pull
|
||||
remote_datasource:
|
||||
type: api
|
||||
handler_configs:
|
||||
contract_settlement_detail:
|
||||
load_method: aggregate
|
||||
```
|
||||
|
||||
含义:
|
||||
- `key=ensure_production/climb_production/challenge_production` 时,从远端拉回本地。
|
||||
- 其余 key 继续按默认 `update_direction=push` 处理。
|
||||
这类参数应该写在 `datasource.handler_configs.<node_type>`,不要错写到 strategy 里。
|
||||
|
||||
## 5. 阶段行为与配置关系
|
||||
### 10.3 项目过滤显式下沉到 handler
|
||||
|
||||
- bind 阶段
|
||||
- `depend_fields` 决定是否做依赖校验。
|
||||
- `pre_bind_data_id` 先按显式 local/remote data_id 映射做预绑定。
|
||||
- `auto_bind/auto_bind_fields` 决定是否继续进行业务键自动绑定。
|
||||
- create 阶段
|
||||
- `local_orphan_action` / `remote_orphan_action` 决定孤儿是否创建。
|
||||
- update 阶段
|
||||
- `update_direction` 决定是否和向哪侧更新。
|
||||
- 写入后 reload(Pipeline 内置)
|
||||
- 对 API / 需要重新拉取权威状态的数据源,create 或 update 成功后仍会按 node_type reload。
|
||||
- 对纯 JSONL -> JSONL pipeline,reload 会跳过,直接复用当前内存状态。
|
||||
- 目的:避免对离线 JSONL 数据做无意义重复扫描,同时保留 API 数据源的权威状态刷新。
|
||||
- 同步后一致性检查(Pipeline 内置)
|
||||
- 每个 node_type 在本轮同步后会基于绑定对执行数据一致性校验(比较时忽略 `id/_id` 字段)。
|
||||
- 若发现差异,会在日志中打印差异样本,并在汇总表新增 `Consistent` 列展示 `Y` / `N(差异数)`。
|
||||
- 全局跳过
|
||||
- `local_orphan_action=none` + `remote_orphan_action=none` + `update_direction=none`
|
||||
可将该类型收敛为“只做 load/bind,不做 create/update 写入”的只读同步策略。
|
||||
例如:
|
||||
|
||||
## 6. 覆盖配置建议
|
||||
```yaml
|
||||
handler_configs:
|
||||
project:
|
||||
data_id_filter:
|
||||
- project-id-1
|
||||
- project-id-2
|
||||
```
|
||||
|
||||
- 先在 default 跑通一轮,再做 local 覆盖。
|
||||
- 每次只改一类问题相关字段(例如只改 `project` 的依赖与自动绑定)。
|
||||
- 覆盖后优先看 `sync_log` 是否出现预期状态迁移。
|
||||
不要在 pipeline 外层再做一层临时过滤逻辑。项目范围应尽量体现在配置里,并由对应 handler 解释。
|
||||
|
||||
## 7. 持久化与日志字段
|
||||
### 10.4 看到报错先修配置,不要做降级
|
||||
|
||||
### `persist`
|
||||
当前系统对配置保持 fail-fast:
|
||||
|
||||
- `enable`:是否开启持久化。
|
||||
- `db_path`:SQLite 路径。
|
||||
- `wipe_on_start`:启动是否清理历史持久化。
|
||||
- datasource type 未注册会报错
|
||||
- legacy strategy key 会报错
|
||||
- `persist` 缺失目标会报错
|
||||
- profile 结构不符合 schema 会报错
|
||||
|
||||
### `logging`
|
||||
这属于预期行为。优先修正配置,而不是在文档或调用侧加 fallback。
|
||||
|
||||
- `file_dir`:日志目录。
|
||||
- `file_name_pattern`:文件命名模板。
|
||||
- `level`:日志级别。
|
||||
## 11. 参考文件
|
||||
|
||||
排障建议:运行失败时优先看日志中的 `状态机流转` 与 `reason=`,再对照 `docs/state_machine.md` 与 `docs/node_state_definition.md` 理解含义。
|
||||
如果你想直接对照真实样例,优先看这些文件:
|
||||
|
||||
- `run_profiles/jsonl_to_api.yaml`
|
||||
- `run_profiles/jsonl_to_jsonl.multi_project.yaml`
|
||||
- backend 中的 `local_tests/sync_system_tests/depm_to_ecm_bootstrap_pull_then_project_push.two_projects.sqlalchemy.pipeline.yaml`
|
||||
|
||||
如果你想看配置模型定义,直接看:
|
||||
|
||||
- `sync_state_machine/config/pipeline_config.py`
|
||||
- `sync_state_machine/config/datasource_config.py`
|
||||
- `sync_state_machine/config/persist_config.py`
|
||||
- `sync_state_machine/config/multi_project_config.py`
|
||||
- `sync_state_machine/config/strategy_config.py`
|
||||
|
||||
排障时优先顺序:
|
||||
|
||||
1. 看输入 YAML。
|
||||
2. 看启动日志里的 `Resolved Full Config`。
|
||||
3. 看对应 datasource type 是否真的已经注册。
|
||||
4. 再看运行日志和状态机结果。
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
- `S12C/S12U/S12D`:按动作拆分的执行失败状态
|
||||
- `S13`:对侧创建等待态(`PEER_CREATING/NONE/PENDING`)
|
||||
- `S14`:UPDATE 运行时降级为 `NONE` 并跳过
|
||||
- `S17`:bootstrap 加载异常隔离(存在绑定,但 datasource refresh 未返回最新节点)
|
||||
|
||||
## 3. 事件到代码映射
|
||||
|
||||
@@ -43,14 +44,15 @@
|
||||
- `E40D_START/E40D`:DELETE 执行入口与结果回写
|
||||
|
||||
## 4. strategy 调用链(当前)
|
||||
- bind 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.bind()` -> `sync_state_machine/sync_system/strategy_ops/bind_ops.py::run_bind()`
|
||||
- create 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.create()` -> `sync_state_machine/sync_system/strategy_ops/create_ops.py::run_create()`
|
||||
- update 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.update()` -> `sync_state_machine/sync_system/strategy_ops/update_ops.py::run_update()`
|
||||
- reset 主入口:`sync_state_machine/sync_system/strategy.py::BaseSyncStrategy.run_reset()` -> `sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`
|
||||
- bind 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.bind()`
|
||||
- create 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.create()`
|
||||
- update 主入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.update()`
|
||||
- reset 主入口:`sync_state_machine/pipeline/full_sync_pipeline.py::_load_persistence_state()` -> `sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`
|
||||
|
||||
## 5. context 字段(以 YAML 为准)
|
||||
|
||||
### bind 相关
|
||||
- `bootstrap_source_present`
|
||||
- `has_binding_record`
|
||||
- `core_binding_result` / `peer_core_binding_result` (`NORMAL|WARNING|ABNORMAL`)
|
||||
- `data_present`
|
||||
@@ -75,6 +77,7 @@
|
||||
## 6. 语义边界
|
||||
- strategy 负责业务语义计算(如依赖是否满足、自动绑定结果、差异检测)。
|
||||
- 状态机只负责迁移判定与不变量校验,不直接访问数据源。
|
||||
- bootstrap 阶段的数据源刷新由 pipeline 完成,状态机只消费 `bootstrap_source_present / has_binding_record / is_create_zombie` 这些已计算上下文。
|
||||
|
||||
## 7. 不变量
|
||||
- `INV-1_UNCHECKED_ACTION_NONE`
|
||||
|
||||
+35
-12
@@ -11,42 +11,65 @@
|
||||
|
||||
## 2. reset 时序
|
||||
|
||||
### 2.1 加载时 reset(phase1 语义)
|
||||
### 2.1 加载时恢复
|
||||
- 在 `sync_state_machine/common/collection.py::load_from_persistence()` 执行。
|
||||
- 语义:仅恢复持久化状态(不在此阶段做状态归并)。
|
||||
- 语义:仅恢复持久化状态,并记录本轮 `sync_log` 的 `persistence` 恢复来源;不在此阶段做状态归并。
|
||||
- 恢复后保留 `data/origin_data`,等待 bootstrap 阶段的 datasource 刷新覆盖最新数据。
|
||||
|
||||
### 2.2 加载后 cleanup(phase2 语义)
|
||||
- pipeline 调用 `BaseSyncStrategy.run_reset()`。
|
||||
### 2.2 bootstrap datasource refresh
|
||||
- pipeline 在持久化恢复后、`E01` 之前,按已恢复到内存的 `node_type` 做一次 datasource 刷新。
|
||||
- 实际实现:`sync_state_machine/pipeline/full_sync_pipeline.py::_bootstrap_refresh_latest_data()`。
|
||||
- 语义:
|
||||
- 如果 datasource 命中节点,则用最新数据覆盖持久化快照,并清除 `_loaded_from_persistence` 标记。
|
||||
- 如果 datasource 未命中节点,则节点保留持久化快照,供 `E01` 判断它是 stale zombie 还是 bootstrap 异常隔离对象。
|
||||
|
||||
### 2.3 加载后 cleanup
|
||||
- pipeline 在 datasource refresh 完成后调用 `run_phase2_cleanup()`。
|
||||
- 实际实现:`sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`。
|
||||
- 核心事件:`E01`(`S16 -> S00/S15`)。
|
||||
- `S16` 表示“持久化加载输入态”:本轮从持久化恢复出的节点集合(可包含上轮遗留节点,也可包含上轮已稳定节点)。
|
||||
- `S16` 表示“bootstrap 输入态”:持久化恢复出的节点,叠加 datasource refresh 后仍待 `E01` 判定的节点集合。
|
||||
- 判定:
|
||||
- `is_create_zombie=true` -> `S15`,节点删除(并尝试解绑);
|
||||
- `is_create_zombie=false` -> `S00`,节点归并为 `UNCHECKED/NONE/PENDING` 并清空 `error`。
|
||||
- `bootstrap_source_present=false && has_binding_record=false` -> `S15`,说明该节点仅残留在 persistence 中且未绑定,按 stale zombie 清理;
|
||||
- `bootstrap_source_present=false && has_binding_record=true` -> `S17`,说明该节点存在绑定,但 datasource refresh 未返回,进入 bootstrap 加载异常隔离;
|
||||
- 其余情况 -> `S00`,节点归并为 `UNCHECKED/NONE/PENDING` 并清空 `error`。
|
||||
|
||||
### 2.2.1 `enable_persistence` 与 bootstrap 关系
|
||||
- `enable_persistence=true`:先 `load_from_persistence()`,再对加载出的节点执行 `E01`(`S16` 域内处理)。
|
||||
- `enable_persistence=true`:先 `load_from_persistence()`,再做 bootstrap datasource refresh,最后对输入节点执行 `E01`(`S16` 域内处理)。
|
||||
- `enable_persistence=false`:本轮没有持久化输入,`S16/E01` 不参与;随后 Stage1 新加载节点直接进入本轮 bind/create/update/sync 主流程。
|
||||
|
||||
### 2.3 加载异常隔离(S17)
|
||||
### 2.4 加载异常隔离(S17)
|
||||
- 持久化中若出现非法枚举值(`action/status/binding_status`),节点标记为 `bootstrap_blocked`(`S17` 语义)。
|
||||
- 持久化恢复后若节点存在绑定,但 bootstrap datasource refresh 未返回该节点,也会进入 `S17` 语义并隔离。
|
||||
- 这类节点保留 `error`,用于人工排查。
|
||||
- 这类节点不会进入 `E01`,也不会参与 bind/create/update/sync 自动流程。
|
||||
|
||||
## 3. bind 流程(strategy_ops)
|
||||
- 入口:`sync_state_machine/sync_system/strategy_ops/bind_ops.py::run_bind()`
|
||||
- 入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.bind()`
|
||||
- 入口状态:`S00`
|
||||
|
||||
### bind 编排步骤
|
||||
- strategy 先通过 `bind_ops.collect_bind_entry_nodes()` 收集两侧 `S00` 节点,作为 bind 入口工作集。
|
||||
- `phase_core_state()` 对入口工作集触发 `E10`,把“已有绑定记录”和“无绑定记录”的节点分流到 `S01/S03/S04/S02`。
|
||||
- `phase_dependency_check()` 对仍需继续处理的缺绑节点触发 `E15`,把缺数据、依赖不满足、语义错误的节点拦到 `S03/S05/S04`。
|
||||
- strategy 再通过 `bind_ops.collect_auto_bind_ready_nodes()` 收窄工作集,只保留当前 `S02 + binding_status=MISSING` 的节点进入自动绑定判定。
|
||||
- strategy 在顶层显式构造 orphan create 控制参数,并传给 `bind_ops.plan_auto_bind_updates()` 生成自动绑定/自动创建决策。
|
||||
- `apply_auto_bind_updates()` 对这些决策逐个触发 `E16`,把节点迁移到 `S01/S04/S05/S13`。
|
||||
- `refresh_bind_data_id()` 最后按绑定关系回填两侧节点的 `context.bind_data_id`。
|
||||
|
||||
### 核心绑定判定
|
||||
- `phase_core_state()` 内触发 `E10`。
|
||||
- 有绑定记录:根据本端/对端有效性进入 `S01/S03/S04`。
|
||||
- 无绑定记录:进入 `S02`。
|
||||
|
||||
注意:如果节点在 bootstrap 阶段已被隔离为 `S17` 语义(`bootstrap_blocked=true`),默认不会进入 bind 工作集。
|
||||
|
||||
### 依赖检查
|
||||
- `phase_dependency_check()` -> `check_dependencies_for_nodes()` 内触发 `E15`。
|
||||
- `data_present=true` 且 `dep_satisfied=true` -> 不迁移,节点保持在 `S02`,继续进入自动绑定判定。
|
||||
- `data_present=false` -> `S03`。
|
||||
- `dep_satisfied=false` -> `S05`。
|
||||
- `depend_fields` 为空 -> `S04`(无法自动处理,阻断后续自动创建)。
|
||||
- `depend_fields` 为空本身不会触发阻断;若节点数据存在,则等价于“当前依赖满足”,继续留在 `S02`。
|
||||
|
||||
### 自动绑定
|
||||
- `phase_auto_binding()` 内触发 `E16`。
|
||||
@@ -57,7 +80,7 @@
|
||||
## 4. create/update 准备
|
||||
|
||||
### create
|
||||
- 入口:`sync_state_machine/sync_system/strategy_ops/create_ops.py::run_create()`
|
||||
- 入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.create()`
|
||||
- 运行时事件:`E20`(`E21/E22` 为配置保留分支,当前 strategy 未接入自动执行)
|
||||
- 运行时入口状态:`S13`
|
||||
- 说明:`E16` 负责识别“需自动创建”并将源节点置为 `S13`;业务代码执行对侧 spawn+bind 后,再通过 `E20` 将 `S13` 提交到 `S01`(成功);若 spawn 失败,`E20` 不迁移并保留在 `S13` 等待人工/重试。
|
||||
@@ -66,7 +89,7 @@
|
||||
- 新创建目标节点在 `S06/S10C` 阶段允许 `data_id` 为空;`CREATE SUCCESS`(`S11C`)必须携带非空 `result_data_id`,否则事件直接报错。
|
||||
|
||||
### update
|
||||
- 入口:`sync_state_machine/sync_system/strategy_ops/update_ops.py::run_update()`
|
||||
- 入口:`sync_state_machine/sync_system/strategy.py::DefaultSyncStrategy.update()`
|
||||
- 事件:`E30`
|
||||
- 入口状态(source 节点):`S01`
|
||||
- `target_has_data_id=false` 或 `update_id_resolve_ok=false` -> `S08`
|
||||
|
||||
@@ -10,6 +10,8 @@ dependencies = [
|
||||
"PyYAML>=6.0,<7",
|
||||
"httpx>=0.27,<1",
|
||||
"aiosqlite>=0.20,<1",
|
||||
"SQLAlchemy>=2.0,<3",
|
||||
"greenlet>=3,<4",
|
||||
"pytz>=2024.1,<2027",
|
||||
"fastapi>=0.115,<1",
|
||||
"uvicorn>=0.30,<1",
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
@@ -10,50 +7,7 @@ PROJECT_ROOT = Path(__file__).resolve().parent
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from sync_state_machine.config import load_overrides_from_file
|
||||
from sync_state_machine.pipeline import run_profile_from_file
|
||||
|
||||
|
||||
def _resolve_config_path(raw_path: str) -> Path:
|
||||
path = Path(raw_path)
|
||||
if not path.is_absolute():
|
||||
path = PROJECT_ROOT / path
|
||||
return path.resolve()
|
||||
|
||||
|
||||
async def _main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Run sync pipeline from config override file")
|
||||
parser.add_argument(
|
||||
"--config_path",
|
||||
required=True,
|
||||
help="Path to run profile file (JSON/YAML, supports relative path, e.g. run_profiles/jsonl_to_jsonl.local.json)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
cfg_path = _resolve_config_path(args.config_path)
|
||||
if not cfg_path.exists() or not cfg_path.is_file():
|
||||
raise FileNotFoundError(f"Config file not found: {cfg_path}")
|
||||
|
||||
print(f"🧩 Loaded config file: {cfg_path}")
|
||||
await run_profile_from_file(
|
||||
project_root=PROJECT_ROOT,
|
||||
config_path=cfg_path,
|
||||
print_summary=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
return asyncio.run(_main())
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted by user")
|
||||
return 130
|
||||
except Exception as exc:
|
||||
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
|
||||
return 1
|
||||
finally:
|
||||
logging.shutdown()
|
||||
from sync_state_machine.cli import main
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import HTTPException
|
||||
from pydantic import Field, BaseModel, field_validator
|
||||
from pydantic import Field, BaseModel, field_validator, model_validator
|
||||
from ..common.push_system import BaseUpdateRequestSchema
|
||||
from ..common.base import BaseResponseWithID
|
||||
from ..common.datetime import get_current_date
|
||||
@@ -73,12 +73,12 @@ class ProjectResponseBase(BaseResponseWithID):
|
||||
progress_type: ProgressType | None = Field(None, description="推进分类", examples=[ProgressType.A])
|
||||
key_constraints: KeyConstraints = Field(KeyConstraints.NONE, description="关键制约", examples=[KeyConstraints.NONE])
|
||||
key_constraints_remark: Optional[str] = Field("", description="关键制约备注", examples=["无"])
|
||||
status: Optional[str] = Field(None, description="项目状态")
|
||||
# status: Optional[str] = Field(None, description="项目状态")
|
||||
apply_file_ids: List[str] = Field(default_factory=list, description="申请表附件ID列表", examples=[[]])
|
||||
|
||||
# 拓展信息
|
||||
dbtc_score_list: Optional[list] = Field(default_factory=list, description='达标投产评分列表') # 这里项目用的是list而不是List
|
||||
approval_status: Optional[str] = Field(None, description="审核状态", examples=["pending"])
|
||||
# approval_status: Optional[str] = Field(None, description="审核状态", examples=["pending"])
|
||||
|
||||
|
||||
|
||||
@@ -266,11 +266,35 @@ class ProjectProductionCapacity(BaseModel):
|
||||
"""
|
||||
plan_production_date: DateStr | None = Field(None, description="集团公司计划投产日期", examples=["2024-08-01"])
|
||||
plan_full_production_date: DateStr | None = Field(None, description="集团公司计划全部投产日期", examples=["2024-09-01"])
|
||||
ic_plan_first_production_date: DateStr | None = Field(None, description="区域公司计划首批投产日期(只有新能源需要)",examples=["2024-08-01"])
|
||||
ic_plan_full_production_date: DateStr | None = Field(None, description="区域公司计划全投日期(只有新能源需要)",examples=["2024-09-01"])
|
||||
actual_production_date: DateStr | None = Field(None, description="实际投产日期(只有新能源需要)", examples=["2024-08-15"])
|
||||
actual_full_production_date: DateStr | None = Field(None, description="实际全投日期(只有新能源需要)", examples=["2024-09-10"])
|
||||
actual_production_list : list[_ActualProductionNode] | None = Field(None, description="实际投产节点列表(每次必须填写全部)(只有新能源需要)")
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def reject_ic_plan_dates(cls, data):
|
||||
if not isinstance(data, dict):
|
||||
return data
|
||||
|
||||
blocked_fields = [
|
||||
field for field in ("ic_plan_first_production_date", "ic_plan_full_production_date")
|
||||
if field in data
|
||||
]
|
||||
if blocked_fields:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"/production_capacity 接口不支持更新字段: {', '.join(blocked_fields)}"
|
||||
)
|
||||
return data
|
||||
|
||||
@property
|
||||
def ic_plan_first_production_date(self) -> str:
|
||||
return ""
|
||||
|
||||
@property
|
||||
def ic_plan_full_production_date(self) -> str:
|
||||
return ""
|
||||
|
||||
@field_validator("actual_production_date")
|
||||
def validate_actual_production_date(cls, v):
|
||||
if v is None:
|
||||
|
||||
@@ -36,4 +36,4 @@ class PreparationDetail(BaseResponseWithID):
|
||||
# approval_status: Optional[str] = Field(None, description="审批状态")
|
||||
name: Optional[str] = Field(None, description="里程碑名称")
|
||||
group: Optional[str] = Field(None, description="分组条件", examples=['KGTJLS'])
|
||||
is_acquired: Optional[int] = Field(0, description="是否取得")
|
||||
# is_acquired: Optional[int] = Field(0, description="是否取得")
|
||||
|
||||
@@ -36,4 +36,4 @@ class ProductionDetail(BaseResponseWithID):
|
||||
is_exist: Optional[bool] = Field(None, description="是否取得")
|
||||
# approval_status: Optional[str] = Field(None, description="审批状态")
|
||||
name: Optional[str] = Field(None, description="投产节点名称")
|
||||
is_acquired: Optional[int] = Field(0, description="是否取得")
|
||||
# is_acquired: Optional[int] = Field(0, description="是否取得")
|
||||
|
||||
@@ -48,15 +48,9 @@ class SupplierDetailResponse(BaseResponseWithID):
|
||||
status: str = Field(..., description="供应商状态", examples=["active"])
|
||||
sort: int = Field(..., description="排序", examples=[1])
|
||||
credit_code: Optional[str] = Field(None, description="信用代码", examples=["914403007109310121"])
|
||||
created_by: str = Field(..., description="创建者ID", examples=["user:001"])
|
||||
created_by_name: str = Field(..., description="创建者名称", examples=["张三"])
|
||||
created_at: str = Field(..., description="创建时间", examples=["2023-10-01T12:00:00Z"])
|
||||
updated_by: str = Field(..., description="更新者ID", examples=["user:001"])
|
||||
updated_by_name: str = Field(..., description="更新者名称", examples=["张三"])
|
||||
updated_at: str = Field(..., description="更新时间", examples=["2023-10-01T12:00:00Z"])
|
||||
province: Optional[str] = Field(None, description="省份编码", examples=[None])
|
||||
city: Optional[str] = Field(None, description="城市编码", examples=[None])
|
||||
company_nature: List[Optional[str]] = Field(None, description="供应商性质数组", examples=[['supplier']])
|
||||
company_nature: List[Optional[str]] = Field([], description="供应商性质数组", examples=[['supplier']])
|
||||
city_name: Optional[str] = Field(None, description="城市名称", examples=[None])
|
||||
province_name: Optional[str] = Field(None, description="省份名称", examples=[None])
|
||||
region_path: Optional[list] = Field(default_factory=list, description="供应商国家城市地址编码列表")
|
||||
|
||||
@@ -309,11 +309,11 @@ def build_temp_run_profile_for_project_auto_bind(
|
||||
project_strategy["update_direction"] = project_update_direction or "none"
|
||||
|
||||
if project_detail_push_keys:
|
||||
project_detail_config = profile.setdefault("strategies", {}).setdefault("project_detail_data", {}).setdefault("config", {})
|
||||
overrides = dict(project_detail_config.get("field_direction_overrides", {}))
|
||||
for key in project_detail_push_keys:
|
||||
overrides.pop(key, None)
|
||||
project_detail_config["field_direction_overrides"] = overrides
|
||||
fixed_pull_keys = {"ensure_production", "climb_production", "challenge_production"}
|
||||
if fixed_pull_keys.intersection(project_detail_push_keys):
|
||||
project_detail_runtime = profile.setdefault("strategies", {}).setdefault("project_detail_data", {})
|
||||
domain_option = project_detail_runtime.setdefault("domain_option", {})
|
||||
domain_option["pull_group_production_plans"] = False
|
||||
|
||||
runtime_dir = Path(report_root) / "runtime_configs"
|
||||
runtime_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
+56
-11
@@ -15,6 +15,8 @@ from urllib import error, request
|
||||
|
||||
import yaml
|
||||
|
||||
from sync_state_machine.common.persistence import PersistenceBackend
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
@@ -312,26 +314,69 @@ def sqlite_rows(db_path: str | Path, sql: str, params: tuple[Any, ...] = ()) ->
|
||||
return [{key: row[key] for key in row.keys()} for row in rows]
|
||||
|
||||
|
||||
def persistence_summary(db_path: str | Path, node_type: str | None = None) -> dict[str, Any]:
|
||||
def persistence_rows(
|
||||
*,
|
||||
backend: str,
|
||||
sql: str,
|
||||
db_path: str | Path | None = None,
|
||||
url: str | None = None,
|
||||
limit: int = 200,
|
||||
) -> list[dict[str, Any]]:
|
||||
_, rows = PersistenceBackend.query_records(
|
||||
backend=backend,
|
||||
db_path=str(db_path) if db_path is not None else None,
|
||||
url=url,
|
||||
sql=sql,
|
||||
limit=limit,
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def persistence_summary(
|
||||
db_path: str | Path | None,
|
||||
node_type: str | None = None,
|
||||
*,
|
||||
backend: str = "sqlite",
|
||||
url: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
records = PersistenceBackend.records_summary(
|
||||
backend=backend,
|
||||
db_path=str(db_path) if db_path is not None else None,
|
||||
url=url,
|
||||
)
|
||||
summary: dict[str, Any] = {
|
||||
"backend": backend,
|
||||
"db_path": str(db_path),
|
||||
"db_exists": Path(db_path).exists(),
|
||||
"url": url,
|
||||
"db_exists": records.get("db_exists", False),
|
||||
"bindings": [],
|
||||
"node_status": [],
|
||||
}
|
||||
if not Path(db_path).exists():
|
||||
if not summary["db_exists"]:
|
||||
return summary
|
||||
|
||||
target_db_path = str(db_path) if db_path is not None else None
|
||||
if node_type:
|
||||
summary["bindings"] = sqlite_rows(
|
||||
db_path,
|
||||
"SELECT node_type, local_id, remote_id, last_updated FROM bindings WHERE node_type = ? ORDER BY last_updated DESC LIMIT 20",
|
||||
(node_type,),
|
||||
escaped_node_type = node_type.replace("'", "''")
|
||||
summary["bindings"] = persistence_rows(
|
||||
backend=backend,
|
||||
db_path=target_db_path,
|
||||
url=url,
|
||||
sql=(
|
||||
"SELECT node_type, local_id, remote_id, last_updated "
|
||||
f"FROM bindings WHERE node_type = '{escaped_node_type}' ORDER BY last_updated DESC LIMIT 20"
|
||||
),
|
||||
)
|
||||
summary["node_status"] = sqlite_rows(
|
||||
db_path,
|
||||
"SELECT collection_id, node_type, action, status, binding_status, COUNT(1) AS count FROM nodes WHERE node_type = ? GROUP BY collection_id, node_type, action, status, binding_status ORDER BY collection_id, action, status",
|
||||
(node_type,),
|
||||
summary["node_status"] = persistence_rows(
|
||||
backend=backend,
|
||||
db_path=target_db_path,
|
||||
url=url,
|
||||
sql=(
|
||||
"SELECT collection_id, node_type, action, status, binding_status, COUNT(1) AS count "
|
||||
f"FROM nodes WHERE node_type = '{escaped_node_type}' "
|
||||
"GROUP BY collection_id, node_type, action, status, binding_status "
|
||||
"ORDER BY collection_id, action, status"
|
||||
),
|
||||
)
|
||||
return summary
|
||||
|
||||
|
||||
@@ -15,6 +15,7 @@ if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
import sync_state_machine.domain # noqa: F401, E402
|
||||
from sync_state_machine.logging import configure_app_logging, get_logger # noqa: E402
|
||||
from sync_state_machine.pipeline import ( # noqa: E402
|
||||
build_config,
|
||||
run_pipeline_from_config,
|
||||
@@ -23,15 +24,17 @@ from sync_state_machine.config import load_named_preset_overrides # noqa: E402
|
||||
|
||||
|
||||
PRESET_NAME = "jsonl_sm"
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
try:
|
||||
overrides, preset_file, used_default = load_named_preset_overrides(PROJECT_ROOT, PRESET_NAME)
|
||||
source_label = "default" if used_default else "custom"
|
||||
print(f"🧩 Loaded preset ({source_label}): {preset_file}")
|
||||
|
||||
config = build_config(project_root=PROJECT_ROOT, overrides=overrides)
|
||||
configure_app_logging(config.logging, replace_handlers=True)
|
||||
logger.info("🧩 Loaded preset (%s): %s", source_label, preset_file)
|
||||
await run_pipeline_from_config(
|
||||
config,
|
||||
title="sync_state_machine JSONL -> JSONL pipeline",
|
||||
@@ -40,12 +43,10 @@ async def main() -> int:
|
||||
return 0
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted by user")
|
||||
logger.warning("⚠️ Interrupted by user")
|
||||
return 130
|
||||
except Exception as exc:
|
||||
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
logger.exception("❌ Pipeline failed: %s: %s", type(exc).__name__, exc)
|
||||
return 1
|
||||
|
||||
|
||||
|
||||
@@ -15,6 +15,7 @@ if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
import sync_state_machine.domain # noqa: F401, E402
|
||||
from sync_state_machine.logging import configure_app_logging, get_logger # noqa: E402
|
||||
from sync_state_machine.pipeline import ( # noqa: E402
|
||||
build_config,
|
||||
run_pipeline_from_config,
|
||||
@@ -23,15 +24,17 @@ from sync_state_machine.config import load_named_preset_overrides # noqa: E402
|
||||
|
||||
|
||||
PRESET_NAME = "simple_sm"
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
try:
|
||||
overrides, preset_file, used_default = load_named_preset_overrides(PROJECT_ROOT, PRESET_NAME)
|
||||
source_label = "default" if used_default else "custom"
|
||||
print(f"🧩 Loaded preset ({source_label}): {preset_file}")
|
||||
|
||||
config = build_config(project_root=PROJECT_ROOT, overrides=overrides)
|
||||
configure_app_logging(config.logging, replace_handlers=True)
|
||||
logger.info("🧩 Loaded preset (%s): %s", source_label, preset_file)
|
||||
await run_pipeline_from_config(
|
||||
config,
|
||||
title="sync_state_machine JSONL -> API pipeline",
|
||||
@@ -40,12 +43,10 @@ async def main() -> int:
|
||||
return 0
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted by user")
|
||||
logger.warning("⚠️ Interrupted by user")
|
||||
return 130
|
||||
except Exception as exc:
|
||||
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
logger.exception("❌ Pipeline failed: %s: %s", type(exc).__name__, exc)
|
||||
return 1
|
||||
|
||||
|
||||
|
||||
@@ -5,9 +5,13 @@ import asyncio
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from .logging import get_logger, init_app_logger
|
||||
from .pipeline import run_profile_from_file
|
||||
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def package_project_root() -> Path:
|
||||
return Path(__file__).resolve().parents[1]
|
||||
|
||||
@@ -20,6 +24,7 @@ def resolve_config_path(raw_path: str, *, project_root: Path) -> Path:
|
||||
|
||||
|
||||
async def _main() -> int:
|
||||
init_app_logger(replace_handlers=True)
|
||||
project_root = package_project_root()
|
||||
|
||||
parser = argparse.ArgumentParser(description="Run sync pipeline from config override file")
|
||||
@@ -34,7 +39,7 @@ async def _main() -> int:
|
||||
if not cfg_path.exists() or not cfg_path.is_file():
|
||||
raise FileNotFoundError(f"Config file not found: {cfg_path}")
|
||||
|
||||
print(f"🧩 Loaded config file: {cfg_path}")
|
||||
logger.info("🧩 Loaded config file: %s", cfg_path)
|
||||
await run_profile_from_file(
|
||||
project_root=project_root,
|
||||
config_path=cfg_path,
|
||||
@@ -47,10 +52,10 @@ def main() -> int:
|
||||
try:
|
||||
return asyncio.run(_main())
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted by user")
|
||||
logger.warning("⚠️ Interrupted by user")
|
||||
return 130
|
||||
except Exception as exc:
|
||||
print(f"\n❌ Pipeline failed: {type(exc).__name__}: {exc}")
|
||||
logger.exception("❌ Pipeline failed: %s: %s", type(exc).__name__, exc)
|
||||
return 1
|
||||
finally:
|
||||
logging.shutdown()
|
||||
|
||||
@@ -6,7 +6,7 @@ from .persistence import (
|
||||
get_registered_persistence_backends,
|
||||
register_persistence_backend,
|
||||
)
|
||||
from .persistence_backends import SQLitePersistenceBackend
|
||||
from .persistence_backends import SQLAlchemyPersistenceBackend, SQLitePersistenceBackend
|
||||
from .collection import DataCollection
|
||||
from .binding import BindingManager
|
||||
|
||||
@@ -17,6 +17,7 @@ __all__ = [
|
||||
"SyncNode",
|
||||
"PersistenceBackend",
|
||||
"SQLitePersistenceBackend",
|
||||
"SQLAlchemyPersistenceBackend",
|
||||
"create_persistence_backend",
|
||||
"register_persistence_backend",
|
||||
"get_registered_persistence_backends",
|
||||
|
||||
@@ -1,10 +1,13 @@
|
||||
import json
|
||||
import logging
|
||||
from typing import Optional, Dict, Any, List
|
||||
|
||||
from ..logging import get_logger, get_scope_display_name
|
||||
from .persistence import PersistenceBackend
|
||||
from .persistence_service import ScopedPersistenceView
|
||||
from .types import BindingStatus, BindingRecord
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class BindingManager:
|
||||
@@ -12,14 +15,53 @@ class BindingManager:
|
||||
BindingManager 负责维护不同 DataCollection 间节点的绑定关系。
|
||||
所有查询均在内存中进行,持久化仅作为备份。
|
||||
"""
|
||||
def __init__(self, persistence: PersistenceBackend, auto_persist: bool = True, scope: str = "global"):
|
||||
def __init__(
|
||||
self,
|
||||
persistence: PersistenceBackend | ScopedPersistenceView,
|
||||
auto_persist: bool = False,
|
||||
scope: str = "global",
|
||||
global_fallback_manager: Optional["BindingManager"] = None,
|
||||
):
|
||||
self.persistence = persistence
|
||||
self.auto_persist = auto_persist
|
||||
self.scope = scope
|
||||
self._global_fallback_manager = global_fallback_manager
|
||||
# { node_type: { local_id: remote_id } }
|
||||
self._bindings: Dict[str, Dict[str, Optional[str]]] = {}
|
||||
self._deleted: Dict[str, set[str]] = {}
|
||||
|
||||
def set_global_fallback_manager(self, manager: Optional["BindingManager"]) -> None:
|
||||
self._global_fallback_manager = manager
|
||||
|
||||
async def _persistence_save_binding(self, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.save_binding(node_type, local_id, payload_dict)
|
||||
return
|
||||
await self.persistence.save_binding(self.scope, node_type, local_id, payload_dict)
|
||||
|
||||
async def _persistence_delete_binding(self, node_type: str, local_id: str) -> None:
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.delete_binding(node_type, local_id)
|
||||
return
|
||||
await self.persistence.delete_binding(self.scope, node_type, local_id)
|
||||
|
||||
async def _persistence_delete_bindings_bulk(self, node_type: str, local_ids: List[str]) -> None:
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.delete_bindings_bulk(node_type, local_ids)
|
||||
return
|
||||
await self.persistence.delete_bindings_bulk(self.scope, node_type, local_ids)
|
||||
|
||||
async def _persistence_save_bindings_bulk(self, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.save_bindings_bulk(node_type, bindings)
|
||||
return
|
||||
await self.persistence.save_bindings_bulk(self.scope, node_type, bindings)
|
||||
|
||||
async def _persistence_load_bindings(self, node_type: str) -> List[Dict[str, Any]]:
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
return await self.persistence.load_bindings(node_type)
|
||||
return await self.persistence.load_bindings(self.scope, node_type)
|
||||
|
||||
def _get_type_bindings(self, node_type: str) -> Dict[str, Optional[str]]:
|
||||
if node_type not in self._bindings:
|
||||
self._bindings[node_type] = {}
|
||||
@@ -56,7 +98,7 @@ class BindingManager:
|
||||
logger.debug(log_line)
|
||||
|
||||
if self.auto_persist:
|
||||
await self.persistence.save_binding(self.scope, node_type, local_id, {"remote_id": remote_id})
|
||||
await self._persistence_save_binding(node_type, local_id, {"remote_id": remote_id})
|
||||
|
||||
async def unbind(self, node_type: str, local_id: str, manual: bool = False):
|
||||
"""
|
||||
@@ -81,14 +123,19 @@ class BindingManager:
|
||||
)
|
||||
|
||||
if self.auto_persist:
|
||||
await self.persistence.delete_binding(self.scope, node_type, local_id)
|
||||
await self._persistence_delete_binding(node_type, local_id)
|
||||
else:
|
||||
self._get_deleted(node_type).add(local_id)
|
||||
|
||||
async def get_remote_id(self, node_type: str, local_id: str) -> Optional[str]:
|
||||
"""查询本地节点对应的远程 ID"""
|
||||
bindings = self._get_type_bindings(node_type)
|
||||
return bindings.get(local_id)
|
||||
remote_id = bindings.get(local_id)
|
||||
if remote_id is not None or local_id in bindings:
|
||||
return remote_id
|
||||
if self._global_fallback_manager is None:
|
||||
return None
|
||||
return await self._global_fallback_manager.get_remote_id(node_type, local_id)
|
||||
|
||||
async def get_local_id(self, node_type: str, remote_id: str) -> Optional[str]:
|
||||
"""查询远程节点对应的本地 ID (内存遍历)"""
|
||||
@@ -96,11 +143,13 @@ class BindingManager:
|
||||
for lid, rid in bindings.items():
|
||||
if rid == remote_id:
|
||||
return lid
|
||||
return None
|
||||
if self._global_fallback_manager is None:
|
||||
return None
|
||||
return await self._global_fallback_manager.get_local_id(node_type, remote_id)
|
||||
|
||||
async def get_record(self, node_type: str, local_id: str) -> Optional[BindingRecord]:
|
||||
"""获取结构化绑定记录"""
|
||||
remote_id = self._get_type_bindings(node_type).get(local_id)
|
||||
remote_id = await self.get_remote_id(node_type, local_id)
|
||||
if remote_id is None:
|
||||
return None
|
||||
return BindingRecord(
|
||||
@@ -120,7 +169,7 @@ class BindingManager:
|
||||
|
||||
async def load_from_persistence(self, node_type: str):
|
||||
"""从持久化层加载特定类型的绑定关系到内存"""
|
||||
raw_bindings = await self.persistence.load_bindings(self.scope, node_type)
|
||||
raw_bindings = await self._persistence_load_bindings(node_type)
|
||||
bindings = self._get_type_bindings(node_type)
|
||||
bindings.clear()
|
||||
self._get_deleted(node_type).clear()
|
||||
@@ -163,7 +212,7 @@ class BindingManager:
|
||||
del bindings[local_id]
|
||||
deleted_count += 1
|
||||
if self.auto_persist:
|
||||
await self.persistence.delete_binding(self.scope, node_type, local_id)
|
||||
await self._persistence_delete_binding(node_type, local_id)
|
||||
else:
|
||||
self._get_deleted(node_type).add(local_id)
|
||||
|
||||
@@ -196,13 +245,19 @@ class BindingManager:
|
||||
touched_types.append(node_type)
|
||||
|
||||
if deleted:
|
||||
await self.persistence.delete_bindings_bulk(self.scope, node_type, deleted)
|
||||
await self._persistence_delete_bindings_bulk(node_type, deleted)
|
||||
self._get_deleted(node_type).clear()
|
||||
|
||||
if bindings:
|
||||
await self.persistence.save_bindings_bulk(self.scope, node_type, bindings)
|
||||
await self._persistence_save_bindings_bulk(node_type, bindings)
|
||||
|
||||
touched_text = ",".join(touched_types) if touched_types else "none"
|
||||
logger.info(
|
||||
f"🔍 [BindingManager.persist] 完成: scope={self.scope}, types={type_count}, saved={saved_total}, deleted={deleted_total}, touched=[{touched_text}]"
|
||||
f"🔍 [BindingManager.persist] 完成: scope={get_scope_display_name(self.scope)}, types={type_count}, saved={saved_total}, deleted={deleted_total}, touched=[{touched_text}]"
|
||||
)
|
||||
|
||||
async def unload(self, *, node_types: Optional[List[str]] = None) -> None:
|
||||
target_types = list(node_types) if node_types is not None else list(self._bindings.keys())
|
||||
for node_type in target_types:
|
||||
self._bindings.pop(node_type, None)
|
||||
self._deleted.pop(node_type, None)
|
||||
|
||||
@@ -5,6 +5,7 @@ from typing import Dict, List, Optional, Any, Set, Callable, Literal
|
||||
from .sync_node import SyncNode
|
||||
from .types import SyncStatus
|
||||
from .persistence import PersistenceBackend
|
||||
from .persistence_service import ScopedPersistenceView
|
||||
|
||||
|
||||
class DataCollection:
|
||||
@@ -24,18 +25,22 @@ class DataCollection:
|
||||
self,
|
||||
collection_id: str,
|
||||
scope: str = "global",
|
||||
persistence: Optional[PersistenceBackend] = None,
|
||||
persistence: Optional[PersistenceBackend | ScopedPersistenceView] = None,
|
||||
auto_persist: bool = False,
|
||||
sm_runtime: Optional[Any] = None,
|
||||
global_fallback_collection: Optional["DataCollection"] = None,
|
||||
):
|
||||
self.collection_id = collection_id
|
||||
self.scope = scope
|
||||
self.persistence = persistence
|
||||
self.auto_persist = auto_persist
|
||||
self._sm_runtime = sm_runtime
|
||||
self._global_fallback_collection = global_fallback_collection
|
||||
self._nodes: Dict[str, SyncNode] = {} # 内存缓存
|
||||
# data_id 全局唯一(按 collection 维度),用于 O(1) 反查 node_id
|
||||
self._data_id_to_node_id: Dict[str, str] = {}
|
||||
# node_id -> data_id 反向索引,用于 O(1) 维护 data_id 更新
|
||||
self._node_id_to_data_id: Dict[str, str] = {}
|
||||
# auto_persist=False 时,delete() 不会立即落库;这里记录删除集合,persist() 时统一写回
|
||||
self._deleted_node_ids: set[str] = set()
|
||||
|
||||
@@ -51,6 +56,83 @@ class DataCollection:
|
||||
"""设置 Collection 默认状态机 runtime。"""
|
||||
self._sm_runtime = runtime
|
||||
|
||||
def set_global_fallback_collection(self, collection: Optional["DataCollection"]) -> None:
|
||||
self._global_fallback_collection = collection
|
||||
|
||||
def _get_global_fallback(self) -> Optional["DataCollection"]:
|
||||
return self._global_fallback_collection
|
||||
|
||||
async def _persistence_save_node(self, node: SyncNode) -> None:
|
||||
if not self.persistence:
|
||||
return
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.save_node(self.collection_id, node)
|
||||
return
|
||||
await self.persistence.save_node(self.collection_id, self.scope, node)
|
||||
|
||||
async def _persistence_delete_node(self, node_id: str) -> None:
|
||||
if not self.persistence:
|
||||
return
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.delete_node(self.collection_id, node_id)
|
||||
return
|
||||
await self.persistence.delete_node(self.collection_id, self.scope, node_id)
|
||||
|
||||
async def _persistence_delete_nodes_bulk(self, node_ids: List[str]) -> None:
|
||||
if not self.persistence:
|
||||
return
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.delete_nodes_bulk(self.collection_id, node_ids)
|
||||
return
|
||||
await self.persistence.delete_nodes_bulk(self.collection_id, self.scope, node_ids)
|
||||
|
||||
async def _persistence_save_nodes_bulk(self, nodes: List[SyncNode]) -> None:
|
||||
if not self.persistence:
|
||||
return
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
await self.persistence.save_nodes_bulk(self.collection_id, nodes)
|
||||
return
|
||||
await self.persistence.save_nodes_bulk(self.collection_id, self.scope, nodes)
|
||||
|
||||
async def _persistence_load_nodes(
|
||||
self,
|
||||
*,
|
||||
exclude_node_types: Optional[List[str]] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
if not self.persistence:
|
||||
return []
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
return await self.persistence.load_nodes(
|
||||
self.collection_id,
|
||||
exclude_node_types=exclude_node_types,
|
||||
)
|
||||
return await self.persistence.load_nodes(
|
||||
self.collection_id,
|
||||
self.scope,
|
||||
exclude_node_types=exclude_node_types,
|
||||
)
|
||||
|
||||
async def _persistence_load_bind_data_id_overrides(self, node_type: str) -> Dict[str, Optional[str]]:
|
||||
if not self.persistence:
|
||||
return {}
|
||||
if isinstance(self.persistence, ScopedPersistenceView):
|
||||
return await self.persistence.load_bind_data_id_overrides(node_type)
|
||||
return await self.persistence.load_bind_data_id_overrides(node_type, self.scope)
|
||||
|
||||
def _remove_data_id_index(self, node_id: str, data_id: Optional[str] = None) -> None:
|
||||
target_data_id = data_id
|
||||
if target_data_id is None:
|
||||
target_data_id = self._node_id_to_data_id.pop(node_id, None)
|
||||
else:
|
||||
self._node_id_to_data_id.pop(node_id, None)
|
||||
|
||||
if target_data_id and self._data_id_to_node_id.get(target_data_id) == node_id:
|
||||
self._data_id_to_node_id.pop(target_data_id, None)
|
||||
|
||||
def _set_data_id_index(self, node_id: str, data_id: str) -> None:
|
||||
self._data_id_to_node_id[data_id] = node_id
|
||||
self._node_id_to_data_id[node_id] = data_id
|
||||
|
||||
async def add(self, node: SyncNode):
|
||||
# 检查 node_id 唯一性
|
||||
if node.node_id in self._nodes:
|
||||
@@ -65,18 +147,17 @@ class DataCollection:
|
||||
raise ValueError(
|
||||
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
|
||||
)
|
||||
self._data_id_to_node_id[node.data_id] = node.node_id
|
||||
self._set_data_id_index(node.node_id, node.data_id)
|
||||
|
||||
self._nodes[node.node_id] = node
|
||||
self._deleted_node_ids.discard(node.node_id)
|
||||
if self.persistence and self.auto_persist:
|
||||
await self.persistence.save_node(self.collection_id, self.scope, node)
|
||||
await self._persistence_save_node(node)
|
||||
|
||||
async def update(self, node: SyncNode):
|
||||
old_node = self._nodes.get(node.node_id)
|
||||
if old_node and old_node.data_id and old_node.data_id != "" and old_node.data_id != node.data_id:
|
||||
# node_id 不变但 data_id 变更,清理旧索引
|
||||
self._data_id_to_node_id.pop(old_node.data_id, None)
|
||||
old_data_id = self._node_id_to_data_id.get(node.node_id)
|
||||
if old_data_id and old_data_id != node.data_id:
|
||||
self._remove_data_id_index(node.node_id, old_data_id)
|
||||
|
||||
if node.data_id and node.data_id != "":
|
||||
existing_node_id = self._data_id_to_node_id.get(node.data_id)
|
||||
@@ -84,27 +165,33 @@ class DataCollection:
|
||||
raise ValueError(
|
||||
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
|
||||
)
|
||||
self._data_id_to_node_id[node.data_id] = node.node_id
|
||||
self._set_data_id_index(node.node_id, node.data_id)
|
||||
else:
|
||||
self._node_id_to_data_id.pop(node.node_id, None)
|
||||
self._nodes[node.node_id] = node
|
||||
self._deleted_node_ids.discard(node.node_id)
|
||||
if self.persistence and self.auto_persist:
|
||||
await self.persistence.save_node(self.collection_id, self.scope, node)
|
||||
await self._persistence_save_node(node)
|
||||
|
||||
async def delete(self, node_id: str):
|
||||
old_node = self._nodes.get(node_id)
|
||||
if old_node and old_node.data_id and old_node.data_id != "":
|
||||
self._data_id_to_node_id.pop(old_node.data_id, None)
|
||||
self._remove_data_id_index(node_id)
|
||||
|
||||
if node_id in self._nodes:
|
||||
del self._nodes[node_id]
|
||||
if self.persistence:
|
||||
if self.auto_persist:
|
||||
await self.persistence.delete_node(self.collection_id, self.scope, node_id)
|
||||
await self._persistence_delete_node(node_id)
|
||||
else:
|
||||
self._deleted_node_ids.add(node_id)
|
||||
|
||||
def get(self, node_id: str) -> Optional[SyncNode]:
|
||||
return self._nodes.get(node_id)
|
||||
node = self._nodes.get(node_id)
|
||||
if node is not None:
|
||||
return node
|
||||
fallback = self._get_global_fallback()
|
||||
if fallback is None:
|
||||
return None
|
||||
return fallback.get(node_id)
|
||||
|
||||
def get_by_node_id(self, node_id: str) -> Optional[SyncNode]:
|
||||
"""别名方法,与 get() 功能相同"""
|
||||
@@ -116,14 +203,23 @@ class DataCollection:
|
||||
|
||||
def get_node_id_by_data_id(self, data_id: str) -> Optional[str]:
|
||||
"""按 data_id 反查 node_id(O(1))。"""
|
||||
return self._data_id_to_node_id.get(data_id)
|
||||
node_id = self._data_id_to_node_id.get(data_id)
|
||||
if node_id is not None:
|
||||
return node_id
|
||||
fallback = self._get_global_fallback()
|
||||
if fallback is None:
|
||||
return None
|
||||
return fallback.get_node_id_by_data_id(data_id)
|
||||
|
||||
def get_by_data_id_any(self, data_id: str) -> Optional[SyncNode]:
|
||||
"""按 data_id 查找节点(不要求 node_type)(O(1))。"""
|
||||
node_id = self._data_id_to_node_id.get(data_id)
|
||||
if not node_id:
|
||||
if node_id:
|
||||
return self._nodes.get(node_id)
|
||||
fallback = self._get_global_fallback()
|
||||
if fallback is None:
|
||||
return None
|
||||
return self._nodes.get(node_id)
|
||||
return fallback.get_by_data_id_any(data_id)
|
||||
|
||||
def get_by_data_id(self, node_type: str, data_id: str) -> Optional[SyncNode]:
|
||||
"""按 node_type 和 data_id 查找节点"""
|
||||
@@ -362,7 +458,7 @@ class DataCollection:
|
||||
- data/origin_data 保留(由 DataSource 覆盖)
|
||||
|
||||
**初始化归并(E01)**:
|
||||
- 由 Pipeline 在加载后调用 run_reset() 触发 E01
|
||||
- 由 Pipeline 在加载后调用 `run_phase2_cleanup()` 触发 E01
|
||||
- CREATE 僵尸节点进入 S15 并删除
|
||||
- 其余节点进入 S00(UNCHECKED/NONE/PENDING)
|
||||
"""
|
||||
@@ -371,9 +467,10 @@ class DataCollection:
|
||||
|
||||
self._nodes = {}
|
||||
self._data_id_to_node_id = {}
|
||||
self._node_id_to_data_id = {}
|
||||
self._deleted_node_ids.clear()
|
||||
|
||||
node_dicts = await self.persistence.load_nodes(self.collection_id, self.scope)
|
||||
node_dicts = await self._persistence_load_nodes()
|
||||
|
||||
from .types import SyncAction, SyncStatus, BindingStatus
|
||||
from .registry import DomainRegistry
|
||||
@@ -472,6 +569,7 @@ class DataCollection:
|
||||
context=context_value,
|
||||
)
|
||||
node.context["_loaded_from_persistence"] = True
|
||||
node.append_log("加载来源: persistence 恢复节点")
|
||||
except Exception:
|
||||
import logging
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -490,11 +588,11 @@ class DataCollection:
|
||||
raise ValueError(
|
||||
f"Duplicate data_id detected while loading: {node.data_id} already bound to node_id={existing_node_id}"
|
||||
)
|
||||
self._data_id_to_node_id[node.data_id] = node.node_id
|
||||
self._set_data_id_index(node.node_id, node.data_id)
|
||||
|
||||
node_types = {node.node_type for node in self._nodes.values()}
|
||||
for node_type in node_types:
|
||||
override_map = await self.persistence.load_bind_data_id_overrides(node_type, self.scope)
|
||||
override_map = await self._persistence_load_bind_data_id_overrides(node_type)
|
||||
for node in self.filter(node_type=node_type):
|
||||
bind_data_id = override_map.get(node.node_id)
|
||||
if bind_data_id:
|
||||
@@ -509,13 +607,10 @@ class DataCollection:
|
||||
|
||||
self._nodes = {}
|
||||
self._data_id_to_node_id = {}
|
||||
self._node_id_to_data_id = {}
|
||||
self._deleted_node_ids.clear()
|
||||
|
||||
node_dicts = await self.persistence.load_nodes(
|
||||
self.collection_id,
|
||||
self.scope,
|
||||
exclude_node_types=node_types,
|
||||
)
|
||||
node_dicts = await self._persistence_load_nodes(exclude_node_types=node_types)
|
||||
|
||||
from .types import SyncAction, SyncStatus, BindingStatus
|
||||
from .registry import DomainRegistry
|
||||
@@ -606,6 +701,7 @@ class DataCollection:
|
||||
context=context_value,
|
||||
)
|
||||
node.context["_loaded_from_persistence"] = True
|
||||
node.append_log("加载来源: persistence 恢复节点")
|
||||
self._nodes[node.node_id] = node
|
||||
|
||||
if node.data_id and node.data_id != "":
|
||||
@@ -614,11 +710,11 @@ class DataCollection:
|
||||
raise ValueError(
|
||||
f"Duplicate data_id detected while loading: {node.data_id} already bound to node_id={existing_node_id}"
|
||||
)
|
||||
self._data_id_to_node_id[node.data_id] = node.node_id
|
||||
self._set_data_id_index(node.node_id, node.data_id)
|
||||
|
||||
loaded_node_types = {node.node_type for node in self._nodes.values()}
|
||||
for node_type in loaded_node_types:
|
||||
override_map = await self.persistence.load_bind_data_id_overrides(node_type, self.scope)
|
||||
override_map = await self._persistence_load_bind_data_id_overrides(node_type)
|
||||
for node in self.filter(node_type=node_type):
|
||||
bind_data_id = override_map.get(node.node_id)
|
||||
if bind_data_id:
|
||||
@@ -639,7 +735,7 @@ class DataCollection:
|
||||
raise ValueError(
|
||||
f"Duplicate data_id detected: {node.data_id} already bound to node_id={existing_node_id}"
|
||||
)
|
||||
self._data_id_to_node_id[node.data_id] = node.node_id
|
||||
self._set_data_id_index(node.node_id, node.data_id)
|
||||
|
||||
self._nodes[node.node_id] = node
|
||||
self._deleted_node_ids.discard(node.node_id)
|
||||
@@ -651,7 +747,7 @@ class DataCollection:
|
||||
|
||||
# 先落库删除(用于 auto_persist=False 的场景)
|
||||
if self._deleted_node_ids:
|
||||
await self.persistence.delete_nodes_bulk(self.collection_id, self.scope, list(self._deleted_node_ids))
|
||||
await self._persistence_delete_nodes_bulk(list(self._deleted_node_ids))
|
||||
self._deleted_node_ids.clear()
|
||||
|
||||
if not self._nodes:
|
||||
@@ -664,7 +760,13 @@ class DataCollection:
|
||||
if not nodes_to_persist:
|
||||
return
|
||||
|
||||
await self.persistence.save_nodes_bulk(self.collection_id, self.scope, nodes_to_persist)
|
||||
await self._persistence_save_nodes_bulk(nodes_to_persist)
|
||||
|
||||
async def unload(self) -> None:
|
||||
self._nodes.clear()
|
||||
self._data_id_to_node_id.clear()
|
||||
self._node_id_to_data_id.clear()
|
||||
self._deleted_node_ids.clear()
|
||||
|
||||
async def filter_by_project_ids(self, data_id_filter: List[str]) -> int:
|
||||
"""按目标项目 data_id 集合过滤当前 collection,返回删除数量。"""
|
||||
|
||||
@@ -5,6 +5,9 @@ import sqlite3
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any, Callable, ClassVar, Dict, List, Optional, TYPE_CHECKING
|
||||
|
||||
from sqlalchemy import create_engine, inspect, text
|
||||
from sqlalchemy.engine import URL, make_url
|
||||
|
||||
from ..config.persist_config import PersistConfig
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -26,72 +29,153 @@ class PersistenceBackend(ABC):
|
||||
cls,
|
||||
*,
|
||||
backend: str,
|
||||
db_path: str,
|
||||
db_path: str | None = None,
|
||||
url: str | None = None,
|
||||
sql: str,
|
||||
limit: int = 200,
|
||||
) -> tuple[List[str], List[Dict[str, Any]]]:
|
||||
if backend != "sqlite":
|
||||
raise NotImplementedError(f"records query backend not supported yet: {backend}")
|
||||
if not cls._is_read_only_sql(sql):
|
||||
raise ValueError("Only read-only SQL is allowed")
|
||||
|
||||
path = db_path
|
||||
if not os.path.exists(path):
|
||||
raise FileNotFoundError(f"DB not found: {path}")
|
||||
if backend == "sqlite":
|
||||
path = db_path
|
||||
if not path:
|
||||
raise ValueError("db_path is required for backend=sqlite")
|
||||
if not os.path.exists(path):
|
||||
raise FileNotFoundError(f"DB not found: {path}")
|
||||
|
||||
with sqlite3.connect(path) as conn:
|
||||
conn.row_factory = sqlite3.Row
|
||||
cursor = conn.execute(sql)
|
||||
rows = cursor.fetchmany(max(1, min(limit, 2000)))
|
||||
columns = [item[0] for item in (cursor.description or [])]
|
||||
data = [{k: row[k] for k in columns} for row in rows]
|
||||
return columns, data
|
||||
with sqlite3.connect(path) as conn:
|
||||
conn.row_factory = sqlite3.Row
|
||||
cursor = conn.execute(sql)
|
||||
rows = cursor.fetchmany(max(1, min(limit, 2000)))
|
||||
columns = [item[0] for item in (cursor.description or [])]
|
||||
data = [{k: row[k] for k in columns} for row in rows]
|
||||
return columns, data
|
||||
|
||||
if backend == "sqlalchemy":
|
||||
if not url:
|
||||
raise ValueError("url is required for backend=sqlalchemy")
|
||||
engine = create_engine(cls._to_sync_sqlalchemy_url(url))
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
result = conn.execute(text(sql))
|
||||
rows = result.fetchmany(max(1, min(limit, 2000)))
|
||||
columns = list(result.keys())
|
||||
data = [{column: row[index] for index, column in enumerate(columns)} for row in rows]
|
||||
return columns, data
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
raise NotImplementedError(f"records query backend not supported yet: {backend}")
|
||||
|
||||
@classmethod
|
||||
def records_summary(
|
||||
cls,
|
||||
*,
|
||||
backend: str,
|
||||
db_path: str,
|
||||
db_path: str | None = None,
|
||||
url: str | None = None,
|
||||
) -> Dict[str, Any]:
|
||||
db_exists = cls._target_exists(backend=backend, db_path=db_path, url=url)
|
||||
summary: Dict[str, Any] = {
|
||||
"backend": backend,
|
||||
"db_path": db_path,
|
||||
"db_exists": os.path.exists(db_path),
|
||||
"url": url,
|
||||
"db_exists": db_exists,
|
||||
"tables": [],
|
||||
"sample_bindings": [],
|
||||
}
|
||||
if not os.path.exists(db_path):
|
||||
return summary
|
||||
if backend != "sqlite":
|
||||
summary["error"] = f"records summary backend not supported yet: {backend}"
|
||||
if not db_exists:
|
||||
return summary
|
||||
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.row_factory = sqlite3.Row
|
||||
tables = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name"
|
||||
).fetchall()
|
||||
table_names = [str(r["name"]) for r in tables]
|
||||
|
||||
for name in table_names:
|
||||
count = conn.execute(f"SELECT COUNT(1) AS c FROM {name}").fetchone()["c"]
|
||||
summary["tables"].append({"name": name, "count": int(count)})
|
||||
|
||||
if "bindings" in table_names:
|
||||
sample = conn.execute(
|
||||
"SELECT node_type, local_id, remote_id FROM bindings ORDER BY last_updated DESC LIMIT 50"
|
||||
if backend == "sqlite":
|
||||
assert db_path is not None
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.row_factory = sqlite3.Row
|
||||
tables = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name"
|
||||
).fetchall()
|
||||
summary["sample_bindings"] = [
|
||||
{
|
||||
"node_type": row["node_type"],
|
||||
"local_id": row["local_id"],
|
||||
"remote_id": row["remote_id"],
|
||||
}
|
||||
for row in sample
|
||||
]
|
||||
table_names = [str(r["name"]) for r in tables]
|
||||
|
||||
for name in table_names:
|
||||
count = conn.execute(f"SELECT COUNT(1) AS c FROM {name}").fetchone()["c"]
|
||||
summary["tables"].append({"name": name, "count": int(count)})
|
||||
|
||||
if "bindings" in table_names:
|
||||
sample = conn.execute(
|
||||
"SELECT node_type, local_id, remote_id FROM bindings ORDER BY last_updated DESC LIMIT 50"
|
||||
).fetchall()
|
||||
summary["sample_bindings"] = [
|
||||
{
|
||||
"node_type": row["node_type"],
|
||||
"local_id": row["local_id"],
|
||||
"remote_id": row["remote_id"],
|
||||
}
|
||||
for row in sample
|
||||
]
|
||||
return summary
|
||||
|
||||
if backend == "sqlalchemy":
|
||||
if not url:
|
||||
summary["error"] = "url is required for backend=sqlalchemy"
|
||||
return summary
|
||||
engine = create_engine(cls._to_sync_sqlalchemy_url(url))
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
inspector = inspect(conn)
|
||||
table_names = sorted(inspector.get_table_names())
|
||||
|
||||
for name in table_names:
|
||||
count = conn.execute(text(f"SELECT COUNT(1) AS c FROM {name}")).scalar_one()
|
||||
summary["tables"].append({"name": name, "count": int(count)})
|
||||
|
||||
if "bindings" in table_names:
|
||||
sample = conn.execute(
|
||||
text("SELECT node_type, local_id, remote_id FROM bindings ORDER BY last_updated DESC LIMIT 50")
|
||||
).fetchall()
|
||||
summary["sample_bindings"] = [
|
||||
{
|
||||
"node_type": row[0],
|
||||
"local_id": row[1],
|
||||
"remote_id": row[2],
|
||||
}
|
||||
for row in sample
|
||||
]
|
||||
finally:
|
||||
engine.dispose()
|
||||
return summary
|
||||
|
||||
summary["error"] = f"records summary backend not supported yet: {backend}"
|
||||
return summary
|
||||
|
||||
@staticmethod
|
||||
def _target_exists(*, backend: str, db_path: str | None, url: str | None) -> bool:
|
||||
if backend == "sqlite":
|
||||
return bool(db_path) and os.path.exists(db_path)
|
||||
if backend == "sqlalchemy":
|
||||
if not url:
|
||||
return False
|
||||
parsed = make_url(url)
|
||||
if parsed.get_backend_name() == "sqlite":
|
||||
database = parsed.database
|
||||
if not database or database == ":memory:":
|
||||
return True
|
||||
return os.path.exists(database)
|
||||
return True
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def _to_sync_sqlalchemy_url(url: str) -> str:
|
||||
parsed = make_url(url)
|
||||
async_to_sync = {
|
||||
"sqlite+aiosqlite": "sqlite+pysqlite",
|
||||
"postgresql+asyncpg": "postgresql+psycopg",
|
||||
"mysql+aiomysql": "mysql+pymysql",
|
||||
}
|
||||
drivername = async_to_sync.get(parsed.drivername, parsed.drivername)
|
||||
sync_url: URL = parsed.set(drivername=drivername)
|
||||
return str(sync_url)
|
||||
|
||||
@abstractmethod
|
||||
async def initialize(self) -> None:
|
||||
raise NotImplementedError
|
||||
@@ -218,9 +302,11 @@ def create_persistence_backend(
|
||||
|
||||
|
||||
def _register_builtin_persistence_backends() -> None:
|
||||
from .persistence_backends.sqlalchemy_backend import SQLAlchemyPersistenceBackend
|
||||
from .persistence_backends.sqlite_backend import SQLitePersistenceBackend
|
||||
|
||||
register_persistence_backend("sqlite", lambda config: SQLitePersistenceBackend(config), override=True)
|
||||
register_persistence_backend("sqlalchemy", lambda config: SQLAlchemyPersistenceBackend(config), override=True)
|
||||
|
||||
|
||||
_register_builtin_persistence_backends()
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
from .sqlalchemy_backend import SQLAlchemyPersistenceBackend
|
||||
from .sqlite_backend import SQLitePersistenceBackend
|
||||
|
||||
__all__ = ["SQLitePersistenceBackend"]
|
||||
__all__ = ["SQLitePersistenceBackend", "SQLAlchemyPersistenceBackend"]
|
||||
@@ -0,0 +1,428 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import Any, Dict, List, Optional, TYPE_CHECKING
|
||||
|
||||
from sqlalchemy import Column, Index, MetaData, PrimaryKeyConstraint, String, Table, Text, delete, insert, select
|
||||
from sqlalchemy.engine import make_url
|
||||
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncConnection, create_async_engine
|
||||
|
||||
from ...config.persist_config import PersistConfig
|
||||
from ..persistence import PersistenceBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..sync_node import SyncNode
|
||||
|
||||
|
||||
class SQLAlchemyPersistenceBackend(PersistenceBackend):
|
||||
"""Generic SQLAlchemy persistence backend with explicit schema bootstrap."""
|
||||
|
||||
backend_name = "sqlalchemy"
|
||||
bulk_delete_batch_size = 1000
|
||||
collection_id_length = 255
|
||||
scope_length = 255
|
||||
node_id_length = 255
|
||||
node_type_length = 128
|
||||
data_id_length = 255
|
||||
bind_data_id_length = 255
|
||||
action_length = 32
|
||||
status_length = 32
|
||||
binding_status_length = 32
|
||||
remote_id_length = 255
|
||||
last_updated_length = 64
|
||||
|
||||
def __init__(self, persist_config: PersistConfig | str | None = None, **_: Any) -> None:
|
||||
if isinstance(persist_config, PersistConfig):
|
||||
self.persist_config = persist_config
|
||||
self.database_url = persist_config.resolved_url()
|
||||
self.db_path = persist_config.resolved_db_path()
|
||||
self.backend_options = dict(persist_config.backend_options)
|
||||
elif isinstance(persist_config, str):
|
||||
self.persist_config = None
|
||||
self.database_url = persist_config
|
||||
parsed = make_url(persist_config)
|
||||
self.db_path = str(parsed.database) if parsed.get_backend_name() == "sqlite" and parsed.database else None
|
||||
self.backend_options = {}
|
||||
else:
|
||||
raise ValueError("SQLAlchemyPersistenceBackend requires PersistConfig or database URL")
|
||||
|
||||
self.engine: Optional[AsyncEngine] = None
|
||||
self.metadata = MetaData()
|
||||
self.nodes = Table(
|
||||
"nodes",
|
||||
self.metadata,
|
||||
Column("collection_id", String(self.collection_id_length), nullable=False),
|
||||
Column("scope", String(self.scope_length), nullable=False, default="global"),
|
||||
Column("node_id", String(self.node_id_length), nullable=False),
|
||||
Column("node_type", String(self.node_type_length), nullable=False),
|
||||
Column("data_id", String(self.data_id_length), nullable=True),
|
||||
Column("bind_data_id", String(self.bind_data_id_length), nullable=True),
|
||||
Column("data", Text, nullable=True),
|
||||
Column("origin_data", Text, nullable=True),
|
||||
Column("action", String(self.action_length), nullable=True),
|
||||
Column("status", String(self.status_length), nullable=True),
|
||||
Column("binding_status", String(self.binding_status_length), nullable=True),
|
||||
Column("error", Text, nullable=True),
|
||||
Column("sync_log", Text, nullable=True),
|
||||
Column("context", Text, nullable=True),
|
||||
PrimaryKeyConstraint("collection_id", "scope", "node_id", name="pk_nodes"),
|
||||
Index("idx_nodes_collection_scope_type", "collection_id", "scope", "node_type"),
|
||||
Index("idx_nodes_collection_scope_data_id", "collection_id", "scope", "data_id"),
|
||||
Index("idx_nodes_scope_type_node_id", "scope", "node_type", "node_id"),
|
||||
)
|
||||
self.bindings = Table(
|
||||
"bindings",
|
||||
self.metadata,
|
||||
Column("scope", String(self.scope_length), nullable=False, default="global"),
|
||||
Column("node_type", String(self.node_type_length), nullable=False),
|
||||
Column("local_id", String(self.node_id_length), nullable=False),
|
||||
Column("remote_id", String(self.remote_id_length), nullable=True),
|
||||
Column("last_updated", String(self.last_updated_length), nullable=True),
|
||||
PrimaryKeyConstraint("scope", "node_type", "local_id", name="pk_bindings"),
|
||||
Index("idx_bindings_scope_type_remote", "scope", "node_type", "remote_id"),
|
||||
)
|
||||
|
||||
async def initialize(self) -> None:
|
||||
if self.db_path and self.db_path != ":memory:":
|
||||
dir_path = os.path.dirname(self.db_path)
|
||||
if dir_path:
|
||||
os.makedirs(dir_path, exist_ok=True)
|
||||
|
||||
engine_options = {"future": True}
|
||||
engine_options.update(self.backend_options)
|
||||
self.engine = create_async_engine(self.database_url, **engine_options)
|
||||
async with self.engine.begin() as conn:
|
||||
await conn.run_sync(self.metadata.create_all)
|
||||
|
||||
async def save_node(self, collection_id: str, scope: str, node: "SyncNode") -> None:
|
||||
row = self._node_row(collection_id=collection_id, scope=scope, node=node)
|
||||
async with self._begin() as conn:
|
||||
await self._replace_node_rows(conn, collection_id=collection_id, scope=scope, node_ids=[node.node_id])
|
||||
await conn.execute(insert(self.nodes), [row])
|
||||
|
||||
async def save_nodes_bulk(self, collection_id: str, scope: str, nodes: List["SyncNode"]) -> None:
|
||||
if not nodes:
|
||||
return
|
||||
rows = [self._node_row(collection_id=collection_id, scope=scope, node=node) for node in nodes]
|
||||
node_ids = [node.node_id for node in nodes]
|
||||
async with self._begin() as conn:
|
||||
await self._replace_node_rows(conn, collection_id=collection_id, scope=scope, node_ids=node_ids)
|
||||
await conn.execute(insert(self.nodes), rows)
|
||||
|
||||
async def delete_node(self, collection_id: str, scope: str, node_id: str) -> None:
|
||||
async with self._begin() as conn:
|
||||
await conn.execute(
|
||||
delete(self.nodes).where(
|
||||
self.nodes.c.collection_id == collection_id,
|
||||
self.nodes.c.scope == scope,
|
||||
self.nodes.c.node_id == node_id,
|
||||
)
|
||||
)
|
||||
|
||||
async def delete_nodes_bulk(self, collection_id: str, scope: str, node_ids: List[str]) -> None:
|
||||
if not node_ids:
|
||||
return
|
||||
async with self._begin() as conn:
|
||||
for batch in self._chunked(node_ids):
|
||||
await conn.execute(
|
||||
delete(self.nodes).where(
|
||||
self.nodes.c.collection_id == collection_id,
|
||||
self.nodes.c.scope == scope,
|
||||
self.nodes.c.node_id.in_(batch),
|
||||
)
|
||||
)
|
||||
|
||||
async def load_nodes(
|
||||
self,
|
||||
collection_id: str,
|
||||
scope: str,
|
||||
*,
|
||||
exclude_node_types: Optional[List[str]] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
stmt = select(self.nodes).where(
|
||||
self.nodes.c.collection_id == collection_id,
|
||||
self.nodes.c.scope == scope,
|
||||
)
|
||||
filtered_types = [node_type for node_type in (exclude_node_types or []) if node_type]
|
||||
if filtered_types:
|
||||
stmt = stmt.where(self.nodes.c.node_type.not_in(filtered_types))
|
||||
|
||||
async with self._connect() as conn:
|
||||
rows = (await conn.execute(stmt)).mappings().all()
|
||||
return [self._deserialize_node_row(row) for row in rows]
|
||||
|
||||
async def save_binding(self, scope: str, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
|
||||
row = self._binding_row(scope=scope, node_type=node_type, local_id=local_id, remote_id=payload_dict.get("remote_id") if payload_dict else None)
|
||||
async with self._begin() as conn:
|
||||
await self._replace_binding_rows(conn, scope=scope, node_type=node_type, local_ids=[local_id])
|
||||
await conn.execute(insert(self.bindings), [row])
|
||||
|
||||
async def save_bindings_bulk(self, scope: str, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
|
||||
if not bindings:
|
||||
return
|
||||
rows = [
|
||||
self._binding_row(scope=scope, node_type=node_type, local_id=local_id, remote_id=remote_id)
|
||||
for local_id, remote_id in bindings.items()
|
||||
]
|
||||
async with self._begin() as conn:
|
||||
await self._replace_binding_rows(conn, scope=scope, node_type=node_type, local_ids=list(bindings.keys()))
|
||||
await conn.execute(insert(self.bindings), rows)
|
||||
|
||||
async def delete_binding(self, scope: str, node_type: str, local_id: str) -> None:
|
||||
async with self._begin() as conn:
|
||||
await conn.execute(
|
||||
delete(self.bindings).where(
|
||||
self.bindings.c.scope == scope,
|
||||
self.bindings.c.node_type == node_type,
|
||||
self.bindings.c.local_id == local_id,
|
||||
)
|
||||
)
|
||||
|
||||
async def delete_bindings_bulk(self, scope: str, node_type: str, local_ids: List[str]) -> None:
|
||||
if not local_ids:
|
||||
return
|
||||
async with self._begin() as conn:
|
||||
for batch in self._chunked(local_ids):
|
||||
await conn.execute(
|
||||
delete(self.bindings).where(
|
||||
self.bindings.c.scope == scope,
|
||||
self.bindings.c.node_type == node_type,
|
||||
self.bindings.c.local_id.in_(batch),
|
||||
)
|
||||
)
|
||||
|
||||
async def load_bindings(self, scope: str, node_type: str) -> List[Dict[str, Any]]:
|
||||
stmt = select(self.bindings.c.local_id, self.bindings.c.remote_id).where(
|
||||
self.bindings.c.scope == scope,
|
||||
self.bindings.c.node_type == node_type,
|
||||
)
|
||||
async with self._connect() as conn:
|
||||
rows = (await conn.execute(stmt)).mappings().all()
|
||||
return [{"local_id": row["local_id"], "payload": {"remote_id": row["remote_id"]}} for row in rows]
|
||||
|
||||
async def load_bind_data_id_overrides(self, node_type: str, scope: str) -> Dict[str, Optional[str]]:
|
||||
local_join = (
|
||||
select(
|
||||
self.bindings.c.local_id.label("src_node_id"),
|
||||
self.nodes.c.data_id.label("bind_data_id"),
|
||||
)
|
||||
.select_from(
|
||||
self.bindings.outerjoin(
|
||||
self.nodes,
|
||||
(self.nodes.c.node_id == self.bindings.c.remote_id)
|
||||
& (self.nodes.c.node_type == self.bindings.c.node_type)
|
||||
& (self.nodes.c.scope == self.bindings.c.scope),
|
||||
)
|
||||
)
|
||||
.where(
|
||||
self.bindings.c.node_type == node_type,
|
||||
self.bindings.c.scope == scope,
|
||||
)
|
||||
)
|
||||
remote_join = (
|
||||
select(
|
||||
self.bindings.c.remote_id.label("src_node_id"),
|
||||
self.nodes.c.data_id.label("bind_data_id"),
|
||||
)
|
||||
.select_from(
|
||||
self.bindings.outerjoin(
|
||||
self.nodes,
|
||||
(self.nodes.c.node_id == self.bindings.c.local_id)
|
||||
& (self.nodes.c.node_type == self.bindings.c.node_type)
|
||||
& (self.nodes.c.scope == self.bindings.c.scope),
|
||||
)
|
||||
)
|
||||
.where(
|
||||
self.bindings.c.node_type == node_type,
|
||||
self.bindings.c.scope == scope,
|
||||
self.bindings.c.remote_id.is_not(None),
|
||||
)
|
||||
)
|
||||
async with self._connect() as conn:
|
||||
rows = (await conn.execute(local_join.union_all(remote_join))).mappings().all()
|
||||
result: Dict[str, Optional[str]] = {}
|
||||
for row in rows:
|
||||
src_node_id = row["src_node_id"]
|
||||
if src_node_id:
|
||||
result[str(src_node_id)] = row["bind_data_id"]
|
||||
return result
|
||||
|
||||
async def delete_scope_node_types(self, *, scope: str, node_types: List[str]) -> None:
|
||||
filtered_types = [node_type for node_type in node_types if node_type]
|
||||
if not filtered_types:
|
||||
return
|
||||
async with self._begin() as conn:
|
||||
await conn.execute(delete(self.nodes).where(self.nodes.c.scope == scope, self.nodes.c.node_type.in_(filtered_types)))
|
||||
await conn.execute(delete(self.bindings).where(self.bindings.c.scope == scope, self.bindings.c.node_type.in_(filtered_types)))
|
||||
|
||||
async def copy_nodes_between_scopes(
|
||||
self,
|
||||
*,
|
||||
collection_id: str,
|
||||
source_scope: str,
|
||||
target_scope: str,
|
||||
node_types: List[str],
|
||||
) -> None:
|
||||
if source_scope == target_scope or not node_types:
|
||||
return
|
||||
stmt = select(self.nodes).where(
|
||||
self.nodes.c.collection_id == collection_id,
|
||||
self.nodes.c.scope == source_scope,
|
||||
self.nodes.c.node_type.in_(node_types),
|
||||
)
|
||||
async with self._begin() as conn:
|
||||
rows = (await conn.execute(stmt)).mappings().all()
|
||||
target_rows = [dict(row, scope=target_scope) for row in rows]
|
||||
if not target_rows:
|
||||
return
|
||||
await self._replace_node_rows(
|
||||
conn,
|
||||
collection_id=collection_id,
|
||||
scope=target_scope,
|
||||
node_ids=[str(row["node_id"]) for row in target_rows],
|
||||
)
|
||||
await conn.execute(insert(self.nodes), target_rows)
|
||||
|
||||
async def copy_bindings_between_scopes(
|
||||
self,
|
||||
*,
|
||||
source_scope: str,
|
||||
target_scope: str,
|
||||
node_types: List[str],
|
||||
) -> None:
|
||||
if source_scope == target_scope or not node_types:
|
||||
return
|
||||
stmt = select(self.bindings).where(
|
||||
self.bindings.c.scope == source_scope,
|
||||
self.bindings.c.node_type.in_(node_types),
|
||||
)
|
||||
async with self._begin() as conn:
|
||||
rows = (await conn.execute(stmt)).mappings().all()
|
||||
target_rows = [dict(row, scope=target_scope) for row in rows]
|
||||
if not target_rows:
|
||||
return
|
||||
by_type: Dict[str, List[str]] = {}
|
||||
for row in target_rows:
|
||||
by_type.setdefault(str(row["node_type"]), []).append(str(row["local_id"]))
|
||||
for node_type, local_ids in by_type.items():
|
||||
await self._replace_binding_rows(conn, scope=target_scope, node_type=node_type, local_ids=local_ids)
|
||||
await conn.execute(insert(self.bindings), target_rows)
|
||||
|
||||
async def dump_to_runtime(self, filename: str) -> None:
|
||||
raise NotImplementedError("dump_to_runtime is only supported by the sqlite backend")
|
||||
|
||||
async def close(self) -> None:
|
||||
if self.engine is None:
|
||||
return
|
||||
await self.engine.dispose()
|
||||
self.engine = None
|
||||
|
||||
async def _replace_node_rows(
|
||||
self,
|
||||
conn: AsyncConnection,
|
||||
*,
|
||||
collection_id: str,
|
||||
scope: str,
|
||||
node_ids: List[str],
|
||||
) -> None:
|
||||
if not node_ids:
|
||||
return
|
||||
for batch in self._chunked(node_ids):
|
||||
await conn.execute(
|
||||
delete(self.nodes).where(
|
||||
self.nodes.c.collection_id == collection_id,
|
||||
self.nodes.c.scope == scope,
|
||||
self.nodes.c.node_id.in_(batch),
|
||||
)
|
||||
)
|
||||
|
||||
async def _replace_binding_rows(
|
||||
self,
|
||||
conn: AsyncConnection,
|
||||
*,
|
||||
scope: str,
|
||||
node_type: str,
|
||||
local_ids: List[str],
|
||||
) -> None:
|
||||
if not local_ids:
|
||||
return
|
||||
for batch in self._chunked(local_ids):
|
||||
await conn.execute(
|
||||
delete(self.bindings).where(
|
||||
self.bindings.c.scope == scope,
|
||||
self.bindings.c.node_type == node_type,
|
||||
self.bindings.c.local_id.in_(batch),
|
||||
)
|
||||
)
|
||||
|
||||
def _chunked(self, values: List[str]) -> List[List[str]]:
|
||||
if not values:
|
||||
return []
|
||||
batch_size = max(1, int(self.bulk_delete_batch_size))
|
||||
return [values[index : index + batch_size] for index in range(0, len(values), batch_size)]
|
||||
|
||||
def _node_row(self, *, collection_id: str, scope: str, node: "SyncNode") -> Dict[str, Any]:
|
||||
bind_data_id = None
|
||||
if isinstance(node.context, dict):
|
||||
value = node.context.get("bind_data_id")
|
||||
if value is not None and value != "":
|
||||
bind_data_id = str(value)
|
||||
return {
|
||||
"collection_id": collection_id,
|
||||
"scope": scope,
|
||||
"node_id": node.node_id,
|
||||
"node_type": node.node_type,
|
||||
"data_id": node.data_id,
|
||||
"bind_data_id": bind_data_id,
|
||||
"data": json.dumps(node.data) if node.data else None,
|
||||
"origin_data": json.dumps(node.origin_data) if node.origin_data else None,
|
||||
"action": node.action.value,
|
||||
"status": node.status.value,
|
||||
"binding_status": node.binding_status.value,
|
||||
"error": node.error,
|
||||
"sync_log": node.sync_log,
|
||||
"context": json.dumps(node.context) if node.context else None,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _binding_row(*, scope: str, node_type: str, local_id: str, remote_id: Optional[str]) -> Dict[str, Any]:
|
||||
return {
|
||||
"scope": scope,
|
||||
"node_type": node_type,
|
||||
"local_id": local_id,
|
||||
"remote_id": remote_id,
|
||||
"last_updated": None,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _deserialize_node_row(row: Dict[str, Any]) -> Dict[str, Any]:
|
||||
item = {
|
||||
"node_id": row["node_id"],
|
||||
"node_type": row["node_type"],
|
||||
"data_id": row["data_id"],
|
||||
"bind_data_id": row["bind_data_id"],
|
||||
"depend_ids": [],
|
||||
"data": json.loads(row["data"]) if row["data"] else None,
|
||||
"origin_data": json.loads(row["origin_data"]) if row["origin_data"] else None,
|
||||
"action": row["action"],
|
||||
"status": row["status"],
|
||||
"binding_status": row["binding_status"],
|
||||
"error": row["error"],
|
||||
"sync_log": row["sync_log"],
|
||||
"context": json.loads(row["context"]) if row["context"] else {},
|
||||
}
|
||||
if item["bind_data_id"] and isinstance(item["context"], dict):
|
||||
item["context"]["bind_data_id"] = item["bind_data_id"]
|
||||
return item
|
||||
|
||||
def _require_engine(self) -> AsyncEngine:
|
||||
if self.engine is None:
|
||||
raise RuntimeError("SQLAlchemyPersistenceBackend is not initialized")
|
||||
return self.engine
|
||||
|
||||
def _connect(self):
|
||||
return self._require_engine().connect()
|
||||
|
||||
def _begin(self):
|
||||
return self._require_engine().begin()
|
||||
@@ -20,7 +20,10 @@ class SQLitePersistenceBackend(PersistenceBackend):
|
||||
def __init__(self, persist_config: PersistConfig | str | None = None, **_: Any) -> None:
|
||||
if isinstance(persist_config, PersistConfig):
|
||||
self.persist_config = persist_config
|
||||
self.db_path = persist_config.db_path
|
||||
resolved_db_path = persist_config.resolved_db_path()
|
||||
if not resolved_db_path:
|
||||
raise ValueError("sqlite persistence requires a resolvable db_path")
|
||||
self.db_path = resolved_db_path
|
||||
elif isinstance(persist_config, str):
|
||||
self.persist_config = None
|
||||
self.db_path = persist_config
|
||||
@@ -112,6 +115,9 @@ class SQLitePersistenceBackend(PersistenceBackend):
|
||||
await self.conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_nodes_collection_scope_data_id ON nodes (collection_id, scope, data_id)"
|
||||
)
|
||||
await self.conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_nodes_scope_type_node_id ON nodes (scope, node_type, node_id)"
|
||||
)
|
||||
|
||||
async def _ensure_bindings_schema(self) -> None:
|
||||
assert self.conn is not None
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from .persistence import PersistenceBackend
|
||||
|
||||
|
||||
class PersistenceService:
|
||||
"""Shared persistence service that owns the backend lifecycle."""
|
||||
|
||||
def __init__(self, backend: PersistenceBackend):
|
||||
self._backend = backend
|
||||
|
||||
@property
|
||||
def backend(self) -> PersistenceBackend:
|
||||
return self._backend
|
||||
|
||||
async def initialize(self) -> None:
|
||||
await self._backend.initialize()
|
||||
|
||||
def view(self, scope: str) -> "ScopedPersistenceView":
|
||||
return ScopedPersistenceView(self._backend, scope=scope)
|
||||
|
||||
async def close(self) -> None:
|
||||
await self._backend.close()
|
||||
|
||||
|
||||
class ScopedPersistenceView:
|
||||
"""Thin scope-bound persistence view used by project runtimes."""
|
||||
|
||||
def __init__(self, backend: PersistenceBackend, *, scope: str):
|
||||
self._backend = backend
|
||||
self.scope = scope
|
||||
|
||||
@property
|
||||
def backend(self) -> PersistenceBackend:
|
||||
return self._backend
|
||||
|
||||
async def load_nodes(
|
||||
self,
|
||||
collection_id: str,
|
||||
*,
|
||||
exclude_node_types: Optional[List[str]] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
return await self._backend.load_nodes(
|
||||
collection_id,
|
||||
self.scope,
|
||||
exclude_node_types=exclude_node_types,
|
||||
)
|
||||
|
||||
async def save_node(self, collection_id: str, node) -> None:
|
||||
await self._backend.save_node(collection_id, self.scope, node)
|
||||
|
||||
async def save_nodes_bulk(self, collection_id: str, nodes) -> None:
|
||||
await self._backend.save_nodes_bulk(collection_id, self.scope, nodes)
|
||||
|
||||
async def delete_node(self, collection_id: str, node_id: str) -> None:
|
||||
await self._backend.delete_node(collection_id, self.scope, node_id)
|
||||
|
||||
async def delete_nodes_bulk(self, collection_id: str, node_ids: List[str]) -> None:
|
||||
await self._backend.delete_nodes_bulk(collection_id, self.scope, node_ids)
|
||||
|
||||
async def load_bindings(self, node_type: str) -> List[Dict[str, Any]]:
|
||||
return await self._backend.load_bindings(self.scope, node_type)
|
||||
|
||||
async def save_binding(self, node_type: str, local_id: str, payload_dict: Dict[str, Any]) -> None:
|
||||
await self._backend.save_binding(self.scope, node_type, local_id, payload_dict)
|
||||
|
||||
async def save_bindings_bulk(self, node_type: str, bindings: Dict[str, Optional[str]]) -> None:
|
||||
await self._backend.save_bindings_bulk(self.scope, node_type, bindings)
|
||||
|
||||
async def delete_binding(self, node_type: str, local_id: str) -> None:
|
||||
await self._backend.delete_binding(self.scope, node_type, local_id)
|
||||
|
||||
async def delete_bindings_bulk(self, node_type: str, local_ids: List[str]) -> None:
|
||||
await self._backend.delete_bindings_bulk(self.scope, node_type, local_ids)
|
||||
|
||||
async def load_bind_data_id_overrides(self, node_type: str) -> Dict[str, Optional[str]]:
|
||||
return await self._backend.load_bind_data_id_overrides(node_type, self.scope)
|
||||
|
||||
async def delete_scope_node_types(self, *, node_types: List[str]) -> None:
|
||||
await self._backend.delete_scope_node_types(scope=self.scope, node_types=node_types)
|
||||
|
||||
async def close(self) -> None:
|
||||
return None
|
||||
@@ -0,0 +1,3 @@
|
||||
from .persistence_backends.sqlalchemy_backend import SQLAlchemyPersistenceBackend
|
||||
|
||||
__all__ = ["SQLAlchemyPersistenceBackend"]
|
||||
@@ -9,7 +9,7 @@ from ..common.registry import DomainRegistry
|
||||
from .datasource_config import DataSourceConfig
|
||||
from .pipeline_config import PipelineRunConfig
|
||||
from .run_presets import load_overrides_from_file
|
||||
from .strategy_config import ConfigPresets, StrategyRuntimeConfig
|
||||
from .strategy_config import ConfigPresets, StrategyRuntimeConfig, resolve_domain_option_config
|
||||
|
||||
|
||||
def _deep_merge(base: Mapping[str, Any], override: Mapping[str, Any]) -> Dict[str, Any]:
|
||||
@@ -28,19 +28,76 @@ def _deep_merge(base: Mapping[str, Any], override: Mapping[str, Any]) -> Dict[st
|
||||
return merged
|
||||
|
||||
|
||||
def _build_default_strategy_runtime_config(node_type: str) -> StrategyRuntimeConfig | None:
|
||||
strategy_class = DomainRegistry.get_strategy_class(node_type)
|
||||
if strategy_class is None:
|
||||
return None
|
||||
|
||||
config = strategy_class.default_config.model_copy(deep=True)
|
||||
config.domain_option = resolve_domain_option_config(
|
||||
getattr(strategy_class, "domain_option_model", None),
|
||||
config.domain_option,
|
||||
)
|
||||
|
||||
return StrategyRuntimeConfig(
|
||||
node_type=node_type,
|
||||
config=config,
|
||||
)
|
||||
|
||||
|
||||
def _default_strategy_map(node_types: list[str]) -> Dict[str, StrategyRuntimeConfig]:
|
||||
strategy_map: Dict[str, StrategyRuntimeConfig] = {}
|
||||
for node_type in node_types:
|
||||
strategy_class = DomainRegistry.get_strategy_class(node_type)
|
||||
if strategy_class is None:
|
||||
runtime_cfg = _build_default_strategy_runtime_config(node_type)
|
||||
if runtime_cfg is None:
|
||||
continue
|
||||
strategy_map[node_type] = StrategyRuntimeConfig(
|
||||
node_type=node_type,
|
||||
config=strategy_class.default_config.model_copy(deep=True),
|
||||
)
|
||||
strategy_map[node_type] = runtime_cfg
|
||||
return strategy_map
|
||||
|
||||
|
||||
def _merge_strategy_runtime_config(
|
||||
runtime_cfg: StrategyRuntimeConfig,
|
||||
raw_override: Any,
|
||||
) -> StrategyRuntimeConfig:
|
||||
override = dict(raw_override or {})
|
||||
override.pop("node_type", None)
|
||||
|
||||
deprecated_keys = [key for key in ("force_sync", "load_mode") if key in override]
|
||||
if deprecated_keys:
|
||||
joined = ", ".join(deprecated_keys)
|
||||
raise ValueError(
|
||||
f"Strategy override for {runtime_cfg.node_type!r} contains unsupported legacy keys: {joined}. "
|
||||
"Use skip_sync under strategies.<node_type> or datasource.handler_configs instead."
|
||||
)
|
||||
|
||||
preset = override.pop("preset", None)
|
||||
if preset:
|
||||
runtime_cfg.config = runtime_cfg.config.model_validate(
|
||||
_deep_merge(
|
||||
runtime_cfg.config.model_dump(),
|
||||
ConfigPresets.get_preset(str(preset)).model_dump(exclude_defaults=True),
|
||||
)
|
||||
)
|
||||
|
||||
if "config" in override and isinstance(override["config"], dict):
|
||||
runtime_cfg.config = runtime_cfg.config.model_validate(
|
||||
_deep_merge(runtime_cfg.config.model_dump(), override.pop("config"))
|
||||
)
|
||||
|
||||
if override:
|
||||
runtime_cfg.config = runtime_cfg.config.model_validate(
|
||||
_deep_merge(runtime_cfg.config.model_dump(), override)
|
||||
)
|
||||
|
||||
strategy_class = DomainRegistry.get_strategy_class(runtime_cfg.node_type)
|
||||
runtime_cfg.config.domain_option = resolve_domain_option_config(
|
||||
getattr(strategy_class, "domain_option_model", None) if strategy_class is not None else None,
|
||||
runtime_cfg.config.domain_option,
|
||||
)
|
||||
|
||||
return runtime_cfg
|
||||
|
||||
|
||||
def _apply_strategy_overrides(
|
||||
node_types: list[str],
|
||||
strategy_overrides: Any,
|
||||
@@ -49,8 +106,16 @@ def _apply_strategy_overrides(
|
||||
|
||||
if isinstance(strategy_overrides, list):
|
||||
for item in strategy_overrides:
|
||||
runtime_cfg = StrategyRuntimeConfig.model_validate(item)
|
||||
strategy_map[runtime_cfg.node_type] = runtime_cfg
|
||||
raw_item = dict(item or {})
|
||||
node_type = str(raw_item.get("node_type") or "")
|
||||
if not node_type:
|
||||
raise ValueError("Strategy override list item missing required node_type")
|
||||
|
||||
runtime_cfg = strategy_map.get(node_type)
|
||||
if runtime_cfg is None:
|
||||
runtime_cfg = _build_default_strategy_runtime_config(node_type) or StrategyRuntimeConfig(node_type=node_type)
|
||||
|
||||
strategy_map[node_type] = _merge_strategy_runtime_config(runtime_cfg, raw_item)
|
||||
return [strategy_map[n] for n in node_types if n in strategy_map]
|
||||
|
||||
if not isinstance(strategy_overrides, dict):
|
||||
@@ -58,37 +123,10 @@ def _apply_strategy_overrides(
|
||||
|
||||
for node_type, raw_override in strategy_overrides.items():
|
||||
if node_type not in strategy_map:
|
||||
strategy_map[node_type] = StrategyRuntimeConfig(node_type=node_type)
|
||||
strategy_map[node_type] = _build_default_strategy_runtime_config(node_type) or StrategyRuntimeConfig(node_type=node_type)
|
||||
|
||||
runtime_cfg = strategy_map[node_type]
|
||||
override = dict(raw_override or {})
|
||||
|
||||
deprecated_keys = [key for key in ("force_sync", "load_mode") if key in override]
|
||||
if deprecated_keys:
|
||||
joined = ", ".join(deprecated_keys)
|
||||
raise ValueError(
|
||||
f"Strategy override for {node_type!r} contains unsupported legacy keys: {joined}. "
|
||||
"Use skip_sync under strategies.<node_type> or datasource.handler_configs instead."
|
||||
)
|
||||
|
||||
preset = override.pop("preset", None)
|
||||
if preset:
|
||||
runtime_cfg.config = ConfigPresets.get_preset(str(preset))
|
||||
|
||||
skip_sync = override.pop("skip_sync", None)
|
||||
|
||||
if skip_sync is not None:
|
||||
runtime_cfg.skip_sync = bool(skip_sync)
|
||||
|
||||
if "config" in override and isinstance(override["config"], dict):
|
||||
runtime_cfg.config = runtime_cfg.config.model_validate(
|
||||
_deep_merge(runtime_cfg.config.model_dump(), override.pop("config"))
|
||||
)
|
||||
|
||||
if override:
|
||||
runtime_cfg.config = runtime_cfg.config.model_validate(
|
||||
_deep_merge(runtime_cfg.config.model_dump(), override)
|
||||
)
|
||||
strategy_map[node_type] = _merge_strategy_runtime_config(runtime_cfg, raw_override)
|
||||
|
||||
return [strategy_map[n] for n in node_types if n in strategy_map]
|
||||
|
||||
|
||||
@@ -25,6 +25,7 @@ class MultiProjectRunConfig(BaseModel):
|
||||
|
||||
global_config: str
|
||||
default_project_config: str
|
||||
max_concurrency: int = 20
|
||||
project_bootstrap_node_types: List[str] = []
|
||||
projects: List[MultiProjectItemConfig]
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ meta:
|
||||
# represented as separate invocations with different inputs.
|
||||
context:
|
||||
is_create_zombie: { type: bool, desc: "初始化时判定为 CREATE 僵尸:action=CREATE 且 (status=FAILED 或 data_id 为空)" }
|
||||
bootstrap_source_present: { type: bool, desc: "bootstrap 阶段 datasource 刷新后,当前节点是否在最新数据源中命中" }
|
||||
|
||||
has_binding_record: { type: bool, desc: "binding_manager 里存在绑定记录 (remote_id/local_id 非空)" }
|
||||
core_binding_result: { type: enum, values: [NORMAL, WARNING, ABNORMAL], desc: "SyncDecisionEngine.determine_core_status() 给本节点的结果(仅在 has_binding_record=true 时有效)" }
|
||||
@@ -304,19 +305,30 @@ states:
|
||||
status: SKIPPED
|
||||
data_id_exist: true
|
||||
|
||||
S17:
|
||||
binding_status: ABNORMAL
|
||||
action: NONE
|
||||
status: FAILED
|
||||
data_id_exist: [false, true]
|
||||
desc: |
|
||||
bootstrap 加载异常隔离态
|
||||
持久化恢复后存在绑定,但 datasource 刷新未命中最新数据
|
||||
binding_status: ABNORMAL
|
||||
action: NONE
|
||||
status: FAILED
|
||||
data_id_exist: false | true
|
||||
|
||||
pseudo_states:
|
||||
S15:
|
||||
desc: "节点被清理删除(zombie cleanup)"
|
||||
S16:
|
||||
desc: "持久化加载输入态:本轮 E01 的输入节点集合(可包含上轮遗留与已稳定节点)"
|
||||
S17:
|
||||
desc: "加载异常隔离态:持久化枚举非法,保留错误供人工处理,不进入自动流程"
|
||||
|
||||
stages:
|
||||
bootstrap:
|
||||
desc: "启动/初始化(仅针对持久化加载出的节点):僵尸清理与归并到 S00"
|
||||
desc: "启动/初始化:持久化恢复后先做 datasource 刷新,再按 E01 清理僵尸或归并/隔离"
|
||||
entry_states: [S16]
|
||||
exit_states: [S00, S15]
|
||||
exit_states: [S00, S15, S17]
|
||||
|
||||
bind:
|
||||
desc: "绑定判定:core matrix +(可选)依赖检查 +(可选)自动绑定;stage 仅约束边界,不要求内部顺序"
|
||||
@@ -346,13 +358,23 @@ stages:
|
||||
transitions:
|
||||
E01:
|
||||
stage: bootstrap
|
||||
desc: "初始化:删CREATE僵尸;其余重置后进入起点"
|
||||
desc: "初始化:先依据 datasource 刷新结果判定 source 是否存在,再做僵尸清理/隔离/归并"
|
||||
from: [S16]
|
||||
outcomes:
|
||||
- when: { is_create_zombie: true }
|
||||
to: S15
|
||||
- when: { is_create_zombie: false }
|
||||
- when: { is_create_zombie: false, bootstrap_source_present: false, has_binding_record: false }
|
||||
to: S15
|
||||
emit:
|
||||
note: "持久化孤儿节点未命中最新数据源,按僵尸清理"
|
||||
- when: { is_create_zombie: false, bootstrap_source_present: false, has_binding_record: true }
|
||||
to: S17
|
||||
emit:
|
||||
note: "存在绑定但最新数据源未返回节点,进入加载异常隔离"
|
||||
- when: { is_create_zombie: false, bootstrap_source_present: true }
|
||||
to: S00
|
||||
emit:
|
||||
note: "bootstrap 已刷新到最新数据,归并到起点"
|
||||
|
||||
# --- Bind phase1 core ---
|
||||
E10:
|
||||
|
||||
@@ -1,13 +1,70 @@
|
||||
from typing import Any, Dict
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
||||
|
||||
|
||||
class PersistConfig(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", validate_assignment=True)
|
||||
|
||||
backend: str = "sqlite"
|
||||
db_path: str
|
||||
db_path: Optional[str] = None
|
||||
url: Optional[str] = None
|
||||
backend_options: Dict[str, Any] = Field(default_factory=dict)
|
||||
enable: bool = True
|
||||
wipe_on_start: bool = False
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def _validate_target(cls, data: Any) -> Any:
|
||||
if not isinstance(data, dict):
|
||||
return data
|
||||
|
||||
normalized = dict(data)
|
||||
normalized["backend"] = (str(normalized.get("backend") or "sqlite").strip().lower() or "sqlite")
|
||||
normalized["db_path"] = cls._normalize_optional_text(normalized.get("db_path"))
|
||||
normalized["url"] = cls._normalize_optional_text(normalized.get("url"))
|
||||
|
||||
if not normalized["db_path"] and not normalized["url"]:
|
||||
raise ValueError("persist requires either db_path or url")
|
||||
if normalized["backend"] != "sqlite" and not normalized["url"]:
|
||||
raise ValueError(f"persist.url is required for backend={normalized['backend']}")
|
||||
return normalized
|
||||
|
||||
@staticmethod
|
||||
def _normalize_optional_text(value: Optional[str]) -> Optional[str]:
|
||||
if value is None:
|
||||
return None
|
||||
normalized = str(value).strip()
|
||||
return normalized or None
|
||||
|
||||
def resolved_url(self) -> str:
|
||||
if self.url:
|
||||
return self.url
|
||||
if self.backend == "sqlite" and self.db_path:
|
||||
return f"sqlite+aiosqlite:///{self.db_path}"
|
||||
raise ValueError(f"persist.url is required for backend={self.backend}")
|
||||
|
||||
def resolved_db_path(self) -> Optional[str]:
|
||||
if self.db_path:
|
||||
return self.db_path
|
||||
if not self.url:
|
||||
return None
|
||||
|
||||
from sqlalchemy.engine import make_url
|
||||
|
||||
parsed = make_url(self.url)
|
||||
if parsed.get_backend_name() != "sqlite":
|
||||
return None
|
||||
database = parsed.database
|
||||
if not database:
|
||||
return None
|
||||
return str(database)
|
||||
|
||||
def wipe_file_path(self) -> Optional[str]:
|
||||
db_path = self.resolved_db_path()
|
||||
if not db_path or db_path == ":memory:":
|
||||
return None
|
||||
return db_path
|
||||
|
||||
def display_target(self) -> str:
|
||||
return self.url or self.db_path or "<unset>"
|
||||
|
||||
@@ -1,11 +1,21 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from enum import Enum
|
||||
from typing import Dict, List, Optional
|
||||
from typing import Any, Dict, List, Mapping, Type
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
|
||||
def resolve_domain_option_config(
|
||||
domain_option_model: Type[BaseModel] | None,
|
||||
domain_option: Mapping[str, Any] | None,
|
||||
) -> Dict[str, Any]:
|
||||
if domain_option_model is None:
|
||||
return dict(domain_option or {})
|
||||
return domain_option_model.model_validate(domain_option or {}).model_dump()
|
||||
|
||||
|
||||
class OrphanAction(str, Enum):
|
||||
CREATE_REMOTE = "create_remote"
|
||||
CREATE_LOCAL = "create_local"
|
||||
@@ -17,7 +27,6 @@ class OrphanAction(str, Enum):
|
||||
class UpdateDirection(str, Enum):
|
||||
PUSH = "push"
|
||||
PULL = "pull"
|
||||
BIDIRECTIONAL = "bidirectional"
|
||||
NONE = "none"
|
||||
|
||||
|
||||
@@ -40,20 +49,76 @@ class StrategyConfig(BaseModel):
|
||||
remote_orphan_action: OrphanAction = OrphanAction.NONE
|
||||
update_direction: UpdateDirection = UpdateDirection.PUSH
|
||||
|
||||
# 字段级同步方向覆盖。
|
||||
# field_direction_key: 节点 data 中用作查表键的字段名,如 "key"。
|
||||
# field_direction_overrides: 字段值 → UpdateDirection 的映射。
|
||||
# 未命中映射的节点沿用 update_direction 作为默认方向。
|
||||
field_direction_key: Optional[str] = None
|
||||
field_direction_overrides: Dict[str, UpdateDirection] = Field(default_factory=dict)
|
||||
|
||||
# post-check 一致性比较时忽略的顶层字段。
|
||||
# 只影响最终一致性评估,不影响 create/update 的差异检测。
|
||||
post_check_ignore_fields: List[str] = Field(default_factory=list)
|
||||
|
||||
# post-check 一致性比较时,忽略列表型字段内各元素的特定子键。
|
||||
# 格式: {字段名: [子键, ...]},如 {"plan_node": ["is_audit"]}。
|
||||
# 用于排除后端硬写死、永远与本地不一致的字段,避免误报。
|
||||
post_check_ignore_list_item_fields: Dict[str, List[str]] = Field(default_factory=dict)
|
||||
|
||||
# schema diff / validate 时忽略的顶层字段。
|
||||
compare_ignore_fields: List[str] = Field(default_factory=list)
|
||||
|
||||
# 当 auto-bind 命中“多对一”候选时,允许稳定选择一条进行绑定,剩余候选按 orphan/create 流程继续处理。
|
||||
# 默认关闭,避免在一般场景下引入隐式匹配。
|
||||
allow_many_to_one_auto_bind: bool = False
|
||||
|
||||
# 是否跳过该类型的 create/update 阶段。
|
||||
skip_sync: bool = False
|
||||
|
||||
# 是否跳过该类型的 post-check reload 与一致性校验。
|
||||
skip_post_check: bool = False
|
||||
|
||||
# 领域策略运行参数,由各 strategy 的 domain_option_model 负责解释与校验。
|
||||
domain_option: Dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
def log_logic_warnings(self, *, node_type: str, logger: logging.Logger) -> None:
|
||||
direction = self.update_direction
|
||||
local_action = self.local_orphan_action
|
||||
remote_action = self.remote_orphan_action
|
||||
|
||||
if direction == UpdateDirection.PUSH:
|
||||
if local_action == OrphanAction.CREATE_LOCAL:
|
||||
logger.warning(
|
||||
f"[{node_type}] Config inconsistency: "
|
||||
f"update_direction=PUSH but local_orphan_action=CREATE_LOCAL (本地创建)。"
|
||||
f"应使用 CREATE_REMOTE (推送到远程)"
|
||||
)
|
||||
if remote_action == OrphanAction.CREATE_LOCAL:
|
||||
logger.warning(
|
||||
f"[{node_type}] Config inconsistency: "
|
||||
f"update_direction=PUSH but remote_orphan_action=CREATE_LOCAL (本地创建)。"
|
||||
f"PUSH 模式应设为 NONE 或 DELETE_REMOTE"
|
||||
)
|
||||
elif direction == UpdateDirection.PULL:
|
||||
if local_action == OrphanAction.CREATE_REMOTE:
|
||||
logger.warning(
|
||||
f"[{node_type}] Config inconsistency: "
|
||||
f"update_direction=PULL but local_orphan_action=CREATE_REMOTE (推送到远程)。"
|
||||
f"PULL 模式应设为 NONE 或 DELETE_LOCAL"
|
||||
)
|
||||
if remote_action in {OrphanAction.CREATE_REMOTE, OrphanAction.DELETE_LOCAL}:
|
||||
logger.warning(
|
||||
f"[{node_type}] Config inconsistency: "
|
||||
f"update_direction=PULL but remote_orphan_action={remote_action}。"
|
||||
f"PULL 模式建议使用 CREATE_LOCAL 或 NONE"
|
||||
)
|
||||
elif direction == UpdateDirection.NONE:
|
||||
if local_action != OrphanAction.NONE:
|
||||
logger.warning(
|
||||
f"[{node_type}] Config inconsistency: "
|
||||
f"update_direction=NONE but local_orphan_action={local_action}。"
|
||||
f"应设为 NONE"
|
||||
)
|
||||
if remote_action != OrphanAction.NONE:
|
||||
logger.warning(
|
||||
f"[{node_type}] Config inconsistency: "
|
||||
f"update_direction=NONE but remote_orphan_action={remote_action}。"
|
||||
f"应设为 NONE"
|
||||
)
|
||||
|
||||
|
||||
class StrategyRuntimeConfig(BaseModel):
|
||||
"""Pipeline 层策略运行配置(按 node_type)。"""
|
||||
@@ -61,7 +126,6 @@ class StrategyRuntimeConfig(BaseModel):
|
||||
model_config = ConfigDict(validate_assignment=True, extra="forbid")
|
||||
|
||||
node_type: str
|
||||
skip_sync: bool = False
|
||||
config: StrategyConfig = Field(default_factory=StrategyConfig)
|
||||
|
||||
|
||||
|
||||
@@ -23,6 +23,7 @@ from datetime import datetime
|
||||
import httpx
|
||||
|
||||
from schemas.common.push_system import PushIdsSchema
|
||||
from ...logging import get_logger, get_scope_display_name
|
||||
|
||||
|
||||
def _build_default_push_steps() -> List[Dict[str, str]]:
|
||||
@@ -33,6 +34,7 @@ def _build_default_push_steps() -> List[Dict[str, str]]:
|
||||
class ApiClient:
|
||||
"""API HTTP 客户端"""
|
||||
_TRACE_TEXT_MAX_LEN = 1000
|
||||
_REQUEST_STATS_LOG_PREFIX = "📊 ApiClient request stats"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -41,6 +43,7 @@ class ApiClient:
|
||||
secret: str,
|
||||
debug: bool = False,
|
||||
log_file: Optional[Path] = None,
|
||||
session_label: Optional[str] = None,
|
||||
rate_limit_max_requests: int = 30,
|
||||
rate_limit_window_seconds: float = 10.0,
|
||||
):
|
||||
@@ -62,16 +65,58 @@ class ApiClient:
|
||||
self._client: Optional[httpx.AsyncClient] = None
|
||||
self._debug = debug
|
||||
self._log_file = log_file
|
||||
self._scope = ""
|
||||
self._session_label = (session_label or "").strip()
|
||||
self._trace_seq = 0
|
||||
self._trace_by_push_id: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
|
||||
self._trace_by_biz_id: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
|
||||
self._trace_by_op: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
|
||||
self._load_trace_by_item: Dict[tuple[str, str], deque[Dict[str, Any]]] = defaultdict(deque)
|
||||
self._logger = logging.getLogger(__name__)
|
||||
self._logger = get_logger(__name__)
|
||||
self._rate_limit_max_requests = int(rate_limit_max_requests)
|
||||
self._rate_limit_window_seconds = float(rate_limit_window_seconds)
|
||||
self._rate_limit_lock = asyncio.Lock()
|
||||
self._rate_limit_timestamps: deque[float] = deque()
|
||||
self._request_total = 0
|
||||
self._request_success = 0
|
||||
self._request_failure = 0
|
||||
|
||||
def set_scope(self, scope: Optional[str]) -> None:
|
||||
self._scope = str(scope or "").strip()
|
||||
|
||||
def set_session_label(self, label: Optional[str]) -> None:
|
||||
self._session_label = str(label or "").strip()
|
||||
|
||||
def get_log_session_label(self) -> str:
|
||||
if self._session_label:
|
||||
return self._session_label
|
||||
scope_label = get_scope_display_name(self._scope)
|
||||
return scope_label or self._scope
|
||||
|
||||
def _project_name_from_request(self, payload: Optional[Dict[str, Any]]) -> str:
|
||||
if not isinstance(payload, dict):
|
||||
return ""
|
||||
project_id = str(payload.get("project_id") or "").strip()
|
||||
if not project_id:
|
||||
return ""
|
||||
display = get_scope_display_name(f"project_id:{project_id}")
|
||||
if not display or display == f"project_id:{project_id}":
|
||||
return ""
|
||||
return display
|
||||
|
||||
def _resolve_request_session_label(
|
||||
self,
|
||||
*,
|
||||
request_params: Optional[Dict[str, Any]] = None,
|
||||
request_payload: Optional[Dict[str, Any]] = None,
|
||||
) -> str:
|
||||
payload_label = self._project_name_from_request(request_payload)
|
||||
if payload_label:
|
||||
return payload_label
|
||||
params_label = self._project_name_from_request(request_params)
|
||||
if params_label:
|
||||
return params_label
|
||||
return self.get_log_session_label()
|
||||
|
||||
def _is_rate_limit_enabled(self) -> bool:
|
||||
return self._rate_limit_max_requests > 0 and self._rate_limit_window_seconds > 0
|
||||
@@ -105,9 +150,54 @@ class ApiClient:
|
||||
|
||||
async def close(self):
|
||||
"""关闭 HTTP 客户端"""
|
||||
self.log_request_stats(level="INFO")
|
||||
if self._client:
|
||||
await self._client.aclose()
|
||||
self._client = None
|
||||
|
||||
def get_request_stats(self) -> Dict[str, int]:
|
||||
"""返回当前客户端生命周期内的请求统计。"""
|
||||
return {
|
||||
"total": self._request_total,
|
||||
"success": self._request_success,
|
||||
"failure": self._request_failure,
|
||||
}
|
||||
|
||||
def format_request_stats(self) -> str:
|
||||
"""格式化当前客户端生命周期内的请求统计。"""
|
||||
stats = self.get_request_stats()
|
||||
session_text = self.get_log_session_label()
|
||||
session_prefix = f"session={session_text} " if session_text else ""
|
||||
return (
|
||||
f"{self._REQUEST_STATS_LOG_PREFIX}: "
|
||||
f"{session_prefix}"
|
||||
f"total={stats['total']}, success={stats['success']}, failure={stats['failure']}"
|
||||
)
|
||||
|
||||
def log_request_stats(self, level: str = "DEBUG") -> None:
|
||||
"""输出当前客户端生命周期内的请求统计。"""
|
||||
self._log(self.format_request_stats(), level=level)
|
||||
|
||||
def _log_request_summary(
|
||||
self,
|
||||
*,
|
||||
method: str,
|
||||
endpoint: str,
|
||||
elapsed_seconds: float,
|
||||
request_params: Optional[Dict[str, Any]] = None,
|
||||
request_payload: Optional[Dict[str, Any]] = None,
|
||||
) -> None:
|
||||
if not self._debug:
|
||||
return
|
||||
session_text = self._resolve_request_session_label(
|
||||
request_params=request_params,
|
||||
request_payload=request_payload,
|
||||
)
|
||||
session_prefix = f"session={session_text} " if session_text else ""
|
||||
self._log(
|
||||
f"🔎 ApiClient request: {session_prefix}method={method.upper()} endpoint={endpoint} elapsed={elapsed_seconds:.3f}s",
|
||||
level="INFO",
|
||||
)
|
||||
|
||||
def _log(self, message: str, level: str = "DEBUG"):
|
||||
"""
|
||||
@@ -382,6 +472,7 @@ class ApiClient:
|
||||
client = self._get_client()
|
||||
url = f"{self._base_url}{endpoint}"
|
||||
await self._acquire_rate_limit_slot()
|
||||
self._request_total += 1
|
||||
|
||||
# 记录请求信息
|
||||
self._log(f"\n{'='*60}", level="DEBUG")
|
||||
@@ -400,10 +491,10 @@ class ApiClient:
|
||||
if params is None:
|
||||
params = {}
|
||||
params['uid'] = self._uid
|
||||
|
||||
|
||||
# 转换所有参数值为字符串(除了 list/dict)
|
||||
params = {k: str(v) if not isinstance(v, (list, dict)) else v for k, v in params.items()}
|
||||
|
||||
|
||||
sign_params = params.copy()
|
||||
# 生成签名并附加到 params
|
||||
signature = self._generate_signature(sign_params)
|
||||
@@ -433,6 +524,13 @@ class ApiClient:
|
||||
|
||||
# 记录响应时间
|
||||
elapsed = time.time() - start_time
|
||||
self._log_request_summary(
|
||||
method=method,
|
||||
endpoint=endpoint,
|
||||
elapsed_seconds=elapsed,
|
||||
request_params=params,
|
||||
request_payload=data if method.upper() in ['POST', 'PUT', 'DELETE'] else None,
|
||||
)
|
||||
self._log(f"Response Time: {elapsed:.3f}s", level="DEBUG")
|
||||
|
||||
# 检查 HTTP 状态
|
||||
@@ -453,14 +551,23 @@ class ApiClient:
|
||||
response=result,
|
||||
elapsed_ms=int(elapsed * 1000),
|
||||
)
|
||||
self._request_success += 1
|
||||
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
self._request_failure += 1
|
||||
|
||||
# 记录错误
|
||||
elapsed = time.time() - start_time
|
||||
self._log_request_summary(
|
||||
method=method,
|
||||
endpoint=endpoint,
|
||||
elapsed_seconds=elapsed,
|
||||
request_params=params,
|
||||
request_payload=data if method.upper() in ['POST', 'PUT', 'DELETE'] else None,
|
||||
)
|
||||
error_msg = "\n========== [ApiClient HTTP Error] ==========\n"
|
||||
error_msg += f"Request: {method} {url}\n"
|
||||
error_msg += f"Params: {params}\n"
|
||||
|
||||
@@ -15,6 +15,7 @@ from ..datasource import BaseDataSource
|
||||
from ..handler import NodeHandler
|
||||
from ..task_result import TaskResult
|
||||
from .client import ApiClient
|
||||
from ...logging import set_scope_display_name
|
||||
|
||||
|
||||
class ApiDataSource(BaseDataSource):
|
||||
@@ -56,6 +57,29 @@ class ApiDataSource(BaseDataSource):
|
||||
rate_limit_window_seconds=rate_limit_window_seconds,
|
||||
)
|
||||
self.poll_mode = poll_mode
|
||||
|
||||
def set_collection(self, collection) -> None:
|
||||
super().set_collection(collection)
|
||||
self.client.set_scope(getattr(collection, "scope", ""))
|
||||
|
||||
def _update_client_session_label_from_project_node(self, node) -> None:
|
||||
if self._collection is None or getattr(self._collection, "scope", "") == "global":
|
||||
return
|
||||
if getattr(node, "node_type", None) != "project":
|
||||
return
|
||||
if not hasattr(node, "get_data"):
|
||||
return
|
||||
|
||||
data = node.get_data()
|
||||
if not isinstance(data, dict):
|
||||
return
|
||||
|
||||
project_name = str(data.get("name") or "").strip()
|
||||
if not project_name:
|
||||
return
|
||||
|
||||
set_scope_display_name(self._collection.scope, f"project:{project_name}")
|
||||
self.client.set_session_label(f"project:{project_name}")
|
||||
|
||||
async def initialize(self):
|
||||
"""初始化数据源(初始化 HTTP 客户端)"""
|
||||
@@ -65,22 +89,10 @@ class ApiDataSource(BaseDataSource):
|
||||
"""关闭数据源(关闭 HTTP 客户端)"""
|
||||
await self.client.close()
|
||||
|
||||
def add_handler(self, handler: NodeHandler) -> None:
|
||||
"""
|
||||
添加业务 Handler
|
||||
|
||||
Args:
|
||||
handler: API Handler 实例
|
||||
"""
|
||||
def _prepare_handler_for_registration(self, handler: NodeHandler) -> None:
|
||||
handler.set_api_client(self.client)
|
||||
handler.set_poll_mode(self.poll_mode)
|
||||
|
||||
# 调用基类添加方法
|
||||
super().add_handler(handler)
|
||||
|
||||
# 如果 collection 已设置,确保新 handler 拿到引用
|
||||
if self._collection is not None:
|
||||
handler.set_collection(self._collection) # type: ignore[arg-type]
|
||||
super()._prepare_handler_for_registration(handler)
|
||||
|
||||
@staticmethod
|
||||
def _fmt_trace_value(value, max_len: int = 600) -> str:
|
||||
@@ -187,6 +199,7 @@ class ApiDataSource(BaseDataSource):
|
||||
return None
|
||||
|
||||
def _on_node_loaded(self, handler: NodeHandler, node) -> None:
|
||||
self._update_client_session_label_from_project_node(node)
|
||||
data_id = node.data_id or ""
|
||||
if not data_id:
|
||||
return
|
||||
|
||||
@@ -17,9 +17,10 @@ import asyncio
|
||||
from abc import abstractmethod
|
||||
from copy import deepcopy
|
||||
from typing import ClassVar, Dict, List, Any, Optional, TYPE_CHECKING, TypeVar
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, ValidationError
|
||||
|
||||
from ...common.type_safety import get_pydantic_model_fields
|
||||
from ...logging import get_logger
|
||||
from ..handler import BaseNodeHandler
|
||||
from ..task_result import TaskResult
|
||||
from ..handler import ValidationResult # Import ValidationResult
|
||||
@@ -30,6 +31,7 @@ if TYPE_CHECKING:
|
||||
from .client import ApiClient
|
||||
|
||||
T = TypeVar("T", bound=BaseModel)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class BaseApiHandler(BaseNodeHandler[T]):
|
||||
@@ -41,8 +43,17 @@ class BaseApiHandler(BaseNodeHandler[T]):
|
||||
_schema: Any
|
||||
|
||||
def __init__(self, **handler_config: Any):
|
||||
# 子类应该在调用 super().__init__() 之前或之后设置 _node_type, _node_class, _schema
|
||||
super().__init__()
|
||||
# 子类可直接传入 node_class;少数特殊场景可显式覆盖 schema/node_type。
|
||||
node_class = handler_config.pop("node_class", None)
|
||||
node_type = handler_config.pop("node_type", None)
|
||||
schema = handler_config.pop("schema", None)
|
||||
|
||||
if node_class is not None and node_type is None:
|
||||
node_type = getattr(node_class, "node_type", None)
|
||||
if node_class is not None and schema is None:
|
||||
schema = getattr(node_class, "schema", None)
|
||||
|
||||
super().__init__(node_type=node_type, node_class=node_class, schema=schema)
|
||||
self.api_client: 'ApiClient' = None # type: ignore # 由 DataSource 注入
|
||||
self.poll_mode: str = "async" # async | sync
|
||||
self._collection: 'DataCollection' = None # type: ignore # 由 DataSource 注入
|
||||
@@ -51,9 +62,6 @@ class BaseApiHandler(BaseNodeHandler[T]):
|
||||
# post-check 将只比较这些 schema 覆盖的字段,过滤后端只读的派生字段。
|
||||
self.update_schemas: List[type] = []
|
||||
|
||||
def set_collection(self, collection: 'DataCollection') -> None:
|
||||
self._collection = collection
|
||||
|
||||
def get_update_fields(self, *, exclude: frozenset = frozenset({'id'})) -> List[str]:
|
||||
"""从 update_schemas 中提取所有可写字段名,排除标识键(默认排除 id)。"""
|
||||
fields: set = set()
|
||||
@@ -68,7 +76,17 @@ class BaseApiHandler(BaseNodeHandler[T]):
|
||||
|
||||
def set_collection(self, collection: 'DataCollection') -> None:
|
||||
"""设置 Collection(由 DataSource 调用)"""
|
||||
self._collection = collection
|
||||
super().set_collection(collection)
|
||||
|
||||
def ensure_api_client(self) -> 'ApiClient':
|
||||
if self.api_client is None:
|
||||
raise RuntimeError("API client not set. Call set_api_client() before invoking API handler methods.")
|
||||
return self.api_client
|
||||
|
||||
def ensure_collection(self) -> 'DataCollection':
|
||||
if self._collection is None:
|
||||
raise RuntimeError("Collection not set. Call set_collection() before invoking handler methods.")
|
||||
return self._collection
|
||||
|
||||
@property
|
||||
def node_type(self) -> str:
|
||||
@@ -112,8 +130,7 @@ class BaseApiHandler(BaseNodeHandler[T]):
|
||||
def _create_basic_node(self, data: Dict[str, Any], depend_ids: Optional[List[str]] = None) -> "SyncNode":
|
||||
"""创建或复用基础节点(默认使用 data['id'] 或 data['_id'])"""
|
||||
data_id = str(data.get("id") or data.get("_id") or "")
|
||||
if self._collection is None:
|
||||
raise RuntimeError("Collection not set. Call set_collection() before creating nodes.")
|
||||
self.ensure_collection()
|
||||
|
||||
if data_id:
|
||||
existing_node = self._collection.get_by_data_id(self.node_type, data_id)
|
||||
@@ -167,10 +184,6 @@ class BaseApiHandler(BaseNodeHandler[T]):
|
||||
payload["steps"] = self._get_request_steps(node) if node is not None else []
|
||||
return payload
|
||||
|
||||
def _build_full_update_data(self, new_data: Dict[str, Any], schema: Any) -> Dict[str, Any]:
|
||||
validated = schema.model_validate(new_data)
|
||||
return validated.model_dump(exclude_none=True)
|
||||
|
||||
def _mark_approval_snapshot_synced(self, node: 'SyncNode') -> None:
|
||||
snapshot = self._get_active_approval_snapshot(node)
|
||||
if snapshot:
|
||||
@@ -255,8 +268,53 @@ class BaseApiHandler(BaseNodeHandler[T]):
|
||||
return None
|
||||
except Exception as e:
|
||||
return str(e)
|
||||
|
||||
def _get_update_schema_payload(
|
||||
self,
|
||||
schema: Any,
|
||||
data: Dict[str, Any],
|
||||
*,
|
||||
require_valid: bool,
|
||||
) -> Dict[str, Any] | None:
|
||||
"""Update 内部实现:提取指定 schema 下的 payload。
|
||||
|
||||
这里明确只服务“update schema”,不负责 response model 一致性校验。
|
||||
这个方法只给 BaseApiHandler 的 update 入口方法复用,不作为 handler 入口。
|
||||
|
||||
处理顺序:
|
||||
1. 先尝试用 update schema 校验,成功则直接使用校验后的 model_dump。
|
||||
2. 如果校验失败且 require_valid=False,则退化为只抽取 schema 定义的字段。
|
||||
|
||||
之所以允许退化,是因为 update 比较时两边快照常常不完整:
|
||||
某些字段在旧快照缺失,但新数据已经有值。此时应该继续比较 schema
|
||||
相关字段,而不是把它误判成“无差异”。
|
||||
|
||||
空字符串和 None 会被统一过滤掉,因为当前 update handler 并不把它们
|
||||
当成“清空字段”的显式语义,而是当作“不参与本次更新”。
|
||||
"""
|
||||
try:
|
||||
validated = schema.model_validate(data)
|
||||
payload = validated.model_dump(exclude_unset=True)
|
||||
except ValidationError:
|
||||
if require_valid:
|
||||
return None
|
||||
|
||||
schema_fields = get_pydantic_model_fields(schema)
|
||||
if not schema_fields:
|
||||
return {}
|
||||
payload = {
|
||||
field_name: data.get(field_name)
|
||||
for field_name in schema_fields.keys()
|
||||
if field_name in data
|
||||
}
|
||||
|
||||
return {
|
||||
field_name: value
|
||||
for field_name, value in payload.items()
|
||||
if value not in (None, "")
|
||||
}
|
||||
|
||||
def _get_schema_diff(
|
||||
def get_update_schema_diff(
|
||||
self,
|
||||
schema,
|
||||
data: Dict[str, Any],
|
||||
@@ -264,8 +322,7 @@ class BaseApiHandler(BaseNodeHandler[T]):
|
||||
node_id: Optional[str] = None,
|
||||
force_full_schema: bool = False,
|
||||
) -> Dict[str, Any] | None:
|
||||
"""
|
||||
获取数据在指定 schema 下的差异字段
|
||||
"""Update 入口:获取指定 update schema 下的差异字段。
|
||||
|
||||
Args:
|
||||
schema: Pydantic schema 类
|
||||
@@ -276,46 +333,37 @@ class BaseApiHandler(BaseNodeHandler[T]):
|
||||
Returns:
|
||||
Dict[str, Any] | None: 有差异返回差异字段字典,无差异或字段不全返回 None
|
||||
"""
|
||||
from pydantic import ValidationError
|
||||
|
||||
try:
|
||||
new_model = schema.model_validate(data)
|
||||
origin_model = schema.model_validate(origin_data)
|
||||
|
||||
new_dict = new_model.model_dump(exclude_unset=True)
|
||||
origin_dict = origin_model.model_dump(exclude_unset=True)
|
||||
|
||||
if new_dict == origin_dict:
|
||||
if force_full_schema:
|
||||
return new_dict
|
||||
return None
|
||||
|
||||
# 返回有差异的字段
|
||||
diff_fields = {k: v for k, v in new_dict.items() if origin_dict.get(k) != v}
|
||||
|
||||
if diff_fields:
|
||||
# 打印差异详情
|
||||
node_info = f"[{node_id}]" if node_id else ""
|
||||
print(f" [DIFF] {self.node_type}{node_info} schema={schema.__name__}")
|
||||
for k, v in diff_fields.items():
|
||||
orig_v = origin_dict.get(k)
|
||||
print(f" • {k}: {orig_v!r} → {v!r}")
|
||||
|
||||
return diff_fields if diff_fields else None
|
||||
|
||||
except ValidationError:
|
||||
# schema 字段不全,视为无差异
|
||||
# 当前待发送的数据必须能构成合法 update payload;允许旧快照不完整。
|
||||
new_dict = self._get_update_schema_payload(schema, data, require_valid=True)
|
||||
origin_dict = self._get_update_schema_payload(schema, origin_data, require_valid=False)
|
||||
|
||||
if new_dict is None or origin_dict is None:
|
||||
return None
|
||||
|
||||
if new_dict == origin_dict:
|
||||
if force_full_schema:
|
||||
return new_dict
|
||||
return None
|
||||
|
||||
diff_fields = {k: v for k, v in new_dict.items() if origin_dict.get(k) != v}
|
||||
|
||||
if diff_fields:
|
||||
node_info = f"[{node_id}]" if node_id else ""
|
||||
logger.debug(" [DIFF] %s%s schema=%s", self.node_type, node_info, schema.__name__)
|
||||
for k, v in diff_fields.items():
|
||||
orig_v = origin_dict.get(k)
|
||||
logger.debug(" • %s: %r → %r", k, orig_v, v)
|
||||
|
||||
return diff_fields if diff_fields else None
|
||||
|
||||
def _filter_update_data(
|
||||
def build_update_request_data(
|
||||
self,
|
||||
new_data: Dict[str, Any],
|
||||
original_data: Dict[str, Any],
|
||||
schema: Optional[Any] = None,
|
||||
force_full_schema: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
过滤出需要更新的字段
|
||||
"""Update 入口:构造最终要发送给更新接口的请求数据。
|
||||
|
||||
规则:
|
||||
1. 提取 schema 定义的所有字段(包括必填)
|
||||
@@ -342,29 +390,10 @@ class BaseApiHandler(BaseNodeHandler[T]):
|
||||
update_data[key] = new_value
|
||||
return update_data
|
||||
|
||||
# 1. 提取 schema 定义的所有字段
|
||||
schema_fields = set(schema_fields.keys())
|
||||
schema_data = {}
|
||||
has_change = False
|
||||
|
||||
for key in schema_fields:
|
||||
new_value = new_data.get(key)
|
||||
|
||||
# 排除空值
|
||||
if new_value == "" or new_value is None:
|
||||
continue
|
||||
|
||||
schema_data[key] = new_value
|
||||
|
||||
# 检查是否变化
|
||||
original_value = original_data.get(key)
|
||||
if new_value != original_value:
|
||||
# 记录具体差异以便排查
|
||||
# print(f" [DEBUG] Field '{key}' changed: '{original_value}' -> '{new_value}'")
|
||||
has_change = True
|
||||
|
||||
# 2. 只有存在变化时才返回数据
|
||||
if has_change or force_full_schema:
|
||||
schema_data = self._get_update_schema_payload(schema, new_data, require_valid=True) or {}
|
||||
origin_schema_data = self._get_update_schema_payload(schema, original_data, require_valid=False) or {}
|
||||
|
||||
if force_full_schema or schema_data != origin_schema_data:
|
||||
return schema_data
|
||||
return {}
|
||||
|
||||
@@ -374,16 +403,12 @@ class BaseApiHandler(BaseNodeHandler[T]):
|
||||
new_data: Dict[str, Any],
|
||||
origin_data: Dict[str, Any],
|
||||
) -> List[str]:
|
||||
schema_fields = get_pydantic_model_fields(schema)
|
||||
if not schema_fields:
|
||||
return []
|
||||
changed: List[str] = []
|
||||
for field_name in schema_fields.keys():
|
||||
if new_data.get(field_name) != origin_data.get(field_name):
|
||||
changed.append(field_name)
|
||||
return changed
|
||||
new_payload = self._get_update_schema_payload(schema, new_data, require_valid=True) or {}
|
||||
origin_payload = self._get_update_schema_payload(schema, origin_data, require_valid=False) or {}
|
||||
field_names = sorted(set(new_payload) | set(origin_payload))
|
||||
return [field_name for field_name in field_names if new_payload.get(field_name) != origin_payload.get(field_name)]
|
||||
|
||||
def _build_update_schema_status(
|
||||
def build_update_schema_status(
|
||||
self,
|
||||
*,
|
||||
schema_name: str,
|
||||
@@ -393,6 +418,7 @@ class BaseApiHandler(BaseNodeHandler[T]):
|
||||
will_update: bool,
|
||||
update_error: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Update 入口:生成 update schema 的状态说明文本。"""
|
||||
new_error = self._validate_schema(new_data, schema)
|
||||
origin_error = self._validate_schema(origin_data, schema)
|
||||
changed_fields = self._schema_changed_fields(schema, new_data, origin_data)
|
||||
|
||||
@@ -37,9 +37,10 @@ from ..engine import (
|
||||
StateMachineRuntime,
|
||||
e40_sync_execute,
|
||||
)
|
||||
from ..logging import get_logger, get_scope_display_name
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class BaseDataSource(ABC):
|
||||
@@ -74,6 +75,8 @@ class BaseDataSource(ABC):
|
||||
def __init__(self, poll_max_retries: int = 1, poll_interval: float = 0.5):
|
||||
self._handlers: Dict[str, "NodeHandler"] = {}
|
||||
self._collection: Optional["DataCollection"] = None
|
||||
self._endpoint_name: str = "unknown"
|
||||
self._datasource_type: str = self.__class__.__name__.lower()
|
||||
|
||||
# 按节点类型统计信息
|
||||
# {"company": {"loaded": 10, "synced": 10, "success": 10, "failed": 0}, ...}
|
||||
@@ -86,6 +89,17 @@ class BaseDataSource(ABC):
|
||||
self._poll_interval = max(0, poll_interval)
|
||||
self._sm_runtime: Optional[StateMachineRuntime] = None
|
||||
|
||||
def set_runtime_logging_context(self, *, endpoint_name: str, datasource_type: str) -> None:
|
||||
self._endpoint_name = endpoint_name
|
||||
self._datasource_type = datasource_type
|
||||
|
||||
def _datasource_log_prefix(self) -> str:
|
||||
scope = self._collection.scope if self._collection is not None else "unbound"
|
||||
return f"[{get_scope_display_name(scope)}][{self._endpoint_name}:{self._datasource_type}]"
|
||||
|
||||
def _node_log_prefix(self, node_type: str) -> str:
|
||||
return f"{self._datasource_log_prefix()}[{node_type}]"
|
||||
|
||||
def _ensure_sm_runtime(self) -> StateMachineRuntime:
|
||||
if self._sm_runtime is None:
|
||||
cfg_path = Path(__file__).resolve().parents[1] / "config" / "node_state_machine.yaml"
|
||||
@@ -162,6 +176,11 @@ class BaseDataSource(ABC):
|
||||
"""Datasource-specific hook after a FAILED result is applied to node."""
|
||||
return
|
||||
|
||||
def _prepare_handler_for_registration(self, handler: "NodeHandler") -> None:
|
||||
"""Allow subclasses to inject datasource-specific dependencies before registration."""
|
||||
if self._collection is not None:
|
||||
handler.set_collection(self._collection)
|
||||
|
||||
async def _handle_success_result(self, node: "SyncNode", result: "TaskResult") -> None:
|
||||
previous_data_id = node.data_id
|
||||
ok = self._apply_sync_execute_state(
|
||||
@@ -370,6 +389,7 @@ class BaseDataSource(ABC):
|
||||
示例:
|
||||
datasource.add_handler(ContractNodeHandler(api_client))
|
||||
"""
|
||||
self._prepare_handler_for_registration(handler)
|
||||
self._handlers[handler.node_type] = handler
|
||||
|
||||
def get_handler(self, node_type: str) -> "NodeHandler":
|
||||
@@ -418,7 +438,9 @@ class BaseDataSource(ABC):
|
||||
async def load_all(
|
||||
self,
|
||||
order: Optional[List[str]] = None,
|
||||
data_type: Optional[str] = None
|
||||
data_type: Optional[str] = None,
|
||||
*,
|
||||
raise_on_error: bool = False,
|
||||
) -> None:
|
||||
"""
|
||||
按依赖顺序加载所有数据
|
||||
@@ -460,7 +482,7 @@ class BaseDataSource(ABC):
|
||||
# Handler 加载数据并创建节点
|
||||
nodes = await handler.load()
|
||||
|
||||
print(f"Loaded nodes for {node_type}: {len(nodes)}")
|
||||
logger.debug("%s Loaded nodes for %s: %d", self._datasource_log_prefix(), node_type, len(nodes))
|
||||
|
||||
# 将节点写入 Collection:优先按 data_id 命中已有节点并更新
|
||||
await self._upsert_loaded_nodes(handler, node_type, nodes)
|
||||
@@ -468,7 +490,9 @@ class BaseDataSource(ABC):
|
||||
# 统计加载的节点数
|
||||
self._stats[node_type]["loaded"] += len(nodes)
|
||||
except Exception as exc:
|
||||
print(f"❌ [{node_type}] 加载失败: {exc}")
|
||||
logger.error("%s ❌ 加载失败: %s", self._node_log_prefix(node_type), exc)
|
||||
if raise_on_error:
|
||||
raise RuntimeError(f"[{node_type}] 加载失败: {exc}") from exc
|
||||
continue
|
||||
|
||||
# ========== 同步执行 ==========
|
||||
@@ -617,7 +641,7 @@ class BaseDataSource(ABC):
|
||||
|
||||
# 批量创建(传入节点列表)
|
||||
if create_nodes:
|
||||
logger.info(f"[{handler.node_type}] create提交开始: nodes={len(create_nodes)}")
|
||||
logger.debug(f"{self._node_log_prefix(handler.node_type)} create提交开始: nodes={len(create_nodes)}")
|
||||
try:
|
||||
create_results = await handler.create_all(create_nodes)
|
||||
create_success = 0
|
||||
@@ -634,12 +658,12 @@ class BaseDataSource(ABC):
|
||||
elif result.status == _TaskStatus.SKIPPED:
|
||||
create_skipped += 1
|
||||
await self._handle_task_result(handler, result, async_tasks)
|
||||
logger.info(
|
||||
f"[{handler.node_type}] create提交完成: success={create_success}, "
|
||||
logger.debug(
|
||||
f"{self._node_log_prefix(handler.node_type)} create提交完成: success={create_success}, "
|
||||
f"in_progress={create_in_progress}, failed={create_failed}, skipped={create_skipped}"
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"❌ [{handler.node_type}] create_all 异常: {e}")
|
||||
logger.error("❌ [%s] create_all 异常: %s", handler.node_type, e)
|
||||
for node in create_nodes:
|
||||
ok = self._apply_sync_execute_state(
|
||||
node,
|
||||
@@ -656,7 +680,7 @@ class BaseDataSource(ABC):
|
||||
for result in update_results:
|
||||
await self._handle_task_result(handler, result, async_tasks)
|
||||
except Exception as e:
|
||||
print(f"❌ [{handler.node_type}] update_all 异常: {e}")
|
||||
logger.error("❌ [%s] update_all 异常: %s", handler.node_type, e)
|
||||
for node in update_nodes:
|
||||
ok = self._apply_sync_execute_state(
|
||||
node,
|
||||
@@ -670,8 +694,8 @@ class BaseDataSource(ABC):
|
||||
|
||||
# 轮询异步任务
|
||||
if async_tasks and poll_async_tasks:
|
||||
logger.info(
|
||||
f"[{handler.node_type}] create已提交,push_id检查即将开始: pending={len(async_tasks)}, "
|
||||
logger.debug(
|
||||
f"{self._node_log_prefix(handler.node_type)} create已提交,push_id检查即将开始: pending={len(async_tasks)}, "
|
||||
f"first_wait={self._poll_interval:.3f}s"
|
||||
)
|
||||
await self._poll_async_tasks(handler, async_tasks)
|
||||
@@ -821,14 +845,14 @@ class BaseDataSource(ABC):
|
||||
return
|
||||
|
||||
if async_tasks and poll_interval > 0:
|
||||
logger.info(
|
||||
f"[{handler.node_type}] push_id首次检查前等待 {poll_interval:.3f}s, pending={len(async_tasks)}"
|
||||
logger.debug(
|
||||
f"{self._node_log_prefix(handler.node_type)} push_id首次检查前等待 {poll_interval:.3f}s, pending={len(async_tasks)}"
|
||||
)
|
||||
await asyncio.sleep(poll_interval)
|
||||
|
||||
while async_tasks and retry_count < max_retries:
|
||||
logger.info(
|
||||
f"[{handler.node_type}] push_id检查 {retry_count + 1}/{max_retries}, pending={len(async_tasks)}"
|
||||
logger.debug(
|
||||
f"{self._node_log_prefix(handler.node_type)} push_id检查 {retry_count + 1}/{max_retries}, pending={len(async_tasks)}"
|
||||
)
|
||||
# 批量查询
|
||||
task_ids = list(async_tasks.values())
|
||||
|
||||
@@ -43,11 +43,21 @@ class BaseJsonlHandler(BaseNodeHandler):
|
||||
def __init__(
|
||||
self,
|
||||
datasource: JsonlDataSource,
|
||||
node_type: str,
|
||||
node_class: type,
|
||||
schema: type
|
||||
node_class_or_node_type,
|
||||
node_class: type | None = None,
|
||||
schema: type | None = None,
|
||||
node_type: str | None = None,
|
||||
):
|
||||
super().__init__(node_type, node_class, schema)
|
||||
if isinstance(node_class_or_node_type, str):
|
||||
resolved_node_type = node_class_or_node_type
|
||||
resolved_node_class = node_class
|
||||
resolved_schema = schema
|
||||
else:
|
||||
resolved_node_class = node_class_or_node_type
|
||||
resolved_node_type = node_type or getattr(resolved_node_class, "node_type", None)
|
||||
resolved_schema = schema or getattr(resolved_node_class, "schema", None)
|
||||
|
||||
super().__init__(resolved_node_type, resolved_node_class, resolved_schema)
|
||||
self.datasource = datasource
|
||||
|
||||
async def load(self) -> List['SyncNode']:
|
||||
|
||||
@@ -3,8 +3,10 @@ from __future__ import annotations
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from ...logging import get_logger
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def prepare_remote_jsonl_dir(remote_dir: Path, *, project_root: Path) -> Path:
|
||||
|
||||
@@ -17,10 +17,7 @@ class AttachmentApiHandler(BaseApiHandler):
|
||||
"""附件 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "attachment"
|
||||
self._node_class = AttachmentSyncNode
|
||||
self._schema = AttachmentSchema
|
||||
super().__init__(node_class=AttachmentSyncNode, schema=AttachmentSchema)
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载附件列表(依赖 Project)"""
|
||||
|
||||
@@ -9,4 +9,4 @@ from .sync_node import AttachmentSyncNode
|
||||
class AttachmentJsonlHandler(BaseJsonlHandler):
|
||||
"""附件 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "attachment", AttachmentSyncNode, Dict[str, Any])
|
||||
super().__init__(datasource, AttachmentSyncNode, schema=Dict[str, Any])
|
||||
|
||||
@@ -6,7 +6,7 @@ Company API Handler (只读)
|
||||
- 不支持创建/更新/删除
|
||||
"""
|
||||
|
||||
from typing import List, Dict, Any
|
||||
from typing import List
|
||||
from ...datasource.api.handler import BaseApiHandler
|
||||
from ...datasource.task_result import TaskResult
|
||||
from ...common.sync_node import SyncNode
|
||||
@@ -18,10 +18,7 @@ class CompanyApiHandler(BaseApiHandler):
|
||||
"""公司 API Handler (只读)"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "company"
|
||||
self._node_class = CompanySyncNode
|
||||
self._schema = CompanyResponse
|
||||
super().__init__(node_class=CompanySyncNode, schema=CompanyResponse)
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载公司列表(支持分页)"""
|
||||
@@ -85,10 +82,6 @@ class CompanyApiHandler(BaseApiHandler):
|
||||
"""轮询异步任务状态(占位)"""
|
||||
return await super().poll_tasks(task_ids)
|
||||
|
||||
def _create_node(self, data: Dict[str, Any]) -> SyncNode:
|
||||
return self._create_basic_node(data, depend_ids=[])
|
||||
|
||||
|
||||
# ========== 静态 API 函数 ==========
|
||||
|
||||
async def api_get_company_list(client, page: int = 1, page_size: int = 100) -> tuple[List[CompanyResponse], int]:
|
||||
|
||||
@@ -7,5 +7,5 @@ from schemas.company.company import CompanyResponse
|
||||
class CompanyJsonlHandler(BaseJsonlHandler):
|
||||
"""公司 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "company", CompanySyncNode, CompanyResponse)
|
||||
super().__init__(datasource, CompanySyncNode, schema=CompanyResponse)
|
||||
|
||||
|
||||
@@ -28,10 +28,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
"""施工任务 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "construction_task"
|
||||
self._node_class = ConstructionTaskSyncNode
|
||||
self._schema = ConstructionResponse
|
||||
super().__init__(node_class=ConstructionTaskSyncNode, schema=ConstructionResponse)
|
||||
self.update_schemas = [ConstructionCreate, ConstructionUpdate, ConstructionPlanUpdate]
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
@@ -101,7 +98,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
else:
|
||||
results.append(TaskResult.in_progress(node_id=node.node_id, task_id=push_id))
|
||||
except Exception as e:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e), push_id=push_id))
|
||||
|
||||
return results
|
||||
|
||||
@@ -135,6 +132,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
|
||||
node_updated = False
|
||||
error = None
|
||||
push_failure_id = None
|
||||
schema_status_lines: List[str] = []
|
||||
|
||||
# 1. 基础信息更新 (ConstructionUpdate)
|
||||
@@ -186,11 +184,12 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
except Exception as e:
|
||||
error = str(e)
|
||||
base_update_error = str(e)
|
||||
push_failure_id = push_id
|
||||
else:
|
||||
base_update_error = "show_name changed but task_type != QT"
|
||||
|
||||
schema_status_lines.append(
|
||||
self._build_update_schema_status(
|
||||
self.build_update_schema_status(
|
||||
schema_name="ConstructionUpdate",
|
||||
schema=ConstructionUpdate,
|
||||
new_data=data,
|
||||
@@ -236,9 +235,10 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
except Exception as e:
|
||||
error = str(e)
|
||||
plan_update_error = str(e)
|
||||
push_failure_id = push_id
|
||||
|
||||
schema_status_lines.append(
|
||||
self._build_update_schema_status(
|
||||
self.build_update_schema_status(
|
||||
schema_name="ConstructionPlanUpdate",
|
||||
schema=ConstructionPlanUpdate,
|
||||
new_data=data,
|
||||
@@ -253,7 +253,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
|
||||
# 汇总结果
|
||||
if error:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=error))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=error, push_id=push_failure_id))
|
||||
elif node_updated:
|
||||
results.append(TaskResult.success(node_id=node.node_id))
|
||||
else:
|
||||
@@ -279,7 +279,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
await api_delete_construction_task(self.api_client, biz_id, push_id)
|
||||
results.append(TaskResult.success(node_id=node.node_id))
|
||||
except Exception as e:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e), push_id=push_id))
|
||||
|
||||
return results
|
||||
|
||||
|
||||
@@ -7,4 +7,4 @@ from schemas.construction.construction import ConstructionResponse
|
||||
class ConstructionTaskJsonlHandler(BaseJsonlHandler):
|
||||
"""施工任务 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "construction_task", ConstructionTaskSyncNode, ConstructionResponse)
|
||||
super().__init__(datasource, ConstructionTaskSyncNode, schema=ConstructionResponse)
|
||||
|
||||
@@ -19,7 +19,8 @@ class ConstructionTaskSyncStrategy(DefaultSyncStrategy[ConstructionResponse]):
|
||||
update_direction=UpdateDirection.PUSH,
|
||||
)
|
||||
|
||||
def preprocess_compare_payload(self, payload: dict) -> dict:
|
||||
def normalize_compare_payload(self, data: dict | None, data_id_map=None) -> dict:
|
||||
payload = super().normalize_compare_payload(data, data_id_map)
|
||||
plan_nodes = payload.get("plan_node")
|
||||
if not isinstance(plan_nodes, list):
|
||||
return payload
|
||||
|
||||
@@ -17,39 +17,68 @@ from schemas.construction.construction_detail import (
|
||||
ConstructionDetailCreate,
|
||||
ConstructionDetailUpdate
|
||||
)
|
||||
from ..project.api_handler import ProjectApiHandler
|
||||
|
||||
|
||||
class ConstructionTaskDetailApiHandler(BaseApiHandler):
|
||||
"""施工明细 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "construction_task_detail"
|
||||
self._node_class = ConstructionTaskDetailSyncNode
|
||||
self._schema = ConstructionDetailResponse
|
||||
def __init__(self, **handler_config: Any):
|
||||
super().__init__(node_class=ConstructionTaskDetailSyncNode, schema=ConstructionDetailResponse, **handler_config)
|
||||
self.update_schemas = [ConstructionDetailCreate, ConstructionDetailUpdate]
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载施工明细列表(依赖 ConstructionTask)"""
|
||||
|
||||
def _use_aggregate_load(self) -> bool:
|
||||
return str(self.handler_config.get("load_method", "legacy")).lower() == "aggregate"
|
||||
|
||||
async def _load_via_project_aggregate(self) -> List[SyncNode]:
|
||||
projects = self._collection.filter(node_type="project")
|
||||
project_ids = [project.data_id for project in projects if project.data_id]
|
||||
if not project_ids:
|
||||
return []
|
||||
|
||||
all_details: List[SyncNode] = []
|
||||
aggregate_limit = int(self.handler_config.get("aggregate_limit", 10000))
|
||||
for project_id in project_ids:
|
||||
aggregate = await ProjectApiHandler.get_aggregate(
|
||||
self.api_client,
|
||||
project_id,
|
||||
domains=["construction_details"],
|
||||
limit=aggregate_limit,
|
||||
)
|
||||
truncated = set(aggregate.get("meta", {}).get("truncated_domains", []))
|
||||
if "construction_details" in truncated:
|
||||
raise RuntimeError(
|
||||
f"project aggregate truncated domain construction_details for project {project_id}"
|
||||
)
|
||||
for detail_data in aggregate.get("data", {}).get("construction_details", []):
|
||||
all_details.append(self._create_node(dict(detail_data)))
|
||||
return all_details
|
||||
|
||||
async def _load_via_legacy_list(self) -> List[SyncNode]:
|
||||
tasks = self._collection.filter(node_type="construction_task")
|
||||
task_ids = [t.data_id for t in tasks if t.data_id]
|
||||
|
||||
|
||||
if not task_ids:
|
||||
return []
|
||||
|
||||
|
||||
all_details = []
|
||||
for task_id in task_ids:
|
||||
try:
|
||||
details = await api_get_construction_task_detail_list(self.api_client, task_id)
|
||||
for detail_model in details:
|
||||
# 已验证的 Pydantic model,转换为 dict
|
||||
detail_data = detail_model.model_dump()
|
||||
all_details.append(self._create_node(detail_data))
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to load construction details for task {task_id}: {e}")
|
||||
|
||||
|
||||
return all_details
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载施工明细列表(依赖 ConstructionTask)"""
|
||||
if self._use_aggregate_load():
|
||||
return await self._load_via_project_aggregate()
|
||||
return await self._load_via_legacy_list()
|
||||
|
||||
async def create_all(self, nodes: List[SyncNode]) -> List[TaskResult]:
|
||||
"""批量创建施工明细"""
|
||||
from pydantic import ValidationError
|
||||
@@ -98,7 +127,7 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
|
||||
else:
|
||||
results.append(TaskResult.in_progress(node_id=node.node_id, task_id=push_id))
|
||||
except Exception as e:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e), push_id=push_id))
|
||||
|
||||
return results
|
||||
|
||||
@@ -124,9 +153,9 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
|
||||
|
||||
origin_data = node.get_origin_data() or {}
|
||||
|
||||
# 使用 _get_schema_diff 检查差异并打印日志
|
||||
# 使用 get_update_schema_diff 作为 update 入口检查差异并打印日志
|
||||
force_steps_update = self._steps_changed(node)
|
||||
diff_fields = self._get_schema_diff(
|
||||
diff_fields = self.get_update_schema_diff(
|
||||
ConstructionDetailUpdate,
|
||||
data,
|
||||
origin_data,
|
||||
@@ -145,7 +174,7 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
|
||||
continue
|
||||
|
||||
# 需要完整的 schema 数据(包括必填字段)
|
||||
update_data = self._filter_update_data(
|
||||
update_data = self.build_update_request_data(
|
||||
data,
|
||||
origin_data,
|
||||
ConstructionDetailUpdate,
|
||||
@@ -190,7 +219,7 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
|
||||
# 整组失败
|
||||
error_msg = str(e)
|
||||
for node, _ in group_items:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=error_msg))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=error_msg, push_id=push_id))
|
||||
|
||||
return results
|
||||
|
||||
@@ -208,7 +237,7 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
|
||||
await api_delete_construction_task_detail(self.api_client, biz_id, push_id)
|
||||
results.append(TaskResult.success(node_id=node.node_id))
|
||||
except Exception as e:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e), push_id=push_id))
|
||||
|
||||
return results
|
||||
|
||||
|
||||
@@ -9,4 +9,4 @@ from schemas.construction.construction_detail import ConstructionDetailResponse
|
||||
class ConstructionTaskDetailJsonlHandler(BaseJsonlHandler):
|
||||
"""施工任务明细 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "construction_task_detail", ConstructionTaskDetailSyncNode, ConstructionDetailResponse)
|
||||
super().__init__(datasource, ConstructionTaskDetailSyncNode, schema=ConstructionDetailResponse)
|
||||
|
||||
@@ -101,10 +101,7 @@ class ContractApiHandler(BaseApiHandler):
|
||||
"""合同 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "contract"
|
||||
self._node_class = ContractSyncNode
|
||||
self._schema = ContractResponse
|
||||
super().__init__(node_class=ContractSyncNode, schema=ContractResponse)
|
||||
self.update_schemas = [ContractCreate, ContractUpdate]
|
||||
|
||||
# ========== Handler 接口实现 ==========
|
||||
@@ -256,7 +253,7 @@ class ContractApiHandler(BaseApiHandler):
|
||||
# 过滤出需要更新的字段(排除空值、未变化的字段)
|
||||
origin_data = node.get_origin_data() or {}
|
||||
force_steps_update = self._steps_changed(node)
|
||||
update_data = self._filter_update_data(
|
||||
update_data = self.build_update_request_data(
|
||||
new_data=data,
|
||||
original_data=origin_data,
|
||||
schema=ContractUpdate,
|
||||
|
||||
@@ -9,6 +9,6 @@ from schemas.contract.contract import ContractResponse
|
||||
class ContractJsonlHandler(BaseJsonlHandler):
|
||||
"""合同 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "contract", ContractSyncNode, ContractResponse)
|
||||
super().__init__(datasource, ContractSyncNode, schema=ContractResponse)
|
||||
|
||||
|
||||
|
||||
@@ -24,11 +24,7 @@ class ContractChangeApiHandler(BaseApiHandler):
|
||||
"""合同变更 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "contract_change"
|
||||
self._node_class = ContractChangeSyncNode
|
||||
# TODO: ContractChangeResponse schema in schemas/project_extensions/contract_change.py is missing 'project_id'
|
||||
self._schema = ContractChangeResponse
|
||||
super().__init__(node_class=ContractChangeSyncNode, schema=ContractChangeResponse)
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载合同变更数据"""
|
||||
@@ -93,7 +89,7 @@ class ContractChangeApiHandler(BaseApiHandler):
|
||||
|
||||
origin_data = node.get_origin_data() or {}
|
||||
force_steps_update = self._steps_changed(node)
|
||||
update_data = self._filter_update_data(
|
||||
update_data = self.build_update_request_data(
|
||||
node_data,
|
||||
origin_data,
|
||||
PostContractChange,
|
||||
|
||||
@@ -7,4 +7,4 @@ from schemas.project_extensions.contract_change import ContractChangeResponse
|
||||
class ContractChangeJsonlHandler(BaseJsonlHandler):
|
||||
"""合同变更 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "contract_change", ContractChangeSyncNode, ContractChangeResponse)
|
||||
super().__init__(datasource, ContractChangeSyncNode, schema=ContractChangeResponse)
|
||||
|
||||
@@ -28,10 +28,7 @@ class ContractSettlementApiHandler(BaseApiHandler):
|
||||
"""合同结算 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "contract_settlement"
|
||||
self._node_class = ContractSettlementSyncNode
|
||||
self._schema = ContractSettlementResponse
|
||||
super().__init__(node_class=ContractSettlementSyncNode, schema=ContractSettlementResponse)
|
||||
self.update_schemas = [ContractSettlementCreate, ContractSettlementUpdate, ContractSettlementPlanUpdate]
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
@@ -140,7 +137,7 @@ class ContractSettlementApiHandler(BaseApiHandler):
|
||||
|
||||
# 1. 基础信息更新 (ContractSettlementUpdate)
|
||||
force_steps_update = self._steps_changed(node)
|
||||
base_diff = self._get_schema_diff(
|
||||
base_diff = self.get_update_schema_diff(
|
||||
ContractSettlementUpdate,
|
||||
data,
|
||||
origin_data,
|
||||
@@ -167,7 +164,7 @@ class ContractSettlementApiHandler(BaseApiHandler):
|
||||
error = f"API error: {e}"
|
||||
|
||||
# 2. 计划信息更新 (ContractSettlementPlanUpdate)
|
||||
plan_diff = self._get_schema_diff(ContractSettlementPlanUpdate, data, origin_data, node.node_id)
|
||||
plan_diff = self.get_update_schema_diff(ContractSettlementPlanUpdate, data, origin_data, node.node_id)
|
||||
if not error and plan_diff:
|
||||
try:
|
||||
plan_model = ContractSettlementPlanUpdate.model_validate(data)
|
||||
|
||||
@@ -7,4 +7,4 @@ from schemas.contract_settlement.contract_settlement import ContractSettlementRe
|
||||
class ContractSettlementJsonlHandler(BaseJsonlHandler):
|
||||
"""合同结算 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "contract_settlement", ContractSettlementSyncNode, ContractSettlementResponse)
|
||||
super().__init__(datasource, ContractSettlementSyncNode, schema=ContractSettlementResponse)
|
||||
|
||||
@@ -19,7 +19,8 @@ class ContractSettlementSyncStrategy(DefaultSyncStrategy[ContractSettlementRespo
|
||||
update_direction=UpdateDirection.PUSH,
|
||||
)
|
||||
|
||||
def preprocess_compare_payload(self, payload: dict) -> dict:
|
||||
def normalize_compare_payload(self, data: dict | None, data_id_map=None) -> dict:
|
||||
payload = super().normalize_compare_payload(data, data_id_map)
|
||||
plan_nodes = payload.get("plan_node")
|
||||
if not isinstance(plan_nodes, list):
|
||||
return payload
|
||||
|
||||
@@ -17,26 +17,50 @@ from schemas.contract_settlement.contract_settlement_detail import (
|
||||
ContractSettlementDetailCreate,
|
||||
ContractSettlementDetailUpdate
|
||||
)
|
||||
from ..project.api_handler import ProjectApiHandler
|
||||
|
||||
|
||||
class ContractSettlementDetailApiHandler(BaseApiHandler):
|
||||
"""合同结算明细 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "contract_settlement_detail"
|
||||
self._node_class = ContractSettlementDetailSyncNode
|
||||
self._schema = ContractSettlementDetailResponse
|
||||
def __init__(self, **handler_config: Any):
|
||||
super().__init__(node_class=ContractSettlementDetailSyncNode, schema=ContractSettlementDetailResponse, **handler_config)
|
||||
self.update_schemas = [ContractSettlementDetailCreate, ContractSettlementDetailUpdate]
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载合同结算明细列表(依赖 ContractSettlement)"""
|
||||
|
||||
def _use_aggregate_load(self) -> bool:
|
||||
return str(self.handler_config.get("load_method", "legacy")).lower() == "aggregate"
|
||||
|
||||
async def _load_via_project_aggregate(self) -> List[SyncNode]:
|
||||
projects = self._collection.filter(node_type="project")
|
||||
project_ids = [project.data_id for project in projects if project.data_id]
|
||||
if not project_ids:
|
||||
return []
|
||||
|
||||
all_details: List[SyncNode] = []
|
||||
aggregate_limit = int(self.handler_config.get("aggregate_limit", 10000))
|
||||
for project_id in project_ids:
|
||||
aggregate = await ProjectApiHandler.get_aggregate(
|
||||
self.api_client,
|
||||
project_id,
|
||||
domains=["contract_settlement_details"],
|
||||
limit=aggregate_limit,
|
||||
)
|
||||
truncated = set(aggregate.get("meta", {}).get("truncated_domains", []))
|
||||
if "contract_settlement_details" in truncated:
|
||||
raise RuntimeError(
|
||||
f"project aggregate truncated domain contract_settlement_details for project {project_id}"
|
||||
)
|
||||
for detail_data in aggregate.get("data", {}).get("contract_settlement_details", []):
|
||||
all_details.append(self._create_node(dict(detail_data)))
|
||||
return all_details
|
||||
|
||||
async def _load_via_legacy_list(self) -> List[SyncNode]:
|
||||
settlements = self._collection.filter(node_type="contract_settlement")
|
||||
settlement_ids = [s.data_id for s in settlements if s.data_id]
|
||||
|
||||
|
||||
if not settlement_ids:
|
||||
return []
|
||||
|
||||
|
||||
all_details = []
|
||||
for settlement_id in settlement_ids:
|
||||
try:
|
||||
@@ -46,9 +70,15 @@ class ContractSettlementDetailApiHandler(BaseApiHandler):
|
||||
all_details.append(self._create_node(detail_data))
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to load settlement details for {settlement_id}: {e}")
|
||||
|
||||
|
||||
return all_details
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载合同结算明细列表(依赖 ContractSettlement)"""
|
||||
if self._use_aggregate_load():
|
||||
return await self._load_via_project_aggregate()
|
||||
return await self._load_via_legacy_list()
|
||||
|
||||
async def create_all(self, nodes: List[SyncNode]) -> List[TaskResult]:
|
||||
"""批量创建合同结算明细"""
|
||||
from pydantic import ValidationError
|
||||
@@ -122,9 +152,9 @@ class ContractSettlementDetailApiHandler(BaseApiHandler):
|
||||
|
||||
origin_data = node.get_origin_data() or {}
|
||||
|
||||
# 使用 _get_schema_diff 检查差异并打印日志
|
||||
# 使用 get_update_schema_diff 作为 update 入口检查差异并打印日志
|
||||
force_steps_update = self._steps_changed(node)
|
||||
diff_fields = self._get_schema_diff(
|
||||
diff_fields = self.get_update_schema_diff(
|
||||
ContractSettlementDetailUpdate,
|
||||
data,
|
||||
origin_data,
|
||||
@@ -143,7 +173,7 @@ class ContractSettlementDetailApiHandler(BaseApiHandler):
|
||||
continue
|
||||
|
||||
# 需要完整的 schema 数据(包括必填字段)
|
||||
update_data = self._filter_update_data(
|
||||
update_data = self.build_update_request_data(
|
||||
data,
|
||||
origin_data,
|
||||
ContractSettlementDetailUpdate,
|
||||
|
||||
@@ -9,4 +9,4 @@ from schemas.contract_settlement.contract_settlement_detail import ContractSettl
|
||||
class ContractSettlementDetailJsonlHandler(BaseJsonlHandler):
|
||||
"""合同结算明细 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "contract_settlement_detail", ContractSettlementDetailSyncNode, ContractSettlementDetailResponse)
|
||||
super().__init__(datasource, ContractSettlementDetailSyncNode, schema=ContractSettlementDetailResponse)
|
||||
|
||||
@@ -31,10 +31,7 @@ class LarApiHandler(BaseApiHandler):
|
||||
"""移民征地 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "lar"
|
||||
self._node_class = LarSyncNode
|
||||
self._schema = LarDetailSchema
|
||||
super().__init__(node_class=LarSyncNode, schema=LarDetailSchema)
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""
|
||||
@@ -118,7 +115,7 @@ class LarApiHandler(BaseApiHandler):
|
||||
|
||||
for schema, api_func in update_configs:
|
||||
# 获取过滤后的更新数据
|
||||
update_data = self._filter_update_data(
|
||||
update_data = self.build_update_request_data(
|
||||
node_data,
|
||||
origin_data,
|
||||
schema,
|
||||
|
||||
@@ -7,4 +7,4 @@ from schemas.project_extensions.lar import LarDetailSchema
|
||||
class LarJsonlHandler(BaseJsonlHandler):
|
||||
"""移民征地 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "lar", LarSyncNode, LarDetailSchema)
|
||||
super().__init__(datasource, LarSyncNode, schema=LarDetailSchema)
|
||||
|
||||
@@ -28,10 +28,7 @@ class LarChangeApiHandler(BaseApiHandler):
|
||||
"""移民变更 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "lar_change"
|
||||
self._node_class = LarChangeSyncNode
|
||||
self._schema = LarChangeDetailSchema
|
||||
super().__init__(node_class=LarChangeSyncNode, schema=LarChangeDetailSchema)
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""
|
||||
@@ -104,7 +101,7 @@ class LarChangeApiHandler(BaseApiHandler):
|
||||
continue
|
||||
origin_data = node.get_origin_data() or {}
|
||||
force_steps_update = self._steps_changed(node)
|
||||
update_data = self._filter_update_data(
|
||||
update_data = self.build_update_request_data(
|
||||
node_data,
|
||||
origin_data,
|
||||
LarChangeUpdateSchema,
|
||||
|
||||
@@ -9,4 +9,4 @@ class LarChangeJsonlHandler(BaseJsonlHandler):
|
||||
"""LAR Change (移民变更) JSONL 数据处理器"""
|
||||
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "lar_change", LarChangeSyncNode, LarChangeDetailSchema)
|
||||
super().__init__(datasource, LarChangeSyncNode, schema=LarChangeDetailSchema)
|
||||
|
||||
@@ -23,10 +23,7 @@ class MaterialApiHandler(BaseApiHandler):
|
||||
"""物资 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "material"
|
||||
self._node_class = MaterialSyncNode
|
||||
self._schema = MaterialResponse
|
||||
super().__init__(node_class=MaterialSyncNode, schema=MaterialResponse)
|
||||
self.update_schemas = [MaterialCreate, MaterialUpdate, MaterialPlanUpdate]
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
@@ -106,7 +103,7 @@ class MaterialApiHandler(BaseApiHandler):
|
||||
|
||||
# 1. 基础信息更新 (MaterialUpdate)
|
||||
force_steps_update = self._steps_changed(node)
|
||||
base_diff = self._get_schema_diff(
|
||||
base_diff = self.get_update_schema_diff(
|
||||
MaterialUpdate,
|
||||
data,
|
||||
origin_data,
|
||||
@@ -131,7 +128,7 @@ class MaterialApiHandler(BaseApiHandler):
|
||||
error = f"{ERROR_VALIDATION_FAILED}: {e}"
|
||||
|
||||
# 2. 计划信息更新 (MaterialPlanUpdate)
|
||||
plan_diff = self._get_schema_diff(MaterialPlanUpdate, data, origin_data, node.node_id)
|
||||
plan_diff = self.get_update_schema_diff(MaterialPlanUpdate, data, origin_data, node.node_id)
|
||||
if not error and plan_diff:
|
||||
try:
|
||||
plan_model = MaterialPlanUpdate.model_validate(data)
|
||||
|
||||
@@ -7,4 +7,4 @@ from schemas.material.material import MaterialResponse
|
||||
class MaterialJsonlHandler(BaseJsonlHandler):
|
||||
"""物资 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "material", MaterialSyncNode, MaterialResponse)
|
||||
super().__init__(datasource, MaterialSyncNode, schema=MaterialResponse)
|
||||
|
||||
@@ -19,7 +19,8 @@ class MaterialSyncStrategy(DefaultSyncStrategy[MaterialResponse]):
|
||||
update_direction=UpdateDirection.PUSH,
|
||||
)
|
||||
|
||||
def preprocess_compare_payload(self, payload: dict) -> dict:
|
||||
def normalize_compare_payload(self, data: dict | None, data_id_map=None) -> dict:
|
||||
payload = super().normalize_compare_payload(data, data_id_map)
|
||||
plan_nodes = payload.get("plan_node")
|
||||
if not isinstance(plan_nodes, list):
|
||||
return payload
|
||||
|
||||
@@ -32,49 +32,75 @@ from schemas.material.material_detail import (
|
||||
MaterialDetailCreate,
|
||||
MaterialDetailUpdate
|
||||
)
|
||||
from ..project.api_handler import ProjectApiHandler
|
||||
|
||||
|
||||
class MaterialDetailApiHandler(BaseApiHandler):
|
||||
"""物资明细 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "material_detail"
|
||||
self._node_class = MaterialDetailSyncNode
|
||||
self._schema = MaterialDetailResponse
|
||||
def __init__(self, **handler_config: Any):
|
||||
super().__init__(node_class=MaterialDetailSyncNode, schema=MaterialDetailResponse, **handler_config)
|
||||
self.update_schemas = [MaterialDetailCreate, MaterialDetailUpdate]
|
||||
|
||||
def _use_aggregate_load(self) -> bool:
|
||||
return str(self.handler_config.get("load_method", "legacy")).lower() == "aggregate"
|
||||
|
||||
async def _load_via_project_aggregate(self) -> List[SyncNode]:
|
||||
projects = self._collection.filter(node_type="project")
|
||||
project_ids = [project.data_id for project in projects if project.data_id]
|
||||
if not project_ids:
|
||||
return []
|
||||
|
||||
all_nodes: List[SyncNode] = []
|
||||
aggregate_limit = int(self.handler_config.get("aggregate_limit", 10000))
|
||||
for project_id in project_ids:
|
||||
aggregate = await ProjectApiHandler.get_aggregate(
|
||||
self.api_client,
|
||||
project_id,
|
||||
domains=["material_details"],
|
||||
limit=aggregate_limit,
|
||||
)
|
||||
truncated = set(aggregate.get("meta", {}).get("truncated_domains", []))
|
||||
if "material_details" in truncated:
|
||||
raise RuntimeError(
|
||||
f"project aggregate truncated domain material_details for project {project_id}"
|
||||
)
|
||||
for detail_data in aggregate.get("data", {}).get("material_details", []):
|
||||
node = self._create_node(dict(detail_data))
|
||||
all_nodes.append(node)
|
||||
return all_nodes
|
||||
|
||||
async def _load_via_legacy_list(self) -> List[SyncNode]:
|
||||
materials = self._collection.filter(node_type="material")
|
||||
material_ids = [m.data_id for m in materials if m.data_id]
|
||||
|
||||
if not material_ids:
|
||||
return []
|
||||
|
||||
all_nodes = []
|
||||
for material_id in material_ids:
|
||||
try:
|
||||
details = await api_get_material_detail_list(self.api_client, material_id)
|
||||
for detail_model in details:
|
||||
detail_data = detail_model.model_dump()
|
||||
node = self._create_node(detail_data)
|
||||
all_nodes.append(node)
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to load material details for {material_id}: {e}")
|
||||
|
||||
return all_nodes
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载物资明细列表(依赖 Material)"""
|
||||
try:
|
||||
materials = self._collection.filter(node_type="material")
|
||||
material_ids = [m.data_id for m in materials if m.data_id]
|
||||
|
||||
if not material_ids:
|
||||
return []
|
||||
|
||||
all_nodes = []
|
||||
for material_id in material_ids:
|
||||
try:
|
||||
details = await api_get_material_detail_list(self.api_client, material_id)
|
||||
for detail_model in details:
|
||||
# 已验证的 Pydantic model,转换为 dict
|
||||
detail_data = detail_model.model_dump()
|
||||
node = self._create_node(detail_data)
|
||||
all_nodes.append(node)
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to load material details for {material_id}: {e}")
|
||||
|
||||
return all_nodes
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to load material details: {e}")
|
||||
return []
|
||||
if self._use_aggregate_load():
|
||||
return await self._load_via_project_aggregate()
|
||||
return await self._load_via_legacy_list()
|
||||
|
||||
async def create_all(self, nodes: List[SyncNode]) -> List[TaskResult]:
|
||||
"""批量创建物资明细"""
|
||||
results = []
|
||||
for node in nodes:
|
||||
push_id: str | None = None
|
||||
try:
|
||||
data = node.get_data()
|
||||
if not data:
|
||||
@@ -118,7 +144,13 @@ class MaterialDetailApiHandler(BaseApiHandler):
|
||||
else:
|
||||
results.append(TaskResult.in_progress(node_id=node.node_id, task_id=push_id))
|
||||
except Exception as e:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=f"Create failed: {str(e)}"))
|
||||
results.append(
|
||||
TaskResult.failed(
|
||||
node_id=node.node_id,
|
||||
error=f"Create failed: {str(e)}",
|
||||
push_id=push_id,
|
||||
)
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
@@ -144,9 +176,9 @@ class MaterialDetailApiHandler(BaseApiHandler):
|
||||
|
||||
origin_data = node.get_origin_data() or {}
|
||||
|
||||
# 使用 _get_schema_diff 检查差异并打印日志
|
||||
# 使用 get_update_schema_diff 作为 update 入口检查差异并打印日志
|
||||
force_steps_update = self._steps_changed(node)
|
||||
diff_fields = self._get_schema_diff(
|
||||
diff_fields = self.get_update_schema_diff(
|
||||
MaterialDetailUpdate,
|
||||
data,
|
||||
origin_data,
|
||||
@@ -165,7 +197,7 @@ class MaterialDetailApiHandler(BaseApiHandler):
|
||||
continue
|
||||
|
||||
# 需要完整的 schema 数据(包括必填字段)
|
||||
update_data = self._filter_update_data(
|
||||
update_data = self.build_update_request_data(
|
||||
data,
|
||||
origin_data,
|
||||
MaterialDetailUpdate,
|
||||
@@ -209,7 +241,7 @@ class MaterialDetailApiHandler(BaseApiHandler):
|
||||
# 整组失败
|
||||
error_msg = str(e)
|
||||
for node, _ in group_items:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=error_msg))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=error_msg, push_id=push_id))
|
||||
|
||||
return results
|
||||
|
||||
|
||||
@@ -9,4 +9,4 @@ from schemas.material.material_detail import MaterialDetailResponse
|
||||
class MaterialDetailJsonlHandler(BaseJsonlHandler):
|
||||
"""物资明细 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "material_detail", MaterialDetailSyncNode, MaterialDetailResponse)
|
||||
super().__init__(datasource, MaterialDetailSyncNode, schema=MaterialDetailResponse)
|
||||
|
||||
@@ -19,10 +19,7 @@ class PersonnelApiHandler(BaseApiHandler):
|
||||
"""人员 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "personnel"
|
||||
self._node_class = PersonnelSyncNode
|
||||
self._schema = PowerResponse
|
||||
super().__init__(node_class=PersonnelSyncNode, schema=PowerResponse)
|
||||
self.update_schemas = [PowerCreate, PowerUpdate, PowerPlanUpdate]
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
@@ -121,7 +118,7 @@ class PersonnelApiHandler(BaseApiHandler):
|
||||
|
||||
# 1. 基础信息更新 (PowerUpdate)
|
||||
force_steps_update = self._steps_changed(node)
|
||||
base_diff = self._get_schema_diff(
|
||||
base_diff = self.get_update_schema_diff(
|
||||
PowerUpdate,
|
||||
data,
|
||||
origin_data,
|
||||
|
||||
@@ -7,4 +7,4 @@ from schemas.power.power import PowerResponse
|
||||
class PersonnelJsonlHandler(BaseJsonlHandler):
|
||||
"""人员 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "personnel", PersonnelSyncNode, PowerResponse)
|
||||
super().__init__(datasource, PersonnelSyncNode, schema=PowerResponse)
|
||||
|
||||
@@ -13,26 +13,51 @@ from ...common.sync_node import SyncNode
|
||||
from ...datasource.api.errors import ERROR_NODE_DATA_NONE, ERROR_ID_NOT_FOUND
|
||||
from .sync_node import PersonnelDetailSyncNode
|
||||
from schemas.power.power_detail import PowerDetail, PowerDetailCreate, PowerDetailUpdate
|
||||
from ..project.api_handler import ProjectApiHandler
|
||||
|
||||
|
||||
class PersonnelDetailApiHandler(BaseApiHandler):
|
||||
"""人员明细 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "personnel_detail"
|
||||
self._node_class = PersonnelDetailSyncNode
|
||||
self._schema = PowerDetail
|
||||
def __init__(self, **handler_config: Any):
|
||||
super().__init__(node_class=PersonnelDetailSyncNode, schema=PowerDetail, **handler_config)
|
||||
self.update_schemas = [PowerDetailCreate, PowerDetailUpdate]
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载人员明细列表(依赖 Personnel)"""
|
||||
|
||||
def _use_aggregate_load(self) -> bool:
|
||||
return str(self.handler_config.get("load_method", "legacy")).lower() == "aggregate"
|
||||
|
||||
async def _load_via_project_aggregate(self) -> List[SyncNode]:
|
||||
projects = self._collection.filter(node_type="project")
|
||||
project_ids = [project.data_id for project in projects if project.data_id]
|
||||
if not project_ids:
|
||||
return []
|
||||
|
||||
all_details: List[SyncNode] = []
|
||||
aggregate_limit = int(self.handler_config.get("aggregate_limit", 10000))
|
||||
for project_id in project_ids:
|
||||
aggregate = await ProjectApiHandler.get_aggregate(
|
||||
self.api_client,
|
||||
project_id,
|
||||
domains=["power_details"],
|
||||
limit=aggregate_limit,
|
||||
)
|
||||
truncated = set(aggregate.get("meta", {}).get("truncated_domains", []))
|
||||
if "power_details" in truncated:
|
||||
raise RuntimeError(
|
||||
f"project aggregate truncated domain power_details for project {project_id}"
|
||||
)
|
||||
for detail_data in aggregate.get("data", {}).get("power_details", []):
|
||||
validated = PowerDetail.model_validate(detail_data)
|
||||
all_details.append(self._create_node(validated.model_dump()))
|
||||
return all_details
|
||||
|
||||
async def _load_via_legacy_list(self) -> List[SyncNode]:
|
||||
personnel_list = self._collection.filter(node_type="personnel")
|
||||
personnel_ids = [p.data_id for p in personnel_list if p.data_id]
|
||||
|
||||
|
||||
if not personnel_ids:
|
||||
return []
|
||||
|
||||
|
||||
all_details = []
|
||||
for personnel_id in personnel_ids:
|
||||
try:
|
||||
@@ -42,9 +67,15 @@ class PersonnelDetailApiHandler(BaseApiHandler):
|
||||
all_details.append(self._create_node(validated.model_dump()))
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to load personnel details for {personnel_id}: {e}")
|
||||
|
||||
|
||||
return all_details
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载人员明细列表(依赖 Personnel)"""
|
||||
if self._use_aggregate_load():
|
||||
return await self._load_via_project_aggregate()
|
||||
return await self._load_via_legacy_list()
|
||||
|
||||
async def create_all(self, nodes: List[SyncNode]) -> List[TaskResult]:
|
||||
"""批量创建人员明细"""
|
||||
results = []
|
||||
@@ -111,9 +142,9 @@ class PersonnelDetailApiHandler(BaseApiHandler):
|
||||
|
||||
origin_data = node.get_origin_data() or {}
|
||||
|
||||
# 使用 _get_schema_diff 检查差异并打印日志
|
||||
# 使用 get_update_schema_diff 作为 update 入口检查差异并打印日志
|
||||
force_steps_update = self._steps_changed(node)
|
||||
diff_fields = self._get_schema_diff(
|
||||
diff_fields = self.get_update_schema_diff(
|
||||
PowerDetailUpdate,
|
||||
data,
|
||||
origin_data,
|
||||
@@ -132,7 +163,7 @@ class PersonnelDetailApiHandler(BaseApiHandler):
|
||||
continue
|
||||
|
||||
# 需要完整的 schema 数据(包括必填字段)
|
||||
update_data = self._filter_update_data(
|
||||
update_data = self.build_update_request_data(
|
||||
data,
|
||||
origin_data,
|
||||
PowerDetailUpdate,
|
||||
|
||||
@@ -9,4 +9,4 @@ from schemas.power.power_detail import PowerDetail
|
||||
class PersonnelDetailJsonlHandler(BaseJsonlHandler):
|
||||
"""人员明细 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "personnel_detail", PersonnelDetailSyncNode, PowerDetail)
|
||||
super().__init__(datasource, PersonnelDetailSyncNode, schema=PowerDetail)
|
||||
|
||||
@@ -10,7 +10,7 @@ PUT:
|
||||
- /preparation - 更新里程碑数据
|
||||
"""
|
||||
|
||||
from typing import List, Dict, Any
|
||||
from typing import List, Dict
|
||||
from ...datasource.api.handler import BaseApiHandler
|
||||
from ...datasource.task_result import TaskResult
|
||||
from ...common.sync_node import SyncNode
|
||||
@@ -23,10 +23,7 @@ class PreparationApiHandler(BaseApiHandler):
|
||||
"""里程碑 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "preparation"
|
||||
self._node_class = PreparationSyncNode
|
||||
self._schema = PreparationDetail
|
||||
super().__init__(node_class=PreparationSyncNode, schema=PreparationDetail)
|
||||
self.update_schemas = [PostPreparation]
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
@@ -94,7 +91,7 @@ class PreparationApiHandler(BaseApiHandler):
|
||||
|
||||
origin_data = node.get_origin_data() or {}
|
||||
force_steps_update = self._steps_changed(node)
|
||||
update_data = self._filter_update_data(
|
||||
update_data = self.build_update_request_data(
|
||||
node_data,
|
||||
origin_data,
|
||||
PostPreparation,
|
||||
@@ -141,10 +138,6 @@ class PreparationApiHandler(BaseApiHandler):
|
||||
for node in nodes
|
||||
]
|
||||
|
||||
def _create_node(self, data: Dict[str, Any]) -> SyncNode:
|
||||
return self._create_basic_node(data, depend_ids=[])
|
||||
|
||||
|
||||
# ========== 静态 API 函数 ==========
|
||||
|
||||
async def api_update_preparation(
|
||||
|
||||
@@ -8,4 +8,4 @@ from schemas.project_extensions.preparation import PreparationDetail
|
||||
class PreparationJsonlHandler(BaseJsonlHandler):
|
||||
"""里程碑 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "preparation", PreparationSyncNode, PreparationDetail)
|
||||
super().__init__(datasource, PreparationSyncNode, schema=PreparationDetail)
|
||||
|
||||
@@ -20,16 +20,13 @@ class PreparationSyncStrategy(DefaultSyncStrategy[PreparationDetail]):
|
||||
update_direction=UpdateDirection.PUSH,
|
||||
)
|
||||
|
||||
def _needs_update(self, source_node, target_node, resolved_data=None, data_id_map=None) -> bool:
|
||||
if resolved_data is None:
|
||||
raise ValueError(f"[{self.node_type}] resolved_data is required for differential sync.")
|
||||
|
||||
target_data = target_node.get_data()
|
||||
def should_update_pair(self, source_node, target_node, *, source_data, target_data, data_id_map=None) -> bool:
|
||||
if target_data is None:
|
||||
return False
|
||||
|
||||
# 这里只按 PostPreparation 可写字段判定是否需要 update,避免 response-only 字段差异触发空更新。
|
||||
update_fields = set(PostPreparation.model_fields.keys())
|
||||
source_payload = {key: value for key, value in resolved_data.items() if key in update_fields}
|
||||
source_payload = {key: value for key, value in source_data.items() if key in update_fields}
|
||||
target_payload = {key: value for key, value in target_data.items() if key in update_fields}
|
||||
|
||||
normalized_source = normalized_data_for_compare(source_payload, data_id_map or {})
|
||||
|
||||
@@ -23,10 +23,7 @@ class ProductionApiHandler(BaseApiHandler):
|
||||
"""投产节点 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "production"
|
||||
self._node_class = ProductionSyncNode
|
||||
self._schema = ProductionDetail
|
||||
super().__init__(node_class=ProductionSyncNode, schema=ProductionDetail)
|
||||
self.update_schemas = [PostProduction]
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
@@ -99,7 +96,7 @@ class ProductionApiHandler(BaseApiHandler):
|
||||
|
||||
# 获取过滤后的更新数据
|
||||
force_steps_update = self._steps_changed(node)
|
||||
update_data = self._filter_update_data(
|
||||
update_data = self.build_update_request_data(
|
||||
node_data,
|
||||
origin_data,
|
||||
PostProduction,
|
||||
|
||||
@@ -7,4 +7,4 @@ from schemas.project_extensions.production import ProductionDetail
|
||||
class ProductionJsonlHandler(BaseJsonlHandler):
|
||||
"""投产节点 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "production", ProductionSyncNode, ProductionDetail)
|
||||
super().__init__(datasource, ProductionSyncNode, schema=ProductionDetail)
|
||||
|
||||
@@ -20,16 +20,13 @@ class ProductionSyncStrategy(DefaultSyncStrategy[ProductionDetail]):
|
||||
update_direction=UpdateDirection.PUSH,
|
||||
)
|
||||
|
||||
def _needs_update(self, source_node, target_node, resolved_data=None, data_id_map=None) -> bool:
|
||||
if resolved_data is None:
|
||||
raise ValueError(f"[{self.node_type}] resolved_data is required for differential sync.")
|
||||
|
||||
target_data = target_node.get_data()
|
||||
def should_update_pair(self, source_node, target_node, *, source_data, target_data, data_id_map=None) -> bool:
|
||||
if target_data is None:
|
||||
return False
|
||||
|
||||
# 这里只按 PostProduction 可写字段判定是否需要 update,避免 response-only 字段差异触发空更新。
|
||||
update_fields = set(PostProduction.model_fields.keys())
|
||||
source_payload = {key: value for key, value in resolved_data.items() if key in update_fields}
|
||||
source_payload = {key: value for key, value in source_data.items() if key in update_fields}
|
||||
target_payload = {key: value for key, value in target_data.items() if key in update_fields}
|
||||
|
||||
normalized_source = normalized_data_for_compare(source_payload, data_id_map or {})
|
||||
|
||||
@@ -34,6 +34,7 @@ from schemas.project.project_base import (
|
||||
ProjectSettlementInvestment
|
||||
)
|
||||
from schemas.project.project import ProjectInfoResponse
|
||||
from schemas.project.project_aggregate import ProjectAggregateResponse
|
||||
|
||||
|
||||
_PROJECT_EXTENSION_FIELD_NAMES = {
|
||||
@@ -72,10 +73,7 @@ class ProjectApiHandler(BaseApiHandler):
|
||||
_all_info_cache: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
def __init__(self, **handler_config: Any):
|
||||
super().__init__(**handler_config)
|
||||
self._node_type = "project"
|
||||
self._node_class = ProjectSyncNode
|
||||
self._schema = ProjectResponseBase
|
||||
super().__init__(node_class=ProjectSyncNode, schema=ProjectResponseBase, **handler_config)
|
||||
|
||||
@classmethod
|
||||
async def get_all_info(cls, client, project_id: str) -> Dict[str, Any]:
|
||||
@@ -94,6 +92,23 @@ class ProjectApiHandler(BaseApiHandler):
|
||||
all_info = await api_get_project_all_info(client, project_id)
|
||||
cls._all_info_cache[project_id] = all_info
|
||||
return all_info
|
||||
|
||||
@classmethod
|
||||
async def get_aggregate(
|
||||
cls,
|
||||
client,
|
||||
project_id: str,
|
||||
*,
|
||||
domains: List[str],
|
||||
limit: int = 10000,
|
||||
) -> Dict[str, Any]:
|
||||
normalized_domains = tuple(dict.fromkeys(domains))
|
||||
return await api_get_project_aggregate(
|
||||
client,
|
||||
project_id,
|
||||
domains=list(normalized_domains),
|
||||
limit=limit,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def clear_cache(cls):
|
||||
@@ -107,6 +122,8 @@ class ProjectApiHandler(BaseApiHandler):
|
||||
project_id: 可选,指定项目ID列表则只加载这些项目(优先级高于构造函数的filter_project_id)
|
||||
"""
|
||||
try:
|
||||
self.ensure_api_client()
|
||||
self.ensure_collection()
|
||||
# 清空旧缓存
|
||||
self.clear_cache()
|
||||
|
||||
@@ -234,7 +251,7 @@ class ProjectApiHandler(BaseApiHandler):
|
||||
|
||||
# 遍历所有更新 schema,检查差异并更新
|
||||
for schema, api_func in update_configs:
|
||||
update_data = self._get_schema_diff(
|
||||
update_data = self.get_update_schema_diff(
|
||||
schema,
|
||||
data,
|
||||
original_data,
|
||||
@@ -244,9 +261,9 @@ class ProjectApiHandler(BaseApiHandler):
|
||||
if not update_data:
|
||||
continue
|
||||
|
||||
# 执行更新(差异已由 _get_schema_diff 打印)
|
||||
# 执行更新(差异已由 get_update_schema_diff 打印)
|
||||
try:
|
||||
full_update_data = self._filter_update_data(
|
||||
full_update_data = self.build_update_request_data(
|
||||
data,
|
||||
original_data,
|
||||
schema,
|
||||
@@ -405,6 +422,27 @@ async def api_get_project_all_info(client, project_id: str) -> Dict[str, Any]:
|
||||
return validated.model_dump()
|
||||
|
||||
|
||||
async def api_get_project_aggregate(
|
||||
client,
|
||||
project_id: str,
|
||||
*,
|
||||
domains: List[str],
|
||||
limit: int = 10000,
|
||||
) -> Dict[str, Any]:
|
||||
"""POST /project/aggregate - 获取项目聚合数据(校验为 ProjectAggregateResponse)"""
|
||||
response = await client.request(
|
||||
method="POST",
|
||||
endpoint="/project/aggregate",
|
||||
data={"project_id": project_id, "domains": domains, "limit": limit},
|
||||
)
|
||||
code = response.get("code")
|
||||
if code != 200:
|
||||
raise Exception(f"API error {code}: {response.get('message', 'Unknown error')}")
|
||||
|
||||
validated = ProjectAggregateResponse.model_validate(response.get("data", {}))
|
||||
return validated.model_dump()
|
||||
|
||||
|
||||
async def api_update_project_base_info(client, data: Dict[str, Any], push_id: str, biz_id: str, *, steps: List[Dict[str, Any]] | None = None) -> None:
|
||||
"""PUT /project - 更新项目基本信息"""
|
||||
ProjectBaseInfoUpdate.model_validate(data)
|
||||
|
||||
@@ -9,4 +9,4 @@ from schemas.project.project_base import ProjectResponseBase
|
||||
class ProjectJsonlHandler(BaseJsonlHandler):
|
||||
"""项目 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "project", ProjectSyncNode, ProjectResponseBase)
|
||||
super().__init__(datasource, ProjectSyncNode, schema=ProjectResponseBase)
|
||||
|
||||
@@ -21,6 +21,8 @@ class ProjectSyncSchema(ProjectResponseBase):
|
||||
completed_investment: float | None = Field(None, description="已完成投资")
|
||||
total_contract_quantity: float | None = Field(None, description="合同总量")
|
||||
completed_settlement_quantity: float | None = Field(None, description="已结算量")
|
||||
ic_plan_first_production_date: str | None = Field(None, description="集团计划首批投产日期")
|
||||
ic_plan_full_production_date: str | None = Field(None, description="集团计划全容量投产日期")
|
||||
|
||||
|
||||
class ProjectSyncNode(SyncNode[ProjectSyncSchema]):
|
||||
|
||||
@@ -1,8 +1,16 @@
|
||||
import copy
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ...sync_system.strategy import DefaultSyncStrategy
|
||||
from ...sync_system.config import StrategyConfig, OrphanAction, UpdateDirection
|
||||
from schemas.project.project_base import ProjectResponseBase
|
||||
|
||||
|
||||
class ProjectDomainOption(BaseModel):
|
||||
pull_group_plan_production_date: bool = False
|
||||
|
||||
|
||||
class ProjectSyncStrategy(DefaultSyncStrategy[ProjectResponseBase]):
|
||||
"""
|
||||
Project 同步策略
|
||||
@@ -17,6 +25,12 @@ class ProjectSyncStrategy(DefaultSyncStrategy[ProjectResponseBase]):
|
||||
|
||||
# 类变量:schema 固定为 ProjectResponseBase
|
||||
schema = ProjectResponseBase
|
||||
domain_option_model = ProjectDomainOption
|
||||
|
||||
_PULL_ONLY_FIELDS = [
|
||||
"ic_plan_first_production_date",
|
||||
"ic_plan_full_production_date",
|
||||
]
|
||||
|
||||
# 类变量:默认配置
|
||||
default_config = StrategyConfig(
|
||||
@@ -26,4 +40,89 @@ class ProjectSyncStrategy(DefaultSyncStrategy[ProjectResponseBase]):
|
||||
local_orphan_action=OrphanAction.CREATE_REMOTE,
|
||||
remote_orphan_action=OrphanAction.NONE,
|
||||
update_direction=UpdateDirection.PUSH,
|
||||
)
|
||||
)
|
||||
|
||||
@property
|
||||
def domain_option(self) -> ProjectDomainOption:
|
||||
return ProjectDomainOption.model_validate(self.config.domain_option)
|
||||
|
||||
def get_node_update_payload(self, node):
|
||||
payload = dict(node.get_data() or {})
|
||||
context = getattr(node, "context", None)
|
||||
if not isinstance(context, dict):
|
||||
return payload
|
||||
|
||||
all_info = context.get("all_info")
|
||||
if not isinstance(all_info, dict):
|
||||
return payload
|
||||
|
||||
for section_name, section_data in all_info.items():
|
||||
if not section_name.endswith("_extension") or not isinstance(section_data, dict):
|
||||
continue
|
||||
for field_name in self._PULL_ONLY_FIELDS:
|
||||
if payload.get(field_name) not in (None, ""):
|
||||
continue
|
||||
if field_name in section_data:
|
||||
payload[field_name] = section_data.get(field_name)
|
||||
|
||||
return payload
|
||||
|
||||
@staticmethod
|
||||
def _snapshot_node(node):
|
||||
# project 是一个很特殊的双向更新场景:
|
||||
# 1. 先做一次 PULL,把远端 ic_plan_* 回填到本地;
|
||||
# 2. 再做一次常规 PUSH,把本地其余字段推到远端。
|
||||
#
|
||||
# 但 prepare_update_for_direction -> e30_update_prepare 会校验
|
||||
# source_node 当前仍然处于可进入更新准备的稳定态(S01)。
|
||||
# 如果直接复用同一个真实节点作为两次 update 的 source,第一次
|
||||
# 准备完成后真实节点的 action/status 已经变化,第二次就不再满足
|
||||
# “source 仍在 S01”的前置条件,导致第二次更新被状态机拒绝。
|
||||
#
|
||||
# 这里复制一个只用于“提供源数据/源状态视图”的快照,让两次 update
|
||||
# 都从各自的原始稳定态出发做判断;真正被推进到 S07 / 写入 payload
|
||||
# 的仍然是目标侧真实节点,而不是这个 snapshot。
|
||||
snapshot = copy.deepcopy(node)
|
||||
if hasattr(snapshot, "sync_log"):
|
||||
snapshot.sync_log = None
|
||||
return snapshot
|
||||
|
||||
async def update_pair(self, local_node, remote_node, data_id_map=None):
|
||||
updated_by_id = {}
|
||||
# 注意:这两个 snapshot 只会在各自方向上充当 source_node。
|
||||
# - PULL: source=remote_snapshot, target=local_node
|
||||
# - PUSH: source=local_snapshot, target=remote_node
|
||||
#
|
||||
# 因此真实 project 节点的状态变更仍只发生在 target 节点上,
|
||||
# 不会因为传入 snapshot 而把本体节点替换掉,也不会把本体节点的
|
||||
# action/status/error/sync_log 写到 snapshot 上后丢失掉。
|
||||
local_snapshot = self._snapshot_node(local_node)
|
||||
remote_snapshot = self._snapshot_node(remote_node)
|
||||
|
||||
if self.domain_option.pull_group_plan_production_date:
|
||||
pull_update = await self.prepare_update_for_direction(
|
||||
local_node,
|
||||
remote_snapshot,
|
||||
UpdateDirection.PULL,
|
||||
data_id_map,
|
||||
include_fields=self._PULL_ONLY_FIELDS,
|
||||
)
|
||||
if pull_update is not None:
|
||||
updated_by_id[pull_update.node_id] = pull_update
|
||||
|
||||
generic_exclude_fields = self._PULL_ONLY_FIELDS
|
||||
else:
|
||||
generic_exclude_fields = None
|
||||
|
||||
if self.config.update_direction != UpdateDirection.NONE:
|
||||
generic_update = await self.prepare_update_for_direction(
|
||||
local_snapshot,
|
||||
remote_node,
|
||||
self.config.update_direction,
|
||||
data_id_map,
|
||||
exclude_fields=generic_exclude_fields,
|
||||
)
|
||||
if generic_update is not None:
|
||||
updated_by_id[generic_update.node_id] = generic_update
|
||||
|
||||
return list(updated_by_id.values())
|
||||
@@ -40,8 +40,6 @@ from schemas.project.project_base import (
|
||||
_PRODUCTION_PROJECT_FIELDS = [
|
||||
"plan_production_date",
|
||||
"plan_full_production_date",
|
||||
"ic_plan_first_production_date",
|
||||
"ic_plan_full_production_date",
|
||||
"actual_production_date",
|
||||
"actual_full_production_date",
|
||||
]
|
||||
@@ -51,10 +49,7 @@ class ProjectDetailApiHandler(BaseApiHandler):
|
||||
"""项目容量信息 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "project_detail_data"
|
||||
self._node_class = ProjectDetailSyncNode
|
||||
self._schema = ProjectDetailProject
|
||||
super().__init__(node_class=ProjectDetailSyncNode, schema=ProjectDetailProject)
|
||||
self._pending_push_context: Dict[str, Dict[str, str]] = {}
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
@@ -185,6 +180,9 @@ class ProjectDetailApiHandler(BaseApiHandler):
|
||||
except ValidationError as e:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=f"{ERROR_VALIDATION_FAILED}: {e}"))
|
||||
continue
|
||||
except Exception as e:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
|
||||
continue
|
||||
|
||||
push_id = self._generate_push_id()
|
||||
biz_id = project_id
|
||||
@@ -207,7 +205,16 @@ class ProjectDetailApiHandler(BaseApiHandler):
|
||||
self._mark_approval_snapshot_synced(node)
|
||||
except Exception as e:
|
||||
self._pending_push_context.pop(push_id, None)
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
|
||||
results.append(
|
||||
TaskResult.failed(
|
||||
node_id=node.node_id,
|
||||
error=str(e),
|
||||
push_id=push_id,
|
||||
biz_id=biz_id,
|
||||
key=str(key),
|
||||
project_id=str(project_id),
|
||||
)
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
@@ -299,46 +306,13 @@ class ProjectDetailApiHandler(BaseApiHandler):
|
||||
|
||||
def _create_node(self, data: Dict[str, Any]) -> SyncNode:
|
||||
project_id_raw = data.get("project_id")
|
||||
key_raw = self._normalize_key(data.get("key"))
|
||||
project_id = str(project_id_raw) if project_id_raw else ""
|
||||
key = str(key_raw) if key_raw else ""
|
||||
loaded_data_id = str(data.get("id") or data.get("_id") or "")
|
||||
|
||||
existing_by_pair = self._find_existing_by_project_key(project_id=project_id, key=key)
|
||||
if existing_by_pair is not None:
|
||||
if loaded_data_id and existing_by_pair.data_id != loaded_data_id:
|
||||
with existing_by_pair.allow_core_state_write("project_detail_pair_rebind"):
|
||||
existing_by_pair.data_id = loaded_data_id
|
||||
existing_by_pair.set_data(data)
|
||||
existing_by_pair.set_origin_data(data)
|
||||
if project_id:
|
||||
existing_by_pair.context["project_id"] = project_id
|
||||
return existing_by_pair
|
||||
|
||||
node = self._create_basic_node(data, depend_ids=[])
|
||||
if project_id:
|
||||
node.context["project_id"] = project_id
|
||||
return node
|
||||
|
||||
def _find_existing_by_project_key(self, *, project_id: str, key: str) -> Optional[SyncNode]:
|
||||
if not project_id or not key or self._collection is None:
|
||||
return None
|
||||
|
||||
candidates = self._collection.filter(
|
||||
node_type=self.node_type,
|
||||
node_filter=lambda n: (
|
||||
((n.get_data() or {}).get("project_id") == project_id)
|
||||
and (self._normalize_key((n.get_data() or {}).get("key")) == key)
|
||||
),
|
||||
)
|
||||
if not candidates:
|
||||
return None
|
||||
if len(candidates) > 1:
|
||||
raise RuntimeError(
|
||||
f"Duplicate project_detail_data detected: project_id={project_id}, key={key}, count={len(candidates)}"
|
||||
)
|
||||
return candidates[0]
|
||||
|
||||
async def _get_project_all_info(self, project_id: str, *, force_refresh: bool = False) -> Dict[str, Any]:
|
||||
from ..project.api_handler import ProjectApiHandler
|
||||
|
||||
|
||||
@@ -7,4 +7,4 @@ from schemas.project_extensions.project_detail import ProjectDetailProject
|
||||
class ProjectDetailJsonlHandler(BaseJsonlHandler):
|
||||
"""项目容量信息 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "project_detail_data", ProjectDetailSyncNode, ProjectDetailProject)
|
||||
super().__init__(datasource, ProjectDetailSyncNode, schema=ProjectDetailProject)
|
||||
|
||||
@@ -1,6 +1,25 @@
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ...common.sync_node import SyncNode
|
||||
from ...sync_system.strategy import DefaultSyncStrategy
|
||||
from ...sync_system.config import StrategyConfig, OrphanAction, UpdateDirection
|
||||
from schemas.project_extensions.project_detail import ProjectDetailProject
|
||||
from ...sync_system.strategy_ops.bind_ops import (
|
||||
apply_auto_bind_updates,
|
||||
collect_auto_bind_ready_nodes,
|
||||
collect_bind_entry_nodes,
|
||||
plan_auto_bind_updates,
|
||||
phase_core_state,
|
||||
phase_dependency_check,
|
||||
refresh_bind_data_id,
|
||||
)
|
||||
from ...sync_system.strategy_ops import BoundNodePair
|
||||
from schemas.project_extensions.project_detail import ProjectDetailKey, ProjectDetailProject
|
||||
|
||||
|
||||
class ProjectDetailDomainOption(BaseModel):
|
||||
pull_group_production_plans: bool = Field(True, description="固定将集团保供/爬坡/挑战产能计划按 pull 处理")
|
||||
|
||||
|
||||
class ProjectDetailSyncStrategy(DefaultSyncStrategy[ProjectDetailProject]):
|
||||
@@ -13,6 +32,18 @@ class ProjectDetailSyncStrategy(DefaultSyncStrategy[ProjectDetailProject]):
|
||||
"""
|
||||
|
||||
schema = ProjectDetailProject
|
||||
domain_option_model = ProjectDetailDomainOption
|
||||
_GROUP_PRODUCTION_PLAN_KEYS = {
|
||||
"ensure_production",
|
||||
"climb_production",
|
||||
"challenge_production",
|
||||
}
|
||||
_PRODUCTION_PROJECT_FIELDS = (
|
||||
"plan_production_date",
|
||||
"plan_full_production_date",
|
||||
"actual_production_date",
|
||||
"actual_full_production_date",
|
||||
)
|
||||
|
||||
default_config = StrategyConfig(
|
||||
auto_bind=True,
|
||||
@@ -21,10 +52,232 @@ class ProjectDetailSyncStrategy(DefaultSyncStrategy[ProjectDetailProject]):
|
||||
local_orphan_action=OrphanAction.NONE,
|
||||
remote_orphan_action=OrphanAction.NONE,
|
||||
update_direction=UpdateDirection.PUSH,
|
||||
field_direction_key="key",
|
||||
field_direction_overrides={
|
||||
"ensure_production": UpdateDirection.PULL,
|
||||
"climb_production": UpdateDirection.PULL,
|
||||
"challenge_production": UpdateDirection.PULL,
|
||||
},
|
||||
)
|
||||
|
||||
async def bind(self) -> None:
|
||||
runtime = self.ensure_runtime()
|
||||
local_nodes, remote_nodes = collect_bind_entry_nodes(
|
||||
node_type=self.node_type,
|
||||
local_collection=self.local_collection,
|
||||
remote_collection=self.remote_collection,
|
||||
)
|
||||
await phase_core_state(
|
||||
node_type=self.node_type,
|
||||
runtime=runtime,
|
||||
binding_manager=self.binding_manager,
|
||||
local_nodes=local_nodes,
|
||||
remote_nodes=remote_nodes,
|
||||
)
|
||||
await phase_dependency_check(
|
||||
node_type=self.node_type,
|
||||
runtime=runtime,
|
||||
depend_fields=dict(self.config.depend_fields),
|
||||
local_nodes=local_nodes,
|
||||
remote_nodes=remote_nodes,
|
||||
local_collection=self.local_collection,
|
||||
remote_collection=self.remote_collection,
|
||||
)
|
||||
local_nodes, remote_nodes = collect_auto_bind_ready_nodes(
|
||||
node_type=self.node_type,
|
||||
local_collection=self.local_collection,
|
||||
remote_collection=self.remote_collection,
|
||||
)
|
||||
|
||||
local_create_enabled = self.config.local_orphan_action == OrphanAction.CREATE_REMOTE
|
||||
remote_create_enabled = self.config.remote_orphan_action == OrphanAction.CREATE_LOCAL
|
||||
local_orphan_create_enabled_by_node = {
|
||||
node.node_id: local_create_enabled for node in local_nodes
|
||||
}
|
||||
remote_orphan_create_enabled_by_node = {
|
||||
node.node_id: remote_create_enabled for node in remote_nodes
|
||||
}
|
||||
|
||||
if self.domain_option.pull_group_production_plans:
|
||||
for node in local_nodes:
|
||||
if self._extract_node_key(node) in self._GROUP_PRODUCTION_PLAN_KEYS:
|
||||
local_orphan_create_enabled_by_node[node.node_id] = False
|
||||
for node in remote_nodes:
|
||||
if self._extract_node_key(node) in self._GROUP_PRODUCTION_PLAN_KEYS:
|
||||
remote_orphan_create_enabled_by_node[node.node_id] = True
|
||||
|
||||
pending_updates = await plan_auto_bind_updates(
|
||||
node_type=self.node_type,
|
||||
config=self.config,
|
||||
local_collection=self.local_collection,
|
||||
remote_collection=self.remote_collection,
|
||||
binding_manager=self.binding_manager,
|
||||
local_nodes=local_nodes,
|
||||
remote_nodes=remote_nodes,
|
||||
id_field_hints=dict(self.config.depend_fields),
|
||||
local_orphan_create_enabled_by_node=local_orphan_create_enabled_by_node,
|
||||
remote_orphan_create_enabled_by_node=remote_orphan_create_enabled_by_node,
|
||||
)
|
||||
await apply_auto_bind_updates(
|
||||
node_type=self.node_type,
|
||||
runtime=runtime,
|
||||
binding_manager=self.binding_manager,
|
||||
pending_auto_bind_updates=pending_updates,
|
||||
)
|
||||
await refresh_bind_data_id(
|
||||
node_type=self.node_type,
|
||||
local_collection=self.local_collection,
|
||||
remote_collection=self.remote_collection,
|
||||
binding_manager=self.binding_manager,
|
||||
)
|
||||
|
||||
async def process_update_pairs(
|
||||
self,
|
||||
pairs: list[BoundNodePair],
|
||||
data_id_map: dict[str, str] | None = None,
|
||||
) -> dict[str, SyncNode]:
|
||||
pull_keys = self._GROUP_PRODUCTION_PLAN_KEYS if self.domain_option.pull_group_production_plans else set()
|
||||
pull_pairs = []
|
||||
generic_pairs = []
|
||||
for pair in pairs:
|
||||
pair_key = self._extract_pair_key(pair.local_node, pair.remote_node)
|
||||
if pair_key in pull_keys:
|
||||
pull_pairs.append(pair)
|
||||
else:
|
||||
generic_pairs.append(pair)
|
||||
|
||||
updated_by_id: dict[str, SyncNode] = {}
|
||||
for pair in pull_pairs:
|
||||
updated_node = await self.prepare_update_for_direction(
|
||||
pair.local_node,
|
||||
pair.remote_node,
|
||||
UpdateDirection.PULL,
|
||||
data_id_map,
|
||||
)
|
||||
if updated_node is not None:
|
||||
updated_by_id[updated_node.node_id] = updated_node
|
||||
|
||||
for pair in generic_pairs:
|
||||
for updated_node in await self.update_pair(pair.local_node, pair.remote_node, data_id_map):
|
||||
updated_by_id[updated_node.node_id] = updated_node
|
||||
|
||||
return updated_by_id
|
||||
|
||||
@property
|
||||
def domain_option(self) -> ProjectDetailDomainOption:
|
||||
return ProjectDetailDomainOption.model_validate(self.config.domain_option)
|
||||
|
||||
async def create_for_direction(self, *, source_is_local: bool) -> list[SyncNode]:
|
||||
created_nodes = await super().create_for_direction(source_is_local=source_is_local)
|
||||
await self._enrich_created_production_nodes(created_nodes, source_is_local=source_is_local)
|
||||
return created_nodes
|
||||
|
||||
def normalize_compare_payload(self, data: dict | None, data_id_map=None) -> dict:
|
||||
normalized = super().normalize_compare_payload(data, data_id_map)
|
||||
key = normalized.get("key")
|
||||
if isinstance(key, ProjectDetailKey):
|
||||
key = key.value
|
||||
normalized["key"] = key
|
||||
if isinstance(key, str):
|
||||
normalized["value"] = self._normalize_detail_value_for_compare(key, normalized.get("value"))
|
||||
return normalized
|
||||
|
||||
@staticmethod
|
||||
def _normalize_detail_value_for_compare(key: str, value: object) -> list[dict[str, object]]:
|
||||
if not isinstance(value, list):
|
||||
return []
|
||||
|
||||
items: list[dict[str, object]] = []
|
||||
for item in value:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
|
||||
normalized_item: dict[str, object] = {}
|
||||
if "capacity" in item:
|
||||
normalized_item["capacity"] = item.get("capacity")
|
||||
|
||||
if key in {"plan_construction", "actual_construction", "production"}:
|
||||
date_value = item.get("date")
|
||||
if date_value in (None, ""):
|
||||
continue
|
||||
normalized_item["date"] = date_value
|
||||
elif key in {"ensure_production", "climb_production", "challenge_production"}:
|
||||
year_value = item.get("year")
|
||||
if year_value in (None, ""):
|
||||
date_value = item.get("date")
|
||||
if isinstance(date_value, str) and len(date_value) >= 4 and date_value[:4].isdigit():
|
||||
year_value = int(date_value[:4])
|
||||
if year_value in (None, ""):
|
||||
continue
|
||||
normalized_item["year"] = year_value
|
||||
else:
|
||||
if item.get("year") not in (None, ""):
|
||||
normalized_item["year"] = item.get("year")
|
||||
if item.get("date") not in (None, ""):
|
||||
normalized_item["date"] = item.get("date")
|
||||
|
||||
items.append(normalized_item)
|
||||
|
||||
return items
|
||||
|
||||
@staticmethod
|
||||
def _extract_node_key(node: SyncNode) -> str | None:
|
||||
data = node.get_data() or {}
|
||||
key = data.get("key")
|
||||
if isinstance(key, ProjectDetailKey):
|
||||
return key.value
|
||||
if isinstance(key, str):
|
||||
return key
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _extract_pair_key(local_node, remote_node) -> str | None:
|
||||
local_data = local_node.get_data() or {}
|
||||
remote_data = remote_node.get_data() or {}
|
||||
|
||||
key = local_data.get("key", remote_data.get("key"))
|
||||
if isinstance(key, ProjectDetailKey):
|
||||
return key.value
|
||||
if isinstance(key, str):
|
||||
return key
|
||||
return None
|
||||
|
||||
async def _enrich_created_production_nodes(self, created_nodes: list[SyncNode], *, source_is_local: bool) -> None:
|
||||
if not created_nodes:
|
||||
return
|
||||
|
||||
source_collection = self.local_collection if source_is_local else self.remote_collection
|
||||
|
||||
for created_node in created_nodes:
|
||||
created_data = created_node.get_data() or {}
|
||||
if self._extract_node_key(created_node) != "production":
|
||||
continue
|
||||
|
||||
if all(created_data.get(field) not in (None, "") for field in self._PRODUCTION_PROJECT_FIELDS):
|
||||
continue
|
||||
|
||||
if source_is_local:
|
||||
source_node_id = await self.binding_manager.get_local_id(self.node_type, created_node.node_id)
|
||||
else:
|
||||
source_node_id = await self.binding_manager.get_remote_id(self.node_type, created_node.node_id)
|
||||
if not source_node_id:
|
||||
continue
|
||||
|
||||
source_node = source_collection.get(source_node_id)
|
||||
if source_node is None:
|
||||
continue
|
||||
|
||||
source_data = source_node.get_data() or {}
|
||||
project_id = source_data.get("project_id")
|
||||
if not project_id:
|
||||
continue
|
||||
|
||||
project_node = source_collection.get_by_data_id("project", str(project_id))
|
||||
if project_node is None:
|
||||
continue
|
||||
|
||||
project_data = project_node.get_data() or {}
|
||||
updated_data: dict[str, Any] = dict(created_data)
|
||||
changed = False
|
||||
for field in self._PRODUCTION_PROJECT_FIELDS:
|
||||
if updated_data.get(field) in (None, "") and project_data.get(field) not in (None, ""):
|
||||
updated_data[field] = project_data.get(field)
|
||||
changed = True
|
||||
|
||||
if changed:
|
||||
created_node.set_data(updated_data)
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ Supplier API Handler (只读)
|
||||
- 不支持创建/更新/删除
|
||||
"""
|
||||
|
||||
from typing import List, Dict, Any
|
||||
from typing import List, Dict
|
||||
from ...datasource.api.handler import BaseApiHandler
|
||||
from ...datasource.task_result import TaskResult
|
||||
from ...common.sync_node import SyncNode
|
||||
@@ -18,10 +18,7 @@ class SupplierApiHandler(BaseApiHandler):
|
||||
"""供应商 API Handler (只读)"""
|
||||
|
||||
def __init__(self, load_mode: str = "api", max_total: int | None = None):
|
||||
super().__init__()
|
||||
self._node_type = "supplier"
|
||||
self._node_class = SupplierSyncNode
|
||||
self._schema = SupplierDetailResponse
|
||||
super().__init__(node_class=SupplierSyncNode, schema=SupplierDetailResponse)
|
||||
self._load_mode = load_mode
|
||||
self._max_total = max_total
|
||||
|
||||
@@ -99,10 +96,6 @@ class SupplierApiHandler(BaseApiHandler):
|
||||
"""轮询异步任务状态(占位)"""
|
||||
return await super().poll_tasks(task_ids)
|
||||
|
||||
def _create_node(self, data: Dict[str, Any]) -> SyncNode:
|
||||
return self._create_basic_node(data, depend_ids=[])
|
||||
|
||||
|
||||
# ========== 静态 API 函数 ==========
|
||||
|
||||
async def api_get_supplier_list(client, page: int = 1, page_size: int = 100) -> tuple[List[SupplierDetailResponse], int]:
|
||||
|
||||
@@ -9,4 +9,4 @@ from schemas.supplier.supplier import SupplierDetailResponse
|
||||
class SupplierJsonlHandler(BaseJsonlHandler):
|
||||
"""供应商 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "supplier", SupplierSyncNode, SupplierDetailResponse)
|
||||
super().__init__(datasource, SupplierSyncNode, schema=SupplierDetailResponse)
|
||||
|
||||
@@ -24,19 +24,16 @@ class SupplierSyncStrategy(DefaultSyncStrategy[SupplierDetailResponse]):
|
||||
update_direction=UpdateDirection.PULL,
|
||||
)
|
||||
|
||||
def _needs_update(
|
||||
def should_update_pair(
|
||||
self,
|
||||
source_node,
|
||||
target_node,
|
||||
resolved_data=None,
|
||||
*,
|
||||
source_data,
|
||||
target_data,
|
||||
data_id_map=None,
|
||||
) -> bool:
|
||||
"""Supplier 更新忽略 id 字段差异。"""
|
||||
if resolved_data is None:
|
||||
raise ValueError(f"[{self.node_type}] resolved_data is required for differential sync.")
|
||||
|
||||
source_data = resolved_data
|
||||
target_data = target_node.get_data()
|
||||
if source_data is None or target_data is None:
|
||||
return False
|
||||
|
||||
|
||||
@@ -23,10 +23,7 @@ class UnitsApiHandler(BaseApiHandler):
|
||||
"""机组 API Handler"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "units"
|
||||
self._node_class = UnitsSyncNode
|
||||
self._schema = GeneratorUnitProject
|
||||
super().__init__(node_class=UnitsSyncNode, schema=GeneratorUnitProject)
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载机组数据(从 project all_info 中提取)"""
|
||||
@@ -94,7 +91,7 @@ class UnitsApiHandler(BaseApiHandler):
|
||||
|
||||
origin_data = node.get_origin_data() or {}
|
||||
force_steps_update = self._steps_changed(node)
|
||||
update_data = self._filter_update_data(
|
||||
update_data = self.build_update_request_data(
|
||||
node_data,
|
||||
origin_data,
|
||||
PostGeneratorUnits,
|
||||
|
||||
@@ -11,7 +11,7 @@ from schemas.project_extensions.generator_unit import GeneratorUnitProject
|
||||
class UnitsJsonlHandler(BaseJsonlHandler):
|
||||
"""机组 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "units", UnitsSyncNode, GeneratorUnitProject)
|
||||
super().__init__(datasource, UnitsSyncNode, schema=GeneratorUnitProject)
|
||||
|
||||
async def load(self) -> List['SyncNode']:
|
||||
"""加载机组数据(支持 generator_unit_N.jsonl 与 units_N.jsonl)"""
|
||||
|
||||
@@ -10,7 +10,7 @@ GET:
|
||||
- /user/list - 获取用户列表
|
||||
"""
|
||||
|
||||
from typing import List, Dict, Any
|
||||
from typing import List, Dict
|
||||
from ...datasource.api.handler import BaseApiHandler
|
||||
from ...datasource.task_result import TaskResult
|
||||
from ...common.sync_node import SyncNode
|
||||
@@ -22,10 +22,7 @@ class UserApiHandler(BaseApiHandler):
|
||||
"""用户 API Handler (只读)"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._node_type = "user"
|
||||
self._node_class = UserSyncNode
|
||||
self._schema = UserListDetailItem
|
||||
super().__init__(node_class=UserSyncNode, schema=UserListDetailItem)
|
||||
|
||||
async def load(self) -> List[SyncNode]:
|
||||
"""加载用户列表(支持分页)"""
|
||||
@@ -87,10 +84,6 @@ class UserApiHandler(BaseApiHandler):
|
||||
"""轮询异步任务状态(占位)"""
|
||||
return await super().poll_tasks(task_ids)
|
||||
|
||||
def _create_node(self, data: Dict[str, Any]) -> SyncNode:
|
||||
return self._create_basic_node(data, depend_ids=[])
|
||||
|
||||
|
||||
# ========== 静态 API 函数 ==========
|
||||
|
||||
async def api_get_user_list(client, page: int = 1, page_size: int = 100) -> tuple[List[UserListDetailItem], int]:
|
||||
|
||||
@@ -7,4 +7,4 @@ from schemas.user.user import UserListDetailItem
|
||||
class UserJsonlHandler(BaseJsonlHandler):
|
||||
"""用户 JSONL Handler"""
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "user", UserSyncNode, UserListDetailItem)
|
||||
super().__init__(datasource, UserSyncNode, schema=UserListDetailItem)
|
||||
|
||||
@@ -276,8 +276,14 @@ def e01_bootstrap(
|
||||
*,
|
||||
node: "SyncNode",
|
||||
is_create_zombie: bool,
|
||||
bootstrap_source_present: bool = True,
|
||||
has_binding_record: bool = False,
|
||||
) -> Decision:
|
||||
context = {"is_create_zombie": is_create_zombie}
|
||||
context = {
|
||||
"is_create_zombie": is_create_zombie,
|
||||
"bootstrap_source_present": bootstrap_source_present,
|
||||
"has_binding_record": has_binding_record,
|
||||
}
|
||||
decision = _dispatch_required(
|
||||
runtime,
|
||||
node=node,
|
||||
@@ -289,6 +295,15 @@ def e01_bootstrap(
|
||||
_apply_decision(runtime, node=node, decision=decision, fields={"binding_status", "action", "status"})
|
||||
node.error = None
|
||||
return decision
|
||||
if decision.to_state == "S17":
|
||||
_apply_decision(runtime, node=node, decision=decision, fields={"binding_status", "action", "status"})
|
||||
reason_text = decision_note(decision) or "<no-reason>"
|
||||
node.context["bootstrap_blocked"] = True
|
||||
node.context["bootstrap_block_reason"] = "source_missing_after_persistence_refresh"
|
||||
node.context["bootstrap_block_errors"] = [reason_text]
|
||||
node.context["bootstrap_source_present"] = False
|
||||
node.error = f"BOOTSTRAP_SOURCE_MISSING: {reason_text}"
|
||||
return decision
|
||||
if decision.to_state == "S15":
|
||||
reason_text = decision_note(decision) or "<no-reason>"
|
||||
node.append_log(
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
APP_LOGGER_NAME = "sync_state_machine"
|
||||
_SCOPE_DISPLAY_NAMES: dict[str, str] = {}
|
||||
|
||||
|
||||
def set_scope_display_name(scope: str, display_name: str) -> None:
|
||||
normalized_scope = str(scope).strip()
|
||||
normalized_name = str(display_name).strip()
|
||||
if not normalized_scope:
|
||||
return
|
||||
if not normalized_name:
|
||||
_SCOPE_DISPLAY_NAMES.pop(normalized_scope, None)
|
||||
return
|
||||
_SCOPE_DISPLAY_NAMES[normalized_scope] = normalized_name
|
||||
|
||||
|
||||
def get_scope_display_name(scope: Optional[str]) -> str:
|
||||
normalized_scope = str(scope or "").strip()
|
||||
if not normalized_scope:
|
||||
return ""
|
||||
return _SCOPE_DISPLAY_NAMES.get(normalized_scope, normalized_scope)
|
||||
|
||||
|
||||
def clear_scope_display_names() -> None:
|
||||
_SCOPE_DISPLAY_NAMES.clear()
|
||||
|
||||
|
||||
class ScopeLoggerAdapter(logging.LoggerAdapter):
|
||||
def process(self, msg, kwargs):
|
||||
scope = get_scope_display_name(self.extra.get("scope"))
|
||||
if scope and not str(msg).startswith("["):
|
||||
msg = f"[{scope}] {msg}"
|
||||
return msg, kwargs
|
||||
|
||||
|
||||
class _NodeLogFilter(logging.Filter):
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
message = record.getMessage()
|
||||
if "node=" in message:
|
||||
return False
|
||||
if "节点 " in message and "状态" in message:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def get_app_logger() -> logging.Logger:
|
||||
return logging.getLogger(APP_LOGGER_NAME)
|
||||
|
||||
|
||||
def get_logger(name: Optional[str] = None) -> logging.Logger:
|
||||
if not name or name == "__main__":
|
||||
return get_app_logger()
|
||||
if name == APP_LOGGER_NAME or name.startswith(f"{APP_LOGGER_NAME}."):
|
||||
return logging.getLogger(name)
|
||||
return logging.getLogger(f"{APP_LOGGER_NAME}.{name}")
|
||||
|
||||
|
||||
def get_scope_logger(scope: str, name: Optional[str] = None) -> logging.LoggerAdapter:
|
||||
return ScopeLoggerAdapter(get_logger(name), {"scope": scope})
|
||||
|
||||
|
||||
def clear_app_logger_handlers(logger: Optional[logging.Logger] = None) -> None:
|
||||
target = logger or get_app_logger()
|
||||
handlers = list(target.handlers)
|
||||
for handler in handlers:
|
||||
target.removeHandler(handler)
|
||||
try:
|
||||
handler.close()
|
||||
except Exception:
|
||||
pass
|
||||
target.propagate = True
|
||||
|
||||
|
||||
def init_app_logger(
|
||||
*,
|
||||
log_file_path: Optional[str] = None,
|
||||
level: int = logging.INFO,
|
||||
mirror_console: bool = True,
|
||||
replace_handlers: bool = True,
|
||||
suppress_node_logs: bool = True,
|
||||
) -> logging.Logger:
|
||||
logger = get_app_logger()
|
||||
logger.setLevel(level)
|
||||
logger.propagate = False
|
||||
|
||||
if logger.handlers and not replace_handlers:
|
||||
return logger
|
||||
|
||||
if replace_handlers:
|
||||
clear_app_logger_handlers(logger)
|
||||
|
||||
formatter = logging.Formatter("%(message)s")
|
||||
node_filter = _NodeLogFilter() if suppress_node_logs else None
|
||||
|
||||
if mirror_console:
|
||||
console_handler = logging.StreamHandler()
|
||||
console_handler.setLevel(level)
|
||||
console_handler.setFormatter(formatter)
|
||||
if node_filter is not None:
|
||||
console_handler.addFilter(node_filter)
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
if log_file_path:
|
||||
path = Path(log_file_path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
file_handler = logging.FileHandler(path, mode="w", encoding="utf-8")
|
||||
file_handler.setLevel(level)
|
||||
file_handler.setFormatter(formatter)
|
||||
if node_filter is not None:
|
||||
file_handler.addFilter(node_filter)
|
||||
logger.addHandler(file_handler)
|
||||
|
||||
return logger
|
||||
|
||||
|
||||
def configure_app_logging(config, *, replace_handlers: bool = False, mirror_console: bool = True) -> logging.Logger:
|
||||
if not getattr(config, "initialize", True):
|
||||
return get_app_logger()
|
||||
|
||||
if not getattr(config, "file_path", None):
|
||||
from .config.logging_config import resolve_log_file_path
|
||||
|
||||
config.file_path = resolve_log_file_path(config)
|
||||
|
||||
return init_app_logger(
|
||||
log_file_path=config.file_path,
|
||||
level=config.level,
|
||||
mirror_console=mirror_console,
|
||||
replace_handlers=replace_handlers,
|
||||
suppress_node_logs=config.suppress_node_logs,
|
||||
)
|
||||
|
||||
|
||||
def ensure_app_logging(config, *, mirror_console: bool = True) -> logging.Logger:
|
||||
logger = get_app_logger()
|
||||
if not getattr(config, "initialize", True):
|
||||
return logger
|
||||
if logger.handlers:
|
||||
return logger
|
||||
return configure_app_logging(config, replace_handlers=False, mirror_console=mirror_console)
|
||||
@@ -17,11 +17,12 @@ import inspect
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
from typing import Any, Dict, List, Literal, Optional
|
||||
|
||||
from ..common.binding import BindingManager
|
||||
from ..common.collection import DataCollection
|
||||
from ..common.persistence import PersistenceBackend, create_persistence_backend
|
||||
from ..common.persistence_service import PersistenceService
|
||||
from ..common.registry import DomainRegistry
|
||||
from ..datasource import ensure_builtin_datasource_types_registered
|
||||
from ..datasource.type_registry import get_datasource_factory
|
||||
@@ -36,21 +37,23 @@ from ..config import (
|
||||
build_config,
|
||||
StrategyRuntimeConfig,
|
||||
build_config_from_file,
|
||||
build_multi_project_config_from_file,
|
||||
apply_config_overrides,
|
||||
OrphanAction,
|
||||
UpdateDirection,
|
||||
resolve_log_file_path,
|
||||
config_snapshot,
|
||||
is_multi_project_payload,
|
||||
load_overrides_from_file,
|
||||
)
|
||||
from ..config.strategy_config import resolve_domain_option_config
|
||||
from ..sync_system.strategy import BaseSyncStrategy
|
||||
from .run_logger import init_pipeline_logger
|
||||
from ..logging import configure_app_logging, ensure_app_logging, get_logger
|
||||
from .full_sync_pipeline import FullSyncPipeline
|
||||
from .project_scope_runtime import ProjectScopeRuntime
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
PipelineType = Literal["auto", "full", "multi-project"]
|
||||
|
||||
|
||||
def _project_root() -> Path:
|
||||
@@ -63,29 +66,6 @@ def _prepare_remote_jsonl_dir(remote_dir: Path, *, project_root: Optional[Path]
|
||||
return prepare_remote_jsonl_dir(remote_dir, project_root=(project_root or _project_root()))
|
||||
|
||||
|
||||
def _ensure_pipeline_logger(
|
||||
*,
|
||||
initialize_logger: bool,
|
||||
logger_file_path: Optional[str],
|
||||
logger_level: int,
|
||||
suppress_node_logs: bool,
|
||||
) -> None:
|
||||
if not initialize_logger:
|
||||
return
|
||||
|
||||
app_logger = logging.getLogger("sync_state_machine")
|
||||
if app_logger.handlers:
|
||||
return
|
||||
|
||||
init_pipeline_logger(
|
||||
log_file_path=logger_file_path,
|
||||
level=logger_level,
|
||||
mirror_console=True,
|
||||
replace_handlers=False,
|
||||
suppress_node_logs=suppress_node_logs,
|
||||
)
|
||||
|
||||
|
||||
def _build_handler_config(
|
||||
node_type: str,
|
||||
ds_config: DataSourceConfig,
|
||||
@@ -139,6 +119,11 @@ async def _create_datasource(
|
||||
datasource = datasource_factory(ds_config, endpoint_name, **factory_kwargs)
|
||||
if inspect.isawaitable(datasource):
|
||||
datasource = await datasource
|
||||
if hasattr(datasource, "set_runtime_logging_context"):
|
||||
datasource.set_runtime_logging_context(
|
||||
endpoint_name=endpoint_name,
|
||||
datasource_type=ds_config.type,
|
||||
)
|
||||
return datasource
|
||||
|
||||
|
||||
@@ -148,6 +133,72 @@ async def _initialize_datasource(datasource, *, endpoint_name: str) -> None:
|
||||
except Exception as exc:
|
||||
raise RuntimeError(f"Failed to initialize {endpoint_name} datasource: {type(exc).__name__}: {exc}") from exc
|
||||
|
||||
|
||||
async def _create_and_initialize_datasource(
|
||||
ds_config: DataSourceConfig,
|
||||
*,
|
||||
datasource_override=None,
|
||||
endpoint_name: str,
|
||||
project_root: Path | None = None,
|
||||
):
|
||||
datasource = await _create_datasource(
|
||||
ds_config,
|
||||
datasource_override=datasource_override,
|
||||
endpoint_name=endpoint_name,
|
||||
project_root=project_root,
|
||||
)
|
||||
await _initialize_datasource(datasource, endpoint_name=endpoint_name)
|
||||
return datasource
|
||||
|
||||
|
||||
async def _safe_close_resource(name: str, resource) -> None:
|
||||
if resource is None:
|
||||
return
|
||||
try:
|
||||
await asyncio.wait_for(resource.close(), timeout=3.0)
|
||||
except Exception as close_exc:
|
||||
logger.warning("Failed to close %s during create cleanup: %s", name, close_exc)
|
||||
|
||||
|
||||
def _resolved_config_summary(config: PipelineRunConfig) -> tuple[str, str]:
|
||||
summary = (
|
||||
"Resolved Full Config Summary: node_types=%d local=%s remote=%s",
|
||||
len(config.node_types),
|
||||
config.local_datasource.type,
|
||||
config.remote_datasource.type,
|
||||
)
|
||||
return summary[0], json.dumps(config_snapshot(config), ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
def _log_resolved_config(config: PipelineRunConfig) -> tuple[str, str]:
|
||||
summary_template, resolved_cfg_text = _resolved_config_summary(config)
|
||||
logger.info(
|
||||
summary_template,
|
||||
len(config.node_types),
|
||||
config.local_datasource.type,
|
||||
config.remote_datasource.type,
|
||||
)
|
||||
logger.info("Resolved Full Config:\n%s", resolved_cfg_text)
|
||||
return summary_template, resolved_cfg_text
|
||||
|
||||
|
||||
def _log_pipeline_start(config: PipelineRunConfig, *, title: str) -> None:
|
||||
logger.info("\n%s", "=" * 80)
|
||||
logger.info("🚀 Creating %s", title)
|
||||
logger.info("📝 Log file: %s", config.logging.file_path or "-")
|
||||
logger.info(
|
||||
"🔧 Resolved Full Config Summary: node_types=%d local=%s remote=%s",
|
||||
len(config.node_types),
|
||||
config.local_datasource.type,
|
||||
config.remote_datasource.type,
|
||||
)
|
||||
logger.info("=" * 80)
|
||||
|
||||
|
||||
def _log_pipeline_end(config: PipelineRunConfig, *, stats: Dict[str, Any]) -> None:
|
||||
logger.info("✅ Pipeline completed: stats=%s", stats)
|
||||
logger.info("🗃️ Persistence Target: %s", config.persist.display_target())
|
||||
|
||||
ensure_builtin_datasource_types_registered()
|
||||
|
||||
|
||||
@@ -187,27 +238,59 @@ def _add_handlers(
|
||||
|
||||
def _build_collections_and_bindings(
|
||||
*,
|
||||
persistence: PersistenceBackend,
|
||||
persistence,
|
||||
scope: str,
|
||||
local_fallback_collection: Optional[DataCollection] = None,
|
||||
remote_fallback_collection: Optional[DataCollection] = None,
|
||||
binding_fallback_manager: Optional[BindingManager] = None,
|
||||
) -> tuple[DataCollection, DataCollection, BindingManager]:
|
||||
local_collection = DataCollection("local", scope=scope, persistence=persistence, auto_persist=False)
|
||||
remote_collection = DataCollection("remote", scope=scope, persistence=persistence, auto_persist=False)
|
||||
binding_manager = BindingManager(persistence, auto_persist=False, scope=scope)
|
||||
local_collection = DataCollection(
|
||||
"local",
|
||||
scope=scope,
|
||||
persistence=persistence,
|
||||
auto_persist=False,
|
||||
global_fallback_collection=local_fallback_collection,
|
||||
)
|
||||
remote_collection = DataCollection(
|
||||
"remote",
|
||||
scope=scope,
|
||||
persistence=persistence,
|
||||
auto_persist=False,
|
||||
global_fallback_collection=remote_fallback_collection,
|
||||
)
|
||||
binding_manager = BindingManager(
|
||||
persistence,
|
||||
auto_persist=False,
|
||||
scope=scope,
|
||||
global_fallback_manager=binding_fallback_manager,
|
||||
)
|
||||
return local_collection, remote_collection, binding_manager
|
||||
|
||||
|
||||
def _resolve_strategy_map(config: PipelineRunConfig, node_types: List[str]) -> Dict[str, StrategyRuntimeConfig]:
|
||||
if config.strategies:
|
||||
return {item.node_type: item for item in config.strategies}
|
||||
resolved_from_config = {item.node_type: item for item in config.strategies}
|
||||
for node_type, runtime_cfg in resolved_from_config.items():
|
||||
strategy_class = DomainRegistry.get_strategy_class(node_type)
|
||||
runtime_cfg.config.domain_option = resolve_domain_option_config(
|
||||
getattr(strategy_class, "domain_option_model", None) if strategy_class is not None else None,
|
||||
runtime_cfg.config.domain_option,
|
||||
)
|
||||
return resolved_from_config
|
||||
|
||||
resolved: Dict[str, StrategyRuntimeConfig] = {}
|
||||
for node_type in node_types:
|
||||
strategy_class = DomainRegistry.get_strategy_class(node_type)
|
||||
if strategy_class is None:
|
||||
continue
|
||||
strategy_config = strategy_class.default_config.model_copy(deep=True)
|
||||
strategy_config.domain_option = resolve_domain_option_config(
|
||||
getattr(strategy_class, "domain_option_model", None),
|
||||
strategy_config.domain_option,
|
||||
)
|
||||
resolved[node_type] = StrategyRuntimeConfig(
|
||||
node_type=node_type,
|
||||
config=strategy_class.default_config.model_copy(deep=True),
|
||||
config=strategy_config,
|
||||
)
|
||||
return resolved
|
||||
|
||||
@@ -251,19 +334,18 @@ def _build_strategies(
|
||||
raise RuntimeError(f"Missing strategy for node_type: {node_type}")
|
||||
strategy = strategy_class(node_type, local_collection, remote_collection, binding_manager)
|
||||
|
||||
if both_jsonl:
|
||||
strategy.update_config(
|
||||
local_orphan_action=OrphanAction.CREATE_REMOTE,
|
||||
remote_orphan_action=OrphanAction.NONE,
|
||||
update_direction=UpdateDirection.PUSH,
|
||||
)
|
||||
if node_type == "contract":
|
||||
strategy.update_config(depend_fields={"project_id": "project"})
|
||||
|
||||
runtime_cfg = strategy_map.get(node_type)
|
||||
if runtime_cfg is not None:
|
||||
strategy.set_config(runtime_cfg.config.model_copy(deep=True))
|
||||
strategy.skip_sync = bool(runtime_cfg.skip_sync)
|
||||
strategy.config = runtime_cfg.config.model_copy(deep=True)
|
||||
strategy.config.log_logic_warnings(node_type=strategy.node_type, logger=logger)
|
||||
|
||||
if both_jsonl:
|
||||
strategy.config.local_orphan_action = OrphanAction.CREATE_REMOTE
|
||||
strategy.config.remote_orphan_action = OrphanAction.NONE
|
||||
strategy.config.update_direction = UpdateDirection.PUSH
|
||||
if node_type == "contract":
|
||||
strategy.config.depend_fields = {"project_id": "project"}
|
||||
strategy.config.log_logic_warnings(node_type=strategy.node_type, logger=logger)
|
||||
|
||||
strategies.append(strategy)
|
||||
return strategies
|
||||
@@ -278,90 +360,129 @@ async def create_pipeline_from_config(
|
||||
ephemeral_node_types: Optional[List[str]] = None,
|
||||
local_datasource_override=None,
|
||||
remote_datasource_override=None,
|
||||
local_fallback_collection: Optional[DataCollection] = None,
|
||||
remote_fallback_collection: Optional[DataCollection] = None,
|
||||
binding_fallback_manager: Optional[BindingManager] = None,
|
||||
persistence_service: Optional[PersistenceService] = None,
|
||||
scope_runtime: Optional[ProjectScopeRuntime] = None,
|
||||
) -> FullSyncPipeline:
|
||||
if config.logging.initialize and not config.logging.file_path:
|
||||
config.logging.file_path = resolve_log_file_path(config.logging)
|
||||
|
||||
_ensure_pipeline_logger(
|
||||
initialize_logger=config.logging.initialize,
|
||||
logger_file_path=config.logging.file_path,
|
||||
logger_level=config.logging.level,
|
||||
suppress_node_logs=config.logging.suppress_node_logs,
|
||||
)
|
||||
ensure_app_logging(config.logging)
|
||||
|
||||
all_types = list(config.node_types)
|
||||
strategy_map = _resolve_strategy_map(config, all_types)
|
||||
|
||||
db_path = Path(config.persist.db_path)
|
||||
if config.persist.wipe_on_start and db_path.exists():
|
||||
db_path.unlink()
|
||||
runtime = scope_runtime
|
||||
close_resources = runtime is None
|
||||
|
||||
persistence: Optional[PersistenceBackend] = None
|
||||
if runtime is None:
|
||||
runtime = await create_scope_runtime_from_config(
|
||||
config,
|
||||
scope=scope,
|
||||
local_datasource_override=local_datasource_override,
|
||||
remote_datasource_override=remote_datasource_override,
|
||||
local_fallback_collection=local_fallback_collection,
|
||||
remote_fallback_collection=remote_fallback_collection,
|
||||
binding_fallback_manager=binding_fallback_manager,
|
||||
persistence_service=persistence_service,
|
||||
)
|
||||
|
||||
strategies = _build_strategies(
|
||||
config,
|
||||
all_types,
|
||||
runtime.local_collection,
|
||||
runtime.remote_collection,
|
||||
runtime.binding_manager,
|
||||
strategy_map,
|
||||
)
|
||||
|
||||
return FullSyncPipeline(
|
||||
local_datasource=runtime.local_datasource,
|
||||
remote_datasource=runtime.remote_datasource,
|
||||
strategies=strategies,
|
||||
node_types=all_types,
|
||||
persistence=runtime.persistence_backend,
|
||||
local_collection=runtime.local_collection,
|
||||
remote_collection=runtime.remote_collection,
|
||||
binding_manager=runtime.binding_manager,
|
||||
scope=scope,
|
||||
pipeline_name=pipeline_name,
|
||||
bootstrap_binding_node_types=bootstrap_binding_node_types,
|
||||
ephemeral_node_types=ephemeral_node_types,
|
||||
enable_persistence=config.persist.enable,
|
||||
scope_runtime=runtime,
|
||||
close_resources=close_resources,
|
||||
)
|
||||
|
||||
|
||||
async def create_scope_runtime_from_config(
|
||||
config: PipelineRunConfig,
|
||||
*,
|
||||
scope: str = "global",
|
||||
local_datasource_override=None,
|
||||
remote_datasource_override=None,
|
||||
local_fallback_collection: Optional[DataCollection] = None,
|
||||
remote_fallback_collection: Optional[DataCollection] = None,
|
||||
binding_fallback_manager: Optional[BindingManager] = None,
|
||||
persistence_service: Optional[PersistenceService] = None,
|
||||
global_runtime: Optional[ProjectScopeRuntime] = None,
|
||||
) -> ProjectScopeRuntime:
|
||||
ensure_app_logging(config.logging)
|
||||
|
||||
owns_persistence_service = persistence_service is None
|
||||
created_backend: Optional[PersistenceBackend] = None
|
||||
local_ds = None
|
||||
remote_ds = None
|
||||
try:
|
||||
persistence = create_persistence_backend(config.persist)
|
||||
await persistence.initialize()
|
||||
if persistence_service is None:
|
||||
wipe_file_path = config.persist.wipe_file_path()
|
||||
if config.persist.wipe_on_start and wipe_file_path:
|
||||
db_path = Path(wipe_file_path)
|
||||
if db_path.exists():
|
||||
db_path.unlink()
|
||||
created_backend = create_persistence_backend(config.persist)
|
||||
persistence_service = PersistenceService(created_backend)
|
||||
await persistence_service.initialize()
|
||||
|
||||
local_ds = await _create_datasource(
|
||||
local_ds = await _create_and_initialize_datasource(
|
||||
config.local_datasource,
|
||||
datasource_override=local_datasource_override,
|
||||
endpoint_name="local",
|
||||
project_root=_project_root(),
|
||||
)
|
||||
await _initialize_datasource(local_ds, endpoint_name="local")
|
||||
remote_ds = await _create_datasource(
|
||||
remote_ds = await _create_and_initialize_datasource(
|
||||
config.remote_datasource,
|
||||
datasource_override=remote_datasource_override,
|
||||
endpoint_name="remote",
|
||||
project_root=_project_root(),
|
||||
)
|
||||
await _initialize_datasource(remote_ds, endpoint_name="remote")
|
||||
_add_handlers(config, local_ds, remote_ds, all_types)
|
||||
_add_handlers(config, local_ds, remote_ds, list(config.node_types))
|
||||
|
||||
persistence_view = persistence_service.view(scope)
|
||||
local_collection, remote_collection, binding_manager = _build_collections_and_bindings(
|
||||
persistence=persistence,
|
||||
persistence=persistence_view,
|
||||
scope=scope,
|
||||
local_fallback_collection=local_fallback_collection,
|
||||
remote_fallback_collection=remote_fallback_collection,
|
||||
binding_fallback_manager=binding_fallback_manager,
|
||||
)
|
||||
|
||||
strategies = _build_strategies(
|
||||
config,
|
||||
all_types,
|
||||
local_collection,
|
||||
remote_collection,
|
||||
binding_manager,
|
||||
strategy_map,
|
||||
)
|
||||
|
||||
pipeline = FullSyncPipeline(
|
||||
local_datasource=local_ds,
|
||||
remote_datasource=remote_ds,
|
||||
strategies=strategies,
|
||||
node_types=all_types,
|
||||
persistence=persistence,
|
||||
return ProjectScopeRuntime(
|
||||
scope=scope,
|
||||
persistence_service=persistence_service,
|
||||
persistence_view=persistence_view,
|
||||
local_collection=local_collection,
|
||||
remote_collection=remote_collection,
|
||||
binding_manager=binding_manager,
|
||||
scope=scope,
|
||||
pipeline_name=pipeline_name,
|
||||
bootstrap_binding_node_types=bootstrap_binding_node_types,
|
||||
ephemeral_node_types=ephemeral_node_types,
|
||||
enable_persistence=config.persist.enable,
|
||||
local_datasource=local_ds,
|
||||
remote_datasource=remote_ds,
|
||||
global_runtime=global_runtime,
|
||||
owns_persistence_service=owns_persistence_service,
|
||||
)
|
||||
|
||||
return pipeline
|
||||
except Exception:
|
||||
async def _safe_close(name: str, resource) -> None:
|
||||
if resource is None:
|
||||
return
|
||||
try:
|
||||
await asyncio.wait_for(resource.close(), timeout=3.0)
|
||||
except Exception as close_exc:
|
||||
logger.warning("Failed to close %s during create cleanup: %s", name, close_exc)
|
||||
|
||||
await _safe_close("remote_datasource", remote_ds)
|
||||
await _safe_close("local_datasource", local_ds)
|
||||
await _safe_close("persistence", persistence)
|
||||
await _safe_close_resource("remote_datasource", remote_ds)
|
||||
await _safe_close_resource("local_datasource", local_ds)
|
||||
if owns_persistence_service:
|
||||
await _safe_close_resource("persistence", persistence_service or created_backend)
|
||||
raise
|
||||
|
||||
|
||||
@@ -374,47 +495,40 @@ async def run_pipeline_from_config(
|
||||
bootstrap_binding_node_types: Optional[List[str]] = None,
|
||||
local_datasource_override=None,
|
||||
remote_datasource_override=None,
|
||||
pipeline_type: PipelineType = "auto",
|
||||
) -> Dict[str, Any]:
|
||||
if config.logging.initialize and not config.logging.file_path:
|
||||
config.logging.file_path = resolve_log_file_path(config.logging)
|
||||
ensure_app_logging(config.logging)
|
||||
|
||||
_ensure_pipeline_logger(
|
||||
initialize_logger=config.logging.initialize,
|
||||
logger_file_path=config.logging.file_path,
|
||||
logger_level=config.logging.level,
|
||||
suppress_node_logs=config.logging.suppress_node_logs,
|
||||
)
|
||||
from .multi_project_pipeline import is_implicit_project_batch_config, run_implicit_project_batch_from_config
|
||||
|
||||
resolved_cfg = config_snapshot(config)
|
||||
resolved_cfg_text = json.dumps(resolved_cfg, ensure_ascii=False, indent=2)
|
||||
logger.info(
|
||||
"Resolved Full Config Summary: node_types=%d local=%s remote=%s",
|
||||
len(config.node_types),
|
||||
config.local_datasource.type,
|
||||
config.remote_datasource.type,
|
||||
)
|
||||
if config.logging.file_path:
|
||||
try:
|
||||
with open(config.logging.file_path, "a", encoding="utf-8") as fp:
|
||||
fp.write("Resolved Full Config:\n")
|
||||
fp.write(resolved_cfg_text)
|
||||
fp.write("\n")
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to write resolved config into log file: %s", exc)
|
||||
implicit_multi_project = is_implicit_project_batch_config(config)
|
||||
if pipeline_type == "full" and implicit_multi_project:
|
||||
logger.info("pipeline_type=full specified, skipping implicit multi-project routing")
|
||||
|
||||
if pipeline_type == "multi-project":
|
||||
_log_resolved_config(config)
|
||||
return await run_implicit_project_batch_from_config(
|
||||
config,
|
||||
title=title,
|
||||
print_summary=print_summary,
|
||||
max_concurrency=getattr(config, "max_concurrency", None),
|
||||
)
|
||||
|
||||
if pipeline_type != "full" and implicit_multi_project:
|
||||
_log_resolved_config(config)
|
||||
return await run_implicit_project_batch_from_config(
|
||||
config,
|
||||
title=title,
|
||||
print_summary=print_summary,
|
||||
max_concurrency=getattr(config, "max_concurrency", None),
|
||||
)
|
||||
|
||||
_log_resolved_config(config)
|
||||
|
||||
pipeline: Optional[FullSyncPipeline] = None
|
||||
try:
|
||||
if print_summary:
|
||||
print("\n" + "=" * 80)
|
||||
print(f"🚀 Creating {title}")
|
||||
print(f"📝 Log file: {config.logging.file_path or '-'}")
|
||||
print(
|
||||
f"🔧 Resolved Full Config Summary: node_types={len(config.node_types)} "
|
||||
f"local={config.local_datasource.type} "
|
||||
f"remote={config.remote_datasource.type}"
|
||||
)
|
||||
print(f"🔧 Resolved Full Config:\n{resolved_cfg_text}")
|
||||
print("=" * 80)
|
||||
_log_pipeline_start(config, title=title)
|
||||
|
||||
pipeline = await create_pipeline_from_config(
|
||||
config,
|
||||
@@ -427,8 +541,7 @@ async def run_pipeline_from_config(
|
||||
stats = await pipeline.run()
|
||||
|
||||
if print_summary:
|
||||
print(f"\n✅ Pipeline completed: stats={stats}")
|
||||
print(f"🗃️ Persistence DB: {config.persist.db_path}")
|
||||
_log_pipeline_end(config, stats=stats)
|
||||
|
||||
return stats
|
||||
finally:
|
||||
@@ -491,17 +604,25 @@ async def run_profile_from_file(
|
||||
cfg_path = Path(config_path)
|
||||
raw = load_overrides_from_file(cfg_path, project_root)
|
||||
if is_multi_project_payload(raw):
|
||||
from .multi_project_pipeline import MultiProjectPipeline
|
||||
from .multi_project_pipeline import MultiProjectPipeline, _resolve_embedded_config_path
|
||||
|
||||
multi_cfg = MultiProjectRunConfig.model_validate(raw)
|
||||
global_cfg_path = _resolve_embedded_config_path(
|
||||
multi_cfg.global_config,
|
||||
project_root=project_root,
|
||||
profile_path=cfg_path,
|
||||
)
|
||||
global_config = build_config_from_file(project_root=project_root, file_path=global_cfg_path)
|
||||
configure_app_logging(global_config.logging, replace_handlers=True)
|
||||
pipeline = MultiProjectPipeline(project_root=project_root, config_path=cfg_path, config=multi_cfg)
|
||||
if print_summary:
|
||||
print("\n" + "=" * 80)
|
||||
print(f"🚀 Creating multi-project pipeline from: {cfg_path}")
|
||||
print("=" * 80)
|
||||
logger.info("\n%s", "=" * 80)
|
||||
logger.info("🚀 Creating multi-project pipeline from: %s", cfg_path)
|
||||
logger.info("=" * 80)
|
||||
return await pipeline.run()
|
||||
|
||||
config = build_config(project_root=project_root, overrides=raw)
|
||||
configure_app_logging(config.logging, replace_handlers=True)
|
||||
mode = f"{config.local_datasource.type}->{config.remote_datasource.type}"
|
||||
return await run_pipeline_from_config(
|
||||
config,
|
||||
|
||||
@@ -5,9 +5,9 @@ Full Sync Pipeline - Implements document-specified end-to-end workflow.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional, TYPE_CHECKING
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, Dict, List, Optional
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..datasource.datasource import BaseDataSource
|
||||
@@ -15,17 +15,31 @@ if TYPE_CHECKING:
|
||||
from ..datasource.jsonl import JsonlDataSource
|
||||
from ..common.collection import DataCollection
|
||||
from ..common.binding import BindingManager
|
||||
from ..logging import get_scope_logger
|
||||
from ..common.persistence import PersistenceBackend
|
||||
from ..sync_system.config import UpdateDirection
|
||||
from ..sync_system.strategy import BaseSyncStrategy
|
||||
from ..sync_system.config import OrphanAction, UpdateDirection
|
||||
from ..engine import StateMachineConfig, StateMachineRuntime
|
||||
from ..engine import e41_post_create_ready
|
||||
from ..sync_system.strategy_ops import finalize_peer_creating_sources
|
||||
from ..sync_system.strategy_ops import finalize_peer_creating_sources, run_phase2_cleanup
|
||||
from ..sync_system.strategy_ops.bind_ops import refresh_bind_data_id
|
||||
from ..sync_system.strategy_ops.post_check_ops import evaluate_consistency_for_node_type
|
||||
from .summary_report import print_pipeline_summary
|
||||
from .project_scope_runtime import ProjectScopeRuntime
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
class NodeTypeLoadError(RuntimeError):
|
||||
"""Raised when a datasource load for a node type fails and the pipeline must stop."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WriteOutcome:
|
||||
local_written: bool = False
|
||||
remote_written: bool = False
|
||||
|
||||
@property
|
||||
def any_written(self) -> bool:
|
||||
return self.local_written or self.remote_written
|
||||
|
||||
|
||||
class FullSyncPipeline:
|
||||
@@ -56,6 +70,8 @@ class FullSyncPipeline:
|
||||
load_order: Optional[List[str]] = None,
|
||||
sync_order: Optional[List[str]] = None,
|
||||
enable_persistence: bool = True,
|
||||
scope_runtime: Optional[ProjectScopeRuntime] = None,
|
||||
close_resources: bool = True,
|
||||
):
|
||||
# 验证必需参数(由工厂函数创建)
|
||||
if strategies is None:
|
||||
@@ -75,6 +91,8 @@ class FullSyncPipeline:
|
||||
self.sync_order = sync_order or self.node_types
|
||||
self.persistence = persistence
|
||||
self.enable_persistence = enable_persistence
|
||||
self.scope_runtime = scope_runtime
|
||||
self.close_resources = close_resources
|
||||
|
||||
self.local_collection = local_collection
|
||||
self.remote_collection = remote_collection
|
||||
@@ -97,12 +115,9 @@ class FullSyncPipeline:
|
||||
"post_check_passed": True,
|
||||
}
|
||||
self._consistency_by_type: Dict[str, Dict[str, Any]] = {}
|
||||
self._logger = logger
|
||||
self._logger = get_scope_logger(scope, __name__)
|
||||
self._closed = False
|
||||
self._loaded_node_types: set[str] = set()
|
||||
self._preloaded_local_nodes: List[Any] = []
|
||||
self._preloaded_remote_nodes: List[Any] = []
|
||||
self._preloaded_bindings: Dict[str, Dict[str, Optional[str]]] = {}
|
||||
|
||||
def _resolve_pipeline_runtime(self) -> StateMachineRuntime:
|
||||
if self.strategies:
|
||||
@@ -113,133 +128,257 @@ class FullSyncPipeline:
|
||||
cfg_path = Path(__file__).resolve().parents[1] / "config" / "node_state_machine.yaml"
|
||||
return StateMachineRuntime(StateMachineConfig.load(cfg_path))
|
||||
|
||||
def _strategy_map(self) -> Dict[str, BaseSyncStrategy]:
|
||||
return {strategy.node_type: strategy for strategy in self.strategies}
|
||||
|
||||
def _iter_sync_strategies(self):
|
||||
strategy_map = self._strategy_map()
|
||||
for node_type in self.sync_order:
|
||||
strategy = strategy_map.get(node_type)
|
||||
if strategy is not None:
|
||||
yield node_type, strategy
|
||||
|
||||
def _get_strategy(self, node_type: str) -> BaseSyncStrategy | None:
|
||||
return next((item for item in self.strategies if item.node_type == node_type), None)
|
||||
|
||||
def _log_phase_completed(self, phase_name: str) -> None:
|
||||
self._logger.info("✅ Phase %s completed (%s)", phase_name, self.pipeline_name)
|
||||
|
||||
def _log_strategy_start(self, node_type: str) -> None:
|
||||
section_width = 96
|
||||
self._logger.info("\n" + "-" * section_width)
|
||||
self._logger.info("🧩 Strategy start: %s", node_type)
|
||||
self._logger.info("-" * section_width)
|
||||
|
||||
def _log_strategy_result(self, action: str, node_type: str, count: int | None = None) -> None:
|
||||
if count is None:
|
||||
self._logger.info("✅ Strategy %s: %s", action, node_type)
|
||||
return
|
||||
self._logger.info("✅ Strategy %s: %s (%s=%d)", action, node_type, action, count)
|
||||
|
||||
def _log_strategy_done(self, node_type: str) -> None:
|
||||
self._logger.info("✅ Strategy done: %s", node_type)
|
||||
|
||||
def _log_strategy_failure(self, stage: str, node_type: str, exc: Exception) -> None:
|
||||
self._logger.error(
|
||||
"❌ Strategy %s failed but pipeline continues: %s | %s: %s",
|
||||
stage,
|
||||
node_type,
|
||||
type(exc).__name__,
|
||||
exc,
|
||||
)
|
||||
|
||||
def _log_reload_skipped(self, node_type: str, reason: str) -> None:
|
||||
self._logger.info("⏭️ Reload skipped for JSONL pipeline: node_type=%s reason=%s", node_type, reason)
|
||||
|
||||
async def _safe_close(self, name: str, close_coro, timeout: float = 3.0) -> None:
|
||||
try:
|
||||
await asyncio.wait_for(close_coro, timeout=timeout)
|
||||
except asyncio.TimeoutError:
|
||||
self._logger.warning("⚠️ close timeout: %s (>%ss)", name, timeout)
|
||||
except Exception as exc:
|
||||
self._logger.warning("⚠️ close failed: %s (%s)", name, exc)
|
||||
|
||||
async def run(self) -> Dict[str, Any]:
|
||||
persistence_started = False
|
||||
try:
|
||||
await self.phase_bootstrap()
|
||||
self._logger.info("✅ Phase bootstrap completed (%s)", self.pipeline_name)
|
||||
self._log_phase_completed("bootstrap")
|
||||
|
||||
await self.phase_sync_by_node_type()
|
||||
self._logger.info("✅ Phase sync-by-node-type completed (%s)", self.pipeline_name)
|
||||
self._log_phase_completed("sync-by-node-type")
|
||||
|
||||
post_ok = await self.phase_post_check()
|
||||
self.stats["post_check_passed"] = post_ok
|
||||
self._logger.info("✅ Phase post-check completed (%s)", self.pipeline_name)
|
||||
self._log_phase_completed("post-check")
|
||||
|
||||
persistence_started = True
|
||||
await self.phase_persistence()
|
||||
self._logger.info("✅ Phase persistence completed (%s)", self.pipeline_name)
|
||||
self._log_phase_completed("persistence")
|
||||
return self.stats
|
||||
except Exception as exc:
|
||||
if not persistence_started:
|
||||
await self._persist_after_failure(exc)
|
||||
raise
|
||||
finally:
|
||||
await self.close()
|
||||
|
||||
async def _persist_after_failure(self, cause: Exception) -> None:
|
||||
if not self.enable_persistence:
|
||||
self._logger.info(
|
||||
"⏭️ Persistence disabled after pipeline failure: %s (%s)",
|
||||
type(cause).__name__,
|
||||
self.pipeline_name,
|
||||
)
|
||||
return
|
||||
|
||||
self._logger.info(
|
||||
"🔄 Pipeline failed before persistence, persisting current state: %s | %s",
|
||||
type(cause).__name__,
|
||||
cause,
|
||||
)
|
||||
try:
|
||||
await self.phase_persistence()
|
||||
self._logger.info("✅ Phase persistence completed after failure (%s)", self.pipeline_name)
|
||||
except Exception as persist_exc:
|
||||
self._logger.error(
|
||||
"❌ Phase persistence failed after pipeline failure: %s | %s",
|
||||
type(persist_exc).__name__,
|
||||
persist_exc,
|
||||
)
|
||||
|
||||
async def phase_sync_by_node_type(self) -> None:
|
||||
self._prepare_datasources()
|
||||
strategy_map = {s.node_type: s for s in self.strategies}
|
||||
for node_type in self.sync_order:
|
||||
strategy = strategy_map.get(node_type)
|
||||
if strategy is None:
|
||||
continue
|
||||
for node_type, strategy in self._iter_sync_strategies():
|
||||
try:
|
||||
section_width = 96
|
||||
self._logger.info("\n" + "-" * section_width)
|
||||
self._logger.info(f"🧩 Strategy start: {node_type}")
|
||||
self._logger.info("-" * section_width)
|
||||
self._log_strategy_start(node_type)
|
||||
|
||||
await self._load_node_type(node_type, reason="pre-strategy refresh")
|
||||
|
||||
# Load failure is a hard stop: create/update cannot safely treat missing data as empty data.
|
||||
await strategy.bind()
|
||||
self._logger.info(f"✅ Strategy bind: {node_type}")
|
||||
self._log_strategy_result("bind", node_type)
|
||||
|
||||
if strategy.skip_sync:
|
||||
self._logger.info(f"⏭️ Skipping create/update for {node_type} (skip_sync=True)")
|
||||
self._logger.info("⏭️ Skipping create/update for %s (skip_sync=True)", node_type)
|
||||
continue
|
||||
|
||||
created = await strategy.create()
|
||||
self._logger.info(f"✅ Strategy create: {node_type} (created={len(created)})")
|
||||
create_written = await self.commit_creates(node_type)
|
||||
self._log_strategy_result("create", node_type, len(created))
|
||||
created_local_node_ids = {
|
||||
node.node_id for node in created if self.local_collection.get_by_node_id(node.node_id) is not None
|
||||
}
|
||||
created_remote_node_ids = {
|
||||
node.node_id for node in created if self.remote_collection.get_by_node_id(node.node_id) is not None
|
||||
}
|
||||
create_outcome = await self.commit_creates(node_type)
|
||||
await self.normalize_create_success_to_update_entry(node_type)
|
||||
if create_written:
|
||||
await self._reload_node_type(node_type, reason="create wrote data")
|
||||
if create_outcome.any_written:
|
||||
await self._reload_node_type_after_write(
|
||||
node_type,
|
||||
reason="create wrote data",
|
||||
reload_local=create_outcome.local_written,
|
||||
reload_remote=create_outcome.remote_written,
|
||||
)
|
||||
await refresh_bind_data_id(
|
||||
node_type=node_type,
|
||||
local_collection=self.local_collection,
|
||||
remote_collection=self.remote_collection,
|
||||
binding_manager=self.binding_manager,
|
||||
local_node_ids=created_local_node_ids,
|
||||
remote_node_ids=created_remote_node_ids,
|
||||
)
|
||||
|
||||
updated = await strategy.update()
|
||||
self._logger.info(f"✅ Strategy update: {node_type} (updated={len(updated)})")
|
||||
update_written = await self.commit_updates(node_type)
|
||||
if update_written:
|
||||
await self._reload_node_type(node_type, reason="update wrote data")
|
||||
self._log_strategy_result("update", node_type, len(updated))
|
||||
update_outcome = await self.commit_updates(node_type)
|
||||
if update_outcome.any_written:
|
||||
await self._reload_node_type_after_write(
|
||||
node_type,
|
||||
reason="update wrote data",
|
||||
reload_local=update_outcome.local_written,
|
||||
reload_remote=update_outcome.remote_written,
|
||||
)
|
||||
|
||||
self._logger.info(f"✅ Strategy done: {node_type}")
|
||||
self._log_strategy_done(node_type)
|
||||
except NodeTypeLoadError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
self.stats["failed"] += 1
|
||||
self._logger.error(f"❌ Strategy failed but pipeline continues: {node_type} | {type(exc).__name__}: {exc}")
|
||||
self._log_strategy_failure("sync", node_type, exc)
|
||||
continue
|
||||
|
||||
async def phase_bootstrap(self) -> None:
|
||||
await self._load_persistence_state()
|
||||
if self.scope_runtime is not None:
|
||||
await self.scope_runtime.load(
|
||||
node_types=self.node_types,
|
||||
bootstrap_binding_node_types=self.bootstrap_binding_node_types,
|
||||
enable_persistence=self.enable_persistence,
|
||||
ephemeral_node_types=self.ephemeral_node_types,
|
||||
)
|
||||
else:
|
||||
await self._load_persistence_state()
|
||||
self._logger.info("✅ Bootstrap: persistence loaded")
|
||||
|
||||
await self._apply_preloaded_state()
|
||||
self._logger.info("✅ Bootstrap: shared preload applied")
|
||||
|
||||
self._prepare_datasources()
|
||||
self._logger.info("✅ Bootstrap: datasource prepared")
|
||||
|
||||
await self._bootstrap_refresh_latest_data()
|
||||
self._logger.info("✅ Bootstrap: datasource refresh completed")
|
||||
|
||||
await self._run_bootstrap_reset_cleanup()
|
||||
self._logger.info("✅ Bootstrap: reset cleanup completed")
|
||||
|
||||
await self._apply_project_scope_cleanup()
|
||||
|
||||
async def phase_bind(self) -> None:
|
||||
self._prepare_datasources()
|
||||
strategy_map = {s.node_type: s for s in self.strategies}
|
||||
for node_type in self.sync_order:
|
||||
strategy = strategy_map.get(node_type)
|
||||
if strategy is None:
|
||||
continue
|
||||
for node_type, strategy in self._iter_sync_strategies():
|
||||
try:
|
||||
await strategy.bind()
|
||||
except Exception as exc:
|
||||
self.stats["failed"] += 1
|
||||
self._logger.error(f"❌ Strategy bind failed but pipeline continues: {node_type} | {type(exc).__name__}: {exc}")
|
||||
self._log_strategy_failure("bind", node_type, exc)
|
||||
continue
|
||||
|
||||
async def phase_create(self) -> None:
|
||||
self._prepare_datasources()
|
||||
strategy_map = {s.node_type: s for s in self.strategies}
|
||||
for node_type in self.sync_order:
|
||||
strategy = strategy_map.get(node_type)
|
||||
if strategy is None:
|
||||
continue
|
||||
for node_type, strategy in self._iter_sync_strategies():
|
||||
if strategy.skip_sync:
|
||||
self._logger.info(f"⏭️ Skipping create for {node_type} (skip_sync=True)")
|
||||
self._logger.info("⏭️ Skipping create for %s (skip_sync=True)", node_type)
|
||||
continue
|
||||
try:
|
||||
created = await strategy.create()
|
||||
self._logger.info(f"✅ Strategy create: {node_type} (created={len(created)})")
|
||||
create_written = await self.commit_creates(node_type)
|
||||
self._log_strategy_result("create", node_type, len(created))
|
||||
created_local_node_ids = {
|
||||
node.node_id for node in created if self.local_collection.get_by_node_id(node.node_id) is not None
|
||||
}
|
||||
created_remote_node_ids = {
|
||||
node.node_id for node in created if self.remote_collection.get_by_node_id(node.node_id) is not None
|
||||
}
|
||||
create_outcome = await self.commit_creates(node_type)
|
||||
await self.normalize_create_success_to_update_entry(node_type)
|
||||
if create_written:
|
||||
await self._reload_node_type(node_type, reason="create wrote data")
|
||||
if create_outcome.any_written:
|
||||
await self._reload_node_type_after_write(
|
||||
node_type,
|
||||
reason="create wrote data",
|
||||
reload_local=create_outcome.local_written,
|
||||
reload_remote=create_outcome.remote_written,
|
||||
)
|
||||
await refresh_bind_data_id(
|
||||
node_type=node_type,
|
||||
local_collection=self.local_collection,
|
||||
remote_collection=self.remote_collection,
|
||||
binding_manager=self.binding_manager,
|
||||
local_node_ids=created_local_node_ids,
|
||||
remote_node_ids=created_remote_node_ids,
|
||||
)
|
||||
except Exception as exc:
|
||||
self.stats["failed"] += 1
|
||||
self._logger.error(f"❌ Strategy create failed but pipeline continues: {node_type} | {type(exc).__name__}: {exc}")
|
||||
self._log_strategy_failure("create", node_type, exc)
|
||||
continue
|
||||
|
||||
async def phase_update(self) -> None:
|
||||
self._prepare_datasources()
|
||||
strategy_map = {s.node_type: s for s in self.strategies}
|
||||
for node_type in self.sync_order:
|
||||
strategy = strategy_map.get(node_type)
|
||||
if strategy is None:
|
||||
continue
|
||||
for node_type, strategy in self._iter_sync_strategies():
|
||||
if strategy.skip_sync:
|
||||
self._logger.info(f"⏭️ Skipping update for {node_type} (skip_sync=True)")
|
||||
self._logger.info("⏭️ Skipping update for %s (skip_sync=True)", node_type)
|
||||
continue
|
||||
try:
|
||||
updated = await strategy.update()
|
||||
self._logger.info(f"✅ Strategy update: {node_type} (updated={len(updated)})")
|
||||
update_written = await self.commit_updates(node_type)
|
||||
if update_written:
|
||||
await self._reload_node_type(node_type, reason="update wrote data")
|
||||
self._log_strategy_result("update", node_type, len(updated))
|
||||
update_outcome = await self.commit_updates(node_type)
|
||||
if update_outcome.any_written:
|
||||
await self._reload_node_type_after_write(
|
||||
node_type,
|
||||
reason="update wrote data",
|
||||
reload_local=update_outcome.local_written,
|
||||
reload_remote=update_outcome.remote_written,
|
||||
)
|
||||
await self._evaluate_consistency_for_node_type(node_type)
|
||||
except Exception as exc:
|
||||
self.stats["failed"] += 1
|
||||
self._logger.error(f"❌ Strategy update failed but pipeline continues: {node_type} | {type(exc).__name__}: {exc}")
|
||||
self._log_strategy_failure("update", node_type, exc)
|
||||
continue
|
||||
|
||||
async def close(self) -> None:
|
||||
@@ -248,20 +387,20 @@ class FullSyncPipeline:
|
||||
return # 已经关闭过,避免重复关闭
|
||||
self._closed = True
|
||||
|
||||
async def _safe_close(name: str, close_coro, timeout: float = 3.0) -> None:
|
||||
try:
|
||||
await asyncio.wait_for(close_coro, timeout=timeout)
|
||||
except asyncio.TimeoutError:
|
||||
self._logger.warning("⚠️ close timeout: %s (>%ss)", name, timeout)
|
||||
except Exception as exc:
|
||||
self._logger.warning("⚠️ close failed: %s (%s)", name, exc)
|
||||
if not self.close_resources:
|
||||
return
|
||||
|
||||
await _safe_close("remote_datasource", self.remote_datasource.close())
|
||||
await _safe_close("local_datasource", self.local_datasource.close())
|
||||
await _safe_close("persistence", self.persistence.close())
|
||||
if self.scope_runtime is not None:
|
||||
await self.scope_runtime.unload()
|
||||
await self.scope_runtime.close()
|
||||
return
|
||||
|
||||
await self._safe_close("remote_datasource", self.remote_datasource.close())
|
||||
await self._safe_close("local_datasource", self.local_datasource.close())
|
||||
await self._safe_close("persistence", self.persistence.close())
|
||||
|
||||
async def _load_persistence_state(self) -> None:
|
||||
"""加载持久化数据并清理僵尸节点"""
|
||||
"""加载持久化数据到内存,等待 bootstrap datasource refresh 与 E01 cleanup。"""
|
||||
if not self.enable_persistence:
|
||||
return
|
||||
|
||||
@@ -278,11 +417,51 @@ class FullSyncPipeline:
|
||||
for node_type in binding_node_types:
|
||||
await self.binding_manager.load_from_persistence(node_type)
|
||||
self._logger.info("🔄 Persistence loaded (state preserved; pending E01 normalization)")
|
||||
|
||||
# 加载后 reset 清理:清理 CREATE 失败的僵尸绑定
|
||||
|
||||
def _resolve_bootstrap_refresh_node_types(self) -> List[str]:
|
||||
local_types = {node.node_type for node in self.local_collection.filter()}
|
||||
remote_types = {node.node_type for node in self.remote_collection.filter()}
|
||||
candidate_types = local_types | remote_types
|
||||
if not candidate_types:
|
||||
return []
|
||||
|
||||
ordered: List[str] = []
|
||||
for node_type in [*self.load_order, *self.bootstrap_binding_node_types, *sorted(candidate_types)]:
|
||||
if node_type in candidate_types and node_type not in ordered:
|
||||
ordered.append(node_type)
|
||||
return ordered
|
||||
|
||||
async def _bootstrap_refresh_latest_data(self) -> None:
|
||||
if not self.enable_persistence:
|
||||
return
|
||||
|
||||
node_types = self._resolve_bootstrap_refresh_node_types()
|
||||
if not node_types:
|
||||
self._logger.info("⏭️ Bootstrap datasource refresh skipped: no persisted node types")
|
||||
return
|
||||
|
||||
for node_type in node_types:
|
||||
self._logger.info("🔄 Bootstrap refresh node_type=%s from datasource", node_type)
|
||||
await self._load_node_type_from_datasource(
|
||||
self.local_datasource,
|
||||
datasource_role="local",
|
||||
node_type=node_type,
|
||||
reason="bootstrap persistence refresh",
|
||||
)
|
||||
await self._load_node_type_from_datasource(
|
||||
self.remote_datasource,
|
||||
datasource_role="remote",
|
||||
node_type=node_type,
|
||||
reason="bootstrap persistence refresh",
|
||||
)
|
||||
|
||||
async def _run_bootstrap_reset_cleanup(self) -> None:
|
||||
if not self.enable_persistence:
|
||||
return
|
||||
if not self.strategies:
|
||||
raise RuntimeError("no strategies configured: cannot run bootstrap event E01")
|
||||
total_cleaned = await BaseSyncStrategy.run_reset(
|
||||
|
||||
total_cleaned = await run_phase2_cleanup(
|
||||
node_types=self.node_types,
|
||||
local_collection=self.local_collection,
|
||||
remote_collection=self.remote_collection,
|
||||
@@ -290,30 +469,7 @@ class FullSyncPipeline:
|
||||
runtime=self.sm_runtime,
|
||||
)
|
||||
if total_cleaned > 0:
|
||||
self._logger.info(f"🧹 Reset cleanup: {total_cleaned} CREATE-failed zombies cleaned")
|
||||
self._logger.info("🔄 Post-load reset cleanup completed")
|
||||
|
||||
async def _apply_preloaded_state(self) -> None:
|
||||
if self._preloaded_local_nodes:
|
||||
await self.local_collection.import_nodes(self._preloaded_local_nodes)
|
||||
if self._preloaded_remote_nodes:
|
||||
await self.remote_collection.import_nodes(self._preloaded_remote_nodes)
|
||||
for node_type, bindings in self._preloaded_bindings.items():
|
||||
self.binding_manager.import_bindings(node_type, bindings)
|
||||
|
||||
def preload_shared_state(
|
||||
self,
|
||||
*,
|
||||
local_nodes: List[Any],
|
||||
remote_nodes: List[Any],
|
||||
bindings_by_type: Dict[str, Dict[str, Optional[str]]],
|
||||
) -> None:
|
||||
self._preloaded_local_nodes = list(local_nodes)
|
||||
self._preloaded_remote_nodes = list(remote_nodes)
|
||||
self._preloaded_bindings = {
|
||||
node_type: dict(bindings)
|
||||
for node_type, bindings in bindings_by_type.items()
|
||||
}
|
||||
self._logger.info(f"🧹 Reset cleanup: {total_cleaned} bootstrap zombies cleaned")
|
||||
|
||||
def _prepare_datasources(self) -> None:
|
||||
"""为后续按 node_type 加载准备 collection 绑定。"""
|
||||
@@ -355,9 +511,19 @@ class FullSyncPipeline:
|
||||
)
|
||||
|
||||
async def _load_node_type(self, node_type: str, *, reason: str) -> None:
|
||||
self._logger.info(f"🔄 Load node_type={node_type}, reason={reason}")
|
||||
await self.local_datasource.load_all(order=[node_type])
|
||||
await self.remote_datasource.load_all(order=[node_type])
|
||||
self._logger.debug(f"🔄 Load node_type={node_type}, reason={reason}")
|
||||
await self._load_node_type_from_datasource(
|
||||
self.local_datasource,
|
||||
datasource_role="local",
|
||||
node_type=node_type,
|
||||
reason=reason,
|
||||
)
|
||||
await self._load_node_type_from_datasource(
|
||||
self.remote_datasource,
|
||||
datasource_role="remote",
|
||||
node_type=node_type,
|
||||
reason=reason,
|
||||
)
|
||||
|
||||
if node_type in self._loaded_node_types:
|
||||
return
|
||||
@@ -366,6 +532,21 @@ class FullSyncPipeline:
|
||||
self.stats["loaded"] += len(self.local_collection.filter(node_type=node_type))
|
||||
self.stats["loaded"] += len(self.remote_collection.filter(node_type=node_type))
|
||||
|
||||
async def _load_node_type_from_datasource(
|
||||
self,
|
||||
datasource: "BaseDataSource",
|
||||
*,
|
||||
datasource_role: str,
|
||||
node_type: str,
|
||||
reason: str,
|
||||
) -> None:
|
||||
try:
|
||||
await datasource.load_all(order=[node_type], raise_on_error=True)
|
||||
except Exception as exc:
|
||||
raise NodeTypeLoadError(
|
||||
f"Load failed for node_type={node_type}, datasource={datasource_role}, reason={reason}: {exc}"
|
||||
) from exc
|
||||
|
||||
def _get_action_success_count(self, datasource: "BaseDataSource", node_type: str, action_key: str) -> int:
|
||||
summary = datasource.get_action_summary().get(node_type, {})
|
||||
action_summary = summary.get(action_key, {}) if isinstance(summary, dict) else {}
|
||||
@@ -376,25 +557,69 @@ class FullSyncPipeline:
|
||||
|
||||
async def _reload_node_type(self, node_type: str, *, reason: str) -> None:
|
||||
if self._should_skip_reload():
|
||||
self._logger.info("⏭️ Reload skipped for JSONL pipeline: node_type=%s reason=%s", node_type, reason)
|
||||
self._log_reload_skipped(node_type, reason)
|
||||
return
|
||||
await self._load_node_type(node_type, reason=reason)
|
||||
|
||||
async def _reload_node_type_after_write(
|
||||
self,
|
||||
node_type: str,
|
||||
*,
|
||||
reason: str,
|
||||
reload_local: bool,
|
||||
reload_remote: bool,
|
||||
) -> None:
|
||||
if not reload_local and not reload_remote:
|
||||
return
|
||||
|
||||
if self._should_skip_reload():
|
||||
self._log_reload_skipped(node_type, reason)
|
||||
return
|
||||
|
||||
self._logger.debug(
|
||||
"🔄 Reload node_type=%s after write, reason=%s, local=%s, remote=%s",
|
||||
node_type,
|
||||
reason,
|
||||
reload_local,
|
||||
reload_remote,
|
||||
)
|
||||
if reload_local:
|
||||
await self._load_node_type_from_datasource(
|
||||
self.local_datasource,
|
||||
datasource_role="local",
|
||||
node_type=node_type,
|
||||
reason=reason,
|
||||
)
|
||||
if reload_remote:
|
||||
await self._load_node_type_from_datasource(
|
||||
self.remote_datasource,
|
||||
datasource_role="remote",
|
||||
node_type=node_type,
|
||||
reason=reason,
|
||||
)
|
||||
|
||||
async def _evaluate_consistency_for_node_type(self, node_type: str) -> Dict[str, Any]:
|
||||
strategy = next((item for item in self.strategies if item.node_type == node_type), None)
|
||||
if strategy is None:
|
||||
raise RuntimeError(f"Missing strategy for node_type={node_type}")
|
||||
depend_fields = dict(strategy.config.depend_fields) if strategy and strategy.config.depend_fields else {}
|
||||
compare_to_remote = True
|
||||
if strategy and strategy.config.update_direction == UpdateDirection.PULL:
|
||||
compare_to_remote = False
|
||||
ignore_fields = list(strategy.config.post_check_ignore_fields) if strategy else []
|
||||
ignore_fields = list(
|
||||
dict.fromkeys([
|
||||
*strategy.config.compare_ignore_fields,
|
||||
*strategy.config.post_check_ignore_fields,
|
||||
])
|
||||
)
|
||||
|
||||
result = await evaluate_consistency_for_node_type(
|
||||
node_type=node_type,
|
||||
strategy=strategy,
|
||||
local_collection=self.local_collection,
|
||||
remote_collection=self.remote_collection,
|
||||
binding_manager=self.binding_manager,
|
||||
logger=self._logger,
|
||||
normalize_compare_payload=strategy.normalize_compare_payload,
|
||||
depend_fields=depend_fields,
|
||||
compare_to_remote=compare_to_remote,
|
||||
ignore_fields=ignore_fields,
|
||||
@@ -402,7 +627,7 @@ class FullSyncPipeline:
|
||||
self._consistency_by_type[node_type] = result
|
||||
return result
|
||||
|
||||
async def commit_creates(self, node_type: str) -> bool:
|
||||
async def commit_creates(self, node_type: str) -> WriteOutcome:
|
||||
"""提交 CREATE 操作(包含异步轮询)"""
|
||||
# 执行同步(sync_all 内部会根据 action 筛选节点)
|
||||
await self.local_datasource.sync_all(
|
||||
@@ -427,8 +652,10 @@ class FullSyncPipeline:
|
||||
|
||||
local_success = self._get_action_success_count(self.local_datasource, node_type, "create")
|
||||
remote_success = self._get_action_success_count(self.remote_datasource, node_type, "create")
|
||||
wrote = (local_success + remote_success) > 0
|
||||
return wrote
|
||||
return WriteOutcome(
|
||||
local_written=local_success > 0,
|
||||
remote_written=remote_success > 0,
|
||||
)
|
||||
|
||||
async def _finalize_peer_creating_sources(self, node_type: str) -> None:
|
||||
finalized_success, finalized_failed = await finalize_peer_creating_sources(
|
||||
@@ -440,7 +667,7 @@ class FullSyncPipeline:
|
||||
)
|
||||
|
||||
if finalized_success or finalized_failed:
|
||||
self._logger.info(
|
||||
self._logger.debug(
|
||||
f"🔁 Peer-creating finalize: node_type={node_type}, success={finalized_success}, failed={finalized_failed}"
|
||||
)
|
||||
|
||||
@@ -463,9 +690,9 @@ class FullSyncPipeline:
|
||||
normalized += 1
|
||||
|
||||
if normalized > 0:
|
||||
self._logger.info(f"🔁 Post-create normalization: {normalized} nodes moved S11C->S01 for {node_type}")
|
||||
self._logger.debug(f"🔁 Post-create normalization: {normalized} nodes moved S11C->S01 for {node_type}")
|
||||
|
||||
async def commit_updates(self, node_type: str) -> bool:
|
||||
async def commit_updates(self, node_type: str) -> WriteOutcome:
|
||||
"""提交 UPDATE 操作(包含异步轮询)"""
|
||||
# 执行同步(sync_all 内部会根据 action 筛选节点)
|
||||
await self.local_datasource.sync_all(
|
||||
@@ -488,8 +715,10 @@ class FullSyncPipeline:
|
||||
|
||||
local_success = self._get_action_success_count(self.local_datasource, node_type, "update")
|
||||
remote_success = self._get_action_success_count(self.remote_datasource, node_type, "update")
|
||||
wrote = (local_success + remote_success) > 0
|
||||
return wrote
|
||||
return WriteOutcome(
|
||||
local_written=local_success > 0,
|
||||
remote_written=remote_success > 0,
|
||||
)
|
||||
|
||||
async def phase_post_check(self) -> bool:
|
||||
"""同步结果报告。
|
||||
@@ -502,8 +731,22 @@ class FullSyncPipeline:
|
||||
else:
|
||||
self._logger.info("🔄 Post-check: reloading all node types for final consistency evaluation...")
|
||||
for node_type in self.node_types:
|
||||
strategy = self._get_strategy(node_type)
|
||||
if strategy is not None and strategy.skip_post_check:
|
||||
self._logger.info("⏭️ Post-check reload skipped: node_type=%s", node_type)
|
||||
continue
|
||||
await self._reload_node_type(node_type, reason="post-check final reload")
|
||||
for node_type in self.node_types:
|
||||
strategy = self._get_strategy(node_type)
|
||||
if strategy is not None and strategy.skip_post_check:
|
||||
self._logger.info("⏭️ Post-check skipped: node_type=%s", node_type)
|
||||
self._consistency_by_type[node_type] = {
|
||||
"ok": True,
|
||||
"checked_pairs": 0,
|
||||
"mismatch_count": 0,
|
||||
"skipped": True,
|
||||
}
|
||||
continue
|
||||
await self._evaluate_consistency_for_node_type(node_type)
|
||||
|
||||
self.stats["consistency"] = {
|
||||
@@ -519,6 +762,18 @@ class FullSyncPipeline:
|
||||
|
||||
async def phase_persistence(self) -> None:
|
||||
"""持久化所有数据。"""
|
||||
if self.scope_runtime is not None:
|
||||
if not self.enable_persistence:
|
||||
self._logger.info("⏭️ Persistence disabled, skipping...")
|
||||
return
|
||||
self._logger.info("🔍 Persisting scope runtime...")
|
||||
await self.scope_runtime.persist(
|
||||
enable_persistence=self.enable_persistence,
|
||||
ephemeral_node_types=self.ephemeral_node_types,
|
||||
)
|
||||
self._logger.info("✅ All data persisted successfully")
|
||||
return
|
||||
|
||||
if not self.enable_persistence:
|
||||
self._logger.info("⏭️ Persistence disabled, skipping...")
|
||||
return
|
||||
|
||||
@@ -1,27 +1,34 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import copy
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional, List, TypedDict, Any
|
||||
from typing import Any, Dict, Optional, List
|
||||
|
||||
from ..config import (
|
||||
MultiProjectItemConfig,
|
||||
MultiProjectRunConfig,
|
||||
PipelineRunConfig,
|
||||
StrategyRuntimeConfig,
|
||||
apply_config_overrides,
|
||||
build_config_from_file,
|
||||
)
|
||||
from .factory import create_pipeline_from_config
|
||||
from ..logging import get_scope_logger, get_scope_display_name
|
||||
from .factory import create_pipeline_from_config, create_scope_runtime_from_config
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_scope_logger("multi-project", __name__)
|
||||
|
||||
|
||||
class _SharedStateSnapshot(TypedDict):
|
||||
local_nodes: List[Any]
|
||||
remote_nodes: List[Any]
|
||||
bindings_by_type: Dict[str, Dict[str, Optional[str]]]
|
||||
@dataclass(frozen=True)
|
||||
class ImplicitProjectItem:
|
||||
local_project_id: str
|
||||
remote_project_id: str
|
||||
|
||||
@property
|
||||
def scope(self) -> str:
|
||||
return f"project_id:{self.local_project_id}"
|
||||
|
||||
|
||||
def _resolve_embedded_config_path(raw_path: str, *, project_root: Path, profile_path: Path) -> Path:
|
||||
@@ -64,7 +71,215 @@ def _resolve_project_bootstrap_node_types(
|
||||
return list(dict.fromkeys(shared_node_types))
|
||||
|
||||
|
||||
class MultiProjectPipeline:
|
||||
def _normalize_project_id_items(raw_items: Any) -> list[str]:
|
||||
items = raw_items if isinstance(raw_items, list) else []
|
||||
normalized: list[str] = []
|
||||
for item in items:
|
||||
value = str(item).strip()
|
||||
if value and value not in normalized:
|
||||
normalized.append(value)
|
||||
return normalized
|
||||
|
||||
|
||||
def _get_project_handler_filters(config: PipelineRunConfig) -> tuple[list[str], list[str]]:
|
||||
local_filters = _normalize_project_id_items(
|
||||
config.local_datasource.handler_configs.get("project", {}).get("data_id_filter", [])
|
||||
)
|
||||
remote_filters = _normalize_project_id_items(
|
||||
config.remote_datasource.handler_configs.get("project", {}).get("data_id_filter", [])
|
||||
)
|
||||
return local_filters, remote_filters
|
||||
|
||||
|
||||
def _get_project_strategy_runtime(config: PipelineRunConfig) -> StrategyRuntimeConfig | None:
|
||||
for item in config.strategies:
|
||||
if item.node_type == "project":
|
||||
return item
|
||||
return None
|
||||
|
||||
|
||||
def _get_project_prebind_pairs(config: PipelineRunConfig) -> list[tuple[str, str]]:
|
||||
runtime_cfg = _get_project_strategy_runtime(config)
|
||||
if runtime_cfg is None:
|
||||
return []
|
||||
pairs: list[tuple[str, str]] = []
|
||||
for item in runtime_cfg.config.pre_bind_data_id:
|
||||
local_id = str(item.local_data_id).strip()
|
||||
remote_id = str(item.remote_data_id).strip()
|
||||
if not local_id or not remote_id:
|
||||
continue
|
||||
pairs.append((local_id, remote_id))
|
||||
return pairs
|
||||
|
||||
|
||||
def is_implicit_project_batch_config(config: PipelineRunConfig) -> bool:
|
||||
if "project" not in config.node_types:
|
||||
return False
|
||||
local_filters, remote_filters = _get_project_handler_filters(config)
|
||||
prebind_pairs = _get_project_prebind_pairs(config)
|
||||
return max(len(local_filters), len(remote_filters), len(prebind_pairs)) > 1
|
||||
|
||||
|
||||
def _extract_implicit_project_items(config: PipelineRunConfig) -> list[ImplicitProjectItem]:
|
||||
local_filters, remote_filters = _get_project_handler_filters(config)
|
||||
prebind_pairs = _get_project_prebind_pairs(config)
|
||||
if not prebind_pairs:
|
||||
raise ValueError("implicit multi-project config requires strategies.project.config.pre_bind_data_id")
|
||||
|
||||
pair_local_ids = [local_id for local_id, _ in prebind_pairs]
|
||||
pair_remote_ids = [remote_id for _, remote_id in prebind_pairs]
|
||||
|
||||
effective_local_filters = local_filters or list(dict.fromkeys(pair_local_ids))
|
||||
effective_remote_filters = remote_filters or list(dict.fromkeys(pair_remote_ids))
|
||||
|
||||
if set(effective_local_filters) != set(pair_local_ids):
|
||||
raise ValueError(
|
||||
"implicit multi-project config mismatch: local project data_id_filter must match project pre_bind_data_id local_data_id"
|
||||
)
|
||||
if set(effective_remote_filters) != set(pair_remote_ids):
|
||||
raise ValueError(
|
||||
"implicit multi-project config mismatch: remote project data_id_filter must match project pre_bind_data_id remote_data_id"
|
||||
)
|
||||
|
||||
seen_local_ids: set[str] = set()
|
||||
seen_remote_ids: set[str] = set()
|
||||
items: list[ImplicitProjectItem] = []
|
||||
for local_id, remote_id in prebind_pairs:
|
||||
if local_id in seen_local_ids:
|
||||
raise ValueError(f"duplicate local project id in implicit multi-project config: {local_id}")
|
||||
if remote_id in seen_remote_ids:
|
||||
raise ValueError(f"duplicate remote project id in implicit multi-project config: {remote_id}")
|
||||
seen_local_ids.add(local_id)
|
||||
seen_remote_ids.add(remote_id)
|
||||
items.append(ImplicitProjectItem(local_project_id=local_id, remote_project_id=remote_id))
|
||||
return items
|
||||
|
||||
|
||||
def _split_implicit_project_node_types(node_types: list[str]) -> tuple[list[str], list[str]]:
|
||||
if "project" not in node_types:
|
||||
raise ValueError("implicit multi-project config requires node_types to include 'project'")
|
||||
project_index = node_types.index("project")
|
||||
return node_types[:project_index], node_types[project_index:]
|
||||
|
||||
|
||||
def _build_implicit_project_config(
|
||||
base_config: PipelineRunConfig,
|
||||
*,
|
||||
item: ImplicitProjectItem,
|
||||
project_node_types: list[str],
|
||||
) -> PipelineRunConfig:
|
||||
return apply_config_overrides(
|
||||
base_config,
|
||||
{
|
||||
"node_types": project_node_types,
|
||||
"persist": {"wipe_on_start": False},
|
||||
"local_datasource": {
|
||||
"handler_configs": {
|
||||
"project": {"data_id_filter": [item.local_project_id]},
|
||||
}
|
||||
},
|
||||
"remote_datasource": {
|
||||
"handler_configs": {
|
||||
"project": {"data_id_filter": [item.remote_project_id]},
|
||||
}
|
||||
},
|
||||
"strategies": {
|
||||
"project": {
|
||||
"config": {
|
||||
"pre_bind_data_id": [
|
||||
{
|
||||
"local_data_id": item.local_project_id,
|
||||
"remote_data_id": item.remote_project_id,
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
async def run_implicit_project_batch_from_config(
|
||||
config: PipelineRunConfig,
|
||||
*,
|
||||
title: str,
|
||||
print_summary: bool = True,
|
||||
max_concurrency: int | None = None,
|
||||
) -> Dict[str, Dict[str, object]]:
|
||||
from .factory import create_pipeline_from_config, create_scope_runtime_from_config
|
||||
|
||||
project_items = _extract_implicit_project_items(config)
|
||||
shared_node_types, project_node_types = _split_implicit_project_node_types(list(config.node_types))
|
||||
results: Dict[str, Dict[str, object]] = {}
|
||||
|
||||
global_runtime = None
|
||||
global_pipeline = None
|
||||
global_config = None
|
||||
|
||||
if print_summary:
|
||||
logger.info("\n%s", "=" * 80)
|
||||
logger.info("🚀 Creating implicit multi-project pipeline from in-memory config")
|
||||
logger.info("📦 Projects=%d", len(project_items))
|
||||
logger.info("=" * 80)
|
||||
|
||||
try:
|
||||
if shared_node_types:
|
||||
global_config = apply_config_overrides(
|
||||
config,
|
||||
{
|
||||
"node_types": shared_node_types,
|
||||
},
|
||||
)
|
||||
global_runtime = await create_scope_runtime_from_config(global_config, scope="global")
|
||||
global_pipeline = await create_pipeline_from_config(
|
||||
global_config,
|
||||
scope="global",
|
||||
pipeline_name=f"{title} [global]",
|
||||
scope_runtime=global_runtime,
|
||||
)
|
||||
results["global"] = await global_pipeline.run()
|
||||
|
||||
semaphore = asyncio.Semaphore(max(1, max_concurrency or len(project_items)))
|
||||
|
||||
async def _run_project(item: ImplicitProjectItem) -> None:
|
||||
project_config = _build_implicit_project_config(
|
||||
config,
|
||||
item=item,
|
||||
project_node_types=project_node_types,
|
||||
)
|
||||
async with semaphore:
|
||||
project_runtime = await create_scope_runtime_from_config(
|
||||
project_config,
|
||||
scope=item.scope,
|
||||
persistence_service=global_runtime.persistence_service if global_runtime is not None else None,
|
||||
local_fallback_collection=global_runtime.local_collection if global_runtime is not None else None,
|
||||
remote_fallback_collection=global_runtime.remote_collection if global_runtime is not None else None,
|
||||
binding_fallback_manager=global_runtime.binding_manager if global_runtime is not None else None,
|
||||
global_runtime=global_runtime,
|
||||
)
|
||||
try:
|
||||
project_pipeline = await create_pipeline_from_config(
|
||||
project_config,
|
||||
scope=item.scope,
|
||||
pipeline_name=f"{title} [{item.scope}]",
|
||||
bootstrap_binding_node_types=shared_node_types,
|
||||
ephemeral_node_types=shared_node_types,
|
||||
scope_runtime=project_runtime,
|
||||
)
|
||||
results[item.scope] = await project_pipeline.run()
|
||||
finally:
|
||||
await project_runtime.unload()
|
||||
await project_runtime.close()
|
||||
|
||||
await asyncio.gather(*[_run_project(item) for item in project_items])
|
||||
return results
|
||||
finally:
|
||||
if global_runtime is not None:
|
||||
await global_runtime.unload()
|
||||
await global_runtime.close()
|
||||
|
||||
|
||||
class ProjectBatchPipeline:
|
||||
def __init__(self, *, project_root: Path, config_path: Path, config: MultiProjectRunConfig):
|
||||
self.project_root = project_root
|
||||
self.config_path = config_path
|
||||
@@ -86,27 +301,6 @@ class MultiProjectPipeline:
|
||||
cfg = apply_config_overrides(cfg, {"persist": {**shared_persist, "wipe_on_start": False}})
|
||||
return apply_config_overrides(cfg, _project_scope_overrides(item, node_types=list(cfg.node_types)))
|
||||
|
||||
async def _snapshot_shared_state(self, *, pipeline, node_types: List[str]) -> _SharedStateSnapshot:
|
||||
local_nodes = [
|
||||
copy.deepcopy(node)
|
||||
for node_type in node_types
|
||||
for node in pipeline.local_collection.filter(node_type=node_type)
|
||||
]
|
||||
remote_nodes = [
|
||||
copy.deepcopy(node)
|
||||
for node_type in node_types
|
||||
for node in pipeline.remote_collection.filter(node_type=node_type)
|
||||
]
|
||||
bindings_by_type: Dict[str, Dict[str, Optional[str]]] = {}
|
||||
for node_type in node_types:
|
||||
bindings = await pipeline.binding_manager.get_all_bindings(node_type)
|
||||
bindings_by_type[node_type] = {local_id: remote_id for local_id, remote_id in bindings}
|
||||
return {
|
||||
"local_nodes": local_nodes,
|
||||
"remote_nodes": remote_nodes,
|
||||
"bindings_by_type": bindings_by_type,
|
||||
}
|
||||
|
||||
async def run(self) -> Dict[str, Dict[str, object]]:
|
||||
results: Dict[str, Dict[str, object]] = {}
|
||||
|
||||
@@ -118,49 +312,67 @@ class MultiProjectPipeline:
|
||||
shared_node_types=shared_node_types,
|
||||
)
|
||||
logger.info("🚀 Multi-project pipeline start: projects=%d", len(self.config.projects))
|
||||
global_pipeline = await create_pipeline_from_config(
|
||||
global_runtime = await create_scope_runtime_from_config(
|
||||
global_config,
|
||||
scope="global",
|
||||
pipeline_name="multi-project pipeline [global]",
|
||||
)
|
||||
global_pipeline = None
|
||||
try:
|
||||
global_pipeline = await create_pipeline_from_config(
|
||||
global_config,
|
||||
scope="global",
|
||||
pipeline_name="multi-project pipeline [global]",
|
||||
scope_runtime=global_runtime,
|
||||
)
|
||||
results["global"] = await global_pipeline.run()
|
||||
shared_state = await self._snapshot_shared_state(
|
||||
pipeline=global_pipeline,
|
||||
node_types=project_bootstrap_node_types,
|
||||
)
|
||||
finally:
|
||||
await global_pipeline.close()
|
||||
semaphore = asyncio.Semaphore(self.config.max_concurrency)
|
||||
|
||||
for item in self.config.projects:
|
||||
project_config = self._prepare_project_config(
|
||||
shared_persist=shared_persist,
|
||||
item=item,
|
||||
base_path=item.config_path,
|
||||
)
|
||||
scope = item.scope
|
||||
title = f"multi-project pipeline [{scope}]"
|
||||
logger.info(
|
||||
"🚀 Project pipeline start: scope=%s local_project_id=%s remote_project_id=%s",
|
||||
scope,
|
||||
item.local_project_id,
|
||||
item.remote_project_id,
|
||||
)
|
||||
project_pipeline = await create_pipeline_from_config(
|
||||
project_config,
|
||||
scope=scope,
|
||||
pipeline_name=title,
|
||||
bootstrap_binding_node_types=project_bootstrap_node_types,
|
||||
ephemeral_node_types=project_bootstrap_node_types,
|
||||
)
|
||||
try:
|
||||
project_pipeline.preload_shared_state(
|
||||
local_nodes=shared_state["local_nodes"],
|
||||
remote_nodes=shared_state["remote_nodes"],
|
||||
bindings_by_type=shared_state["bindings_by_type"],
|
||||
async def _run_project(item: MultiProjectItemConfig) -> None:
|
||||
project_config = self._prepare_project_config(
|
||||
shared_persist=shared_persist,
|
||||
item=item,
|
||||
base_path=item.config_path,
|
||||
)
|
||||
results[scope] = await project_pipeline.run()
|
||||
finally:
|
||||
await project_pipeline.close()
|
||||
scope = item.scope
|
||||
title = f"multi-project pipeline [{scope}]"
|
||||
logger.info(
|
||||
"🚀 Project pipeline start: scope=%s local_project_id=%s remote_project_id=%s",
|
||||
get_scope_display_name(scope),
|
||||
item.local_project_id,
|
||||
item.remote_project_id,
|
||||
)
|
||||
async with semaphore:
|
||||
project_runtime = await create_scope_runtime_from_config(
|
||||
project_config,
|
||||
scope=scope,
|
||||
persistence_service=global_runtime.persistence_service,
|
||||
local_fallback_collection=global_runtime.local_collection,
|
||||
remote_fallback_collection=global_runtime.remote_collection,
|
||||
binding_fallback_manager=global_runtime.binding_manager,
|
||||
global_runtime=global_runtime,
|
||||
)
|
||||
project_pipeline = None
|
||||
try:
|
||||
project_pipeline = await create_pipeline_from_config(
|
||||
project_config,
|
||||
scope=scope,
|
||||
pipeline_name=title,
|
||||
bootstrap_binding_node_types=project_bootstrap_node_types,
|
||||
ephemeral_node_types=project_bootstrap_node_types,
|
||||
scope_runtime=project_runtime,
|
||||
)
|
||||
results[scope] = await project_pipeline.run()
|
||||
finally:
|
||||
await project_runtime.unload()
|
||||
await project_runtime.close()
|
||||
|
||||
return results
|
||||
await asyncio.gather(*[_run_project(item) for item in self.config.projects])
|
||||
finally:
|
||||
await global_runtime.unload()
|
||||
await global_runtime.close()
|
||||
|
||||
return results
|
||||
|
||||
|
||||
class MultiProjectPipeline(ProjectBatchPipeline):
|
||||
pass
|
||||
@@ -0,0 +1,102 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from typing import List, Optional
|
||||
|
||||
from ..common.binding import BindingManager
|
||||
from ..common.collection import DataCollection
|
||||
from ..common.persistence_service import PersistenceService, ScopedPersistenceView
|
||||
|
||||
|
||||
class ProjectScopeRuntime:
|
||||
"""Thin runtime container for one scope execution."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
scope: str,
|
||||
persistence_service: PersistenceService,
|
||||
persistence_view: ScopedPersistenceView,
|
||||
local_collection: DataCollection,
|
||||
remote_collection: DataCollection,
|
||||
binding_manager: BindingManager,
|
||||
local_datasource,
|
||||
remote_datasource,
|
||||
global_runtime: Optional["ProjectScopeRuntime"] = None,
|
||||
owns_persistence_service: bool = False,
|
||||
) -> None:
|
||||
self.scope = scope
|
||||
self.persistence_service = persistence_service
|
||||
self.persistence_view = persistence_view
|
||||
self.local_collection = local_collection
|
||||
self.remote_collection = remote_collection
|
||||
self.binding_manager = binding_manager
|
||||
self.local_datasource = local_datasource
|
||||
self.remote_datasource = remote_datasource
|
||||
self.global_runtime = global_runtime
|
||||
self.owns_persistence_service = owns_persistence_service
|
||||
self._closed = False
|
||||
|
||||
@property
|
||||
def persistence_backend(self):
|
||||
return self.persistence_service.backend
|
||||
|
||||
async def load(
|
||||
self,
|
||||
*,
|
||||
node_types: List[str],
|
||||
bootstrap_binding_node_types: Optional[List[str]] = None,
|
||||
enable_persistence: bool = True,
|
||||
ephemeral_node_types: Optional[List[str]] = None,
|
||||
) -> None:
|
||||
if not enable_persistence:
|
||||
return
|
||||
|
||||
excluded_node_types = list(dict.fromkeys(ephemeral_node_types or [])) or None
|
||||
await self.local_collection.load_from_persistence_excluding(excluded_node_types)
|
||||
await self.remote_collection.load_from_persistence_excluding(excluded_node_types)
|
||||
|
||||
excluded = set(ephemeral_node_types or [])
|
||||
binding_node_types = [
|
||||
node_type
|
||||
for node_type in dict.fromkeys([*node_types, *(bootstrap_binding_node_types or [])])
|
||||
if node_type not in excluded
|
||||
]
|
||||
for node_type in binding_node_types:
|
||||
await self.binding_manager.load_from_persistence(node_type)
|
||||
|
||||
async def persist(
|
||||
self,
|
||||
*,
|
||||
enable_persistence: bool = True,
|
||||
ephemeral_node_types: Optional[List[str]] = None,
|
||||
) -> None:
|
||||
if not enable_persistence:
|
||||
return
|
||||
|
||||
await self.local_collection.persist(exclude_node_types=ephemeral_node_types)
|
||||
await self.remote_collection.persist(exclude_node_types=ephemeral_node_types)
|
||||
await self.binding_manager.persist(exclude_node_types=ephemeral_node_types)
|
||||
|
||||
async def unload(self) -> None:
|
||||
await self.local_collection.unload()
|
||||
await self.remote_collection.unload()
|
||||
await self.binding_manager.unload()
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._closed:
|
||||
return
|
||||
self._closed = True
|
||||
|
||||
await self._safe_close("remote_datasource", self.remote_datasource.close())
|
||||
await self._safe_close("local_datasource", self.local_datasource.close())
|
||||
if self.owns_persistence_service:
|
||||
await self._safe_close("persistence_service", self.persistence_service.close())
|
||||
|
||||
async def _safe_close(self, name: str, close_coro, timeout: float = 3.0) -> None:
|
||||
try:
|
||||
await asyncio.wait_for(close_coro, timeout=timeout)
|
||||
except asyncio.TimeoutError:
|
||||
return
|
||||
except Exception:
|
||||
return
|
||||
@@ -1,18 +1,9 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
class _NodeLogFilter(logging.Filter):
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
message = record.getMessage()
|
||||
if "node=" in message:
|
||||
return False
|
||||
if "节点 " in message and "状态" in message:
|
||||
return False
|
||||
return True
|
||||
from ..logging import clear_app_logger_handlers, init_app_logger
|
||||
|
||||
|
||||
def init_pipeline_logger(
|
||||
@@ -23,43 +14,14 @@ def init_pipeline_logger(
|
||||
replace_handlers: bool = True,
|
||||
suppress_node_logs: bool = True,
|
||||
) -> logging.Logger:
|
||||
logger = logging.getLogger("sync_state_machine")
|
||||
logger.setLevel(level)
|
||||
logger.propagate = False
|
||||
|
||||
if replace_handlers:
|
||||
clear_pipeline_logger_handlers(logger)
|
||||
|
||||
formatter = logging.Formatter("%(message)s")
|
||||
node_filter = _NodeLogFilter() if suppress_node_logs else None
|
||||
|
||||
if mirror_console:
|
||||
console_handler = logging.StreamHandler()
|
||||
console_handler.setLevel(level)
|
||||
console_handler.setFormatter(formatter)
|
||||
if node_filter is not None:
|
||||
console_handler.addFilter(node_filter)
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
if log_file_path:
|
||||
path = Path(log_file_path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
file_handler = logging.FileHandler(path, mode="w", encoding="utf-8")
|
||||
file_handler.setLevel(level)
|
||||
file_handler.setFormatter(formatter)
|
||||
if node_filter is not None:
|
||||
file_handler.addFilter(node_filter)
|
||||
logger.addHandler(file_handler)
|
||||
|
||||
return logger
|
||||
return init_app_logger(
|
||||
log_file_path=log_file_path,
|
||||
level=level,
|
||||
mirror_console=mirror_console,
|
||||
replace_handlers=replace_handlers,
|
||||
suppress_node_logs=suppress_node_logs,
|
||||
)
|
||||
|
||||
|
||||
def clear_pipeline_logger_handlers(logger: Optional[logging.Logger] = None) -> None:
|
||||
target = logger or logging.getLogger("sync_state_machine")
|
||||
handlers = list(target.handlers)
|
||||
for handler in handlers:
|
||||
target.removeHandler(handler)
|
||||
try:
|
||||
handler.close()
|
||||
except Exception:
|
||||
pass
|
||||
clear_app_logger_handlers(logger)
|
||||
|
||||
@@ -1,7 +1,18 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
|
||||
async def print_pipeline_summary(*, logger, node_types, binding_manager, local_datasource, remote_datasource, local_collection, remote_collection, consistency_by_type, stats) -> None:
|
||||
scoped_logger = logger
|
||||
plain_logger = logger.logger if isinstance(logger, logging.LoggerAdapter) else logger
|
||||
|
||||
def _log_header(message: str) -> None:
|
||||
scoped_logger.info(message)
|
||||
|
||||
def _log_body(message: str) -> None:
|
||||
plain_logger.info(message)
|
||||
|
||||
binding_counts = {}
|
||||
for node_type in node_types:
|
||||
records = await binding_manager.get_all_records(node_type)
|
||||
@@ -36,31 +47,31 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
|
||||
remote_text = _clip(repr(sample.get("remote")), 72)
|
||||
return f"{local_text}/{remote_text}"
|
||||
|
||||
logger.info(f"\n{'=' * line_width}")
|
||||
logger.info("🔎 Consistency Summary")
|
||||
logger.info(f"{'=' * line_width}")
|
||||
_log_header(f"\n{'=' * line_width}")
|
||||
_log_header("🔎 Consistency Summary")
|
||||
_log_header(f"{'=' * line_width}")
|
||||
|
||||
for node_type in node_types:
|
||||
item = consistency_by_type.get(node_type, {}) or {}
|
||||
checked_pairs = int(item.get("checked_pairs", 0))
|
||||
mismatch_count = int(item.get("mismatch_count", 0))
|
||||
logger.info(f"{node_type} (checked_pairs={checked_pairs}, mismatches={mismatch_count})")
|
||||
_log_body(f"{node_type} (checked_pairs={checked_pairs}, mismatches={mismatch_count})")
|
||||
|
||||
fields = item.get("fields", []) or []
|
||||
differing_fields = [field for field in fields if int(field.get("mismatch_count", 0)) > 0]
|
||||
if not differing_fields:
|
||||
logger.info(" pass")
|
||||
_log_body(" pass")
|
||||
continue
|
||||
|
||||
logger.info(" field mismatch/total samples")
|
||||
logger.info(" -----------------------------------------------------------")
|
||||
_log_body(" field mismatch/total samples")
|
||||
_log_body(" -----------------------------------------------------------")
|
||||
for field_report in differing_fields:
|
||||
field_name = str(field_report.get("field_name", ""))
|
||||
field_mismatch = int(field_report.get("mismatch_count", 0))
|
||||
field_total = int(field_report.get("total_count", 0))
|
||||
samples = field_report.get("samples", []) or []
|
||||
sample_text = _format_sample(samples[0]) if samples else "-"
|
||||
logger.info(
|
||||
_log_body(
|
||||
f" {_clip(field_name, 30):<30} {field_mismatch:>4}/{field_total:<4} "
|
||||
f"{sample_text}"
|
||||
)
|
||||
@@ -74,15 +85,15 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
|
||||
action_w = 16
|
||||
consistency_w = 18
|
||||
|
||||
logger.info(f"\n{'='*line_width}")
|
||||
logger.info(f"📊 {title} Summary")
|
||||
logger.info(f"{'='*line_width}")
|
||||
logger.info("说明: Create/Update/Delete = 成功/失败/总数 (失败=FAILED+SKIPPED)")
|
||||
logger.info("说明: Consistent = 一致对数/不一致对数/已检查绑定对数")
|
||||
logger.info(
|
||||
_log_header(f"\n{'='*line_width}")
|
||||
_log_header(f"📊 {title} Summary")
|
||||
_log_header(f"{'='*line_width}")
|
||||
_log_body("说明: Create/Update/Delete = 成功/失败/总数 (失败=FAILED+SKIPPED)")
|
||||
_log_body("说明: Consistent = 一致对数/不一致对数/已检查绑定对数")
|
||||
_log_body(
|
||||
f"{'Node Type':<{node_type_w}} {'Bound/Rec/Load':<{bound_w}} {'Create(S/F/T)':<{action_w}} {'Update(S/F/T)':<{action_w}} {'Delete(S/F/T)':<{action_w}} {'Consistent(S/F/T)':<{consistency_w}}"
|
||||
)
|
||||
logger.info(f"{'-'*line_width}")
|
||||
_log_body(f"{'-'*line_width}")
|
||||
|
||||
total_bound_normal = 0
|
||||
total_bindings = 0
|
||||
@@ -105,7 +116,7 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
|
||||
consistency_str = format_consistency_sft(node_type)
|
||||
|
||||
bound_str = f"{bound_normal_count}/{bindings}/{loaded_count}"
|
||||
logger.info(
|
||||
_log_body(
|
||||
f"{_clip(node_type, node_type_w):<{node_type_w}} "
|
||||
f"{_clip(bound_str, bound_w):<{bound_w}} "
|
||||
f"{_clip(create_str, action_w):<{action_w}} "
|
||||
@@ -131,14 +142,14 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
|
||||
total_consistency["checked"] += int(consistency.get("checked_pairs", 0))
|
||||
total_consistency["mismatch"] += int(consistency.get("mismatch_count", 0))
|
||||
|
||||
logger.info(f"{'-'*line_width}")
|
||||
_log_body(f"{'-'*line_width}")
|
||||
total_consistent = max(0, total_consistency["checked"] - total_consistency["mismatch"])
|
||||
total_create_sft = f"{total_create['success']}/{total_create['failed_effective']}/{total_create['total']}"
|
||||
total_update_sft = f"{total_update['success']}/{total_update['failed_effective']}/{total_update['total']}"
|
||||
total_delete_sft = f"{total_delete['success']}/{total_delete['failed_effective']}/{total_delete['total']}"
|
||||
total_consistency_sft = f"{total_consistent}/{total_consistency['mismatch']}/{total_consistency['checked']}"
|
||||
total_bound_str = f"{total_bound_normal}/{total_bindings}/{total_loaded}"
|
||||
logger.info(
|
||||
_log_body(
|
||||
f"{'TOTAL':<{node_type_w}} "
|
||||
f"{_clip(total_bound_str, bound_w):<{bound_w}} "
|
||||
f"{_clip(total_create_sft, action_w):<{action_w}} "
|
||||
@@ -158,13 +169,13 @@ async def print_pipeline_summary(*, logger, node_types, binding_manager, local_d
|
||||
failed = int(stats.get("failed", 0))
|
||||
post_check_passed = bool(stats.get("post_check_passed", True))
|
||||
|
||||
logger.info("\n" + "=" * 96)
|
||||
logger.info(
|
||||
_log_header("\n" + "=" * 96)
|
||||
_log_header(
|
||||
f"📈 Pipeline Stats: loaded={loaded}, synced={synced}, failed={failed}, post_check_passed={post_check_passed}, mismatch_types={len(mismatch_node_types)}"
|
||||
)
|
||||
if mismatch_node_types:
|
||||
logger.info(f"🔎 Mismatch Node Types: {', '.join(mismatch_node_types)}")
|
||||
logger.info("=" * 96)
|
||||
_log_body(f"🔎 Mismatch Node Types: {', '.join(mismatch_node_types)}")
|
||||
_log_header("=" * 96)
|
||||
|
||||
print_consistency_summary()
|
||||
print_collection_summary("Local", local_datasource, local_collection)
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user