整理配置和代码入口,增加部分测试。

This commit is contained in:
strepsiades
2026-03-10 12:22:54 +08:00
parent 515c3d9388
commit a638e4203b
35 changed files with 1314 additions and 199 deletions
+2 -1
View File
@@ -94,4 +94,5 @@ run_profiles/*.yaml
run_profiles/*.yml run_profiles/*.yml
# Sync System Runtime Data # Sync System Runtime Data
_runtime/ _runtime/
test_results/ test_results/
*.code-workspace
+147
View File
@@ -0,0 +1,147 @@
# Factory / Registry / Adapter 初始化流程改造方案
## 目标
这次改造只做三件事:
1. 简化 `factory`,让它只负责编排,不再关心过多介质细节。
2. 统一 handler 初始化流程,让 `jsonl` / `db` / `api` 三类路径走同一套装配逻辑。
3. 让适配层尽量极简:适配层只补充少量 handler / datasource 实现,不再复制初始化流程。
## 当前问题
### 1. `factory` 里存在大量按类型分支的重复逻辑
当前 [sync_state_machine/pipeline/factory.py](sync_state_machine/pipeline/factory.py) 同时负责:
- 创建 local / remote datasource
- 判断 datasource 类型
- 选择不同 handler getter
- 注入一部分 handler 配置
- 注册 handler
这些逻辑对 local / remote 各写了一遍,重复明显。
### 2. `DomainRegistry` 暴露了过多介质细节
当前 registry 暴露:
- `get_jsonl_handler()`
- `get_db_handler()`
- `get_api_handler()`
这导致编排层必须知道 handler 是哪一类,从而在 `factory` 里出现大量 `if isinstance(...)`
### 3. handler 配置注入不统一
目前只有 API handler 在 factory 里显式吃到了 `handler_configs`
- `api`:支持 per-side 配置
- `db`:当前没有统一注入入口
- `jsonl`:当前也没有统一注入入口
这使得“同一类 datasource,两侧配置不同”只能在部分路径下工作。
### 4. 适配层初始化成本偏高
如果适配层要补充一种 handler,目前需要理解:
- factory 的类型分支
- registry 的不同 getter
- 某些配置为什么只在 API 生效
这和“适配层极简”的目标不一致。
## 本次改造范围
本次只做最小闭环改造,不碰更大的业务逻辑:
### Step 1. 统一 `DomainRegistry` 的 handler 查询接口
状态:已完成
新增统一接口:
- `get_handler(node_type, datasource_type)`
保留旧接口作为兼容壳,但 factory 新逻辑只使用统一接口。
### Step 2. 给所有 handler 基类增加统一配置注入入口
状态:已完成
新增统一入口:
- `set_handler_config(config)`
这样 `jsonl` / `db` / `api` 三类 handler 都可以在实例化后走同一套配置注入流程。
### Step 3. 重写 factory 的 datasource / handler 初始化流程
状态:已完成
把当前分散的:
- `_create_datasources()`
- `_register_handlers()`
- `_api_handler_kwargs()`
收敛成按 endpoint 的统一初始化流程:
- endpoint 创建 datasource
- endpoint 根据 datasource type 向 registry 查询 handler class
- endpoint 创建 handler
- endpoint 统一注入:
- `handler_config`
- `target_project_ids`
- datasource-specific runtime object(如 api client
最终让 `factory` 只负责:
- persistence 初始化
- endpoint 初始化
- strategy 初始化
- pipeline 组装
## 不在本次范围内
以下内容先不动,只在文档中记为后续方向:
1. 显式 bootstrap 内建 domain 与适配层 domain 的注册时机
2. 用更强的 `EndpointConfig` / `DataSourceDriver` 抽象进一步收敛 factory
3. 清理 `NodeHandlerRegistry` 这类历史遗留接口
## 测试策略
每一步都补对应 pytest,并保证通过:
### Step 1 测试
- registry 可以通过统一接口拿到对应 handler class
- 旧接口仍可兼容使用
### Step 2 测试
- handler 可以收到统一的 `handler_config`
- 同一 datasource 类型在 local / remote 两侧可以拿到不同配置
### Step 3 测试
- factory 不再依赖 `get_jsonl_handler/get_db_handler/get_api_handler`
- 统一 endpoint 初始化路径可正常创建并注册 handler
- 原有 db handler 注册测试仍通过
## 预期结果
改造完成后:
1. `factory` 中关于 datasource / handler 初始化的重复逻辑显著减少。
2. `DomainRegistry` 对编排层只暴露统一 handler 查询方式。
3. handler 配置注入完全统一。
4. 适配层只需:
- 提供 datasource(如需要)
- 提供 handler
- 注册 handler
- 调用统一 runner
这就是“适配极简”的基础。
+8 -3
View File
@@ -296,9 +296,14 @@ def build_temp_run_profile_for_project_auto_bind(
profile.setdefault("remote_datasource", {})["target_project_ids"] = [remote_project_id] profile.setdefault("remote_datasource", {})["target_project_ids"] = [remote_project_id]
project_strategy = profile.setdefault("strategies", {}).setdefault("project", {}).setdefault("config", {}) project_strategy = profile.setdefault("strategies", {}).setdefault("project", {}).setdefault("config", {})
project_strategy["auto_bind"] = True project_strategy["auto_bind"] = False
project_strategy["auto_bind_fields"] = ["name"] project_strategy["auto_bind_fields"] = []
project_strategy["pre_bind_data_id"] = [] project_strategy["pre_bind_data_id"] = [
{
"local_data_id": local_project_id,
"remote_data_id": remote_project_id,
}
]
project_strategy["local_orphan_action"] = project_local_orphan_action or "none" project_strategy["local_orphan_action"] = project_local_orphan_action or "none"
project_strategy["remote_orphan_action"] = project_remote_orphan_action or "none" project_strategy["remote_orphan_action"] = project_remote_orphan_action or "none"
project_strategy["update_direction"] = project_update_direction or "none" project_strategy["update_direction"] = project_update_direction or "none"
+1 -7
View File
@@ -141,9 +141,6 @@ def run_case(config: dict[str, Any], report_root) -> dict[str, Any]:
"investment_decision_date", "investment_decision_date",
"complete_investment", "complete_investment",
"dynamic_investment", "dynamic_investment",
"implementation_estimate",
"static_total_investment",
"completed_investment",
], ],
} }
}, },
@@ -164,9 +161,6 @@ def run_case(config: dict[str, Any], report_root) -> dict[str, Any]:
"investment_decision_date": "2021-08-15", "investment_decision_date": "2021-08-15",
"complete_investment": 125000.0, "complete_investment": 125000.0,
"dynamic_investment": 1413000.5, "dynamic_investment": 1413000.5,
"implementation_estimate": 1225000.0,
"static_total_investment": 1224300.0,
"completed_investment": 194500.0,
} }
} }
} }
@@ -180,7 +174,7 @@ def run_case(config: dict[str, Any], report_root) -> dict[str, Any]:
"PUT /project/dynamic_investment", "PUT /project/dynamic_investment",
"PUT /project/settlement_investment", "PUT /project/settlement_investment",
], ],
"notes": "单轮覆盖 project 剩余 4 条投资类更新接口。", "notes": "单轮覆盖 project 剩余 4 条投资类更新接口。当前回归仅校验后端可稳定读回的投资决策/完成投资/动态投资字段;结算投资接口仍保留覆盖,但扩展字段读回语义以后端修正为准。",
}, },
) )
rounds.append(round_3) rounds.append(round_3)
@@ -15,12 +15,14 @@ def run_case(config: dict[str, Any], report_root) -> dict[str, Any]:
- `actual_construction` → `PUT /project/actual_construction` - `actual_construction` → `PUT /project/actual_construction`
- `ensure_production` → `PUT /project/ensure_capacity` - `ensure_production` → `PUT /project/ensure_capacity`
- `climb_production` → `PUT /project/climb_capacity` - `climb_production` → `PUT /project/climb_capacity`
- `challenge_production` → `PUT /project/challenge_capacity` - `challenge_production` → `PUT /project/challenge_capacity`
- `production` → `PUT /project/production_capacity`
其中 `production` 在当前水电项目上主要验证“推送后 project 本身字段变更”; `ensure/climb/challenge` 默认策略是 pull,本用例会在临时 profile 中仅对当前 key 切到 push
`ensure/climb/challenge` 默认策略是 pull,本用例会在临时 profile 中仅对当前 key 切到 push
以确认对应后端更新接口真实可用,并在报告中展示字段校验结果。 以确认对应后端更新接口真实可用,并在报告中展示字段校验结果。
`production` 当前按文档约束不混入这组统一回归:
后端确认该接口会直接更新 `project` 字段,但不保证创建/更新可轮询的
`project_detail_data.production` 记录,需单列方案再测。
""" """
dataset_root = f"{config['project_root']}/tests/fixtures/auto_sync/datasource/project_detail_data" dataset_root = f"{config['project_root']}/tests/fixtures/auto_sync/datasource/project_detail_data"
@@ -31,14 +33,12 @@ def run_case(config: dict[str, Any], report_root) -> dict[str, Any]:
] ]
remote_project_id = "f3e02e84-e81c-4aa6-88d4-f5aa108c8227" remote_project_id = "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
remote_actual_construction_id = "c99ae2e4-809e-47dd-8b86-61674b6a5bea" remote_actual_construction_id = "c99ae2e4-809e-47dd-8b86-61674b6a5bea"
remote_production_id = "0b3f7e2d-9dc7-4f71-aac9-c9e7093df64f"
seed_ensure_production_id = "11111111-1111-4111-8111-111111111111" seed_ensure_production_id = "11111111-1111-4111-8111-111111111111"
seed_climb_production_id = "22222222-2222-4222-8222-222222222222" seed_climb_production_id = "22222222-2222-4222-8222-222222222222"
seed_challenge_production_id = "33333333-3333-4333-8333-333333333333" seed_challenge_production_id = "33333333-3333-4333-8333-333333333333"
seed_plan_construction_id = "4bc5e9db-c9dc-41f1-a77d-574b4fe37b01" seed_plan_construction_id = "4bc5e9db-c9dc-41f1-a77d-574b4fe37b01"
local_project_id = make_local_data_id("project", remote_project_id) local_project_id = make_local_data_id("project", remote_project_id)
local_actual_construction_id = make_local_data_id("project_detail_data", remote_actual_construction_id) local_actual_construction_id = make_local_data_id("project_detail_data", remote_actual_construction_id)
local_production_id = make_local_data_id("project_detail_data", remote_production_id)
local_plan_construction_id = make_local_data_id("project_detail_data", seed_plan_construction_id) local_plan_construction_id = make_local_data_id("project_detail_data", seed_plan_construction_id)
local_ensure_production_id = make_local_data_id("project_detail_data", seed_ensure_production_id) local_ensure_production_id = make_local_data_id("project_detail_data", seed_ensure_production_id)
local_climb_production_id = make_local_data_id("project_detail_data", seed_climb_production_id) local_climb_production_id = make_local_data_id("project_detail_data", seed_climb_production_id)
@@ -365,63 +365,8 @@ def run_case(config: dict[str, Any], report_root) -> dict[str, Any]:
) )
rounds.append(round_7) rounds.append(round_7)
round_4 = run_pipeline_round(
config=config,
dataset_dir=f"{dataset_root}/round_4_production_update",
previous_dataset_dir=f"{dataset_root}/round_3_plan_construction_create",
base_dataset_dir=filtered_root,
base_dataset_files=base_dataset_files,
domain_name="project_detail_data",
round_name="round_4_production_update",
purpose="推送 production,验证 /project/production_capacity 在当前水电项目上会更新 project 投产计划字段",
expected_changes={
"project": {
local_project_id: [
"plan_production_date",
"plan_full_production_date",
],
}
},
required_bindings={
"project": [local_project_id],
"project_detail_data": [local_production_id],
},
expected_binding_pairs={
"project": {local_project_id: remote_project_id},
},
require_distinct_binding_ids={
"project": [local_project_id],
"project_detail_data": [local_production_id],
},
expected_bound_fields={
"remote": {
"project": {
local_project_id: {
"plan_production_date": "2026-04-01",
"plan_full_production_date": "2026-06-01",
}
},
"project_detail_data": {
local_production_id: {
"key": "production",
"value.0.capacity": 51.0,
"value.0.date": "2025-12-27",
}
},
}
},
project_id_remap=project_id_remap,
report_root=report_root,
coverage={
"keys": ["production"],
"api": ["PUT /project/production_capacity"],
"notes": "在可访问项目上复用现有 production 明细记录,验证生产接口会更新 project 投产计划字段并保持 detail 绑定可解析。",
},
)
rounds.append(round_4)
return build_domain_report( return build_domain_report(
domain="project_detail_data", domain="project_detail_data",
description="项目详情 domain 回归。覆盖 plan_construction / actual_construction / ensure_production / climb_production / challenge_production / production 六类 key,并在报告中展示对应接口与 project 字段联动校验。", description="项目详情 domain 回归。覆盖 plan_construction / actual_construction / ensure_production / climb_production / challenge_production 类 key,并在报告中展示对应接口与 project 字段联动校验。production 按文档约束单列,不混入本组统一回归。",
rounds=rounds, rounds=rounds,
) )
+5 -1
View File
@@ -14,6 +14,7 @@ from . import domain as _domain
from .common.registry import DomainRegistry from .common.registry import DomainRegistry
from .config import ( from .config import (
ApiDataSourceConfig, ApiDataSourceConfig,
DbDataSourceConfig,
JsonlDataSourceConfig, JsonlDataSourceConfig,
PipelineRunConfig, PipelineRunConfig,
build_config, build_config,
@@ -21,7 +22,7 @@ from .config import (
load_named_preset_overrides, load_named_preset_overrides,
load_overrides_from_file, load_overrides_from_file,
) )
from .datasource import ApiClient, ApiDataSource, BaseApiHandler, BaseJsonlHandler, JsonlDataSource from .datasource import ApiClient, ApiDataSource, BaseApiHandler, BaseDbHandler, BaseJsonlHandler, DbDataSource, JsonlDataSource
from .pipeline import FullSyncPipeline, create_pipeline_from_config, run_pipeline_from_config from .pipeline import FullSyncPipeline, create_pipeline_from_config, run_pipeline_from_config
try: try:
@@ -41,6 +42,7 @@ __all__ = [
"DomainRegistry", "DomainRegistry",
"PipelineRunConfig", "PipelineRunConfig",
"ApiDataSourceConfig", "ApiDataSourceConfig",
"DbDataSourceConfig",
"JsonlDataSourceConfig", "JsonlDataSourceConfig",
"build_config", "build_config",
"build_default_config", "build_default_config",
@@ -48,8 +50,10 @@ __all__ = [
"load_named_preset_overrides", "load_named_preset_overrides",
"ApiClient", "ApiClient",
"ApiDataSource", "ApiDataSource",
"DbDataSource",
"JsonlDataSource", "JsonlDataSource",
"BaseApiHandler", "BaseApiHandler",
"BaseDbHandler",
"BaseJsonlHandler", "BaseJsonlHandler",
"FullSyncPipeline", "FullSyncPipeline",
"create_pipeline_from_config", "create_pipeline_from_config",
+24 -24
View File
@@ -26,7 +26,7 @@ Domain 注册中心
# DataSource 查询 Handler # DataSource 查询 Handler
handler_class = DomainRegistry.get_jsonl_handler("contract") handler_class = DomainRegistry.get_jsonl_handler("contract")
""" """
from typing import Dict, Type, Optional, Any from typing import Dict, Type, Optional, Any, Literal
from dataclasses import dataclass, field from dataclasses import dataclass, field
from pydantic import BaseModel from pydantic import BaseModel
@@ -64,6 +64,12 @@ class DomainRegistry:
""" """
_registry: Dict[str, DomainRegistration] = {} _registry: Dict[str, DomainRegistration] = {}
_HANDLER_ATTRS: Dict[str, str] = {
"jsonl": "jsonl_handler_class",
"api": "api_handler_class",
"db": "db_handler_class",
}
@classmethod @classmethod
def register( def register(
@@ -143,36 +149,30 @@ class DomainRegistry:
return reg.strategy_class return reg.strategy_class
@classmethod @classmethod
def get_jsonl_handler(cls, node_type: str) -> Optional[Type[Any]]: def get_handler(
""" cls,
获取 JSONL Handler 类(用于 DataSource node_type: str,
datasource_type: Literal["jsonl", "api", "db"],
Args: ) -> Optional[Type[Any]]:
node_type: 节点类型 """按 datasource 类型获取 Handler 类。"""
Returns:
Handler 类,如果未注册则返回 None
"""
reg = cls._registry.get(node_type) reg = cls._registry.get(node_type)
if not reg: if not reg:
raise ValueError(f"Unknown node_type: '{node_type}'") raise ValueError(f"Unknown node_type: '{node_type}'")
return reg.jsonl_handler_class attr_name = cls._HANDLER_ATTRS.get(datasource_type)
if attr_name is None:
raise ValueError(
f"Unsupported datasource_type: '{datasource_type}'. "
f"Available: {sorted(cls._HANDLER_ATTRS.keys())}"
)
return getattr(reg, attr_name)
@classmethod @classmethod
def get_api_handler(cls, node_type: str) -> Optional[Type[Any]]: def register_db_handler(cls, node_type: str, db_handler_class: Type[Any]) -> None:
"""获取 API Handler""" """为已注册 domain 补充 DB Handler"""
reg = cls._registry.get(node_type) reg = cls._registry.get(node_type)
if not reg: if not reg:
raise ValueError(f"Unknown node_type: '{node_type}'") raise ValueError(f"Unknown node_type: '{node_type}'")
return reg.api_handler_class reg.db_handler_class = db_handler_class
@classmethod
def get_db_handler(cls, node_type: str) -> Optional[Type[Any]]:
"""获取 DB Handler 类"""
reg = cls._registry.get(node_type)
if not reg:
raise ValueError(f"Unknown node_type: '{node_type}'")
return reg.db_handler_class
@classmethod @classmethod
def get_schema(cls, node_type: str) -> Type[BaseModel]: def get_schema(cls, node_type: str) -> Type[BaseModel]:
+2 -1
View File
@@ -1,4 +1,4 @@
from .datasource_config import ApiDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig from .datasource_config import ApiDataSourceConfig, DbDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
from .persist_config import PersistConfig from .persist_config import PersistConfig
from .logging_config import LoggingConfig, resolve_log_file_path from .logging_config import LoggingConfig, resolve_log_file_path
from .run_presets import ( from .run_presets import (
@@ -26,6 +26,7 @@ from .builder import (
__all__ = [ __all__ = [
"ApiDataSourceConfig", "ApiDataSourceConfig",
"DbDataSourceConfig",
"JsonlDataSourceConfig", "JsonlDataSourceConfig",
"PipelineRunConfig", "PipelineRunConfig",
"DataSourceConfig", "DataSourceConfig",
+12 -1
View File
@@ -15,6 +15,14 @@ class JsonlDataSourceConfig(BaseModel):
handler_configs: Dict[str, Dict[str, Any]] = Field(default_factory=dict) handler_configs: Dict[str, Dict[str, Any]] = Field(default_factory=dict)
class DbDataSourceConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True)
type: Literal["db"] = "db"
target_project_ids: List[str] = Field(default_factory=list)
handler_configs: Dict[str, Dict[str, Any]] = Field(default_factory=dict)
class ApiDataSourceConfig(BaseModel): class ApiDataSourceConfig(BaseModel):
model_config = ConfigDict(extra="forbid", validate_assignment=True) model_config = ConfigDict(extra="forbid", validate_assignment=True)
@@ -31,4 +39,7 @@ class ApiDataSourceConfig(BaseModel):
handler_configs: Dict[str, Dict[str, Any]] = Field(default_factory=dict) handler_configs: Dict[str, Dict[str, Any]] = Field(default_factory=dict)
DataSourceConfig = Annotated[Union[JsonlDataSourceConfig, ApiDataSourceConfig], Field(discriminator="type")] DataSourceConfig = Annotated[
Union[JsonlDataSourceConfig, DbDataSourceConfig, ApiDataSourceConfig],
Field(discriminator="type"),
]
+2 -1
View File
@@ -3,7 +3,7 @@
Prefer importing from `sync_state_machine.config` directly. Prefer importing from `sync_state_machine.config` directly.
""" """
from .datasource_config import ApiDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig from .datasource_config import ApiDataSourceConfig, DbDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
from .persist_config import PersistConfig from .persist_config import PersistConfig
from .logging_config import LoggingConfig from .logging_config import LoggingConfig
from .strategy_config import ( from .strategy_config import (
@@ -24,6 +24,7 @@ from .builder import (
__all__ = [ __all__ = [
"ApiDataSourceConfig", "ApiDataSourceConfig",
"DbDataSourceConfig",
"JsonlDataSourceConfig", "JsonlDataSourceConfig",
"DataSourceConfig", "DataSourceConfig",
"PersistConfig", "PersistConfig",
@@ -17,6 +17,9 @@ from .task_result import TaskResult, TaskStatus
# API DataSource # API DataSource
from .api import ApiClient, ApiDataSource, BaseApiHandler from .api import ApiClient, ApiDataSource, BaseApiHandler
# DB DataSource
from .db import DbDataSource, BaseDbHandler
# JSONL DataSource # JSONL DataSource
from .jsonl import JsonlDataSource, BaseJsonlHandler from .jsonl import JsonlDataSource, BaseJsonlHandler
@@ -34,6 +37,10 @@ __all__ = [
"ApiClient", "ApiClient",
"ApiDataSource", "ApiDataSource",
"BaseApiHandler", "BaseApiHandler",
# DB DataSource
"DbDataSource",
"BaseDbHandler",
# JSONL DataSource # JSONL DataSource
"JsonlDataSource", "JsonlDataSource",
@@ -44,6 +44,7 @@ class BaseApiHandler(NodeHandler[T]):
self.api_client: 'ApiClient' = None # type: ignore # 由 DataSource 注入 self.api_client: 'ApiClient' = None # type: ignore # 由 DataSource 注入
self.poll_mode: str = "async" # async | sync self.poll_mode: str = "async" # async | sync
self._collection: 'DataCollection' = None # type: ignore # 由 DataSource 注入 self._collection: 'DataCollection' = None # type: ignore # 由 DataSource 注入
self.handler_config: Dict[str, Any] = {}
# 子类在 __init__ 中设置此列表以声明 create/update 中涉及的 Pydantic schema 类; # 子类在 __init__ 中设置此列表以声明 create/update 中涉及的 Pydantic schema 类;
# post-check 将只比较这些 schema 覆盖的字段,过滤后端只读的派生字段。 # post-check 将只比较这些 schema 覆盖的字段,过滤后端只读的派生字段。
self.update_schemas: List[type] = [] self.update_schemas: List[type] = []
@@ -63,6 +64,13 @@ class BaseApiHandler(NodeHandler[T]):
def set_collection(self, collection: 'DataCollection') -> None: def set_collection(self, collection: 'DataCollection') -> None:
"""设置 Collection(由 DataSource 调用)""" """设置 Collection(由 DataSource 调用)"""
self._collection = collection self._collection = collection
def set_handler_config(self, config: Dict[str, Any]) -> None:
self.handler_config = dict(config)
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""默认忽略项目过滤列表,具体子类按需覆盖。"""
return
@property @property
def node_type(self) -> str: def node_type(self) -> str:
@@ -0,0 +1,6 @@
"""DB DataSource 模块。"""
from .datasource import DbDataSource
from .handler import BaseDbHandler
__all__ = ["DbDataSource", "BaseDbHandler"]
@@ -0,0 +1,22 @@
"""DbDataSource - 数据库数据源协调层。"""
from __future__ import annotations
from ..datasource import BaseDataSource
class DbDataSource(BaseDataSource):
"""基于数据库 Handler 的数据源实现。"""
async def close(self) -> None:
return None
def _on_node_loaded(self, handler, node) -> None:
node.append_log(
f"DB链路[load] node_type={node.node_type} data_id={node.data_id or '<empty>'}"
)
def _on_in_progress_result(self, handler, node, result) -> None:
node.append_log(
f"DB链路[submit] action={node.action.value} data_id={node.data_id or '<empty>'}"
)
@@ -0,0 +1,68 @@
"""BaseDbHandler - DB Handler 基类。"""
from __future__ import annotations
from typing import Any, Dict, List, Optional, Set, TypeVar, TYPE_CHECKING
from pydantic import BaseModel
from ...common.type_safety import get_pydantic_model_fields
from ..handler import BaseNodeHandler
if TYPE_CHECKING:
from ...common.sync_node import SyncNode
from .datasource import DbDataSource
T = TypeVar("T", bound=BaseModel)
class BaseDbHandler(BaseNodeHandler[T]):
"""数据库 Handler 基类。"""
def __init__(
self,
datasource: "DbDataSource",
node_type: str,
node_class: type["SyncNode[T]"],
schema: type[T],
):
super().__init__(node_type, node_class, schema)
self.datasource = datasource
self.update_schemas: List[type] = []
self._target_project_ids: Optional[Set[str]] = None
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
if target_project_ids:
self._target_project_ids = {str(item) for item in target_project_ids if str(item)}
else:
self._target_project_ids = None
def get_update_fields(self, *, exclude: frozenset = frozenset({"id"})) -> List[str]:
fields: set[str] = set()
for schema in self.update_schemas:
for field_name in get_pydantic_model_fields(schema):
if field_name not in exclude:
fields.add(field_name)
return sorted(fields)
def _create_basic_node(self, data: Dict[str, Any], depend_ids: Optional[List[str]] = None) -> "SyncNode[T]":
data_id = self.extract_id(data)
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() before creating nodes.")
if data_id:
existing_node = self._collection.get_by_data_id(self.node_type, data_id)
if existing_node is not None:
existing_node.depend_ids = list(depend_ids or [])
existing_node.set_data(data.copy())
existing_node.set_origin_data(data.copy())
return existing_node
node_id = self._collection.generate_node_id()
return self.node_class(
node_id=node_id,
data_id=data_id,
data=data.copy(),
depend_ids=depend_ids or [],
origin_data=data.copy(),
)
+14
View File
@@ -70,6 +70,10 @@ class NodeHandler(Protocol[T]):
""" """
... ...
def set_handler_config(self, config: Dict[str, Any]) -> None:
"""注入 handler 级别配置。默认可忽略。"""
...
def set_target_project_ids(self, target_project_ids: List[str]) -> None: def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""注入项目过滤列表。默认可忽略。""" """注入项目过滤列表。默认可忽略。"""
... ...
@@ -345,6 +349,7 @@ class BaseNodeHandler(ABC, Generic[T]):
self._node_class = node_class self._node_class = node_class
self._schema = schema self._schema = schema
self._collection: Optional["DataCollection"] = None self._collection: Optional["DataCollection"] = None
self.handler_config: Dict[str, Any] = {}
def set_collection(self, collection: "DataCollection") -> None: def set_collection(self, collection: "DataCollection") -> None:
""" """
@@ -359,6 +364,10 @@ class BaseNodeHandler(ABC, Generic[T]):
"""默认忽略项目过滤能力。""" """默认忽略项目过滤能力。"""
return return
def set_handler_config(self, config: Dict[str, Any]) -> None:
"""默认保存 handler 级别配置,供子类按需读取。"""
self.handler_config = dict(config)
def set_api_client(self, client: "ApiClient") -> None: def set_api_client(self, client: "ApiClient") -> None:
"""默认忽略 API 客户端注入。""" """默认忽略 API 客户端注入。"""
return return
@@ -491,6 +500,11 @@ class CompatibleNodeHandler(Generic[T]):
def set_collection(self, collection: "DataCollection") -> None: def set_collection(self, collection: "DataCollection") -> None:
self._handler.set_collection(collection) self._handler.set_collection(collection)
def set_handler_config(self, config: Dict[str, Any]) -> None:
method = getattr(self._handler, "set_handler_config", None)
if callable(method):
method(config)
def set_target_project_ids(self, target_project_ids: List[str]) -> None: def set_target_project_ids(self, target_project_ids: List[str]) -> None:
method = getattr(self._handler, "set_target_project_ids", None) method = getattr(self._handler, "set_target_project_ids", None)
if callable(method): if callable(method):
@@ -188,7 +188,7 @@ class MaterialDetailApiHandler(BaseApiHandler):
) )
# 整组成功 # 整组成功
for node, _ in group_items: for node, _ in group_items:
results.append(TaskResult.success(node_id=node.node_id)) results.append(TaskResult.in_progress(node_id=node.node_id, task_id=push_id))
except Exception as e: except Exception as e:
# 整组失败 # 整组失败
error_msg = str(e) error_msg = str(e)
@@ -18,6 +18,7 @@ PUT (6个更新接口):
不支持: CREATE, DELETE 不支持: CREATE, DELETE
""" """
import asyncio
from typing import List, Dict, Any from typing import List, Dict, Any
from ...datasource.api.handler import BaseApiHandler from ...datasource.api.handler import BaseApiHandler
from ...datasource.task_result import TaskResult from ...datasource.task_result import TaskResult
@@ -35,6 +36,35 @@ from schemas.project.project_base import (
from schemas.project.project import ProjectInfoResponse from schemas.project.project import ProjectInfoResponse
_PROJECT_EXTENSION_FIELD_NAMES = {
"implementation_estimate",
"static_total_investment",
"completed_investment",
"total_contract_quantity",
"completed_settlement_quantity",
}
def _merge_project_all_info_fields(
project_data: Dict[str, Any],
all_info: Dict[str, Any],
) -> Dict[str, Any]:
"""把 all_info 中项目扩展字段拍平到 project 主数据上。"""
merged = dict(project_data)
for key, value in all_info.items():
if not key.endswith("_extension") or not isinstance(value, dict):
continue
for field_name in _PROJECT_EXTENSION_FIELD_NAMES:
if field_name not in value:
continue
current_value = merged.get(field_name)
if current_value in (None, ""):
merged[field_name] = value.get(field_name)
return merged
class ProjectApiHandler(BaseApiHandler): class ProjectApiHandler(BaseApiHandler):
"""项目 API Handler""" """项目 API Handler"""
@@ -136,12 +166,15 @@ class ProjectApiHandler(BaseApiHandler):
# 处理所有项目 # 处理所有项目
for project_data in projects: for project_data in projects:
node = self._create_node(project_data.model_dump())
# 设置 context
node.context["project_id"] = project_data.id
# 预加载并缓存 all_info,同时存储到 node.context # 预加载并缓存 all_info,同时存储到 node.context
all_info = await self.get_all_info(self.api_client, project_data.id) all_info = await self.get_all_info(self.api_client, project_data.id)
normalized_project_data = _merge_project_all_info_fields(
project_data.model_dump(),
all_info,
)
node = self._create_node(normalized_project_data)
# 设置 context
node.context["project_id"] = project_data.id
node.context["all_info"] = all_info node.context["all_info"] = all_info
nodes.append(node) nodes.append(node)
@@ -217,10 +250,12 @@ class ProjectApiHandler(BaseApiHandler):
# 执行更新(差异已由 _get_schema_diff 打印) # 执行更新(差异已由 _get_schema_diff 打印)
try: try:
push_id = self._generate_push_id()
# project 需要使用 _filter_update_data 返回的完整 schema 数据
full_update_data = self._filter_update_data(data, original_data, schema) full_update_data = self._filter_update_data(data, original_data, schema)
await api_func(self.api_client, full_update_data, push_id, biz_id) await self._submit_update_and_wait(
api_func=api_func,
data=full_update_data,
biz_id=biz_id,
)
node_updated = True node_updated = True
except Exception as e: except Exception as e:
error = str(e) error = str(e)
@@ -249,13 +284,39 @@ class ProjectApiHandler(BaseApiHandler):
async def poll_tasks(self, task_ids: List[str]) -> Dict[str, TaskResult]: async def poll_tasks(self, task_ids: List[str]) -> Dict[str, TaskResult]:
"""轮询异步任务状态""" """轮询异步任务状态"""
return await super().poll_tasks(task_ids) return await super().poll_tasks(task_ids)
async def _submit_update_and_wait(self, *, api_func, data: Dict[str, Any], biz_id: str) -> None:
push_id = self._generate_push_id()
await api_func(self.api_client, data, push_id, biz_id)
max_attempts = 20
poll_interval = 0.2
last_result: TaskResult | None = None
for attempt in range(max_attempts):
polled = await super().poll_tasks([push_id])
result = polled.get(push_id)
last_result = result
if result is None:
raise RuntimeError(f"Project update poll missing result: push_id={push_id}")
if result.status == result.status.SUCCESS:
return
if result.status == result.status.FAILED:
raise RuntimeError(result.error or f"Project update failed: push_id={push_id}")
if attempt < max_attempts - 1:
await asyncio.sleep(poll_interval)
raise RuntimeError(
f"Project update poll timeout: push_id={push_id}, last_status={getattr(last_result, 'status', None)}"
)
def _create_node(self, data: Dict[str, Any]) -> SyncNode: def _create_node(self, data: Dict[str, Any]) -> SyncNode:
node = self._create_basic_node(data, depend_ids=[]) node = self._create_basic_node(data, depend_ids=[])
# 设置 context # 设置 context
node.context["project_id"] = node.data_id node.context["project_id"] = node.data_id
return node return node
@@ -1,7 +1,6 @@
"""Project domain registration""" """Project domain registration"""
from ...common.registry import DomainRegistry from ...common.registry import DomainRegistry
from schemas.project.project_base import ProjectResponseBase from .sync_node import ProjectSyncNode, ProjectSyncSchema
from .sync_node import ProjectSyncNode
from .sync_strategy import ProjectSyncStrategy from .sync_strategy import ProjectSyncStrategy
from .jsonl_handler import ProjectJsonlHandler from .jsonl_handler import ProjectJsonlHandler
from .api_handler import ProjectApiHandler from .api_handler import ProjectApiHandler
@@ -9,7 +8,7 @@ from .api_handler import ProjectApiHandler
# 注册 project domain # 注册 project domain
DomainRegistry.register( DomainRegistry.register(
node_type="project", node_type="project",
schema=ProjectResponseBase, schema=ProjectSyncSchema,
node_class=ProjectSyncNode, node_class=ProjectSyncNode,
strategy_class=ProjectSyncStrategy, strategy_class=ProjectSyncStrategy,
jsonl_handler_class=ProjectJsonlHandler, jsonl_handler_class=ProjectJsonlHandler,
+14 -2
View File
@@ -1,11 +1,23 @@
from pydantic import Field
from ...common.sync_node import SyncNode from ...common.sync_node import SyncNode
from schemas.project.project_base import ProjectResponseBase from schemas.project.project_base import ProjectResponseBase
class ProjectSyncNode(SyncNode[ProjectResponseBase]): class ProjectSyncSchema(ProjectResponseBase):
"""项目同步内部 schema,补充 all_info 扩展字段。"""
implementation_estimate: float | None = Field(None, description="执行概算")
static_total_investment: float | None = Field(None, description="静态总投资")
completed_investment: float | None = Field(None, description="已完成投资")
total_contract_quantity: float | None = Field(None, description="合同总量")
completed_settlement_quantity: float | None = Field(None, description="已结算量")
class ProjectSyncNode(SyncNode[ProjectSyncSchema]):
"""项目同步节点""" """项目同步节点"""
node_type = "project" node_type = "project"
schema = ProjectResponseBase schema = ProjectSyncSchema
def __init__(self, **kwargs): def __init__(self, **kwargs):
super().__init__(**kwargs) super().__init__(**kwargs)
@@ -17,6 +17,7 @@ PUT:
不支持: CREATE, DELETE 不支持: CREATE, DELETE
""" """
import asyncio
from typing import List, Dict, Any, Optional, Tuple, Set from typing import List, Dict, Any, Optional, Tuple, Set
from pydantic import ValidationError from pydantic import ValidationError
@@ -270,6 +271,8 @@ class ProjectDetailApiHandler(BaseApiHandler):
try: try:
resolved_data_id = self._resolve_data_id_from_all_info(all_info, project_id=project_id, key=key) resolved_data_id = self._resolve_data_id_from_all_info(all_info, project_id=project_id, key=key)
if not resolved_data_id:
resolved_data_id = await self._resolve_data_id_with_retry(project_id=project_id, key=key)
except Exception as exc: except Exception as exc:
self._pending_push_context.pop(task_id, None) self._pending_push_context.pop(task_id, None)
results[task_id] = TaskResult.failed(error=f"Poll resolve data_id failed: {exc}", push_id=task_id) results[task_id] = TaskResult.failed(error=f"Poll resolve data_id failed: {exc}", push_id=task_id)
@@ -342,6 +345,9 @@ class ProjectDetailApiHandler(BaseApiHandler):
if isinstance(all_info, dict): if isinstance(all_info, dict):
return all_info return all_info
if force_refresh:
ProjectApiHandler._all_info_cache.pop(project_id, None)
all_info = await ProjectApiHandler.get_all_info(self.api_client, project_id) all_info = await ProjectApiHandler.get_all_info(self.api_client, project_id)
if project_node is not None: if project_node is not None:
project_node.context["all_info"] = all_info project_node.context["all_info"] = all_info
@@ -370,6 +376,20 @@ class ProjectDetailApiHandler(BaseApiHandler):
return matched[0] return matched[0]
return None return None
async def _resolve_data_id_with_retry(self, *, project_id: str, key: str) -> Optional[str]:
attempts = 6
delay = 0.2
for attempt in range(attempts):
all_info = await self._get_project_all_info(project_id, force_refresh=True)
resolved = self._resolve_data_id_from_all_info(all_info, project_id=project_id, key=key)
if resolved:
return resolved
if attempt < attempts - 1:
await asyncio.sleep(delay)
return None
async def _resolve_data_id_from_project_all_info(self, project_id: str, key: str) -> Optional[str]: async def _resolve_data_id_from_project_all_info(self, project_id: str, key: str) -> Optional[str]:
all_info = await self._get_project_all_info(project_id) all_info = await self._get_project_all_info(project_id)
return self._resolve_data_id_from_all_info(all_info, project_id=project_id, key=key) return self._resolve_data_id_from_all_info(all_info, project_id=project_id, key=key)
+2
View File
@@ -7,6 +7,7 @@ from .full_sync_pipeline import FullSyncPipeline
from ..config import ( from ..config import (
PipelineRunConfig, PipelineRunConfig,
ApiDataSourceConfig, ApiDataSourceConfig,
DbDataSourceConfig,
JsonlDataSourceConfig, JsonlDataSourceConfig,
DataSourceConfig, DataSourceConfig,
PersistConfig, PersistConfig,
@@ -35,6 +36,7 @@ __all__ = [
"run_pipeline_from_config", "run_pipeline_from_config",
"PipelineRunConfig", "PipelineRunConfig",
"ApiDataSourceConfig", "ApiDataSourceConfig",
"DbDataSourceConfig",
"JsonlDataSourceConfig", "JsonlDataSourceConfig",
"DataSourceConfig", "DataSourceConfig",
"PersistConfig", "PersistConfig",
+114 -83
View File
@@ -23,11 +23,13 @@ from ..common.collection import DataCollection
from ..common.persistence import PersistenceBackend from ..common.persistence import PersistenceBackend
from ..common.registry import DomainRegistry from ..common.registry import DomainRegistry
from ..datasource.api.datasource import ApiDataSource from ..datasource.api.datasource import ApiDataSource
from ..datasource.db.datasource import DbDataSource
from ..datasource.jsonl.datasource import JsonlDataSource from ..datasource.jsonl.datasource import JsonlDataSource
from ..config import ( from ..config import (
PipelineRunConfig, PipelineRunConfig,
DataSourceConfig, DataSourceConfig,
ApiDataSourceConfig, ApiDataSourceConfig,
DbDataSourceConfig,
JsonlDataSourceConfig, JsonlDataSourceConfig,
PersistConfig, PersistConfig,
LoggingConfig, LoggingConfig,
@@ -97,16 +99,41 @@ def _ensure_pipeline_logger(
) )
def _api_handler_kwargs( def _build_handler_config(
node_type: str, node_type: str,
ds_config: DataSourceConfig, ds_config: DataSourceConfig,
target_project_ids: List[str], target_project_ids: List[str],
) -> Dict[str, Any]: ) -> Dict[str, Any]:
kwargs = dict(ds_config.handler_configs.get(node_type, {})) config = dict(ds_config.handler_configs.get(node_type, {}))
if node_type == "project" and target_project_ids: if node_type == "project" and target_project_ids:
kwargs.setdefault("filter_project_id", target_project_ids) config.setdefault("filter_project_id", target_project_ids)
return kwargs return config
def _create_handler(
*,
node_type: str,
ds_config: DataSourceConfig,
datasource,
target_project_ids: List[str],
):
datasource_type = ds_config.type
handler_class = DomainRegistry.get_handler(node_type, datasource_type)
if handler_class is None:
raise RuntimeError(
f"Missing {datasource_type.upper()} handler for node_type: {node_type}"
)
handler_config = _build_handler_config(node_type, ds_config, target_project_ids)
if datasource_type == "api":
handler = handler_class(**handler_config)
else:
handler = handler_class(datasource)
if hasattr(handler, "set_handler_config"):
handler.set_handler_config(handler_config)
return handler
def _resolve_datasource_target_project_ids(config: PipelineRunConfig) -> tuple[List[str], List[str]]: def _resolve_datasource_target_project_ids(config: PipelineRunConfig) -> tuple[List[str], List[str]]:
@@ -120,48 +147,58 @@ def _resolve_datasource_target_project_ids(config: PipelineRunConfig) -> tuple[L
return local_target_ids, remote_target_ids return local_target_ids, remote_target_ids
async def _create_datasources(config: PipelineRunConfig): async def _create_datasource(
local_cfg = config.local_datasource ds_config: DataSourceConfig,
remote_cfg = config.remote_datasource *,
datasource_override=None,
endpoint_name: str,
):
if datasource_override is not None:
return datasource_override
if isinstance(local_cfg, JsonlDataSourceConfig): if isinstance(ds_config, JsonlDataSourceConfig):
local_dir = Path(local_cfg.jsonl_dir) dir_path = Path(ds_config.jsonl_dir)
if not local_dir.exists() or not local_dir.is_dir(): if endpoint_name == "remote":
dir_path = _prepare_remote_jsonl_dir(dir_path)
elif not dir_path.exists() or not dir_path.is_dir():
raise FileNotFoundError( raise FileNotFoundError(
f"local_datasource.jsonl_dir does not exist or is not a directory: {local_dir}" f"{endpoint_name}_datasource.jsonl_dir does not exist or is not a directory: {dir_path}"
) )
local_ds = JsonlDataSource(local_dir, read_only=local_cfg.read_only) return JsonlDataSource(dir_path, read_only=ds_config.read_only)
else:
local_ds = ApiDataSource(
base_url=local_cfg.api_base_url,
uid=local_cfg.api_uid,
secret=local_cfg.api_secret,
debug=local_cfg.api_debug,
poll_max_retries=local_cfg.poll_max_retries,
poll_interval=local_cfg.poll_interval,
rate_limit_max_requests=local_cfg.api_rate_limit_max_requests,
rate_limit_window_seconds=local_cfg.api_rate_limit_window_seconds,
)
await local_ds.initialize()
if isinstance(remote_cfg, JsonlDataSourceConfig): if isinstance(ds_config, DbDataSourceConfig):
remote_dir = Path(remote_cfg.jsonl_dir) return DbDataSource()
remote_dir = _prepare_remote_jsonl_dir(remote_dir)
remote_ds = JsonlDataSource(remote_dir, read_only=remote_cfg.read_only)
else:
remote_ds = ApiDataSource(
base_url=remote_cfg.api_base_url,
uid=remote_cfg.api_uid,
secret=remote_cfg.api_secret,
debug=remote_cfg.api_debug,
poll_max_retries=remote_cfg.poll_max_retries,
poll_interval=remote_cfg.poll_interval,
rate_limit_max_requests=remote_cfg.api_rate_limit_max_requests,
rate_limit_window_seconds=remote_cfg.api_rate_limit_window_seconds,
)
await remote_ds.initialize()
return local_ds, remote_ds datasource = ApiDataSource(
base_url=ds_config.api_base_url,
uid=ds_config.api_uid,
secret=ds_config.api_secret,
debug=ds_config.api_debug,
poll_max_retries=ds_config.poll_max_retries,
poll_interval=ds_config.poll_interval,
rate_limit_max_requests=ds_config.api_rate_limit_max_requests,
rate_limit_window_seconds=ds_config.api_rate_limit_window_seconds,
)
await datasource.initialize()
return datasource
def _register_handlers_for_endpoint(
*,
datasource,
ds_config: DataSourceConfig,
node_types: List[str],
target_project_ids: List[str],
) -> None:
for node_type in node_types:
datasource.register_handler(
_create_handler(
node_type=node_type,
ds_config=ds_config,
datasource=datasource,
target_project_ids=target_project_ids,
)
)
def _register_handlers( def _register_handlers(
@@ -170,44 +207,18 @@ def _register_handlers(
remote_ds, remote_ds,
all_types: List[str], all_types: List[str],
) -> None: ) -> None:
for node_type in all_types: _register_handlers_for_endpoint(
if isinstance(config.local_datasource, JsonlDataSourceConfig): datasource=local_ds,
jsonl_handler_class = DomainRegistry.get_jsonl_handler(node_type) ds_config=config.local_datasource,
if jsonl_handler_class is None: node_types=all_types,
raise RuntimeError(f"Missing JSONL handler for node_type: {node_type}") target_project_ids=[str(pid) for pid in config.local_datasource.target_project_ids if str(pid)],
local_ds.register_handler(jsonl_handler_class(local_ds)) )
else: _register_handlers_for_endpoint(
api_handler_class = DomainRegistry.get_api_handler(node_type) datasource=remote_ds,
if api_handler_class is None: ds_config=config.remote_datasource,
raise RuntimeError(f"Missing API handler for node_type: {node_type}") node_types=all_types,
local_ds.register_handler( target_project_ids=[str(pid) for pid in config.remote_datasource.target_project_ids if str(pid)],
api_handler_class( )
**_api_handler_kwargs(
node_type,
config.local_datasource,
config.target_project_ids,
)
)
)
if isinstance(config.remote_datasource, JsonlDataSourceConfig):
jsonl_handler_class = DomainRegistry.get_jsonl_handler(node_type)
if jsonl_handler_class is None:
raise RuntimeError(f"Missing JSONL handler for node_type: {node_type}")
remote_ds.register_handler(jsonl_handler_class(remote_ds))
else:
api_handler_class = DomainRegistry.get_api_handler(node_type)
if api_handler_class is None:
raise RuntimeError(f"Missing API handler for node_type: {node_type}")
remote_ds.register_handler(
api_handler_class(
**_api_handler_kwargs(
node_type,
config.remote_datasource,
config.target_project_ids,
)
)
)
def _resolve_strategy_map(config: PipelineRunConfig, node_types: List[str]) -> Dict[str, StrategyRuntimeConfig]: def _resolve_strategy_map(config: PipelineRunConfig, node_types: List[str]) -> Dict[str, StrategyRuntimeConfig]:
@@ -260,7 +271,12 @@ def _build_strategies(
return strategies return strategies
async def create_pipeline_from_config(config: PipelineRunConfig) -> FullSyncPipeline: async def create_pipeline_from_config(
config: PipelineRunConfig,
*,
local_datasource_override=None,
remote_datasource_override=None,
) -> FullSyncPipeline:
if config.logging.initialize and not config.logging.file_path: if config.logging.initialize and not config.logging.file_path:
config.logging.file_path = resolve_log_file_path(config.logging) config.logging.file_path = resolve_log_file_path(config.logging)
@@ -289,7 +305,16 @@ async def create_pipeline_from_config(config: PipelineRunConfig) -> FullSyncPipe
persistence = PersistenceBackend(str(config.persist.db_path), backend=config.persist.backend) persistence = PersistenceBackend(str(config.persist.db_path), backend=config.persist.backend)
await persistence.initialize() await persistence.initialize()
local_ds, remote_ds = await _create_datasources(config) local_ds = await _create_datasource(
config.local_datasource,
datasource_override=local_datasource_override,
endpoint_name="local",
)
remote_ds = await _create_datasource(
config.remote_datasource,
datasource_override=remote_datasource_override,
endpoint_name="remote",
)
local_cfg_with_target = config.local_datasource.model_copy( local_cfg_with_target = config.local_datasource.model_copy(
update={"target_project_ids": list(local_target_project_ids)}, update={"target_project_ids": list(local_target_project_ids)},
deep=True, deep=True,
@@ -340,7 +365,7 @@ async def create_pipeline_from_config(config: PipelineRunConfig) -> FullSyncPipe
return pipeline return pipeline
except Exception: except Exception:
async def _safe_close(name: str, resource: ApiDataSource | JsonlDataSource | PersistenceBackend | None) -> None: async def _safe_close(name: str, resource) -> None:
if resource is None: if resource is None:
return return
try: try:
@@ -359,6 +384,8 @@ async def run_pipeline_from_config(
*, *,
title: str = "sync_state_machine pipeline", title: str = "sync_state_machine pipeline",
print_summary: bool = True, print_summary: bool = True,
local_datasource_override=None,
remote_datasource_override=None,
) -> Dict[str, Any]: ) -> Dict[str, Any]:
if config.logging.initialize and not config.logging.file_path: if config.logging.initialize and not config.logging.file_path:
config.logging.file_path = resolve_log_file_path(config.logging) config.logging.file_path = resolve_log_file_path(config.logging)
@@ -401,7 +428,11 @@ async def run_pipeline_from_config(
print(f"🔧 Resolved Config:\n{resolved_cfg_text}") print(f"🔧 Resolved Config:\n{resolved_cfg_text}")
print("=" * 80) print("=" * 80)
pipeline = await create_pipeline_from_config(config) pipeline = await create_pipeline_from_config(
config,
local_datasource_override=local_datasource_override,
remote_datasource_override=remote_datasource_override,
)
stats = await pipeline.run() stats = await pipeline.run()
if print_summary: if print_summary:
@@ -355,3 +355,55 @@ async def test_poll_tasks_queries_all_info_once_per_project_for_multiple_keys(mo
assert polled[task_ids[1]].status == TaskStatus.SUCCESS assert polled[task_ids[1]].status == TaskStatus.SUCCESS
assert polled[task_ids[1]].data_id == "remote-detail-2" assert polled[task_ids[1]].data_id == "remote-detail-2"
assert client.all_info_calls == ["project-1"] assert client.all_info_calls == ["project-1"]
@pytest.mark.asyncio
async def test_poll_tasks_retries_until_project_detail_appears(monkeypatch: pytest.MonkeyPatch) -> None:
collection = DataCollection("remote")
project = _build_project_node(node_id="p-node", project_id="project-1", all_info={"project_detail": []})
await collection.add(project)
handler = ProjectDetailApiHandler()
handler.set_collection(collection)
handler.api_client = _MockApiClient()
call_count = {"value": 0}
async def _fake_get_all_info(cls, _client, project_id: str):
call_count["value"] += 1
if call_count["value"] >= 2:
return {
"project_detail": [
{
"id": "remote-detail-1",
"project_id": project_id,
"key": "ensure_production",
"value": [{"year": 2026, "capacity": 21.0}],
}
]
}
return {"project_detail": []}
monkeypatch.setattr(ProjectApiHandler, "get_all_info", classmethod(_fake_get_all_info))
node = ProjectDetailSyncNode(
node_id="detail-node-1",
data_id="local-detail-1",
data={
"id": "local-detail-1",
"project_id": "project-1",
"key": "ensure_production",
"value": [{"year": 2026, "capacity": 21.0}],
},
)
create_result = (await handler.create_all([node]))[0]
assert create_result.task_id
handler.api_client.push_logs[create_result.task_id] = {"code": 200, "data": {"biz_id": "project-1"}}
polled = await handler.poll_tasks([create_result.task_id])
final = polled[create_result.task_id]
assert final.status == TaskStatus.SUCCESS
assert final.data_id == "remote-detail-1"
assert call_count["value"] >= 2
@@ -0,0 +1,65 @@
from __future__ import annotations
import importlib.util
from pathlib import Path
import sys
import yaml
def _load_base_module():
scripts_dir = Path(__file__).resolve().parents[2] / "scripts"
if str(scripts_dir) not in sys.path:
sys.path.insert(0, str(scripts_dir))
file_path = Path(__file__).resolve().parents[2] / "scripts" / "auto_test_domains" / "base.py"
spec = importlib.util.spec_from_file_location("auto_test_domain_base_for_test", file_path)
if spec is None or spec.loader is None:
raise RuntimeError(f"Unable to load module from {file_path}")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def test_build_temp_run_profile_uses_explicit_project_prebind(tmp_path: Path) -> None:
module = _load_base_module()
source_profile = tmp_path / "source.yaml"
source_profile.write_text(
yaml.safe_dump(
{
"local_datasource": {"target_project_ids": ["old-local"]},
"remote_datasource": {"target_project_ids": ["old-remote"]},
"strategies": {"project": {"config": {}}},
},
allow_unicode=True,
sort_keys=False,
),
encoding="utf-8",
)
config = {
"pipeline": {
"run_profile": str(source_profile),
"command": ["python", "run.py", "--config_path", "placeholder.yaml"],
}
}
profile_path, command = module.build_temp_run_profile_for_project_auto_bind(
config=config,
report_root=tmp_path,
profile_name="demo",
local_project_id="local-project",
remote_project_id="remote-project",
)
profile = yaml.safe_load(Path(profile_path).read_text(encoding="utf-8"))
project_config = profile["strategies"]["project"]["config"]
assert profile["local_datasource"]["target_project_ids"] == ["local-project"]
assert profile["remote_datasource"]["target_project_ids"] == ["remote-project"]
assert project_config["auto_bind"] is False
assert project_config["auto_bind_fields"] == []
assert project_config["pre_bind_data_id"] == [
{"local_data_id": "local-project", "remote_data_id": "remote-project"}
]
assert command[-1] == profile_path
@@ -7,6 +7,7 @@ from sync_state_machine.config import (
PipelineRunConfig, PipelineRunConfig,
JsonlDataSourceConfig, JsonlDataSourceConfig,
ApiDataSourceConfig, ApiDataSourceConfig,
DbDataSourceConfig,
PersistConfig, PersistConfig,
LoggingConfig, LoggingConfig,
) )
@@ -101,3 +102,11 @@ def test_api_datasource_rate_limit_config_fields() -> None:
) )
assert overridden.api_rate_limit_max_requests == 5 assert overridden.api_rate_limit_max_requests == 5
assert overridden.api_rate_limit_window_seconds == 2.5 assert overridden.api_rate_limit_window_seconds == 2.5
def test_db_datasource_accepts_minimal_config() -> None:
cfg = DbDataSourceConfig(type="db")
assert cfg.type == "db"
assert cfg.target_project_ids == []
assert cfg.handler_configs == {}
@@ -0,0 +1,89 @@
from __future__ import annotations
from pathlib import Path
from pydantic import BaseModel
from sync_state_machine.common.registry import DomainRegistry
from sync_state_machine.common.sync_node import SyncNode
from sync_state_machine.config import DbDataSourceConfig, JsonlDataSourceConfig, LoggingConfig, PersistConfig, PipelineRunConfig
from sync_state_machine.datasource.db import BaseDbHandler, DbDataSource
from sync_state_machine.datasource.jsonl import BaseJsonlHandler, JsonlDataSource
from sync_state_machine.pipeline.factory import _register_handlers
from sync_state_machine.sync_system.strategy import DefaultSyncStrategy
class TestDbSchema(BaseModel):
__test__ = False
id: str
name: str
class TestDbNode(SyncNode[TestDbSchema]):
__test__ = False
node_type = "test_db_handler"
schema = TestDbSchema
class TestDbStrategy(DefaultSyncStrategy[TestDbSchema]):
__test__ = False
schema = TestDbSchema
class TestDbHandler(BaseDbHandler[TestDbSchema]):
__test__ = False
def __init__(self, datasource: DbDataSource):
super().__init__(datasource, "test_db_handler", TestDbNode, TestDbSchema)
def extract_id(self, data: dict[str, str]) -> str:
return str(data.get("id", ""))
async def load(self):
return []
async def create_all(self, nodes):
return []
async def update_all(self, nodes):
return []
class TestJsonlHandler(BaseJsonlHandler):
__test__ = False
def __init__(self, datasource: JsonlDataSource):
super().__init__(datasource, "test_db_handler", TestDbNode, TestDbSchema)
def test_register_handlers_uses_db_handler_for_db_datasource(tmp_path: Path) -> None:
if not DomainRegistry.is_registered("test_db_handler"):
DomainRegistry.register(
node_type="test_db_handler",
schema=TestDbSchema,
node_class=TestDbNode,
strategy_class=TestDbStrategy,
jsonl_handler_class=TestJsonlHandler,
)
DomainRegistry.register_db_handler("test_db_handler", TestDbHandler)
config = PipelineRunConfig(
node_types=["test_db_handler"],
local_datasource=DbDataSourceConfig(type="db"),
remote_datasource=JsonlDataSourceConfig(
type="jsonl",
jsonl_dir=str(tmp_path),
target_project_ids=[],
),
strategies=[],
persist=PersistConfig(db_path=str(tmp_path / "pipeline.db")),
logging=LoggingConfig(initialize=False),
)
local_ds = DbDataSource()
remote_ds = JsonlDataSource(tmp_path)
_register_handlers(config, local_ds, remote_ds, ["test_db_handler"])
assert isinstance(local_ds.get_handler("test_db_handler"), TestDbHandler)
assert isinstance(remote_ds.get_handler("test_db_handler"), TestJsonlHandler)
@@ -0,0 +1,105 @@
from __future__ import annotations
from pathlib import Path
import pytest
from pydantic import BaseModel
from sync_state_machine.common.registry import DomainRegistry
from sync_state_machine.common.sync_node import SyncNode
from sync_state_machine.config import DbDataSourceConfig, LoggingConfig, PersistConfig, PipelineRunConfig
from sync_state_machine.datasource.db import BaseDbHandler, DbDataSource
from sync_state_machine.pipeline.factory import create_pipeline_from_config
from sync_state_machine.sync_system.strategy import DefaultSyncStrategy
class EndpointInitSchema(BaseModel):
__test__ = False
id: str
name: str | None = None
class EndpointInitNode(SyncNode[EndpointInitSchema]):
__test__ = False
node_type = "factory_endpoint_demo"
schema = EndpointInitSchema
class EndpointInitStrategy(DefaultSyncStrategy[EndpointInitSchema]):
__test__ = False
schema = EndpointInitSchema
class EndpointInitDbHandler(BaseDbHandler[EndpointInitSchema]):
__test__ = False
def __init__(self, datasource: DbDataSource):
super().__init__(datasource, "factory_endpoint_demo", EndpointInitNode, EndpointInitSchema)
def extract_id(self, data: dict[str, str]) -> str:
return str(data.get("id", ""))
async def load(self):
return []
async def create_all(self, nodes):
return []
async def update_all(self, nodes):
return []
def _ensure_registered() -> None:
node_type = "factory_endpoint_demo"
if DomainRegistry.is_registered(node_type):
reg = DomainRegistry.get_registration(node_type)
reg.schema = EndpointInitSchema
reg.node_class = EndpointInitNode
reg.strategy_class = EndpointInitStrategy
reg.db_handler_class = EndpointInitDbHandler
return
DomainRegistry.register(
node_type=node_type,
schema=EndpointInitSchema,
node_class=EndpointInitNode,
strategy_class=EndpointInitStrategy,
db_handler_class=EndpointInitDbHandler,
)
@pytest.mark.asyncio
async def test_factory_initializes_same_datasource_type_with_distinct_handler_configs(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
_ensure_registered()
config = PipelineRunConfig(
node_types=["factory_endpoint_demo"],
local_datasource=DbDataSourceConfig(
type="db",
handler_configs={"factory_endpoint_demo": {"side": "local", "batch_size": 10}},
target_project_ids=["project_local"],
),
remote_datasource=DbDataSourceConfig(
type="db",
handler_configs={"factory_endpoint_demo": {"side": "remote", "batch_size": 20}},
target_project_ids=["project_remote"],
),
strategies=[],
persist=PersistConfig(db_path=str(tmp_path / "factory-init.db")),
logging=LoggingConfig(initialize=False),
)
pipeline = await create_pipeline_from_config(config)
try:
local_handler = pipeline.local_datasource.get_handler("factory_endpoint_demo")
remote_handler = pipeline.remote_datasource.get_handler("factory_endpoint_demo")
assert isinstance(local_handler, EndpointInitDbHandler)
assert isinstance(remote_handler, EndpointInitDbHandler)
assert local_handler.handler_config == {"side": "local", "batch_size": 10}
assert remote_handler.handler_config == {"side": "remote", "batch_size": 20}
finally:
await pipeline.close()
+86
View File
@@ -0,0 +1,86 @@
from __future__ import annotations
from typing import Any
from pydantic import BaseModel
from sync_state_machine.common.sync_node import SyncNode
from sync_state_machine.datasource.api.handler import BaseApiHandler
from sync_state_machine.datasource.handler import BaseNodeHandler, CompatibleNodeHandler
from sync_state_machine.datasource.task_result import TaskResult
class SharedConfigSchema(BaseModel):
__test__ = False
id: str
class SharedConfigNode(SyncNode[SharedConfigSchema]):
__test__ = False
node_type = "shared_config_demo"
schema = SharedConfigSchema
class SharedConfigBaseHandler(BaseNodeHandler[SharedConfigSchema]):
__test__ = False
def __init__(self) -> None:
super().__init__("shared_config_demo", SharedConfigNode, SharedConfigSchema)
async def load(self):
return []
async def create_all(self, nodes):
return []
async def update_all(self, nodes):
return []
def extract_id(self, data: dict[str, Any]) -> str:
return str(data.get("id", ""))
class SharedConfigApiHandler(BaseApiHandler[SharedConfigSchema]):
__test__ = False
def __init__(self) -> None:
super().__init__()
self._node_type = "shared_api_demo"
self._node_class = SharedConfigNode
self._schema = SharedConfigSchema
async def load(self):
return []
async def create_all(self, nodes):
return [TaskResult.success(node_id=node.node_id) for node in nodes]
async def update_all(self, nodes):
return [TaskResult.success(node_id=node.node_id) for node in nodes]
async def delete_all(self, nodes):
return [TaskResult.success(node_id=node.node_id) for node in nodes]
def test_base_handler_supports_shared_handler_config() -> None:
handler = SharedConfigBaseHandler()
handler.set_handler_config({"mode": "local", "batch_size": 10})
assert handler.handler_config == {"mode": "local", "batch_size": 10}
def test_compatible_handler_forwards_shared_handler_config() -> None:
handler = CompatibleNodeHandler(SharedConfigBaseHandler())
handler.set_handler_config({"mode": "remote"})
assert handler.handler_config == {"mode": "remote"}
def test_api_handler_supports_shared_handler_config() -> None:
handler = SharedConfigApiHandler()
handler.set_handler_config({"filter_project_id": ["p1", "p2"]})
assert handler.handler_config == {"filter_project_id": ["p1", "p2"]}
@@ -0,0 +1,62 @@
from __future__ import annotations
from typing import Any
import pytest
from pydantic import BaseModel
from sync_state_machine.common.sync_node import SyncNode
from sync_state_machine.domain.material_detail.api_handler import MaterialDetailApiHandler
from sync_state_machine.datasource.task_result import TaskStatus
class _MaterialDetailSchema(BaseModel):
id: str
parent_id: str
quantity: float
remark: str | None = None
class _MaterialDetailNode(SyncNode[_MaterialDetailSchema]):
node_type = "material_detail"
schema = _MaterialDetailSchema
class _MockApiClient:
def __init__(self) -> None:
self.requests: list[dict[str, Any]] = []
async def request(self, method: str, endpoint: str, **kwargs: Any) -> dict[str, Any]:
self.requests.append({"method": method, "endpoint": endpoint, **kwargs})
return {"code": 200, "message": "OK", "data": {"biz_id": kwargs.get("data", {}).get("biz_id")}}
@pytest.mark.asyncio
async def test_material_detail_update_returns_in_progress_with_shared_push_id() -> None:
handler = MaterialDetailApiHandler()
handler.api_client = _MockApiClient()
node = _MaterialDetailNode(
node_id="detail-node-1",
data_id="detail-1",
data={
"id": "detail-1",
"parent_id": "material-1",
"quantity": 10.0,
"remark": "new",
},
origin_data={
"id": "detail-1",
"parent_id": "material-1",
"quantity": 5.0,
"remark": None,
},
context={"material_id": "material-1"},
)
results = await handler.update_all([node])
assert len(results) == 1
assert results[0].status == TaskStatus.IN_PROGRESS
assert results[0].task_id
assert handler.api_client.requests[0]["endpoint"] == "/material/detail/update"
@@ -0,0 +1,84 @@
from __future__ import annotations
from uuid import uuid4
from sync_state_machine.domain.project.api_handler import _merge_project_all_info_fields
from sync_state_machine.domain.project.sync_node import ProjectSyncNode
def test_merge_project_all_info_fields_flattens_extension_investment_fields() -> None:
project_data = {
"id": "project-1",
"name": "Demo Project",
"dynamic_investment": 123.0,
}
all_info = {
"project": {"id": "project-1"},
"hydropower_extension": {
"implementation_estimate": 1000.0,
"static_total_investment": 990.0,
"completed_investment": 120.0,
"total_contract_quantity": 50.0,
},
}
merged = _merge_project_all_info_fields(project_data, all_info)
assert merged["dynamic_investment"] == 123.0
assert merged["implementation_estimate"] == 1000.0
assert merged["static_total_investment"] == 990.0
assert merged["completed_investment"] == 120.0
assert merged["total_contract_quantity"] == 50.0
def test_merge_project_all_info_fields_keeps_existing_project_values() -> None:
project_data = {
"id": "project-1",
"implementation_estimate": 2000.0,
"static_total_investment": 1800.0,
}
all_info = {
"hydropower_extension": {
"implementation_estimate": 1000.0,
"static_total_investment": 990.0,
"completed_investment": 120.0,
},
}
merged = _merge_project_all_info_fields(project_data, all_info)
assert merged["implementation_estimate"] == 2000.0
assert merged["static_total_investment"] == 1800.0
assert merged["completed_investment"] == 120.0
def test_project_sync_node_preserves_extension_backed_fields() -> None:
project_id = str(uuid4())
data = {
"_id": project_id,
"id": project_id,
"project_type": "hydropower",
"region_name": ["中国", "测试省"],
"region_path": ["100000", "110000"],
"province": "110000",
"is_domestic": True,
"company_id": str(uuid4()),
"company_name": "测试公司",
"address": "测试地址",
"code": "P-001",
"name": "测试项目",
"progress_type": "A",
"dynamic_investment": 100.0,
"complete_investment": 80.0,
"implementation_estimate": 120.0,
"static_total_investment": 110.0,
"completed_investment": 70.0,
}
node = ProjectSyncNode(node_id=str(uuid4()), data_id=data["id"], data=data)
normalized = node.get_data()
assert normalized is not None
assert normalized["implementation_estimate"] == 120.0
assert normalized["static_total_investment"] == 110.0
assert normalized["completed_investment"] == 70.0
@@ -0,0 +1,56 @@
from __future__ import annotations
from typing import Any
import pytest
from pydantic import BaseModel
from sync_state_machine.common.sync_node import SyncNode
from sync_state_machine.domain.project.api_handler import ProjectApiHandler
from sync_state_machine.datasource.task_result import TaskStatus
class _ProjectUpdateSchema(BaseModel):
id: str
dynamic_investment: float
class _ProjectUpdateNode(SyncNode[_ProjectUpdateSchema]):
node_type = "project"
schema = _ProjectUpdateSchema
class _MockApiClient:
def __init__(self) -> None:
self.requests: list[dict[str, Any]] = []
self._poll_count = 0
async def request(self, method: str, endpoint: str, **kwargs: Any) -> dict[str, Any]:
self.requests.append({"method": method, "endpoint": endpoint, **kwargs})
return {"code": 200, "message": "OK", "data": {"biz_id": kwargs.get("data", {}).get("biz_id")}}
async def get_push_logs(self, task_ids: list[str]) -> dict[str, dict[str, Any]]:
self._poll_count += 1
if self._poll_count == 1:
return {task_ids[0]: {}}
return {task_ids[0]: {"code": 200, "data": {"biz_id": "project-1"}}}
@pytest.mark.asyncio
async def test_project_update_waits_for_async_push_completion() -> None:
handler = ProjectApiHandler(filter_project_id=["project-1"])
handler.api_client = _MockApiClient()
node = _ProjectUpdateNode(
node_id="project-node-1",
data_id="project-1",
data={"id": "project-1", "dynamic_investment": 200.0},
origin_data={"id": "project-1", "dynamic_investment": 100.0},
)
results = await handler.update_all([node])
assert len(results) == 1
assert results[0].status == TaskStatus.SUCCESS
assert handler.api_client.requests[0]["endpoint"] == "/project/dynamic_investment"
assert handler.api_client._poll_count == 2
+53
View File
@@ -0,0 +1,53 @@
from __future__ import annotations
import importlib.util
from pathlib import Path
import sys
def _load_case_module():
scripts_dir = Path(__file__).resolve().parents[2] / "scripts"
if str(scripts_dir) not in sys.path:
sys.path.insert(0, str(scripts_dir))
file_path = scripts_dir / "auto_test_domains" / "project_case.py"
spec = importlib.util.spec_from_file_location("project_case_for_test", file_path)
if spec is None or spec.loader is None:
raise RuntimeError(f"Unable to load module from {file_path}")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def test_project_case_round_3_focuses_on_stable_readback_fields(monkeypatch, tmp_path: Path) -> None:
module = _load_case_module()
captured_rounds: dict[str, dict] = {}
def _fake_run_pipeline_round(**kwargs):
captured_rounds[kwargs["round_name"]] = kwargs
return {"name": kwargs["round_name"], "passed": True, "issues": []}
def _fake_build_domain_report(**kwargs):
return kwargs
monkeypatch.setattr(module, "run_pipeline_round", _fake_run_pipeline_round)
monkeypatch.setattr(module, "build_domain_report", _fake_build_domain_report)
module.run_case(config={"project_root": str(tmp_path)}, report_root=tmp_path)
round_3 = captured_rounds["round_3_investment_update"]
expected_fields = round_3["expected_bound_fields"]["remote"]["project"][module.LOCAL_PROJECT_ID]
changed_fields = round_3["expected_changes"]["project"][module.LOCAL_PROJECT_ID]
assert set(changed_fields) == {
"investment_decision_capacity",
"investment_decision_date",
"complete_investment",
"dynamic_investment",
}
assert set(expected_fields.keys()) == {
"investment_decision_capacity",
"investment_decision_date",
"complete_investment",
"dynamic_investment",
}
assert "PUT /project/settlement_investment" in round_3["coverage"]["api"]
@@ -0,0 +1,46 @@
from __future__ import annotations
import importlib.util
from pathlib import Path
import sys
def _load_case_module():
scripts_dir = Path(__file__).resolve().parents[2] / "scripts"
if str(scripts_dir) not in sys.path:
sys.path.insert(0, str(scripts_dir))
file_path = scripts_dir / "auto_test_domains" / "project_detail_data_case.py"
spec = importlib.util.spec_from_file_location("project_detail_data_case_for_test", file_path)
if spec is None or spec.loader is None:
raise RuntimeError(f"Unable to load module from {file_path}")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def test_project_detail_case_excludes_production_from_unified_rounds(monkeypatch, tmp_path: Path) -> None:
module = _load_case_module()
recorded_rounds: list[str] = []
def _fake_run_pipeline_round(**kwargs):
recorded_rounds.append(kwargs["round_name"])
return {"name": kwargs["round_name"], "passed": True, "issues": []}
def _fake_build_domain_report(**kwargs):
return kwargs
monkeypatch.setattr(module, "run_pipeline_round", _fake_run_pipeline_round)
monkeypatch.setattr(module, "build_domain_report", _fake_build_domain_report)
report = module.run_case(config={"project_root": str(tmp_path)}, report_root=tmp_path)
assert recorded_rounds == [
"round_1_create",
"round_2_update",
"round_3_plan_construction_create",
"round_5_ensure_production_push",
"round_6_climb_production_push",
"round_7_challenge_production_push",
]
assert "production" in report["description"]
assert "不混入本组统一回归" in report["description"]
@@ -0,0 +1,49 @@
from __future__ import annotations
from pydantic import BaseModel
from sync_state_machine.common.registry import DomainRegistry
class RegistryLookupSchema(BaseModel):
__test__ = False
id: str
class RegistryLookupNode:
__test__ = False
class RegistryLookupJsonlHandler:
__test__ = False
class RegistryLookupApiHandler:
__test__ = False
class RegistryLookupDbHandler:
__test__ = False
def test_registry_supports_unified_handler_lookup() -> None:
node_type = "registry_lookup_demo"
if DomainRegistry.is_registered(node_type):
reg = DomainRegistry.get_registration(node_type)
reg.jsonl_handler_class = RegistryLookupJsonlHandler
reg.api_handler_class = RegistryLookupApiHandler
reg.db_handler_class = RegistryLookupDbHandler
else:
DomainRegistry.register(
node_type=node_type,
schema=RegistryLookupSchema,
node_class=RegistryLookupNode,
jsonl_handler_class=RegistryLookupJsonlHandler,
api_handler_class=RegistryLookupApiHandler,
db_handler_class=RegistryLookupDbHandler,
)
assert DomainRegistry.get_handler(node_type, "jsonl") is RegistryLookupJsonlHandler
assert DomainRegistry.get_handler(node_type, "api") is RegistryLookupApiHandler
assert DomainRegistry.get_handler(node_type, "db") is RegistryLookupDbHandler