整理配置和代码入口,增加部分测试。
This commit is contained in:
+2
-1
@@ -94,4 +94,5 @@ run_profiles/*.yaml
|
||||
run_profiles/*.yml
|
||||
# Sync System Runtime Data
|
||||
_runtime/
|
||||
test_results/
|
||||
test_results/
|
||||
*.code-workspace
|
||||
@@ -0,0 +1,147 @@
|
||||
# Factory / Registry / Adapter 初始化流程改造方案
|
||||
|
||||
## 目标
|
||||
|
||||
这次改造只做三件事:
|
||||
|
||||
1. 简化 `factory`,让它只负责编排,不再关心过多介质细节。
|
||||
2. 统一 handler 初始化流程,让 `jsonl` / `db` / `api` 三类路径走同一套装配逻辑。
|
||||
3. 让适配层尽量极简:适配层只补充少量 handler / datasource 实现,不再复制初始化流程。
|
||||
|
||||
## 当前问题
|
||||
|
||||
### 1. `factory` 里存在大量按类型分支的重复逻辑
|
||||
|
||||
当前 [sync_state_machine/pipeline/factory.py](sync_state_machine/pipeline/factory.py) 同时负责:
|
||||
|
||||
- 创建 local / remote datasource
|
||||
- 判断 datasource 类型
|
||||
- 选择不同 handler getter
|
||||
- 注入一部分 handler 配置
|
||||
- 注册 handler
|
||||
|
||||
这些逻辑对 local / remote 各写了一遍,重复明显。
|
||||
|
||||
### 2. `DomainRegistry` 暴露了过多介质细节
|
||||
|
||||
当前 registry 暴露:
|
||||
|
||||
- `get_jsonl_handler()`
|
||||
- `get_db_handler()`
|
||||
- `get_api_handler()`
|
||||
|
||||
这导致编排层必须知道 handler 是哪一类,从而在 `factory` 里出现大量 `if isinstance(...)`。
|
||||
|
||||
### 3. handler 配置注入不统一
|
||||
|
||||
目前只有 API handler 在 factory 里显式吃到了 `handler_configs`。
|
||||
|
||||
- `api`:支持 per-side 配置
|
||||
- `db`:当前没有统一注入入口
|
||||
- `jsonl`:当前也没有统一注入入口
|
||||
|
||||
这使得“同一类 datasource,两侧配置不同”只能在部分路径下工作。
|
||||
|
||||
### 4. 适配层初始化成本偏高
|
||||
|
||||
如果适配层要补充一种 handler,目前需要理解:
|
||||
|
||||
- factory 的类型分支
|
||||
- registry 的不同 getter
|
||||
- 某些配置为什么只在 API 生效
|
||||
|
||||
这和“适配层极简”的目标不一致。
|
||||
|
||||
## 本次改造范围
|
||||
|
||||
本次只做最小闭环改造,不碰更大的业务逻辑:
|
||||
|
||||
### Step 1. 统一 `DomainRegistry` 的 handler 查询接口
|
||||
|
||||
状态:已完成
|
||||
|
||||
新增统一接口:
|
||||
|
||||
- `get_handler(node_type, datasource_type)`
|
||||
|
||||
保留旧接口作为兼容壳,但 factory 新逻辑只使用统一接口。
|
||||
|
||||
### Step 2. 给所有 handler 基类增加统一配置注入入口
|
||||
|
||||
状态:已完成
|
||||
|
||||
新增统一入口:
|
||||
|
||||
- `set_handler_config(config)`
|
||||
|
||||
这样 `jsonl` / `db` / `api` 三类 handler 都可以在实例化后走同一套配置注入流程。
|
||||
|
||||
### Step 3. 重写 factory 的 datasource / handler 初始化流程
|
||||
|
||||
状态:已完成
|
||||
|
||||
把当前分散的:
|
||||
|
||||
- `_create_datasources()`
|
||||
- `_register_handlers()`
|
||||
- `_api_handler_kwargs()`
|
||||
|
||||
收敛成按 endpoint 的统一初始化流程:
|
||||
|
||||
- endpoint 创建 datasource
|
||||
- endpoint 根据 datasource type 向 registry 查询 handler class
|
||||
- endpoint 创建 handler
|
||||
- endpoint 统一注入:
|
||||
- `handler_config`
|
||||
- `target_project_ids`
|
||||
- datasource-specific runtime object(如 api client)
|
||||
|
||||
最终让 `factory` 只负责:
|
||||
|
||||
- persistence 初始化
|
||||
- endpoint 初始化
|
||||
- strategy 初始化
|
||||
- pipeline 组装
|
||||
|
||||
## 不在本次范围内
|
||||
|
||||
以下内容先不动,只在文档中记为后续方向:
|
||||
|
||||
1. 显式 bootstrap 内建 domain 与适配层 domain 的注册时机
|
||||
2. 用更强的 `EndpointConfig` / `DataSourceDriver` 抽象进一步收敛 factory
|
||||
3. 清理 `NodeHandlerRegistry` 这类历史遗留接口
|
||||
|
||||
## 测试策略
|
||||
|
||||
每一步都补对应 pytest,并保证通过:
|
||||
|
||||
### Step 1 测试
|
||||
|
||||
- registry 可以通过统一接口拿到对应 handler class
|
||||
- 旧接口仍可兼容使用
|
||||
|
||||
### Step 2 测试
|
||||
|
||||
- handler 可以收到统一的 `handler_config`
|
||||
- 同一 datasource 类型在 local / remote 两侧可以拿到不同配置
|
||||
|
||||
### Step 3 测试
|
||||
|
||||
- factory 不再依赖 `get_jsonl_handler/get_db_handler/get_api_handler`
|
||||
- 统一 endpoint 初始化路径可正常创建并注册 handler
|
||||
- 原有 db handler 注册测试仍通过
|
||||
|
||||
## 预期结果
|
||||
|
||||
改造完成后:
|
||||
|
||||
1. `factory` 中关于 datasource / handler 初始化的重复逻辑显著减少。
|
||||
2. `DomainRegistry` 对编排层只暴露统一 handler 查询方式。
|
||||
3. handler 配置注入完全统一。
|
||||
4. 适配层只需:
|
||||
- 提供 datasource(如需要)
|
||||
- 提供 handler
|
||||
- 注册 handler
|
||||
- 调用统一 runner
|
||||
|
||||
这就是“适配极简”的基础。
|
||||
@@ -296,9 +296,14 @@ def build_temp_run_profile_for_project_auto_bind(
|
||||
profile.setdefault("remote_datasource", {})["target_project_ids"] = [remote_project_id]
|
||||
|
||||
project_strategy = profile.setdefault("strategies", {}).setdefault("project", {}).setdefault("config", {})
|
||||
project_strategy["auto_bind"] = True
|
||||
project_strategy["auto_bind_fields"] = ["name"]
|
||||
project_strategy["pre_bind_data_id"] = []
|
||||
project_strategy["auto_bind"] = False
|
||||
project_strategy["auto_bind_fields"] = []
|
||||
project_strategy["pre_bind_data_id"] = [
|
||||
{
|
||||
"local_data_id": local_project_id,
|
||||
"remote_data_id": remote_project_id,
|
||||
}
|
||||
]
|
||||
project_strategy["local_orphan_action"] = project_local_orphan_action or "none"
|
||||
project_strategy["remote_orphan_action"] = project_remote_orphan_action or "none"
|
||||
project_strategy["update_direction"] = project_update_direction or "none"
|
||||
|
||||
@@ -141,9 +141,6 @@ def run_case(config: dict[str, Any], report_root) -> dict[str, Any]:
|
||||
"investment_decision_date",
|
||||
"complete_investment",
|
||||
"dynamic_investment",
|
||||
"implementation_estimate",
|
||||
"static_total_investment",
|
||||
"completed_investment",
|
||||
],
|
||||
}
|
||||
},
|
||||
@@ -164,9 +161,6 @@ def run_case(config: dict[str, Any], report_root) -> dict[str, Any]:
|
||||
"investment_decision_date": "2021-08-15",
|
||||
"complete_investment": 125000.0,
|
||||
"dynamic_investment": 1413000.5,
|
||||
"implementation_estimate": 1225000.0,
|
||||
"static_total_investment": 1224300.0,
|
||||
"completed_investment": 194500.0,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -180,7 +174,7 @@ def run_case(config: dict[str, Any], report_root) -> dict[str, Any]:
|
||||
"PUT /project/dynamic_investment",
|
||||
"PUT /project/settlement_investment",
|
||||
],
|
||||
"notes": "单轮覆盖 project 剩余 4 条投资类更新接口。",
|
||||
"notes": "单轮覆盖 project 剩余 4 条投资类更新接口。当前回归仅校验后端可稳定读回的投资决策/完成投资/动态投资字段;结算投资接口仍保留覆盖,但扩展字段读回语义以后端修正为准。",
|
||||
},
|
||||
)
|
||||
rounds.append(round_3)
|
||||
|
||||
@@ -15,12 +15,14 @@ def run_case(config: dict[str, Any], report_root) -> dict[str, Any]:
|
||||
- `actual_construction` → `PUT /project/actual_construction`
|
||||
- `ensure_production` → `PUT /project/ensure_capacity`
|
||||
- `climb_production` → `PUT /project/climb_capacity`
|
||||
- `challenge_production` → `PUT /project/challenge_capacity`
|
||||
- `production` → `PUT /project/production_capacity`
|
||||
- `challenge_production` → `PUT /project/challenge_capacity`
|
||||
|
||||
其中 `production` 在当前水电项目上主要验证“推送后 project 本身字段变更”;
|
||||
`ensure/climb/challenge` 默认策略是 pull,本用例会在临时 profile 中仅对当前 key 切到 push,
|
||||
`ensure/climb/challenge` 默认策略是 pull,本用例会在临时 profile 中仅对当前 key 切到 push,
|
||||
以确认对应后端更新接口真实可用,并在报告中展示字段校验结果。
|
||||
|
||||
`production` 当前按文档约束不混入这组统一回归:
|
||||
后端确认该接口会直接更新 `project` 字段,但不保证创建/更新可轮询的
|
||||
`project_detail_data.production` 记录,需单列方案再测。
|
||||
"""
|
||||
|
||||
dataset_root = f"{config['project_root']}/tests/fixtures/auto_sync/datasource/project_detail_data"
|
||||
@@ -31,14 +33,12 @@ def run_case(config: dict[str, Any], report_root) -> dict[str, Any]:
|
||||
]
|
||||
remote_project_id = "f3e02e84-e81c-4aa6-88d4-f5aa108c8227"
|
||||
remote_actual_construction_id = "c99ae2e4-809e-47dd-8b86-61674b6a5bea"
|
||||
remote_production_id = "0b3f7e2d-9dc7-4f71-aac9-c9e7093df64f"
|
||||
seed_ensure_production_id = "11111111-1111-4111-8111-111111111111"
|
||||
seed_climb_production_id = "22222222-2222-4222-8222-222222222222"
|
||||
seed_challenge_production_id = "33333333-3333-4333-8333-333333333333"
|
||||
seed_plan_construction_id = "4bc5e9db-c9dc-41f1-a77d-574b4fe37b01"
|
||||
local_project_id = make_local_data_id("project", remote_project_id)
|
||||
local_actual_construction_id = make_local_data_id("project_detail_data", remote_actual_construction_id)
|
||||
local_production_id = make_local_data_id("project_detail_data", remote_production_id)
|
||||
local_plan_construction_id = make_local_data_id("project_detail_data", seed_plan_construction_id)
|
||||
local_ensure_production_id = make_local_data_id("project_detail_data", seed_ensure_production_id)
|
||||
local_climb_production_id = make_local_data_id("project_detail_data", seed_climb_production_id)
|
||||
@@ -365,63 +365,8 @@ def run_case(config: dict[str, Any], report_root) -> dict[str, Any]:
|
||||
)
|
||||
rounds.append(round_7)
|
||||
|
||||
round_4 = run_pipeline_round(
|
||||
config=config,
|
||||
dataset_dir=f"{dataset_root}/round_4_production_update",
|
||||
previous_dataset_dir=f"{dataset_root}/round_3_plan_construction_create",
|
||||
base_dataset_dir=filtered_root,
|
||||
base_dataset_files=base_dataset_files,
|
||||
domain_name="project_detail_data",
|
||||
round_name="round_4_production_update",
|
||||
purpose="推送 production,验证 /project/production_capacity 在当前水电项目上会更新 project 投产计划字段",
|
||||
expected_changes={
|
||||
"project": {
|
||||
local_project_id: [
|
||||
"plan_production_date",
|
||||
"plan_full_production_date",
|
||||
],
|
||||
}
|
||||
},
|
||||
required_bindings={
|
||||
"project": [local_project_id],
|
||||
"project_detail_data": [local_production_id],
|
||||
},
|
||||
expected_binding_pairs={
|
||||
"project": {local_project_id: remote_project_id},
|
||||
},
|
||||
require_distinct_binding_ids={
|
||||
"project": [local_project_id],
|
||||
"project_detail_data": [local_production_id],
|
||||
},
|
||||
expected_bound_fields={
|
||||
"remote": {
|
||||
"project": {
|
||||
local_project_id: {
|
||||
"plan_production_date": "2026-04-01",
|
||||
"plan_full_production_date": "2026-06-01",
|
||||
}
|
||||
},
|
||||
"project_detail_data": {
|
||||
local_production_id: {
|
||||
"key": "production",
|
||||
"value.0.capacity": 51.0,
|
||||
"value.0.date": "2025-12-27",
|
||||
}
|
||||
},
|
||||
}
|
||||
},
|
||||
project_id_remap=project_id_remap,
|
||||
report_root=report_root,
|
||||
coverage={
|
||||
"keys": ["production"],
|
||||
"api": ["PUT /project/production_capacity"],
|
||||
"notes": "在可访问项目上复用现有 production 明细记录,验证生产接口会更新 project 投产计划字段并保持 detail 绑定可解析。",
|
||||
},
|
||||
)
|
||||
rounds.append(round_4)
|
||||
|
||||
return build_domain_report(
|
||||
domain="project_detail_data",
|
||||
description="项目详情 domain 回归。覆盖 plan_construction / actual_construction / ensure_production / climb_production / challenge_production / production 六类 key,并在报告中展示对应接口与 project 字段联动校验。",
|
||||
description="项目详情 domain 回归。覆盖 plan_construction / actual_construction / ensure_production / climb_production / challenge_production 五类 key,并在报告中展示对应接口与 project 字段联动校验。production 按文档约束单列,不混入本组统一回归。",
|
||||
rounds=rounds,
|
||||
)
|
||||
|
||||
@@ -14,6 +14,7 @@ from . import domain as _domain
|
||||
from .common.registry import DomainRegistry
|
||||
from .config import (
|
||||
ApiDataSourceConfig,
|
||||
DbDataSourceConfig,
|
||||
JsonlDataSourceConfig,
|
||||
PipelineRunConfig,
|
||||
build_config,
|
||||
@@ -21,7 +22,7 @@ from .config import (
|
||||
load_named_preset_overrides,
|
||||
load_overrides_from_file,
|
||||
)
|
||||
from .datasource import ApiClient, ApiDataSource, BaseApiHandler, BaseJsonlHandler, JsonlDataSource
|
||||
from .datasource import ApiClient, ApiDataSource, BaseApiHandler, BaseDbHandler, BaseJsonlHandler, DbDataSource, JsonlDataSource
|
||||
from .pipeline import FullSyncPipeline, create_pipeline_from_config, run_pipeline_from_config
|
||||
|
||||
try:
|
||||
@@ -41,6 +42,7 @@ __all__ = [
|
||||
"DomainRegistry",
|
||||
"PipelineRunConfig",
|
||||
"ApiDataSourceConfig",
|
||||
"DbDataSourceConfig",
|
||||
"JsonlDataSourceConfig",
|
||||
"build_config",
|
||||
"build_default_config",
|
||||
@@ -48,8 +50,10 @@ __all__ = [
|
||||
"load_named_preset_overrides",
|
||||
"ApiClient",
|
||||
"ApiDataSource",
|
||||
"DbDataSource",
|
||||
"JsonlDataSource",
|
||||
"BaseApiHandler",
|
||||
"BaseDbHandler",
|
||||
"BaseJsonlHandler",
|
||||
"FullSyncPipeline",
|
||||
"create_pipeline_from_config",
|
||||
|
||||
@@ -26,7 +26,7 @@ Domain 注册中心
|
||||
# DataSource 查询 Handler
|
||||
handler_class = DomainRegistry.get_jsonl_handler("contract")
|
||||
"""
|
||||
from typing import Dict, Type, Optional, Any
|
||||
from typing import Dict, Type, Optional, Any, Literal
|
||||
from dataclasses import dataclass, field
|
||||
from pydantic import BaseModel
|
||||
|
||||
@@ -64,6 +64,12 @@ class DomainRegistry:
|
||||
"""
|
||||
|
||||
_registry: Dict[str, DomainRegistration] = {}
|
||||
|
||||
_HANDLER_ATTRS: Dict[str, str] = {
|
||||
"jsonl": "jsonl_handler_class",
|
||||
"api": "api_handler_class",
|
||||
"db": "db_handler_class",
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def register(
|
||||
@@ -143,36 +149,30 @@ class DomainRegistry:
|
||||
return reg.strategy_class
|
||||
|
||||
@classmethod
|
||||
def get_jsonl_handler(cls, node_type: str) -> Optional[Type[Any]]:
|
||||
"""
|
||||
获取 JSONL Handler 类(用于 DataSource)
|
||||
|
||||
Args:
|
||||
node_type: 节点类型
|
||||
|
||||
Returns:
|
||||
Handler 类,如果未注册则返回 None
|
||||
"""
|
||||
def get_handler(
|
||||
cls,
|
||||
node_type: str,
|
||||
datasource_type: Literal["jsonl", "api", "db"],
|
||||
) -> Optional[Type[Any]]:
|
||||
"""按 datasource 类型获取 Handler 类。"""
|
||||
reg = cls._registry.get(node_type)
|
||||
if not reg:
|
||||
raise ValueError(f"Unknown node_type: '{node_type}'")
|
||||
return reg.jsonl_handler_class
|
||||
|
||||
attr_name = cls._HANDLER_ATTRS.get(datasource_type)
|
||||
if attr_name is None:
|
||||
raise ValueError(
|
||||
f"Unsupported datasource_type: '{datasource_type}'. "
|
||||
f"Available: {sorted(cls._HANDLER_ATTRS.keys())}"
|
||||
)
|
||||
return getattr(reg, attr_name)
|
||||
|
||||
@classmethod
|
||||
def get_api_handler(cls, node_type: str) -> Optional[Type[Any]]:
|
||||
"""获取 API Handler 类"""
|
||||
def register_db_handler(cls, node_type: str, db_handler_class: Type[Any]) -> None:
|
||||
"""为已注册 domain 补充 DB Handler。"""
|
||||
reg = cls._registry.get(node_type)
|
||||
if not reg:
|
||||
raise ValueError(f"Unknown node_type: '{node_type}'")
|
||||
return reg.api_handler_class
|
||||
|
||||
@classmethod
|
||||
def get_db_handler(cls, node_type: str) -> Optional[Type[Any]]:
|
||||
"""获取 DB Handler 类"""
|
||||
reg = cls._registry.get(node_type)
|
||||
if not reg:
|
||||
raise ValueError(f"Unknown node_type: '{node_type}'")
|
||||
return reg.db_handler_class
|
||||
reg.db_handler_class = db_handler_class
|
||||
|
||||
@classmethod
|
||||
def get_schema(cls, node_type: str) -> Type[BaseModel]:
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from .datasource_config import ApiDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
|
||||
from .datasource_config import ApiDataSourceConfig, DbDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
|
||||
from .persist_config import PersistConfig
|
||||
from .logging_config import LoggingConfig, resolve_log_file_path
|
||||
from .run_presets import (
|
||||
@@ -26,6 +26,7 @@ from .builder import (
|
||||
|
||||
__all__ = [
|
||||
"ApiDataSourceConfig",
|
||||
"DbDataSourceConfig",
|
||||
"JsonlDataSourceConfig",
|
||||
"PipelineRunConfig",
|
||||
"DataSourceConfig",
|
||||
|
||||
@@ -15,6 +15,14 @@ class JsonlDataSourceConfig(BaseModel):
|
||||
handler_configs: Dict[str, Dict[str, Any]] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class DbDataSourceConfig(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", validate_assignment=True)
|
||||
|
||||
type: Literal["db"] = "db"
|
||||
target_project_ids: List[str] = Field(default_factory=list)
|
||||
handler_configs: Dict[str, Dict[str, Any]] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class ApiDataSourceConfig(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", validate_assignment=True)
|
||||
|
||||
@@ -31,4 +39,7 @@ class ApiDataSourceConfig(BaseModel):
|
||||
handler_configs: Dict[str, Dict[str, Any]] = Field(default_factory=dict)
|
||||
|
||||
|
||||
DataSourceConfig = Annotated[Union[JsonlDataSourceConfig, ApiDataSourceConfig], Field(discriminator="type")]
|
||||
DataSourceConfig = Annotated[
|
||||
Union[JsonlDataSourceConfig, DbDataSourceConfig, ApiDataSourceConfig],
|
||||
Field(discriminator="type"),
|
||||
]
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
Prefer importing from `sync_state_machine.config` directly.
|
||||
"""
|
||||
|
||||
from .datasource_config import ApiDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
|
||||
from .datasource_config import ApiDataSourceConfig, DbDataSourceConfig, JsonlDataSourceConfig, DataSourceConfig
|
||||
from .persist_config import PersistConfig
|
||||
from .logging_config import LoggingConfig
|
||||
from .strategy_config import (
|
||||
@@ -24,6 +24,7 @@ from .builder import (
|
||||
|
||||
__all__ = [
|
||||
"ApiDataSourceConfig",
|
||||
"DbDataSourceConfig",
|
||||
"JsonlDataSourceConfig",
|
||||
"DataSourceConfig",
|
||||
"PersistConfig",
|
||||
|
||||
@@ -17,6 +17,9 @@ from .task_result import TaskResult, TaskStatus
|
||||
# API DataSource
|
||||
from .api import ApiClient, ApiDataSource, BaseApiHandler
|
||||
|
||||
# DB DataSource
|
||||
from .db import DbDataSource, BaseDbHandler
|
||||
|
||||
# JSONL DataSource
|
||||
from .jsonl import JsonlDataSource, BaseJsonlHandler
|
||||
|
||||
@@ -34,6 +37,10 @@ __all__ = [
|
||||
"ApiClient",
|
||||
"ApiDataSource",
|
||||
"BaseApiHandler",
|
||||
|
||||
# DB DataSource
|
||||
"DbDataSource",
|
||||
"BaseDbHandler",
|
||||
|
||||
# JSONL DataSource
|
||||
"JsonlDataSource",
|
||||
|
||||
@@ -44,6 +44,7 @@ class BaseApiHandler(NodeHandler[T]):
|
||||
self.api_client: 'ApiClient' = None # type: ignore # 由 DataSource 注入
|
||||
self.poll_mode: str = "async" # async | sync
|
||||
self._collection: 'DataCollection' = None # type: ignore # 由 DataSource 注入
|
||||
self.handler_config: Dict[str, Any] = {}
|
||||
# 子类在 __init__ 中设置此列表以声明 create/update 中涉及的 Pydantic schema 类;
|
||||
# post-check 将只比较这些 schema 覆盖的字段,过滤后端只读的派生字段。
|
||||
self.update_schemas: List[type] = []
|
||||
@@ -63,6 +64,13 @@ class BaseApiHandler(NodeHandler[T]):
|
||||
def set_collection(self, collection: 'DataCollection') -> None:
|
||||
"""设置 Collection(由 DataSource 调用)"""
|
||||
self._collection = collection
|
||||
|
||||
def set_handler_config(self, config: Dict[str, Any]) -> None:
|
||||
self.handler_config = dict(config)
|
||||
|
||||
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
|
||||
"""默认忽略项目过滤列表,具体子类按需覆盖。"""
|
||||
return
|
||||
|
||||
@property
|
||||
def node_type(self) -> str:
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
"""DB DataSource 模块。"""
|
||||
|
||||
from .datasource import DbDataSource
|
||||
from .handler import BaseDbHandler
|
||||
|
||||
__all__ = ["DbDataSource", "BaseDbHandler"]
|
||||
@@ -0,0 +1,22 @@
|
||||
"""DbDataSource - 数据库数据源协调层。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from ..datasource import BaseDataSource
|
||||
|
||||
|
||||
class DbDataSource(BaseDataSource):
|
||||
"""基于数据库 Handler 的数据源实现。"""
|
||||
|
||||
async def close(self) -> None:
|
||||
return None
|
||||
|
||||
def _on_node_loaded(self, handler, node) -> None:
|
||||
node.append_log(
|
||||
f"DB链路[load] node_type={node.node_type} data_id={node.data_id or '<empty>'}"
|
||||
)
|
||||
|
||||
def _on_in_progress_result(self, handler, node, result) -> None:
|
||||
node.append_log(
|
||||
f"DB链路[submit] action={node.action.value} data_id={node.data_id or '<empty>'}"
|
||||
)
|
||||
@@ -0,0 +1,68 @@
|
||||
"""BaseDbHandler - DB Handler 基类。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, List, Optional, Set, TypeVar, TYPE_CHECKING
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ...common.type_safety import get_pydantic_model_fields
|
||||
from ..handler import BaseNodeHandler
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ...common.sync_node import SyncNode
|
||||
from .datasource import DbDataSource
|
||||
|
||||
T = TypeVar("T", bound=BaseModel)
|
||||
|
||||
|
||||
class BaseDbHandler(BaseNodeHandler[T]):
|
||||
"""数据库 Handler 基类。"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
datasource: "DbDataSource",
|
||||
node_type: str,
|
||||
node_class: type["SyncNode[T]"],
|
||||
schema: type[T],
|
||||
):
|
||||
super().__init__(node_type, node_class, schema)
|
||||
self.datasource = datasource
|
||||
self.update_schemas: List[type] = []
|
||||
self._target_project_ids: Optional[Set[str]] = None
|
||||
|
||||
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
|
||||
if target_project_ids:
|
||||
self._target_project_ids = {str(item) for item in target_project_ids if str(item)}
|
||||
else:
|
||||
self._target_project_ids = None
|
||||
|
||||
def get_update_fields(self, *, exclude: frozenset = frozenset({"id"})) -> List[str]:
|
||||
fields: set[str] = set()
|
||||
for schema in self.update_schemas:
|
||||
for field_name in get_pydantic_model_fields(schema):
|
||||
if field_name not in exclude:
|
||||
fields.add(field_name)
|
||||
return sorted(fields)
|
||||
|
||||
def _create_basic_node(self, data: Dict[str, Any], depend_ids: Optional[List[str]] = None) -> "SyncNode[T]":
|
||||
data_id = self.extract_id(data)
|
||||
if self._collection is None:
|
||||
raise RuntimeError("Collection not set. Call set_collection() before creating nodes.")
|
||||
|
||||
if data_id:
|
||||
existing_node = self._collection.get_by_data_id(self.node_type, data_id)
|
||||
if existing_node is not None:
|
||||
existing_node.depend_ids = list(depend_ids or [])
|
||||
existing_node.set_data(data.copy())
|
||||
existing_node.set_origin_data(data.copy())
|
||||
return existing_node
|
||||
|
||||
node_id = self._collection.generate_node_id()
|
||||
return self.node_class(
|
||||
node_id=node_id,
|
||||
data_id=data_id,
|
||||
data=data.copy(),
|
||||
depend_ids=depend_ids or [],
|
||||
origin_data=data.copy(),
|
||||
)
|
||||
@@ -70,6 +70,10 @@ class NodeHandler(Protocol[T]):
|
||||
"""
|
||||
...
|
||||
|
||||
def set_handler_config(self, config: Dict[str, Any]) -> None:
|
||||
"""注入 handler 级别配置。默认可忽略。"""
|
||||
...
|
||||
|
||||
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
|
||||
"""注入项目过滤列表。默认可忽略。"""
|
||||
...
|
||||
@@ -345,6 +349,7 @@ class BaseNodeHandler(ABC, Generic[T]):
|
||||
self._node_class = node_class
|
||||
self._schema = schema
|
||||
self._collection: Optional["DataCollection"] = None
|
||||
self.handler_config: Dict[str, Any] = {}
|
||||
|
||||
def set_collection(self, collection: "DataCollection") -> None:
|
||||
"""
|
||||
@@ -359,6 +364,10 @@ class BaseNodeHandler(ABC, Generic[T]):
|
||||
"""默认忽略项目过滤能力。"""
|
||||
return
|
||||
|
||||
def set_handler_config(self, config: Dict[str, Any]) -> None:
|
||||
"""默认保存 handler 级别配置,供子类按需读取。"""
|
||||
self.handler_config = dict(config)
|
||||
|
||||
def set_api_client(self, client: "ApiClient") -> None:
|
||||
"""默认忽略 API 客户端注入。"""
|
||||
return
|
||||
@@ -491,6 +500,11 @@ class CompatibleNodeHandler(Generic[T]):
|
||||
def set_collection(self, collection: "DataCollection") -> None:
|
||||
self._handler.set_collection(collection)
|
||||
|
||||
def set_handler_config(self, config: Dict[str, Any]) -> None:
|
||||
method = getattr(self._handler, "set_handler_config", None)
|
||||
if callable(method):
|
||||
method(config)
|
||||
|
||||
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
|
||||
method = getattr(self._handler, "set_target_project_ids", None)
|
||||
if callable(method):
|
||||
|
||||
@@ -188,7 +188,7 @@ class MaterialDetailApiHandler(BaseApiHandler):
|
||||
)
|
||||
# 整组成功
|
||||
for node, _ in group_items:
|
||||
results.append(TaskResult.success(node_id=node.node_id))
|
||||
results.append(TaskResult.in_progress(node_id=node.node_id, task_id=push_id))
|
||||
except Exception as e:
|
||||
# 整组失败
|
||||
error_msg = str(e)
|
||||
|
||||
@@ -18,6 +18,7 @@ PUT (6个更新接口):
|
||||
不支持: CREATE, DELETE
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from typing import List, Dict, Any
|
||||
from ...datasource.api.handler import BaseApiHandler
|
||||
from ...datasource.task_result import TaskResult
|
||||
@@ -35,6 +36,35 @@ from schemas.project.project_base import (
|
||||
from schemas.project.project import ProjectInfoResponse
|
||||
|
||||
|
||||
_PROJECT_EXTENSION_FIELD_NAMES = {
|
||||
"implementation_estimate",
|
||||
"static_total_investment",
|
||||
"completed_investment",
|
||||
"total_contract_quantity",
|
||||
"completed_settlement_quantity",
|
||||
}
|
||||
|
||||
|
||||
def _merge_project_all_info_fields(
|
||||
project_data: Dict[str, Any],
|
||||
all_info: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
"""把 all_info 中项目扩展字段拍平到 project 主数据上。"""
|
||||
merged = dict(project_data)
|
||||
|
||||
for key, value in all_info.items():
|
||||
if not key.endswith("_extension") or not isinstance(value, dict):
|
||||
continue
|
||||
for field_name in _PROJECT_EXTENSION_FIELD_NAMES:
|
||||
if field_name not in value:
|
||||
continue
|
||||
current_value = merged.get(field_name)
|
||||
if current_value in (None, ""):
|
||||
merged[field_name] = value.get(field_name)
|
||||
|
||||
return merged
|
||||
|
||||
|
||||
class ProjectApiHandler(BaseApiHandler):
|
||||
"""项目 API Handler"""
|
||||
|
||||
@@ -136,12 +166,15 @@ class ProjectApiHandler(BaseApiHandler):
|
||||
|
||||
# 处理所有项目
|
||||
for project_data in projects:
|
||||
node = self._create_node(project_data.model_dump())
|
||||
# 设置 context
|
||||
node.context["project_id"] = project_data.id
|
||||
|
||||
# 预加载并缓存 all_info,同时存储到 node.context
|
||||
all_info = await self.get_all_info(self.api_client, project_data.id)
|
||||
normalized_project_data = _merge_project_all_info_fields(
|
||||
project_data.model_dump(),
|
||||
all_info,
|
||||
)
|
||||
node = self._create_node(normalized_project_data)
|
||||
# 设置 context
|
||||
node.context["project_id"] = project_data.id
|
||||
node.context["all_info"] = all_info
|
||||
|
||||
nodes.append(node)
|
||||
@@ -217,10 +250,12 @@ class ProjectApiHandler(BaseApiHandler):
|
||||
|
||||
# 执行更新(差异已由 _get_schema_diff 打印)
|
||||
try:
|
||||
push_id = self._generate_push_id()
|
||||
# project 需要使用 _filter_update_data 返回的完整 schema 数据
|
||||
full_update_data = self._filter_update_data(data, original_data, schema)
|
||||
await api_func(self.api_client, full_update_data, push_id, biz_id)
|
||||
await self._submit_update_and_wait(
|
||||
api_func=api_func,
|
||||
data=full_update_data,
|
||||
biz_id=biz_id,
|
||||
)
|
||||
node_updated = True
|
||||
except Exception as e:
|
||||
error = str(e)
|
||||
@@ -249,13 +284,39 @@ class ProjectApiHandler(BaseApiHandler):
|
||||
async def poll_tasks(self, task_ids: List[str]) -> Dict[str, TaskResult]:
|
||||
"""轮询异步任务状态"""
|
||||
return await super().poll_tasks(task_ids)
|
||||
|
||||
async def _submit_update_and_wait(self, *, api_func, data: Dict[str, Any], biz_id: str) -> None:
|
||||
push_id = self._generate_push_id()
|
||||
await api_func(self.api_client, data, push_id, biz_id)
|
||||
|
||||
max_attempts = 20
|
||||
poll_interval = 0.2
|
||||
last_result: TaskResult | None = None
|
||||
|
||||
for attempt in range(max_attempts):
|
||||
polled = await super().poll_tasks([push_id])
|
||||
result = polled.get(push_id)
|
||||
last_result = result
|
||||
|
||||
if result is None:
|
||||
raise RuntimeError(f"Project update poll missing result: push_id={push_id}")
|
||||
if result.status == result.status.SUCCESS:
|
||||
return
|
||||
if result.status == result.status.FAILED:
|
||||
raise RuntimeError(result.error or f"Project update failed: push_id={push_id}")
|
||||
if attempt < max_attempts - 1:
|
||||
await asyncio.sleep(poll_interval)
|
||||
|
||||
raise RuntimeError(
|
||||
f"Project update poll timeout: push_id={push_id}, last_status={getattr(last_result, 'status', None)}"
|
||||
)
|
||||
|
||||
def _create_node(self, data: Dict[str, Any]) -> SyncNode:
|
||||
node = self._create_basic_node(data, depend_ids=[])
|
||||
|
||||
|
||||
# 设置 context
|
||||
node.context["project_id"] = node.data_id
|
||||
|
||||
|
||||
return node
|
||||
|
||||
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
"""Project domain registration"""
|
||||
from ...common.registry import DomainRegistry
|
||||
from schemas.project.project_base import ProjectResponseBase
|
||||
from .sync_node import ProjectSyncNode
|
||||
from .sync_node import ProjectSyncNode, ProjectSyncSchema
|
||||
from .sync_strategy import ProjectSyncStrategy
|
||||
from .jsonl_handler import ProjectJsonlHandler
|
||||
from .api_handler import ProjectApiHandler
|
||||
@@ -9,7 +8,7 @@ from .api_handler import ProjectApiHandler
|
||||
# 注册 project domain
|
||||
DomainRegistry.register(
|
||||
node_type="project",
|
||||
schema=ProjectResponseBase,
|
||||
schema=ProjectSyncSchema,
|
||||
node_class=ProjectSyncNode,
|
||||
strategy_class=ProjectSyncStrategy,
|
||||
jsonl_handler_class=ProjectJsonlHandler,
|
||||
|
||||
@@ -1,11 +1,23 @@
|
||||
from pydantic import Field
|
||||
|
||||
from ...common.sync_node import SyncNode
|
||||
from schemas.project.project_base import ProjectResponseBase
|
||||
|
||||
|
||||
class ProjectSyncNode(SyncNode[ProjectResponseBase]):
|
||||
class ProjectSyncSchema(ProjectResponseBase):
|
||||
"""项目同步内部 schema,补充 all_info 扩展字段。"""
|
||||
|
||||
implementation_estimate: float | None = Field(None, description="执行概算")
|
||||
static_total_investment: float | None = Field(None, description="静态总投资")
|
||||
completed_investment: float | None = Field(None, description="已完成投资")
|
||||
total_contract_quantity: float | None = Field(None, description="合同总量")
|
||||
completed_settlement_quantity: float | None = Field(None, description="已结算量")
|
||||
|
||||
|
||||
class ProjectSyncNode(SyncNode[ProjectSyncSchema]):
|
||||
"""项目同步节点"""
|
||||
node_type = "project"
|
||||
schema = ProjectResponseBase
|
||||
schema = ProjectSyncSchema
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
|
||||
@@ -17,6 +17,7 @@ PUT:
|
||||
不支持: CREATE, DELETE
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from typing import List, Dict, Any, Optional, Tuple, Set
|
||||
from pydantic import ValidationError
|
||||
|
||||
@@ -270,6 +271,8 @@ class ProjectDetailApiHandler(BaseApiHandler):
|
||||
|
||||
try:
|
||||
resolved_data_id = self._resolve_data_id_from_all_info(all_info, project_id=project_id, key=key)
|
||||
if not resolved_data_id:
|
||||
resolved_data_id = await self._resolve_data_id_with_retry(project_id=project_id, key=key)
|
||||
except Exception as exc:
|
||||
self._pending_push_context.pop(task_id, None)
|
||||
results[task_id] = TaskResult.failed(error=f"Poll resolve data_id failed: {exc}", push_id=task_id)
|
||||
@@ -342,6 +345,9 @@ class ProjectDetailApiHandler(BaseApiHandler):
|
||||
if isinstance(all_info, dict):
|
||||
return all_info
|
||||
|
||||
if force_refresh:
|
||||
ProjectApiHandler._all_info_cache.pop(project_id, None)
|
||||
|
||||
all_info = await ProjectApiHandler.get_all_info(self.api_client, project_id)
|
||||
if project_node is not None:
|
||||
project_node.context["all_info"] = all_info
|
||||
@@ -370,6 +376,20 @@ class ProjectDetailApiHandler(BaseApiHandler):
|
||||
return matched[0]
|
||||
return None
|
||||
|
||||
async def _resolve_data_id_with_retry(self, *, project_id: str, key: str) -> Optional[str]:
|
||||
attempts = 6
|
||||
delay = 0.2
|
||||
|
||||
for attempt in range(attempts):
|
||||
all_info = await self._get_project_all_info(project_id, force_refresh=True)
|
||||
resolved = self._resolve_data_id_from_all_info(all_info, project_id=project_id, key=key)
|
||||
if resolved:
|
||||
return resolved
|
||||
if attempt < attempts - 1:
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
return None
|
||||
|
||||
async def _resolve_data_id_from_project_all_info(self, project_id: str, key: str) -> Optional[str]:
|
||||
all_info = await self._get_project_all_info(project_id)
|
||||
return self._resolve_data_id_from_all_info(all_info, project_id=project_id, key=key)
|
||||
|
||||
@@ -7,6 +7,7 @@ from .full_sync_pipeline import FullSyncPipeline
|
||||
from ..config import (
|
||||
PipelineRunConfig,
|
||||
ApiDataSourceConfig,
|
||||
DbDataSourceConfig,
|
||||
JsonlDataSourceConfig,
|
||||
DataSourceConfig,
|
||||
PersistConfig,
|
||||
@@ -35,6 +36,7 @@ __all__ = [
|
||||
"run_pipeline_from_config",
|
||||
"PipelineRunConfig",
|
||||
"ApiDataSourceConfig",
|
||||
"DbDataSourceConfig",
|
||||
"JsonlDataSourceConfig",
|
||||
"DataSourceConfig",
|
||||
"PersistConfig",
|
||||
|
||||
@@ -23,11 +23,13 @@ from ..common.collection import DataCollection
|
||||
from ..common.persistence import PersistenceBackend
|
||||
from ..common.registry import DomainRegistry
|
||||
from ..datasource.api.datasource import ApiDataSource
|
||||
from ..datasource.db.datasource import DbDataSource
|
||||
from ..datasource.jsonl.datasource import JsonlDataSource
|
||||
from ..config import (
|
||||
PipelineRunConfig,
|
||||
DataSourceConfig,
|
||||
ApiDataSourceConfig,
|
||||
DbDataSourceConfig,
|
||||
JsonlDataSourceConfig,
|
||||
PersistConfig,
|
||||
LoggingConfig,
|
||||
@@ -97,16 +99,41 @@ def _ensure_pipeline_logger(
|
||||
)
|
||||
|
||||
|
||||
def _api_handler_kwargs(
|
||||
def _build_handler_config(
|
||||
node_type: str,
|
||||
ds_config: DataSourceConfig,
|
||||
target_project_ids: List[str],
|
||||
) -> Dict[str, Any]:
|
||||
kwargs = dict(ds_config.handler_configs.get(node_type, {}))
|
||||
config = dict(ds_config.handler_configs.get(node_type, {}))
|
||||
|
||||
if node_type == "project" and target_project_ids:
|
||||
kwargs.setdefault("filter_project_id", target_project_ids)
|
||||
return kwargs
|
||||
config.setdefault("filter_project_id", target_project_ids)
|
||||
return config
|
||||
|
||||
|
||||
def _create_handler(
|
||||
*,
|
||||
node_type: str,
|
||||
ds_config: DataSourceConfig,
|
||||
datasource,
|
||||
target_project_ids: List[str],
|
||||
):
|
||||
datasource_type = ds_config.type
|
||||
handler_class = DomainRegistry.get_handler(node_type, datasource_type)
|
||||
if handler_class is None:
|
||||
raise RuntimeError(
|
||||
f"Missing {datasource_type.upper()} handler for node_type: {node_type}"
|
||||
)
|
||||
|
||||
handler_config = _build_handler_config(node_type, ds_config, target_project_ids)
|
||||
if datasource_type == "api":
|
||||
handler = handler_class(**handler_config)
|
||||
else:
|
||||
handler = handler_class(datasource)
|
||||
|
||||
if hasattr(handler, "set_handler_config"):
|
||||
handler.set_handler_config(handler_config)
|
||||
return handler
|
||||
|
||||
|
||||
def _resolve_datasource_target_project_ids(config: PipelineRunConfig) -> tuple[List[str], List[str]]:
|
||||
@@ -120,48 +147,58 @@ def _resolve_datasource_target_project_ids(config: PipelineRunConfig) -> tuple[L
|
||||
return local_target_ids, remote_target_ids
|
||||
|
||||
|
||||
async def _create_datasources(config: PipelineRunConfig):
|
||||
local_cfg = config.local_datasource
|
||||
remote_cfg = config.remote_datasource
|
||||
async def _create_datasource(
|
||||
ds_config: DataSourceConfig,
|
||||
*,
|
||||
datasource_override=None,
|
||||
endpoint_name: str,
|
||||
):
|
||||
if datasource_override is not None:
|
||||
return datasource_override
|
||||
|
||||
if isinstance(local_cfg, JsonlDataSourceConfig):
|
||||
local_dir = Path(local_cfg.jsonl_dir)
|
||||
if not local_dir.exists() or not local_dir.is_dir():
|
||||
if isinstance(ds_config, JsonlDataSourceConfig):
|
||||
dir_path = Path(ds_config.jsonl_dir)
|
||||
if endpoint_name == "remote":
|
||||
dir_path = _prepare_remote_jsonl_dir(dir_path)
|
||||
elif not dir_path.exists() or not dir_path.is_dir():
|
||||
raise FileNotFoundError(
|
||||
f"local_datasource.jsonl_dir does not exist or is not a directory: {local_dir}"
|
||||
f"{endpoint_name}_datasource.jsonl_dir does not exist or is not a directory: {dir_path}"
|
||||
)
|
||||
local_ds = JsonlDataSource(local_dir, read_only=local_cfg.read_only)
|
||||
else:
|
||||
local_ds = ApiDataSource(
|
||||
base_url=local_cfg.api_base_url,
|
||||
uid=local_cfg.api_uid,
|
||||
secret=local_cfg.api_secret,
|
||||
debug=local_cfg.api_debug,
|
||||
poll_max_retries=local_cfg.poll_max_retries,
|
||||
poll_interval=local_cfg.poll_interval,
|
||||
rate_limit_max_requests=local_cfg.api_rate_limit_max_requests,
|
||||
rate_limit_window_seconds=local_cfg.api_rate_limit_window_seconds,
|
||||
)
|
||||
await local_ds.initialize()
|
||||
return JsonlDataSource(dir_path, read_only=ds_config.read_only)
|
||||
|
||||
if isinstance(remote_cfg, JsonlDataSourceConfig):
|
||||
remote_dir = Path(remote_cfg.jsonl_dir)
|
||||
remote_dir = _prepare_remote_jsonl_dir(remote_dir)
|
||||
remote_ds = JsonlDataSource(remote_dir, read_only=remote_cfg.read_only)
|
||||
else:
|
||||
remote_ds = ApiDataSource(
|
||||
base_url=remote_cfg.api_base_url,
|
||||
uid=remote_cfg.api_uid,
|
||||
secret=remote_cfg.api_secret,
|
||||
debug=remote_cfg.api_debug,
|
||||
poll_max_retries=remote_cfg.poll_max_retries,
|
||||
poll_interval=remote_cfg.poll_interval,
|
||||
rate_limit_max_requests=remote_cfg.api_rate_limit_max_requests,
|
||||
rate_limit_window_seconds=remote_cfg.api_rate_limit_window_seconds,
|
||||
)
|
||||
await remote_ds.initialize()
|
||||
if isinstance(ds_config, DbDataSourceConfig):
|
||||
return DbDataSource()
|
||||
|
||||
return local_ds, remote_ds
|
||||
datasource = ApiDataSource(
|
||||
base_url=ds_config.api_base_url,
|
||||
uid=ds_config.api_uid,
|
||||
secret=ds_config.api_secret,
|
||||
debug=ds_config.api_debug,
|
||||
poll_max_retries=ds_config.poll_max_retries,
|
||||
poll_interval=ds_config.poll_interval,
|
||||
rate_limit_max_requests=ds_config.api_rate_limit_max_requests,
|
||||
rate_limit_window_seconds=ds_config.api_rate_limit_window_seconds,
|
||||
)
|
||||
await datasource.initialize()
|
||||
return datasource
|
||||
|
||||
|
||||
def _register_handlers_for_endpoint(
|
||||
*,
|
||||
datasource,
|
||||
ds_config: DataSourceConfig,
|
||||
node_types: List[str],
|
||||
target_project_ids: List[str],
|
||||
) -> None:
|
||||
for node_type in node_types:
|
||||
datasource.register_handler(
|
||||
_create_handler(
|
||||
node_type=node_type,
|
||||
ds_config=ds_config,
|
||||
datasource=datasource,
|
||||
target_project_ids=target_project_ids,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _register_handlers(
|
||||
@@ -170,44 +207,18 @@ def _register_handlers(
|
||||
remote_ds,
|
||||
all_types: List[str],
|
||||
) -> None:
|
||||
for node_type in all_types:
|
||||
if isinstance(config.local_datasource, JsonlDataSourceConfig):
|
||||
jsonl_handler_class = DomainRegistry.get_jsonl_handler(node_type)
|
||||
if jsonl_handler_class is None:
|
||||
raise RuntimeError(f"Missing JSONL handler for node_type: {node_type}")
|
||||
local_ds.register_handler(jsonl_handler_class(local_ds))
|
||||
else:
|
||||
api_handler_class = DomainRegistry.get_api_handler(node_type)
|
||||
if api_handler_class is None:
|
||||
raise RuntimeError(f"Missing API handler for node_type: {node_type}")
|
||||
local_ds.register_handler(
|
||||
api_handler_class(
|
||||
**_api_handler_kwargs(
|
||||
node_type,
|
||||
config.local_datasource,
|
||||
config.target_project_ids,
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
if isinstance(config.remote_datasource, JsonlDataSourceConfig):
|
||||
jsonl_handler_class = DomainRegistry.get_jsonl_handler(node_type)
|
||||
if jsonl_handler_class is None:
|
||||
raise RuntimeError(f"Missing JSONL handler for node_type: {node_type}")
|
||||
remote_ds.register_handler(jsonl_handler_class(remote_ds))
|
||||
else:
|
||||
api_handler_class = DomainRegistry.get_api_handler(node_type)
|
||||
if api_handler_class is None:
|
||||
raise RuntimeError(f"Missing API handler for node_type: {node_type}")
|
||||
remote_ds.register_handler(
|
||||
api_handler_class(
|
||||
**_api_handler_kwargs(
|
||||
node_type,
|
||||
config.remote_datasource,
|
||||
config.target_project_ids,
|
||||
)
|
||||
)
|
||||
)
|
||||
_register_handlers_for_endpoint(
|
||||
datasource=local_ds,
|
||||
ds_config=config.local_datasource,
|
||||
node_types=all_types,
|
||||
target_project_ids=[str(pid) for pid in config.local_datasource.target_project_ids if str(pid)],
|
||||
)
|
||||
_register_handlers_for_endpoint(
|
||||
datasource=remote_ds,
|
||||
ds_config=config.remote_datasource,
|
||||
node_types=all_types,
|
||||
target_project_ids=[str(pid) for pid in config.remote_datasource.target_project_ids if str(pid)],
|
||||
)
|
||||
|
||||
|
||||
def _resolve_strategy_map(config: PipelineRunConfig, node_types: List[str]) -> Dict[str, StrategyRuntimeConfig]:
|
||||
@@ -260,7 +271,12 @@ def _build_strategies(
|
||||
return strategies
|
||||
|
||||
|
||||
async def create_pipeline_from_config(config: PipelineRunConfig) -> FullSyncPipeline:
|
||||
async def create_pipeline_from_config(
|
||||
config: PipelineRunConfig,
|
||||
*,
|
||||
local_datasource_override=None,
|
||||
remote_datasource_override=None,
|
||||
) -> FullSyncPipeline:
|
||||
if config.logging.initialize and not config.logging.file_path:
|
||||
config.logging.file_path = resolve_log_file_path(config.logging)
|
||||
|
||||
@@ -289,7 +305,16 @@ async def create_pipeline_from_config(config: PipelineRunConfig) -> FullSyncPipe
|
||||
persistence = PersistenceBackend(str(config.persist.db_path), backend=config.persist.backend)
|
||||
await persistence.initialize()
|
||||
|
||||
local_ds, remote_ds = await _create_datasources(config)
|
||||
local_ds = await _create_datasource(
|
||||
config.local_datasource,
|
||||
datasource_override=local_datasource_override,
|
||||
endpoint_name="local",
|
||||
)
|
||||
remote_ds = await _create_datasource(
|
||||
config.remote_datasource,
|
||||
datasource_override=remote_datasource_override,
|
||||
endpoint_name="remote",
|
||||
)
|
||||
local_cfg_with_target = config.local_datasource.model_copy(
|
||||
update={"target_project_ids": list(local_target_project_ids)},
|
||||
deep=True,
|
||||
@@ -340,7 +365,7 @@ async def create_pipeline_from_config(config: PipelineRunConfig) -> FullSyncPipe
|
||||
|
||||
return pipeline
|
||||
except Exception:
|
||||
async def _safe_close(name: str, resource: ApiDataSource | JsonlDataSource | PersistenceBackend | None) -> None:
|
||||
async def _safe_close(name: str, resource) -> None:
|
||||
if resource is None:
|
||||
return
|
||||
try:
|
||||
@@ -359,6 +384,8 @@ async def run_pipeline_from_config(
|
||||
*,
|
||||
title: str = "sync_state_machine pipeline",
|
||||
print_summary: bool = True,
|
||||
local_datasource_override=None,
|
||||
remote_datasource_override=None,
|
||||
) -> Dict[str, Any]:
|
||||
if config.logging.initialize and not config.logging.file_path:
|
||||
config.logging.file_path = resolve_log_file_path(config.logging)
|
||||
@@ -401,7 +428,11 @@ async def run_pipeline_from_config(
|
||||
print(f"🔧 Resolved Config:\n{resolved_cfg_text}")
|
||||
print("=" * 80)
|
||||
|
||||
pipeline = await create_pipeline_from_config(config)
|
||||
pipeline = await create_pipeline_from_config(
|
||||
config,
|
||||
local_datasource_override=local_datasource_override,
|
||||
remote_datasource_override=remote_datasource_override,
|
||||
)
|
||||
stats = await pipeline.run()
|
||||
|
||||
if print_summary:
|
||||
|
||||
@@ -355,3 +355,55 @@ async def test_poll_tasks_queries_all_info_once_per_project_for_multiple_keys(mo
|
||||
assert polled[task_ids[1]].status == TaskStatus.SUCCESS
|
||||
assert polled[task_ids[1]].data_id == "remote-detail-2"
|
||||
assert client.all_info_calls == ["project-1"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_poll_tasks_retries_until_project_detail_appears(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
collection = DataCollection("remote")
|
||||
project = _build_project_node(node_id="p-node", project_id="project-1", all_info={"project_detail": []})
|
||||
await collection.add(project)
|
||||
|
||||
handler = ProjectDetailApiHandler()
|
||||
handler.set_collection(collection)
|
||||
handler.api_client = _MockApiClient()
|
||||
|
||||
call_count = {"value": 0}
|
||||
|
||||
async def _fake_get_all_info(cls, _client, project_id: str):
|
||||
call_count["value"] += 1
|
||||
if call_count["value"] >= 2:
|
||||
return {
|
||||
"project_detail": [
|
||||
{
|
||||
"id": "remote-detail-1",
|
||||
"project_id": project_id,
|
||||
"key": "ensure_production",
|
||||
"value": [{"year": 2026, "capacity": 21.0}],
|
||||
}
|
||||
]
|
||||
}
|
||||
return {"project_detail": []}
|
||||
|
||||
monkeypatch.setattr(ProjectApiHandler, "get_all_info", classmethod(_fake_get_all_info))
|
||||
|
||||
node = ProjectDetailSyncNode(
|
||||
node_id="detail-node-1",
|
||||
data_id="local-detail-1",
|
||||
data={
|
||||
"id": "local-detail-1",
|
||||
"project_id": "project-1",
|
||||
"key": "ensure_production",
|
||||
"value": [{"year": 2026, "capacity": 21.0}],
|
||||
},
|
||||
)
|
||||
|
||||
create_result = (await handler.create_all([node]))[0]
|
||||
assert create_result.task_id
|
||||
handler.api_client.push_logs[create_result.task_id] = {"code": 200, "data": {"biz_id": "project-1"}}
|
||||
|
||||
polled = await handler.poll_tasks([create_result.task_id])
|
||||
final = polled[create_result.task_id]
|
||||
|
||||
assert final.status == TaskStatus.SUCCESS
|
||||
assert final.data_id == "remote-detail-1"
|
||||
assert call_count["value"] >= 2
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
def _load_base_module():
|
||||
scripts_dir = Path(__file__).resolve().parents[2] / "scripts"
|
||||
if str(scripts_dir) not in sys.path:
|
||||
sys.path.insert(0, str(scripts_dir))
|
||||
file_path = Path(__file__).resolve().parents[2] / "scripts" / "auto_test_domains" / "base.py"
|
||||
spec = importlib.util.spec_from_file_location("auto_test_domain_base_for_test", file_path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError(f"Unable to load module from {file_path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def test_build_temp_run_profile_uses_explicit_project_prebind(tmp_path: Path) -> None:
|
||||
module = _load_base_module()
|
||||
|
||||
source_profile = tmp_path / "source.yaml"
|
||||
source_profile.write_text(
|
||||
yaml.safe_dump(
|
||||
{
|
||||
"local_datasource": {"target_project_ids": ["old-local"]},
|
||||
"remote_datasource": {"target_project_ids": ["old-remote"]},
|
||||
"strategies": {"project": {"config": {}}},
|
||||
},
|
||||
allow_unicode=True,
|
||||
sort_keys=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
config = {
|
||||
"pipeline": {
|
||||
"run_profile": str(source_profile),
|
||||
"command": ["python", "run.py", "--config_path", "placeholder.yaml"],
|
||||
}
|
||||
}
|
||||
|
||||
profile_path, command = module.build_temp_run_profile_for_project_auto_bind(
|
||||
config=config,
|
||||
report_root=tmp_path,
|
||||
profile_name="demo",
|
||||
local_project_id="local-project",
|
||||
remote_project_id="remote-project",
|
||||
)
|
||||
|
||||
profile = yaml.safe_load(Path(profile_path).read_text(encoding="utf-8"))
|
||||
project_config = profile["strategies"]["project"]["config"]
|
||||
|
||||
assert profile["local_datasource"]["target_project_ids"] == ["local-project"]
|
||||
assert profile["remote_datasource"]["target_project_ids"] == ["remote-project"]
|
||||
assert project_config["auto_bind"] is False
|
||||
assert project_config["auto_bind_fields"] == []
|
||||
assert project_config["pre_bind_data_id"] == [
|
||||
{"local_data_id": "local-project", "remote_data_id": "remote-project"}
|
||||
]
|
||||
assert command[-1] == profile_path
|
||||
@@ -7,6 +7,7 @@ from sync_state_machine.config import (
|
||||
PipelineRunConfig,
|
||||
JsonlDataSourceConfig,
|
||||
ApiDataSourceConfig,
|
||||
DbDataSourceConfig,
|
||||
PersistConfig,
|
||||
LoggingConfig,
|
||||
)
|
||||
@@ -101,3 +102,11 @@ def test_api_datasource_rate_limit_config_fields() -> None:
|
||||
)
|
||||
assert overridden.api_rate_limit_max_requests == 5
|
||||
assert overridden.api_rate_limit_window_seconds == 2.5
|
||||
|
||||
|
||||
def test_db_datasource_accepts_minimal_config() -> None:
|
||||
cfg = DbDataSourceConfig(type="db")
|
||||
|
||||
assert cfg.type == "db"
|
||||
assert cfg.target_project_ids == []
|
||||
assert cfg.handler_configs == {}
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from sync_state_machine.common.registry import DomainRegistry
|
||||
from sync_state_machine.common.sync_node import SyncNode
|
||||
from sync_state_machine.config import DbDataSourceConfig, JsonlDataSourceConfig, LoggingConfig, PersistConfig, PipelineRunConfig
|
||||
from sync_state_machine.datasource.db import BaseDbHandler, DbDataSource
|
||||
from sync_state_machine.datasource.jsonl import BaseJsonlHandler, JsonlDataSource
|
||||
from sync_state_machine.pipeline.factory import _register_handlers
|
||||
from sync_state_machine.sync_system.strategy import DefaultSyncStrategy
|
||||
|
||||
|
||||
class TestDbSchema(BaseModel):
|
||||
__test__ = False
|
||||
id: str
|
||||
name: str
|
||||
|
||||
|
||||
class TestDbNode(SyncNode[TestDbSchema]):
|
||||
__test__ = False
|
||||
node_type = "test_db_handler"
|
||||
schema = TestDbSchema
|
||||
|
||||
|
||||
class TestDbStrategy(DefaultSyncStrategy[TestDbSchema]):
|
||||
__test__ = False
|
||||
schema = TestDbSchema
|
||||
|
||||
|
||||
class TestDbHandler(BaseDbHandler[TestDbSchema]):
|
||||
__test__ = False
|
||||
|
||||
def __init__(self, datasource: DbDataSource):
|
||||
super().__init__(datasource, "test_db_handler", TestDbNode, TestDbSchema)
|
||||
|
||||
def extract_id(self, data: dict[str, str]) -> str:
|
||||
return str(data.get("id", ""))
|
||||
|
||||
async def load(self):
|
||||
return []
|
||||
|
||||
async def create_all(self, nodes):
|
||||
return []
|
||||
|
||||
async def update_all(self, nodes):
|
||||
return []
|
||||
|
||||
|
||||
class TestJsonlHandler(BaseJsonlHandler):
|
||||
__test__ = False
|
||||
|
||||
def __init__(self, datasource: JsonlDataSource):
|
||||
super().__init__(datasource, "test_db_handler", TestDbNode, TestDbSchema)
|
||||
|
||||
|
||||
def test_register_handlers_uses_db_handler_for_db_datasource(tmp_path: Path) -> None:
|
||||
if not DomainRegistry.is_registered("test_db_handler"):
|
||||
DomainRegistry.register(
|
||||
node_type="test_db_handler",
|
||||
schema=TestDbSchema,
|
||||
node_class=TestDbNode,
|
||||
strategy_class=TestDbStrategy,
|
||||
jsonl_handler_class=TestJsonlHandler,
|
||||
)
|
||||
DomainRegistry.register_db_handler("test_db_handler", TestDbHandler)
|
||||
|
||||
config = PipelineRunConfig(
|
||||
node_types=["test_db_handler"],
|
||||
local_datasource=DbDataSourceConfig(type="db"),
|
||||
remote_datasource=JsonlDataSourceConfig(
|
||||
type="jsonl",
|
||||
jsonl_dir=str(tmp_path),
|
||||
target_project_ids=[],
|
||||
),
|
||||
strategies=[],
|
||||
persist=PersistConfig(db_path=str(tmp_path / "pipeline.db")),
|
||||
logging=LoggingConfig(initialize=False),
|
||||
)
|
||||
|
||||
local_ds = DbDataSource()
|
||||
remote_ds = JsonlDataSource(tmp_path)
|
||||
|
||||
_register_handlers(config, local_ds, remote_ds, ["test_db_handler"])
|
||||
|
||||
assert isinstance(local_ds.get_handler("test_db_handler"), TestDbHandler)
|
||||
assert isinstance(remote_ds.get_handler("test_db_handler"), TestJsonlHandler)
|
||||
@@ -0,0 +1,105 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from pydantic import BaseModel
|
||||
|
||||
from sync_state_machine.common.registry import DomainRegistry
|
||||
from sync_state_machine.common.sync_node import SyncNode
|
||||
from sync_state_machine.config import DbDataSourceConfig, LoggingConfig, PersistConfig, PipelineRunConfig
|
||||
from sync_state_machine.datasource.db import BaseDbHandler, DbDataSource
|
||||
from sync_state_machine.pipeline.factory import create_pipeline_from_config
|
||||
from sync_state_machine.sync_system.strategy import DefaultSyncStrategy
|
||||
|
||||
|
||||
class EndpointInitSchema(BaseModel):
|
||||
__test__ = False
|
||||
id: str
|
||||
name: str | None = None
|
||||
|
||||
|
||||
class EndpointInitNode(SyncNode[EndpointInitSchema]):
|
||||
__test__ = False
|
||||
node_type = "factory_endpoint_demo"
|
||||
schema = EndpointInitSchema
|
||||
|
||||
|
||||
class EndpointInitStrategy(DefaultSyncStrategy[EndpointInitSchema]):
|
||||
__test__ = False
|
||||
schema = EndpointInitSchema
|
||||
|
||||
|
||||
class EndpointInitDbHandler(BaseDbHandler[EndpointInitSchema]):
|
||||
__test__ = False
|
||||
|
||||
def __init__(self, datasource: DbDataSource):
|
||||
super().__init__(datasource, "factory_endpoint_demo", EndpointInitNode, EndpointInitSchema)
|
||||
|
||||
def extract_id(self, data: dict[str, str]) -> str:
|
||||
return str(data.get("id", ""))
|
||||
|
||||
async def load(self):
|
||||
return []
|
||||
|
||||
async def create_all(self, nodes):
|
||||
return []
|
||||
|
||||
async def update_all(self, nodes):
|
||||
return []
|
||||
|
||||
|
||||
def _ensure_registered() -> None:
|
||||
node_type = "factory_endpoint_demo"
|
||||
if DomainRegistry.is_registered(node_type):
|
||||
reg = DomainRegistry.get_registration(node_type)
|
||||
reg.schema = EndpointInitSchema
|
||||
reg.node_class = EndpointInitNode
|
||||
reg.strategy_class = EndpointInitStrategy
|
||||
reg.db_handler_class = EndpointInitDbHandler
|
||||
return
|
||||
|
||||
DomainRegistry.register(
|
||||
node_type=node_type,
|
||||
schema=EndpointInitSchema,
|
||||
node_class=EndpointInitNode,
|
||||
strategy_class=EndpointInitStrategy,
|
||||
db_handler_class=EndpointInitDbHandler,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_factory_initializes_same_datasource_type_with_distinct_handler_configs(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
_ensure_registered()
|
||||
|
||||
config = PipelineRunConfig(
|
||||
node_types=["factory_endpoint_demo"],
|
||||
local_datasource=DbDataSourceConfig(
|
||||
type="db",
|
||||
handler_configs={"factory_endpoint_demo": {"side": "local", "batch_size": 10}},
|
||||
target_project_ids=["project_local"],
|
||||
),
|
||||
remote_datasource=DbDataSourceConfig(
|
||||
type="db",
|
||||
handler_configs={"factory_endpoint_demo": {"side": "remote", "batch_size": 20}},
|
||||
target_project_ids=["project_remote"],
|
||||
),
|
||||
strategies=[],
|
||||
persist=PersistConfig(db_path=str(tmp_path / "factory-init.db")),
|
||||
logging=LoggingConfig(initialize=False),
|
||||
)
|
||||
|
||||
pipeline = await create_pipeline_from_config(config)
|
||||
try:
|
||||
local_handler = pipeline.local_datasource.get_handler("factory_endpoint_demo")
|
||||
remote_handler = pipeline.remote_datasource.get_handler("factory_endpoint_demo")
|
||||
|
||||
assert isinstance(local_handler, EndpointInitDbHandler)
|
||||
assert isinstance(remote_handler, EndpointInitDbHandler)
|
||||
assert local_handler.handler_config == {"side": "local", "batch_size": 10}
|
||||
assert remote_handler.handler_config == {"side": "remote", "batch_size": 20}
|
||||
finally:
|
||||
await pipeline.close()
|
||||
@@ -0,0 +1,86 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from sync_state_machine.common.sync_node import SyncNode
|
||||
from sync_state_machine.datasource.api.handler import BaseApiHandler
|
||||
from sync_state_machine.datasource.handler import BaseNodeHandler, CompatibleNodeHandler
|
||||
from sync_state_machine.datasource.task_result import TaskResult
|
||||
|
||||
|
||||
class SharedConfigSchema(BaseModel):
|
||||
__test__ = False
|
||||
id: str
|
||||
|
||||
|
||||
class SharedConfigNode(SyncNode[SharedConfigSchema]):
|
||||
__test__ = False
|
||||
node_type = "shared_config_demo"
|
||||
schema = SharedConfigSchema
|
||||
|
||||
|
||||
class SharedConfigBaseHandler(BaseNodeHandler[SharedConfigSchema]):
|
||||
__test__ = False
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__("shared_config_demo", SharedConfigNode, SharedConfigSchema)
|
||||
|
||||
async def load(self):
|
||||
return []
|
||||
|
||||
async def create_all(self, nodes):
|
||||
return []
|
||||
|
||||
async def update_all(self, nodes):
|
||||
return []
|
||||
|
||||
def extract_id(self, data: dict[str, Any]) -> str:
|
||||
return str(data.get("id", ""))
|
||||
|
||||
|
||||
class SharedConfigApiHandler(BaseApiHandler[SharedConfigSchema]):
|
||||
__test__ = False
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self._node_type = "shared_api_demo"
|
||||
self._node_class = SharedConfigNode
|
||||
self._schema = SharedConfigSchema
|
||||
|
||||
async def load(self):
|
||||
return []
|
||||
|
||||
async def create_all(self, nodes):
|
||||
return [TaskResult.success(node_id=node.node_id) for node in nodes]
|
||||
|
||||
async def update_all(self, nodes):
|
||||
return [TaskResult.success(node_id=node.node_id) for node in nodes]
|
||||
|
||||
async def delete_all(self, nodes):
|
||||
return [TaskResult.success(node_id=node.node_id) for node in nodes]
|
||||
|
||||
|
||||
def test_base_handler_supports_shared_handler_config() -> None:
|
||||
handler = SharedConfigBaseHandler()
|
||||
|
||||
handler.set_handler_config({"mode": "local", "batch_size": 10})
|
||||
|
||||
assert handler.handler_config == {"mode": "local", "batch_size": 10}
|
||||
|
||||
|
||||
def test_compatible_handler_forwards_shared_handler_config() -> None:
|
||||
handler = CompatibleNodeHandler(SharedConfigBaseHandler())
|
||||
|
||||
handler.set_handler_config({"mode": "remote"})
|
||||
|
||||
assert handler.handler_config == {"mode": "remote"}
|
||||
|
||||
|
||||
def test_api_handler_supports_shared_handler_config() -> None:
|
||||
handler = SharedConfigApiHandler()
|
||||
|
||||
handler.set_handler_config({"filter_project_id": ["p1", "p2"]})
|
||||
|
||||
assert handler.handler_config == {"filter_project_id": ["p1", "p2"]}
|
||||
@@ -0,0 +1,62 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from pydantic import BaseModel
|
||||
|
||||
from sync_state_machine.common.sync_node import SyncNode
|
||||
from sync_state_machine.domain.material_detail.api_handler import MaterialDetailApiHandler
|
||||
from sync_state_machine.datasource.task_result import TaskStatus
|
||||
|
||||
|
||||
class _MaterialDetailSchema(BaseModel):
|
||||
id: str
|
||||
parent_id: str
|
||||
quantity: float
|
||||
remark: str | None = None
|
||||
|
||||
|
||||
class _MaterialDetailNode(SyncNode[_MaterialDetailSchema]):
|
||||
node_type = "material_detail"
|
||||
schema = _MaterialDetailSchema
|
||||
|
||||
|
||||
class _MockApiClient:
|
||||
def __init__(self) -> None:
|
||||
self.requests: list[dict[str, Any]] = []
|
||||
|
||||
async def request(self, method: str, endpoint: str, **kwargs: Any) -> dict[str, Any]:
|
||||
self.requests.append({"method": method, "endpoint": endpoint, **kwargs})
|
||||
return {"code": 200, "message": "OK", "data": {"biz_id": kwargs.get("data", {}).get("biz_id")}}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_material_detail_update_returns_in_progress_with_shared_push_id() -> None:
|
||||
handler = MaterialDetailApiHandler()
|
||||
handler.api_client = _MockApiClient()
|
||||
|
||||
node = _MaterialDetailNode(
|
||||
node_id="detail-node-1",
|
||||
data_id="detail-1",
|
||||
data={
|
||||
"id": "detail-1",
|
||||
"parent_id": "material-1",
|
||||
"quantity": 10.0,
|
||||
"remark": "new",
|
||||
},
|
||||
origin_data={
|
||||
"id": "detail-1",
|
||||
"parent_id": "material-1",
|
||||
"quantity": 5.0,
|
||||
"remark": None,
|
||||
},
|
||||
context={"material_id": "material-1"},
|
||||
)
|
||||
|
||||
results = await handler.update_all([node])
|
||||
|
||||
assert len(results) == 1
|
||||
assert results[0].status == TaskStatus.IN_PROGRESS
|
||||
assert results[0].task_id
|
||||
assert handler.api_client.requests[0]["endpoint"] == "/material/detail/update"
|
||||
@@ -0,0 +1,84 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from uuid import uuid4
|
||||
|
||||
from sync_state_machine.domain.project.api_handler import _merge_project_all_info_fields
|
||||
from sync_state_machine.domain.project.sync_node import ProjectSyncNode
|
||||
|
||||
|
||||
def test_merge_project_all_info_fields_flattens_extension_investment_fields() -> None:
|
||||
project_data = {
|
||||
"id": "project-1",
|
||||
"name": "Demo Project",
|
||||
"dynamic_investment": 123.0,
|
||||
}
|
||||
all_info = {
|
||||
"project": {"id": "project-1"},
|
||||
"hydropower_extension": {
|
||||
"implementation_estimate": 1000.0,
|
||||
"static_total_investment": 990.0,
|
||||
"completed_investment": 120.0,
|
||||
"total_contract_quantity": 50.0,
|
||||
},
|
||||
}
|
||||
|
||||
merged = _merge_project_all_info_fields(project_data, all_info)
|
||||
|
||||
assert merged["dynamic_investment"] == 123.0
|
||||
assert merged["implementation_estimate"] == 1000.0
|
||||
assert merged["static_total_investment"] == 990.0
|
||||
assert merged["completed_investment"] == 120.0
|
||||
assert merged["total_contract_quantity"] == 50.0
|
||||
|
||||
|
||||
def test_merge_project_all_info_fields_keeps_existing_project_values() -> None:
|
||||
project_data = {
|
||||
"id": "project-1",
|
||||
"implementation_estimate": 2000.0,
|
||||
"static_total_investment": 1800.0,
|
||||
}
|
||||
all_info = {
|
||||
"hydropower_extension": {
|
||||
"implementation_estimate": 1000.0,
|
||||
"static_total_investment": 990.0,
|
||||
"completed_investment": 120.0,
|
||||
},
|
||||
}
|
||||
|
||||
merged = _merge_project_all_info_fields(project_data, all_info)
|
||||
|
||||
assert merged["implementation_estimate"] == 2000.0
|
||||
assert merged["static_total_investment"] == 1800.0
|
||||
assert merged["completed_investment"] == 120.0
|
||||
|
||||
|
||||
def test_project_sync_node_preserves_extension_backed_fields() -> None:
|
||||
project_id = str(uuid4())
|
||||
data = {
|
||||
"_id": project_id,
|
||||
"id": project_id,
|
||||
"project_type": "hydropower",
|
||||
"region_name": ["中国", "测试省"],
|
||||
"region_path": ["100000", "110000"],
|
||||
"province": "110000",
|
||||
"is_domestic": True,
|
||||
"company_id": str(uuid4()),
|
||||
"company_name": "测试公司",
|
||||
"address": "测试地址",
|
||||
"code": "P-001",
|
||||
"name": "测试项目",
|
||||
"progress_type": "A",
|
||||
"dynamic_investment": 100.0,
|
||||
"complete_investment": 80.0,
|
||||
"implementation_estimate": 120.0,
|
||||
"static_total_investment": 110.0,
|
||||
"completed_investment": 70.0,
|
||||
}
|
||||
|
||||
node = ProjectSyncNode(node_id=str(uuid4()), data_id=data["id"], data=data)
|
||||
normalized = node.get_data()
|
||||
|
||||
assert normalized is not None
|
||||
assert normalized["implementation_estimate"] == 120.0
|
||||
assert normalized["static_total_investment"] == 110.0
|
||||
assert normalized["completed_investment"] == 70.0
|
||||
@@ -0,0 +1,56 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from pydantic import BaseModel
|
||||
|
||||
from sync_state_machine.common.sync_node import SyncNode
|
||||
from sync_state_machine.domain.project.api_handler import ProjectApiHandler
|
||||
from sync_state_machine.datasource.task_result import TaskStatus
|
||||
|
||||
|
||||
class _ProjectUpdateSchema(BaseModel):
|
||||
id: str
|
||||
dynamic_investment: float
|
||||
|
||||
|
||||
class _ProjectUpdateNode(SyncNode[_ProjectUpdateSchema]):
|
||||
node_type = "project"
|
||||
schema = _ProjectUpdateSchema
|
||||
|
||||
|
||||
class _MockApiClient:
|
||||
def __init__(self) -> None:
|
||||
self.requests: list[dict[str, Any]] = []
|
||||
self._poll_count = 0
|
||||
|
||||
async def request(self, method: str, endpoint: str, **kwargs: Any) -> dict[str, Any]:
|
||||
self.requests.append({"method": method, "endpoint": endpoint, **kwargs})
|
||||
return {"code": 200, "message": "OK", "data": {"biz_id": kwargs.get("data", {}).get("biz_id")}}
|
||||
|
||||
async def get_push_logs(self, task_ids: list[str]) -> dict[str, dict[str, Any]]:
|
||||
self._poll_count += 1
|
||||
if self._poll_count == 1:
|
||||
return {task_ids[0]: {}}
|
||||
return {task_ids[0]: {"code": 200, "data": {"biz_id": "project-1"}}}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_project_update_waits_for_async_push_completion() -> None:
|
||||
handler = ProjectApiHandler(filter_project_id=["project-1"])
|
||||
handler.api_client = _MockApiClient()
|
||||
|
||||
node = _ProjectUpdateNode(
|
||||
node_id="project-node-1",
|
||||
data_id="project-1",
|
||||
data={"id": "project-1", "dynamic_investment": 200.0},
|
||||
origin_data={"id": "project-1", "dynamic_investment": 100.0},
|
||||
)
|
||||
|
||||
results = await handler.update_all([node])
|
||||
|
||||
assert len(results) == 1
|
||||
assert results[0].status == TaskStatus.SUCCESS
|
||||
assert handler.api_client.requests[0]["endpoint"] == "/project/dynamic_investment"
|
||||
assert handler.api_client._poll_count == 2
|
||||
@@ -0,0 +1,53 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
|
||||
def _load_case_module():
|
||||
scripts_dir = Path(__file__).resolve().parents[2] / "scripts"
|
||||
if str(scripts_dir) not in sys.path:
|
||||
sys.path.insert(0, str(scripts_dir))
|
||||
file_path = scripts_dir / "auto_test_domains" / "project_case.py"
|
||||
spec = importlib.util.spec_from_file_location("project_case_for_test", file_path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError(f"Unable to load module from {file_path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def test_project_case_round_3_focuses_on_stable_readback_fields(monkeypatch, tmp_path: Path) -> None:
|
||||
module = _load_case_module()
|
||||
captured_rounds: dict[str, dict] = {}
|
||||
|
||||
def _fake_run_pipeline_round(**kwargs):
|
||||
captured_rounds[kwargs["round_name"]] = kwargs
|
||||
return {"name": kwargs["round_name"], "passed": True, "issues": []}
|
||||
|
||||
def _fake_build_domain_report(**kwargs):
|
||||
return kwargs
|
||||
|
||||
monkeypatch.setattr(module, "run_pipeline_round", _fake_run_pipeline_round)
|
||||
monkeypatch.setattr(module, "build_domain_report", _fake_build_domain_report)
|
||||
|
||||
module.run_case(config={"project_root": str(tmp_path)}, report_root=tmp_path)
|
||||
|
||||
round_3 = captured_rounds["round_3_investment_update"]
|
||||
expected_fields = round_3["expected_bound_fields"]["remote"]["project"][module.LOCAL_PROJECT_ID]
|
||||
changed_fields = round_3["expected_changes"]["project"][module.LOCAL_PROJECT_ID]
|
||||
|
||||
assert set(changed_fields) == {
|
||||
"investment_decision_capacity",
|
||||
"investment_decision_date",
|
||||
"complete_investment",
|
||||
"dynamic_investment",
|
||||
}
|
||||
assert set(expected_fields.keys()) == {
|
||||
"investment_decision_capacity",
|
||||
"investment_decision_date",
|
||||
"complete_investment",
|
||||
"dynamic_investment",
|
||||
}
|
||||
assert "PUT /project/settlement_investment" in round_3["coverage"]["api"]
|
||||
@@ -0,0 +1,46 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
|
||||
def _load_case_module():
|
||||
scripts_dir = Path(__file__).resolve().parents[2] / "scripts"
|
||||
if str(scripts_dir) not in sys.path:
|
||||
sys.path.insert(0, str(scripts_dir))
|
||||
file_path = scripts_dir / "auto_test_domains" / "project_detail_data_case.py"
|
||||
spec = importlib.util.spec_from_file_location("project_detail_data_case_for_test", file_path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError(f"Unable to load module from {file_path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def test_project_detail_case_excludes_production_from_unified_rounds(monkeypatch, tmp_path: Path) -> None:
|
||||
module = _load_case_module()
|
||||
recorded_rounds: list[str] = []
|
||||
|
||||
def _fake_run_pipeline_round(**kwargs):
|
||||
recorded_rounds.append(kwargs["round_name"])
|
||||
return {"name": kwargs["round_name"], "passed": True, "issues": []}
|
||||
|
||||
def _fake_build_domain_report(**kwargs):
|
||||
return kwargs
|
||||
|
||||
monkeypatch.setattr(module, "run_pipeline_round", _fake_run_pipeline_round)
|
||||
monkeypatch.setattr(module, "build_domain_report", _fake_build_domain_report)
|
||||
|
||||
report = module.run_case(config={"project_root": str(tmp_path)}, report_root=tmp_path)
|
||||
|
||||
assert recorded_rounds == [
|
||||
"round_1_create",
|
||||
"round_2_update",
|
||||
"round_3_plan_construction_create",
|
||||
"round_5_ensure_production_push",
|
||||
"round_6_climb_production_push",
|
||||
"round_7_challenge_production_push",
|
||||
]
|
||||
assert "production" in report["description"]
|
||||
assert "不混入本组统一回归" in report["description"]
|
||||
@@ -0,0 +1,49 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from sync_state_machine.common.registry import DomainRegistry
|
||||
|
||||
|
||||
class RegistryLookupSchema(BaseModel):
|
||||
__test__ = False
|
||||
id: str
|
||||
|
||||
|
||||
class RegistryLookupNode:
|
||||
__test__ = False
|
||||
|
||||
|
||||
class RegistryLookupJsonlHandler:
|
||||
__test__ = False
|
||||
|
||||
|
||||
class RegistryLookupApiHandler:
|
||||
__test__ = False
|
||||
|
||||
|
||||
class RegistryLookupDbHandler:
|
||||
__test__ = False
|
||||
|
||||
|
||||
def test_registry_supports_unified_handler_lookup() -> None:
|
||||
node_type = "registry_lookup_demo"
|
||||
|
||||
if DomainRegistry.is_registered(node_type):
|
||||
reg = DomainRegistry.get_registration(node_type)
|
||||
reg.jsonl_handler_class = RegistryLookupJsonlHandler
|
||||
reg.api_handler_class = RegistryLookupApiHandler
|
||||
reg.db_handler_class = RegistryLookupDbHandler
|
||||
else:
|
||||
DomainRegistry.register(
|
||||
node_type=node_type,
|
||||
schema=RegistryLookupSchema,
|
||||
node_class=RegistryLookupNode,
|
||||
jsonl_handler_class=RegistryLookupJsonlHandler,
|
||||
api_handler_class=RegistryLookupApiHandler,
|
||||
db_handler_class=RegistryLookupDbHandler,
|
||||
)
|
||||
|
||||
assert DomainRegistry.get_handler(node_type, "jsonl") is RegistryLookupJsonlHandler
|
||||
assert DomainRegistry.get_handler(node_type, "api") is RegistryLookupApiHandler
|
||||
assert DomainRegistry.get_handler(node_type, "db") is RegistryLookupDbHandler
|
||||
Reference in New Issue
Block a user