strepsiades/ecm_sync_system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
strepsiades
2026-03-09 16:31:42 +08:00
commit 4a9c444b10
486 changed files with 52479 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sync_state_machine/datasource/README.md
+252
View File
@@ -0,0 +1,252 @@
# DataSource Layer Implementation
## Overview
This document describes the new datasource layer for `sync_system_new`, which is completely decoupled from the sync strategy and state machine logic.
## Architecture
The datasource layer follows a pure execution model:
1. **Load**: Reads data from backend and creates `SyncNode` instances
2. **Save**: Executes actions (CREATE/UPDATE/DELETE) based on `node.action`
3. **Update Status**: Sets `node.status` to SUCCESS or FAILED after execution
### Key Principles
- **No Strategy Logic**: Datasource doesn't make decisions about what to sync
- **State-Driven**: Only reads `node.action` and `node.status` to determine behavior
- **No ID Resolution**: IDs are already resolved by the strategy layer
- **Backend Agnostic**: Abstract base class supports multiple backends
## Components
### 1. Base Protocol (`base.py`)
#### `DataSourceProtocol`
Protocol defining the interface for all datasource implementations:
```python
async def load_nodes(
node_type: str,
collection: DataCollection,
**filters: Any
) -> None:
"""Load data from backend and create SyncNodes"""
async def save_nodes(
node_type: str,
collection: DataCollection
) -> None:
"""Execute pending actions for nodes"""
```
#### `BaseDataSource`
Abstract base class providing common functionality:
- `load_nodes`: Creates SyncNodes from raw data
- `save_nodes`: Executes actions and updates status
- `_handle_create/update/delete`: Action-specific handlers
Subclasses implement:
- `_load_raw_data`: Load from backend
- `_create_item`: Execute CREATE
- `_update_item`: Execute UPDATE
- `_delete_item`: Execute DELETE
- `_generate_id`: Generate new IDs
### 2. JSONL Implementation (`jsonl_datasource.py`)
#### `JsonlDataSource`
JSONL-based datasource for local testing:
**Features:**
- Loads `.jsonl` files into memory
- Generates UUIDs for new items
- Supports filtering during load
- Supports read-only mode
- Persists changes back to files
**File Structure:**
```
data/
├── contract_1.jsonl
├── project_1.jsonl
└── ...
```
**Usage:**
```python
datasource = JsonlDataSource(Path("data/"))
# Load nodes
await datasource.load_nodes("contract", collection, project_id="P1")
# Execute actions
await datasource.save_nodes("contract", collection)
# Persist to disk
await datasource.save_all()
```
### 3. Domain Handlers
Domain-specific handlers provide business logic (optional):
**Example: `domain/contract/jsonl_handler.py`**
```python
class ContractJsonlHandler:
@staticmethod
def validate_create(data: Any) -> None:
"""Validate before create"""
@staticmethod
def transform_for_create(data: Any) -> Dict[str, Any]:
"""Transform data before create"""
```
## Node Lifecycle
### Loading Phase
1. Datasource reads raw data from backend
2. Creates `SyncNode` for each item:
- `node_id`: Unique session ID (e.g., "contract_C1")
- `data_id`: Backend ID (e.g., "C1")
- `origin_data`: Raw data from backend
- `data`: Copy of origin_data (may be modified by strategy)
- `action`: NONE (initial)
- `status`: PENDING (initial)
### Execution Phase
For each node with `status == PENDING` and `action != NONE`:
1. **CREATE**:
- Generate new ID via `_generate_id`
- Call `_create_item` with new ID and data
- Set `node.data_id` to new ID
- Set `node.status` to SUCCESS
2. **UPDATE**:
- Call `_update_item` with existing `data_id` and data
- Set `node.status` to SUCCESS
3. **DELETE**:
- Call `_delete_item` with `data_id`
- Set `node.status` to SUCCESS
4. **Error Handling**:
- On exception: Set `node.status` to FAILED
- Set `node.error` to exception message
## Integration with Sync System
The datasource is used by the pipeline layer:
```python
# 1. Load data
local_ds = JsonlDataSource(Path("local_data/"))
remote_ds = JsonlDataSource(Path("remote_data/"))
local_collection = DataCollection("local")
remote_collection = DataCollection("remote")
await local_ds.load_nodes("contract", local_collection)
await remote_ds.load_nodes("contract", remote_collection)
# 2. Strategy layer determines actions
strategy = ContractSyncStrategy(...)
await strategy.bind(...) # Sets node.action
# 3. Execute actions
await remote_ds.save_nodes("contract", remote_collection)
await remote_ds.save_all()
```
## Testing
Comprehensive test suite in `tests/test_datasource.py`:
- Basic loading and node creation
- Filtering during load
- CREATE/UPDATE/DELETE actions
- Error handling and status updates
- Read-only mode
- Multiple node types
Run tests:
```bash
python -m pytest sync_system_new/tests/test_datasource.py -v
```
## Future Extensions
### API DataSource
For production use, implement `ApiDataSource`:
```python
class ApiDataSource(BaseDataSource):
async def _load_raw_data(self, node_type: str, **filters):
# Call REST API
async def _create_item(self, node_type: str, item_id: str, data):
# POST request
async def _generate_id(self, node_type: str, data):
# Return None - server generates ID
```
### Database DataSource
For SQL databases:
```python
class DatabaseDataSource(BaseDataSource):
async def _load_raw_data(self, node_type: str, **filters):
# SELECT query
async def _create_item(self, node_type: str, item_id: str, data):
# INSERT query
```
## Comparison with Legacy System
### Legacy System (`sync_system/datasource/`)
- Tightly coupled with RepositoryProtocol
- Mixed concerns (data access + business logic)
- Schema validation in datasource layer
- Different interface for each entity type
### New System (`sync_system_new/datasource/`)
- Decoupled from strategy and state machine
- Pure data access (business logic in domain handlers)
- Generic interface for all entity types
- Direct integration with SyncNode and DataCollection
## Key Differences
| Aspect | Legacy | New |
|--------|--------|-----|
| Coupling | Tight with sync policies | Decoupled from sync logic |
| Interface | Repository per entity | Generic load/save |
| ID Handling | Mixed responsibilities | IDs already resolved |
| State Management | Mixed with data access | SyncNode-based |
| Testing | Complex setup | Simple, focused tests |
## Conclusion
The new datasource layer provides:
1. **Clear Separation**: Data access is isolated from sync logic
2. **Flexibility**: Easy to add new backend types
3. **Testability**: Simple, focused tests
4. **Type Safety**: Full integration with Pydantic models
5. **Maintainability**: Single responsibility per component
sync_state_machine/datasource/__init__.py
+41
View File
@@ -0,0 +1,41 @@
"""
DataSource Layer for sync_state_machine
DataSource 只负责底层 I/O，不关心节点类型。
Handler 负责类型转换和业务逻辑。
"""
from .datasource import BaseDataSource
from .handler import (
NodeHandler,
BaseNodeHandler,
NodeHandlerRegistry,
ValidationResult
)
from .task_result import TaskResult, TaskStatus
# API DataSource
from .api import ApiClient, ApiDataSource, BaseApiHandler
# JSONL DataSource
from .jsonl import JsonlDataSource, BaseJsonlHandler
__all__ = [
# 基础类
"BaseDataSource",
"NodeHandler",
"BaseNodeHandler",
"NodeHandlerRegistry",
"TaskResult",
"TaskStatus",
"ValidationResult",
# API DataSource
"ApiClient",
"ApiDataSource",
"BaseApiHandler",
# JSONL DataSource
"JsonlDataSource",
"BaseJsonlHandler",
]
sync_state_machine/datasource/api/__init__.py
+15
View File
@@ -0,0 +1,15 @@
"""
API DataSource 模块
提供基于 HTTP API 的数据源实现。
"""
from .client import ApiClient
from .datasource import ApiDataSource
from .handler import BaseApiHandler
__all__ = [
"ApiClient",
"ApiDataSource",
"BaseApiHandler",
]
sync_state_machine/datasource/api/client.py
+627
View File
@@ -0,0 +1,627 @@
"""
ApiClient - HTTP 通信层
职责:
- 封装所有 HTTP 通信逻辑
- 管理 httpx.AsyncClient，维护连接池和 session
- 生成请求签名（timestamp, nonce, sign）
- 统一请求入口，自动注入 uid
- 批量 Push Log 查询
"""
import hashlib
import hmac
import time
import uuid
import json as json_lib
import logging
import asyncio
from typing import Dict, List, Any, Optional
from collections import defaultdict, deque
from pathlib import Path
from datetime import datetime
import httpx
from schemas.common.push_system import PushIdsSchema
def _build_default_push_steps() -> List[Dict[str, str]]:
"""构造最小审批步骤字段，满足第三方 v2 push schema。"""
return []
class ApiClient:
"""API HTTP 客户端"""
_TRACE_TEXT_MAX_LEN = 1000
def __init__(
self,
base_url: str,
uid: str,
secret: str,
debug: bool = False,
log_file: Optional[Path] = None,
rate_limit_max_requests: int = 30,
rate_limit_window_seconds: float = 10.0,
):
"""
初始化 API 客户端
Args:
base_url: API 基础 URL（如 https://api.example.com）
uid: 用户标识
secret: 签名密钥
debug: 是否启用调试模式（打印请求详情）
log_file: 日志文件路径（可选）
rate_limit_max_requests: 限流窗口内最大请求数（<=0 表示禁用）
rate_limit_window_seconds: 限流窗口秒数（<=0 表示禁用）
"""
self._base_url = base_url.rstrip('/')
self._uid = uid
self._secret = secret
self._client: Optional[httpx.AsyncClient] = None
self._debug = debug
self._log_file = log_file
self._trace_seq = 0
self._trace_by_push_id: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
self._trace_by_biz_id: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
self._trace_by_op: Dict[str, deque[Dict[str, Any]]] = defaultdict(deque)
self._load_trace_by_item: Dict[tuple[str, str], deque[Dict[str, Any]]] = defaultdict(deque)
self._logger = logging.getLogger(__name__)
self._rate_limit_max_requests = int(rate_limit_max_requests)
self._rate_limit_window_seconds = float(rate_limit_window_seconds)
self._rate_limit_lock = asyncio.Lock()
self._rate_limit_timestamps: deque[float] = deque()
def _is_rate_limit_enabled(self) -> bool:
return self._rate_limit_max_requests > 0 and self._rate_limit_window_seconds > 0
async def _acquire_rate_limit_slot(self) -> None:
if not self._is_rate_limit_enabled():
return
while True:
wait_seconds = 0.0
async with self._rate_limit_lock:
now = time.monotonic()
window_start = now - self._rate_limit_window_seconds
while self._rate_limit_timestamps and self._rate_limit_timestamps[0] <= window_start:
self._rate_limit_timestamps.popleft()
if len(self._rate_limit_timestamps) < self._rate_limit_max_requests:
self._rate_limit_timestamps.append(now)
return
oldest = self._rate_limit_timestamps[0]
wait_seconds = max(0.001, self._rate_limit_window_seconds - (now - oldest))
await asyncio.sleep(wait_seconds)
async def initialize(self):
"""初始化 HTTP 客户端"""
if self._client is None:
# trust_env=False 阻止从环境变量读取代理设置（避免 localhost 走代理导致超时）
self._client = httpx.AsyncClient(timeout=300.0, trust_env=False)
async def close(self):
"""关闭 HTTP 客户端"""
if self._client:
await self._client.aclose()
self._client = None
def _log(self, message: str, level: str = "DEBUG"):
"""
统一日志输出
Args:
message: 日志消息
level: 日志级别（INFO/ERROR）
"""
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
log_line = f"[{timestamp}] [{level}] {message}"
level_num = logging.getLevelNamesMapping().get(level.upper(), logging.DEBUG)
self._logger.log(level_num, log_line)
# 文件输出
if self._log_file:
self._log_file.parent.mkdir(parents=True, exist_ok=True)
with open(self._log_file, 'a', encoding='utf-8') as f:
f.write(log_line + '\n')
def _get_client(self) -> httpx.AsyncClient:
"""获取 HTTP 客户端实例"""
if self._client is None:
raise RuntimeError("ApiClient not initialized. Call 'await client.initialize()' first.")
return self._client
def _derive_op_kind(self, method: str, endpoint: str) -> str:
endpoint_norm = endpoint.lower().strip()
if endpoint_norm == "/push/result":
return "poll"
method_upper = method.upper()
if method_upper == "GET":
if endpoint_norm.endswith("/list"):
return "load"
return "get"
if endpoint_norm.endswith("/create"):
return "create"
if endpoint_norm.endswith("/delete"):
return "delete"
if endpoint_norm.endswith("/update") or endpoint_norm.endswith("/plan"):
return "update"
if method_upper == "POST":
return "create"
if method_upper == "PUT":
return "update"
if method_upper == "DELETE":
return "delete"
return method.lower()
def _derive_node_type_from_list_endpoint(self, endpoint: str) -> Optional[str]:
endpoint_norm = endpoint.strip("/")
if not endpoint_norm.endswith("/list"):
return None
prefix = endpoint_norm[:-5]
if not prefix:
return None
return prefix.replace("/", "_")
def _record_trace(
self,
*,
method: str,
endpoint: str,
url: str,
request_params: Optional[Dict[str, Any]],
request_payload: Optional[Dict[str, Any]],
response: Optional[Dict[str, Any]],
elapsed_ms: int,
) -> None:
self._trace_seq += 1
op_kind = self._derive_op_kind(method, endpoint)
payload = request_payload or {}
response_view = self._build_trace_response_view(op_kind=op_kind, response=response)
params_text = self._to_trace_text(request_params or {}, max_len=self._TRACE_TEXT_MAX_LEN)
payload_text = self._to_trace_text(payload, max_len=self._TRACE_TEXT_MAX_LEN)
response_text = self._to_trace_text(response_view, max_len=self._TRACE_TEXT_MAX_LEN)
trace = {
"trace_id": self._trace_seq,
"op": op_kind,
"method": method.upper(),
"endpoint": endpoint,
"url": url,
"params": request_params or {},
"payload": payload,
"response": response_view,
"params_text": params_text,
"payload_text": payload_text,
"response_text": response_text,
"elapsed_ms": elapsed_ms,
"ts": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
}
push_id = payload.get("push_id")
biz_id = payload.get("biz_id")
if isinstance(push_id, str) and push_id:
self._trace_by_push_id[push_id].append(trace)
if isinstance(biz_id, str) and biz_id:
self._trace_by_biz_id[biz_id].append(trace)
self._trace_by_op[op_kind].append(trace)
if op_kind == "load":
node_type = self._derive_node_type_from_list_endpoint(endpoint)
data = (response or {}).get("data") if isinstance(response, dict) else None
if isinstance(data, list):
items = data
elif isinstance(data, dict):
items = data.get("list", [])
else:
items = []
if node_type and isinstance(items, list):
for item in items:
if not isinstance(item, dict):
continue
item_id = item.get("id") or item.get("_id")
if not item_id:
continue
self._load_trace_by_item[(node_type, str(item_id))].append(trace)
def _to_trace_text(self, value: Any, *, max_len: int) -> str:
try:
text = json_lib.dumps(value, ensure_ascii=False, separators=(",", ":"))
except Exception:
text = str(value)
if len(text) > max_len:
return f"{text[:max_len]}...(已截断)"
return text
def _build_trace_response_view(self, *, op_kind: str, response: Optional[Dict[str, Any]]) -> Any:
return response or {}
def consume_traces_by_push_id(self, push_id: str) -> List[Dict[str, Any]]:
if not push_id:
return []
bucket = self._trace_by_push_id.get(push_id)
if not bucket:
return []
traces = list(bucket)
bucket.clear()
return traces
def consume_latest_trace_by_biz_id(self, biz_id: str, op_kind: Optional[str] = None) -> Optional[Dict[str, Any]]:
if not biz_id:
return None
bucket = self._trace_by_biz_id.get(biz_id)
if not bucket:
return None
if op_kind is None:
return bucket.pop() if bucket else None
for index in range(len(bucket) - 1, -1, -1):
trace = bucket[index]
if trace.get("op") == op_kind:
del bucket[index]
return trace
return None
def peek_latest_trace_by_biz_id(self, biz_id: str, op_kind: Optional[str] = None) -> Optional[Dict[str, Any]]:
if not biz_id:
return None
bucket = self._trace_by_biz_id.get(biz_id)
if not bucket:
return None
if op_kind is None:
return bucket[-1] if bucket else None
for index in range(len(bucket) - 1, -1, -1):
trace = bucket[index]
if trace.get("op") == op_kind:
return trace
return None
def consume_latest_trace_by_op(self, op_kind: str) -> Optional[Dict[str, Any]]:
bucket = self._trace_by_op.get(op_kind)
if not bucket:
return None
return bucket.pop() if bucket else None
def peek_latest_trace_by_op(self, op_kind: str) -> Optional[Dict[str, Any]]:
bucket = self._trace_by_op.get(op_kind)
if not bucket:
return None
return bucket[-1] if bucket else None
def consume_load_traces(self, node_type: str, data_id: str) -> List[Dict[str, Any]]:
key = (node_type, data_id)
bucket = self._load_trace_by_item.get(key)
if not bucket:
return []
traces = list(bucket)
bucket.clear()
return traces
def _generate_signature(self, params: Dict[str, Any]) -> Dict[str, str]:
"""
生成签名参数（兼容旧版签名算法）
Args:
params: 请求参数
Returns:
包含 timestamp, nonce, sign 的字典
"""
timestamp = int(time.time() * 1000) # 毫秒时间戳
nonce = uuid.uuid4().hex[:16] # 16位随机字符串
# 构建签名参数
sign_params = params.copy()
sign_params['timestamp'] = timestamp
sign_params['nonce'] = nonce
# 按key排序
sorted_items = sorted(sign_params.items())
# 构建签名字符串
items = []
for k, v in sorted_items:
if v is None:
v_str = "null"
elif isinstance(v, bool):
v_str = "true" if v else "false"
elif isinstance(v, str):
v_str = v
elif isinstance(v, (int, float)):
v_str = str(v)
elif isinstance(v, (list, dict)):
v_str = json_lib.dumps(v, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
else:
v_str = str(v)
items.append(f"{k}={v_str}")
sign_str = "&".join(items)
# 使用 HMAC-SHA256 生成签名
sign = hmac.new(self._secret.encode("utf-8"), sign_str.encode("utf-8"), hashlib.sha256).hexdigest()
return {
'timestamp': str(timestamp),
'nonce': nonce,
'sign': sign
}
async def request(
self,
method: str,
endpoint: str,
params: Optional[Dict[str, Any]] = None,
data: Optional[Dict[str, Any]] = None,
json_data: Optional[Dict[str, Any]] = None,
json: Optional[Dict[str, Any]] = None
) -> Dict[str, Any]:
"""
统一请求入口，自动签名
Args:
method: HTTP 方法（GET/POST/PUT/DELETE）
endpoint: API 端点（如 /api/v2/project）
params: URL 查询参数
data: 请求体数据（form data）
json_data: JSON 请求体数据
json: JSON 请求体数据（兼容参数）
Returns:
响应 JSON 数据
Raises:
httpx.HTTPStatusError: HTTP 错误
"""
# 兼容json参数
if json is not None and json_data is None:
json_data = json
client = self._get_client()
url = f"{self._base_url}{endpoint}"
await self._acquire_rate_limit_slot()
# 记录请求信息
self._log(f"\n{'='*60}", level="DEBUG")
self._log(f"REQUEST: {method} {endpoint}", level="DEBUG")
self._log(f"Full URL: {url}", level="DEBUG")
if params:
self._log(f"Params: {json_lib.dumps(params, ensure_ascii=False, indent=2)}", level="DEBUG")
if data:
self._log(f"Data: {json_lib.dumps(data, ensure_ascii=False, indent=2)}", level="DEBUG")
if json_data:
self._log(f"JSON: {json_lib.dumps(json_data, ensure_ascii=False, indent=2)}", level="DEBUG")
# 准备签名参数
if method.upper() == 'GET':
# GET 请求：uid 放在 params 中
if params is None:
params = {}
params['uid'] = self._uid
# 转换所有参数值为字符串（除了 list/dict）
params = {k: str(v) if not isinstance(v, (list, dict)) else v for k, v in params.items()}
sign_params = params.copy()
# 生成签名并附加到 params
signature = self._generate_signature(sign_params)
params.update(signature)
else:
# POST/PUT/DELETE：uid 和签名放在 body 中（符合 v2 schema）
if data is None:
data = {}
if 'steps' not in data:
data['steps'] = _build_default_push_steps()
data['uid'] = self._uid
sign_params = data.copy()
if params:
sign_params.update(params)
signature = self._generate_signature(sign_params)
data.update(signature)
# 发送请求并增强异常处理
start_time = time.time()
try:
response = await client.request(
method=method,
url=url,
params=params,
json=data if method.upper() in ['POST', 'PUT'] else None
)
# 记录响应时间
elapsed = time.time() - start_time
self._log(f"Response Time: {elapsed:.3f}s", level="DEBUG")
# 检查 HTTP 状态
response.raise_for_status()
# 记录响应
result = response.json()
self._log(f"Status Code: {response.status_code}", level="DEBUG")
self._log(f"Response: {json_lib.dumps(result, ensure_ascii=False, indent=2)}", level="DEBUG")
self._log(f"{'='*60}\n", level="DEBUG")
self._record_trace(
method=method,
endpoint=endpoint,
url=url,
request_params=params,
request_payload=data if method.upper() in ['POST', 'PUT', 'DELETE'] else None,
response=result,
elapsed_ms=int(elapsed * 1000),
)
return result
except Exception as e:
import traceback
# 记录错误
elapsed = time.time() - start_time
error_msg = "\n========== [ApiClient HTTP Error] ==========\n"
error_msg += f"Request: {method} {url}\n"
error_msg += f"Params: {params}\n"
if method.upper() in ['POST', 'PUT']:
error_msg += f"JSON: {data}\n"
error_msg += f"Exception: {type(e).__name__}: {e}\n"
error_msg += f"Elapsed: {elapsed:.3f}s\n"
response_obj = e.response if isinstance(e, httpx.HTTPStatusError) else None
if response_obj is not None:
error_msg += f"Status Code: {response_obj.status_code}\n"
error_msg += f"Response Headers: {response_obj.headers}\n"
try:
error_msg += f"Response Text: {response_obj.text}\n"
except Exception:
pass
error_msg += traceback.format_exc()
error_msg += "============================================\n"
# 记录到日志文件
self._log(error_msg, level="ERROR")
raise
async def get(self, endpoint: str, params: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
"""
GET 请求
Args:
endpoint: API 端点
params: URL 查询参数
Returns:
响应 JSON 数据
"""
return await self.request('GET', endpoint, params=params)
async def post(self, endpoint: str, data: Dict[str, Any]) -> Dict[str, Any]:
"""
POST 请求，data 中已包含 push_id/biz_id（由 Handler 生成）
Args:
endpoint: API 端点
data: 请求体数据（包含业务数据和 push_id/biz_id）
Returns:
响应 JSON 数据
"""
return await self.request('POST', endpoint, data=data)
async def put(self, endpoint: str, data: Dict[str, Any]) -> Dict[str, Any]:
"""
PUT 请求，data 中已包含 push_id/biz_id
Args:
endpoint: API 端点
data: 请求体数据
Returns:
响应 JSON 数据
"""
return await self.request('PUT', endpoint, data=data)
async def delete(self, endpoint: str, data: Dict[str, Any]) -> Dict[str, Any]:
"""
DELETE 请求，data 中已包含 push_id/biz_id
Args:
endpoint: API 端点
data: 请求体数据
Returns:
响应 JSON 数据
"""
return await self.request('DELETE', endpoint, data=data)
async def get_push_logs(self, push_ids: List[str]) -> Dict[str, Dict[str, Any]]:
"""
批量查询 Push Log
注意：单个查询也传单元素列表
当前实现：用 for 循环模拟，后续等后端实现批量接口
Args:
push_ids: Push ID 列表
Returns:
{push_id: push_log_data} 的映射
"""
result: Dict[str, Dict[str, Any]] = {}
if not push_ids:
return result
try:
payload = PushIdsSchema(push_ids=','.join(push_ids)).model_dump(exclude_none=True)
response = await self.request(
method='POST',
endpoint='/push/result',
data=payload
)
except httpx.HTTPStatusError as e:
return {push_id: {'error': str(e)} for push_id in push_ids}
data_list: List[Dict[str, Any]] = []
if isinstance(response, dict):
raw_data = response.get('data')
if isinstance(raw_data, list):
data_list = [item for item in raw_data if isinstance(item, dict)]
elif isinstance(raw_data, dict):
data_list = [raw_data]
elif 'push_id' in response and 'status' in response:
data_list = [response]
for item in data_list:
push_id = item.get('push_id')
if not push_id:
continue
poll_trace = {
"trace_id": f"poll-{push_id}-{int(time.time() * 1000)}",
"op": "poll",
"method": "POST",
"endpoint": "/push/result",
"url": f"{self._base_url}/push/result",
"params": {},
"payload": {"push_ids": push_id},
"response": item,
"elapsed_ms": 0,
"ts": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
}
self._trace_by_push_id[push_id].append(poll_trace)
status = (item.get('status') or '').lower()
error_msg = item.get('error') or ''
result_data = item.get('result') or {}
if status == 'success':
# result_data 的结构是 {"code": 200, "message": "OK", "data": {"biz_id": "..."}}
# 需要提取 result_data["data"] 作为最终的 data
inner_data = result_data.get('data', {}) if isinstance(result_data, dict) else {}
result[push_id] = {
'code': 200,
'status': 'success',
'data': inner_data
}
elif status in {'failed', 'not_find'}:
result[push_id] = {
'code': 500,
'status': 'failed',
'message': error_msg or status or 'Unknown error',
'error': error_msg or status or 'Unknown error'
}
else:
result[push_id] = {
'status': status or 'processing'
}
for push_id in push_ids:
if push_id not in result:
result[push_id] = {'status': 'processing'}
return result
sync_state_machine/datasource/api/datasource.py
+264
View File
@@ -0,0 +1,264 @@
"""
ApiDataSource - API 数据源协调层
职责：
- 持有 ApiClient 实例，提供给 Handler 使用
- 实现 DataSource 接口（load_all, sync_all）
- 协调多个 Handler 的执行顺序
- 统一异常处理和日志记录
"""
import json
from typing import Optional
from pathlib import Path
from ..datasource import BaseDataSource
from ..handler import NodeHandler, ensure_handler_compat
from ..task_result import TaskResult
from .client import ApiClient
class ApiDataSource(BaseDataSource):
"""基于 API 的数据源实现"""
_TRACE_MAX_PARAMS_LEN = 1000
_TRACE_MAX_RESPONSE_LEN = 1000
def __init__(
self,
base_url: str,
uid: str,
secret: str,
poll_mode: str = "async",
debug: bool = False,
log_file: Optional[Path] = None,
poll_max_retries: int = 1,
poll_interval: float = 0.5,
rate_limit_max_requests: int = 30,
rate_limit_window_seconds: float = 10.0,
):
"""
初始化 API 数据源
Args:
base_url: API 基础 URL
uid: 用户标识
secret: 签名密钥
debug: 是否启用调试模式
log_file: 日志文件路径（可选）
"""
super().__init__(poll_max_retries=poll_max_retries, poll_interval=poll_interval)
self.client = ApiClient(
base_url,
uid,
secret,
debug=debug,
log_file=log_file,
rate_limit_max_requests=rate_limit_max_requests,
rate_limit_window_seconds=rate_limit_window_seconds,
)
self.poll_mode = poll_mode
async def initialize(self):
"""初始化数据源（初始化 HTTP 客户端）"""
await self.client.initialize()
async def close(self):
"""关闭数据源（关闭 HTTP 客户端）"""
await self.client.close()
def register_handler(self, handler: NodeHandler) -> None:
"""
注册业务 Handler
Args:
handler: API Handler 实例
"""
compatible_handler = ensure_handler_compat(handler)
compatible_handler.set_api_client(self.client)
compatible_handler.set_poll_mode(self.poll_mode)
# 调用基类注册方法
super().register_handler(compatible_handler)
# 如果 collection 已设置，确保新 handler 拿到引用
if self._collection is not None:
compatible_handler.set_collection(self._collection) # type: ignore[arg-type]
@staticmethod
def _fmt_trace_value(value, max_len: int = 600) -> str:
try:
text = json.dumps(value, ensure_ascii=False, separators=(",", ":"))
except Exception:
text = str(value)
if len(text) > max_len:
return text[:max_len] + "...<truncated>"
return text
def _append_trace_log(self, node, trace: dict, *, stage: str) -> None:
method = str(trace.get("method", "")).upper()
endpoint = str(trace.get("endpoint", ""))
url = str(trace.get("url", ""))
params = trace.get("params_text")
if not isinstance(params, str):
params = self._fmt_trace_value(trace.get("params", {}), max_len=self._TRACE_MAX_PARAMS_LEN)
payload = trace.get("payload_text")
if not isinstance(payload, str):
payload = self._fmt_trace_value(trace.get("payload", {}), max_len=self._TRACE_MAX_PARAMS_LEN)
response = trace.get("response_text")
if not isinstance(response, str):
response = self._fmt_trace_value(trace.get("response", {}), max_len=self._TRACE_MAX_RESPONSE_LEN)
elapsed_ms = trace.get("elapsed_ms", 0)
node.append_log(f"API链路[{stage}] {method} {endpoint}")
node.append_log(f" url={url}")
if params not in {"{}", "null"}:
node.append_log(f" params={params}")
if payload not in {"{}", "null"}:
node.append_log(f" payload={payload}")
node.append_log(f" response={response}")
node.append_log(f" elapsed_ms={elapsed_ms}")
def _consume_and_append_push_traces(self, node, push_id: str, *, stage: str) -> int:
if not push_id:
return 0
traces = self.client.consume_traces_by_push_id(push_id)
for trace in traces:
self._append_trace_log(node, trace, stage=stage)
return len(traces)
@staticmethod
def _extract_push_id_from_trace(trace: dict | None) -> Optional[str]:
if trace is None:
return None
payload = trace.get("payload")
if not isinstance(payload, dict):
return None
push_id = payload.get("push_id")
return push_id if isinstance(push_id, str) and push_id else None
def _resolve_push_id_for_success(self, node, result: TaskResult) -> Optional[str]:
if result.task_id:
return result.task_id
if result.push_id:
return result.push_id
action_name = node.action.value.lower()
biz_id = node.data_id or ""
if biz_id:
by_biz_and_op = self.client.peek_latest_trace_by_biz_id(biz_id, op_kind=action_name)
if by_biz_and_op is not None:
push_id = self._extract_push_id_from_trace(by_biz_and_op)
if push_id:
self.client.consume_latest_trace_by_biz_id(biz_id, op_kind=action_name)
return push_id
by_biz_any = self.client.peek_latest_trace_by_biz_id(biz_id)
if by_biz_any is not None:
push_id = self._extract_push_id_from_trace(by_biz_any)
if push_id:
self.client.consume_latest_trace_by_biz_id(biz_id)
return push_id
return None
def _append_missing_push_id_debug_trace(self, node) -> None:
action_name = node.action.value.lower()
biz_id = node.data_id or ""
if biz_id:
trace = self.client.peek_latest_trace_by_biz_id(biz_id, op_kind=action_name)
if trace is None:
trace = self.client.peek_latest_trace_by_biz_id(biz_id)
if trace is not None:
self._append_trace_log(node, trace, stage="submit-missing-push-id")
return
node.append_log(
f"API链路[submit-missing-push-id] 未找到匹配 biz_id 的请求记录，action={node.action.value}, data_id={node.data_id or '<empty>'}"
)
return
node.append_log(
f"API链路[submit-missing-push-id] 未找到可用请求记录（已禁用按操作类型兜底），action={node.action.value}, data_id={node.data_id or '<empty>'}"
)
return None
def _on_node_loaded(self, handler: NodeHandler, node) -> None:
data_id = node.data_id or ""
if not data_id:
return
traces = self.client.consume_load_traces(node.node_type, data_id)
for trace in traces:
self._append_trace_log(node, trace, stage="load")
async def _intercept_success_result(self, handler: NodeHandler, node, result: TaskResult, async_tasks: dict) -> bool:
if node.action.value not in {"CREATE", "UPDATE", "DELETE"}:
return False
push_id = self._resolve_push_id_for_success(node, result)
if not push_id:
self._append_missing_push_id_debug_trace(node)
raise RuntimeError(
f"[{node.node_type}] unified polling requires push_id for {node.action.value}, "
f"but none resolved for node={node.node_id}, data_id={node.data_id}"
)
synthetic = TaskResult.in_progress(node_id=node.node_id, task_id=push_id)
appended = self._consume_and_append_push_traces(node, push_id, stage="submit")
if appended == 0:
node.append_log(f"API链路[submit] 未捕获请求记录，push_id={push_id}")
await self._handle_in_progress_result(node, synthetic, async_tasks)
return True
def _on_in_progress_result(self, handler: NodeHandler, node, result: TaskResult) -> None:
if result.task_id:
appended = self._consume_and_append_push_traces(node, result.task_id, stage="submit")
if appended == 0:
node.append_log(f"API链路[submit] 未捕获请求记录，push_id={result.task_id}")
def _on_poll_result(self, handler: NodeHandler, node, task_id: str, result: TaskResult) -> None:
appended = self._consume_and_append_push_traces(node, task_id, stage="poll")
if appended == 0:
node.append_log(f"API链路[poll] 未捕获请求记录，push_id={task_id}")
def _on_failed_result(self, handler: NodeHandler, node, result: TaskResult) -> None:
if node.action.value not in {"CREATE", "UPDATE", "DELETE"}:
return
if result.push_id:
traces = self.client.consume_traces_by_push_id(result.push_id)
if traces:
for trace in traces:
self._append_trace_log(node, trace, stage="submit-failed")
return
node.append_log(f"API链路[submit-failed] 未捕获请求记录，push_id={result.push_id}")
action_name = node.action.value.lower()
biz_id = node.data_id or ""
trace = None
if biz_id:
trace = self.client.consume_latest_trace_by_biz_id(biz_id, op_kind=action_name)
if trace is None:
trace = self.client.consume_latest_trace_by_biz_id(biz_id)
if trace is not None:
self._append_trace_log(node, trace, stage="submit-failed")
return
if biz_id:
node.append_log(
f"API链路[submit-failed] 未找到匹配 biz_id 的请求记录，action={node.action.value}, data_id={biz_id}"
)
return
node.append_log(
f"API链路[submit-failed] 未找到可用请求记录（已禁用按操作类型兜底），action={node.action.value}, data_id=<empty>"
)
# load_all 和 sync_all 继承自 BaseDataSource
# BaseDataSource 会按顺序调用 Handler 的方法
sync_state_machine/datasource/api/errors.py
+25
View File
@@ -0,0 +1,25 @@
"""
API Handler 通用错误常量
"""
# API Client 相关错误
ERROR_API_CLIENT_NOT_INITIALIZED = "api_client not initialized"
ERROR_API_CLIENT_NOT_INJECTED = "api_client not injected"
# Collection 相关错误
ERROR_COLLECTION_NOT_SET = "collection not set"
# 数据验证错误
ERROR_NODE_DATA_NONE = "Node data is None or validation failed"
ERROR_VALIDATION_FAILED = "Validation failed"
# 业务数据错误
ERROR_PROJECT_ID_NOT_FOUND = "project_id not found"
ERROR_CONTRACT_ID_NOT_FOUND = "contract_id not found"
ERROR_ID_NOT_FOUND = "id not found"
# 操作不支持错误
ERROR_OPERATION_NOT_SUPPORTED = "Operation not supported"
ERROR_CREATE_NOT_SUPPORTED = "Create operation not supported"
ERROR_UPDATE_NOT_SUPPORTED = "Update operation not supported"
ERROR_DELETE_NOT_SUPPORTED = "Delete operation not supported"
sync_state_machine/datasource/api/handler.py
+538
View File
@@ -0,0 +1,538 @@
"""
BaseApiHandler - API Handler 基类
职责：
- 实现 Handler 接口（load, create, update, delete, poll）
- 使用 self.api_client 发送 HTTP 请求
- 生成 push_id 和 biz_id（业务标识）
- 从 collection 获取依赖节点数据
- 管理节点的 context（存储 project_id, contract_id 等）
- 实现 poll 方法，调用 api_client.get_push_logs 批量查询
- 提取 created_id（因为不同业务的 id 字段名和位置可能不同）
- 返回 TaskResult（SUCCESS/FAILED/IN_PROGRESS）
"""
import uuid
import asyncio
from abc import abstractmethod
from typing import ClassVar, Dict, List, Any, Optional, TYPE_CHECKING, TypeVar
from pydantic import BaseModel
from ...common.type_safety import get_pydantic_model_fields
from ..handler import NodeHandler
from ..task_result import TaskResult
from ..handler import ValidationResult # Import ValidationResult
if TYPE_CHECKING:
from ...common.sync_node import SyncNode
from ...common.collection import DataCollection
from .client import ApiClient
T = TypeVar("T", bound=BaseModel)
class BaseApiHandler(NodeHandler[T]):
"""API Handler 基类"""
# 子类需要设置这些属性
_node_type: str
_node_class: type['SyncNode']
_schema: Any
def __init__(self):
# 子类应该在调用 super().__init__() 之前或之后设置 _node_type, _node_class, _schema
self.api_client: 'ApiClient' = None # type: ignore # 由 DataSource 注入
self.poll_mode: str = "async" # async | sync
self._collection: 'DataCollection' = None # type: ignore # 由 DataSource 注入
# 子类在 __init__ 中设置此列表以声明 create/update 中涉及的 Pydantic schema 类；
# post-check 将只比较这些 schema 覆盖的字段，过滤后端只读的派生字段。
self.update_schemas: List[type] = []
def get_update_fields(self, *, exclude: frozenset = frozenset({'id'})) -> List[str]:
"""从 update_schemas 中提取所有可写字段名，排除标识键（默认排除 id）。"""
fields: set = set()
for schema in self.update_schemas:
for field_name in get_pydantic_model_fields(schema):
if field_name not in exclude:
fields.add(field_name)
return sorted(fields)
def set_api_client(self, client: 'ApiClient') -> None:
self.api_client = client
def set_collection(self, collection: 'DataCollection') -> None:
"""设置 Collection（由 DataSource 调用）"""
self._collection = collection
@property
def node_type(self) -> str:
"""返回节点类型"""
return self._node_type
@property
def node_class(self) -> type['SyncNode']:
"""返回节点类"""
return self._node_class
@property
def schema(self) -> Any:
"""返回 schema（子类可重写）"""
return self._schema
def _generate_push_id(self) -> str:
"""
生成 push_id（推送唯一标识）
Returns:
UUID 格式的 push_id
"""
return str(uuid.uuid4())
def _generate_biz_id(self, node: 'SyncNode') -> str:
"""
生成 biz_id（业务标识）
子类可重写此方法实现自定义业务 ID 生成逻辑。
默认使用 node_id 作为 biz_id。
Args:
node: 同步节点
Returns:
业务标识符
"""
return node.node_id
def _create_basic_node(self, data: Dict[str, Any], depend_ids: Optional[List[str]] = None) -> "SyncNode":
"""创建或复用基础节点（默认使用 data['id'] 或 data['_id']）"""
data_id = str(data.get("id") or data.get("_id") or "")
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() before creating nodes.")
if data_id:
existing_node = self._collection.get_by_data_id(self.node_type, data_id)
if existing_node is not None:
existing_node.depend_ids = list(depend_ids or [])
existing_node.set_data(data.copy())
existing_node.set_origin_data(data.copy())
return existing_node
node_id = self._collection.generate_node_id()
node = self.node_class(
node_id=node_id,
data_id=data_id,
data=data.copy(),
depend_ids=depend_ids or [],
origin_data=data.copy(),
)
return node
def _create_node(self, data: Dict[str, Any]) -> "SyncNode":
"""默认节点创建（子类可覆盖并补充 context）"""
return self._create_basic_node(data)
def extract_id(self, data: Dict[str, Any]) -> str:
"""从原始数据提取 ID（默认优先 id，其次 _id）"""
return str(data.get("id") or data.get("_id") or "")
def extract_created_id(self, push_log: Dict[str, Any]) -> Optional[str]:
"""
从 push_log 提取 created_id
子类可重写此方法以适配不同的响应结构。
默认从 push_log['data']['biz_id'] 提取。
Args:
push_log: Push Log 响应数据
Returns:
创建的对象 ID，未找到返回 None
"""
try:
data = push_log.get('data', {})
if isinstance(data, dict):
return data.get('biz_id')
return None
except (KeyError, AttributeError):
return None
def set_poll_mode(self, mode: str) -> None:
"""设置轮询模式（sync/async）"""
if mode not in {"sync", "async"}:
raise ValueError(f"Unsupported poll mode: {mode}")
self.poll_mode = mode
def _extract_created_id_from_response(self, response: Dict[str, Any]) -> Optional[str]:
"""从同步创建响应提取创建的 ID（默认 data.biz_id）"""
data = response.get("data") if isinstance(response, dict) else None
if isinstance(data, dict):
return data.get("biz_id")
return None
def _build_create_task_result(
self,
node_id: str,
response: Optional[Dict[str, Any]] = None,
task_id: Optional[str] = None,
fallback_data_id: Optional[str] = None,
) -> TaskResult:
"""根据轮询模式构建 create 的 TaskResult"""
if self.poll_mode == "sync":
created_id = None
if response is not None:
created_id = self._extract_created_id_from_response(response)
if not created_id:
created_id = fallback_data_id
return TaskResult.success(node_id=node_id, data_id=created_id)
if not task_id:
return TaskResult.failed(node_id=node_id, error="Missing task_id for async create")
return TaskResult.in_progress(node_id=node_id, task_id=task_id)
# ========== 通用辅助方法 ==========
def _validate_schema(self, data: Dict[str, Any], schema: Any) -> Optional[str]:
"""
验证数据是否符合 schema
Args:
data: 待验证数据
schema: Pydantic schema
Returns:
错误信息，无错误返回 None
"""
try:
schema.model_validate(data)
return None
except Exception as e:
return str(e)
def _get_schema_diff(
self,
schema,
data: Dict[str, Any],
origin_data: Dict[str, Any],
node_id: Optional[str] = None
) -> Dict[str, Any] | None:
"""
获取数据在指定 schema 下的差异字段
Args:
schema: Pydantic schema 类
data: 新数据
origin_data: 原始数据
node_id: 节点ID（用于日志）
Returns:
Dict[str, Any] | None: 有差异返回差异字段字典，无差异或字段不全返回 None
"""
from pydantic import ValidationError
try:
new_model = schema.model_validate(data)
origin_model = schema.model_validate(origin_data)
new_dict = new_model.model_dump(exclude_unset=True)
origin_dict = origin_model.model_dump(exclude_unset=True)
if new_dict == origin_dict:
return None
# 返回有差异的字段
diff_fields = {k: v for k, v in new_dict.items() if origin_dict.get(k) != v}
if diff_fields:
# 打印差异详情
node_info = f"[{node_id}]" if node_id else ""
print(f" [DIFF] {self.node_type}{node_info} schema={schema.__name__}")
for k, v in diff_fields.items():
orig_v = origin_dict.get(k)
print(f" • {k}: {orig_v!r} → {v!r}")
return diff_fields if diff_fields else None
except ValidationError:
# schema 字段不全，视为无差异
return None
def _filter_update_data(
self,
new_data: Dict[str, Any],
original_data: Dict[str, Any],
schema: Optional[Any] = None
) -> Dict[str, Any]:
"""
过滤出需要更新的字段
规则：
1. 提取 schema 定义的所有字段（包括必填）
2. 检查是否有任何字段发生变化
3. 如果有变化，返回所有 schema 字段；否则返回空
Args:
new_data: 新数据
original_data: 原始数据
schema: 更新 schema（可选）
Returns:
包含 schema 所有字段的数据（如果有变化），否则返回空字典
"""
schema_fields = get_pydantic_model_fields(schema)
if not schema_fields:
# 无 schema，按原逻辑处理
update_data = {}
for key, new_value in new_data.items():
if new_value == "" or new_value is None:
continue
original_value = original_data.get(key)
if new_value != original_value:
update_data[key] = new_value
return update_data
# 1. 提取 schema 定义的所有字段
schema_fields = set(schema_fields.keys())
schema_data = {}
has_change = False
for key in schema_fields:
new_value = new_data.get(key)
# 排除空值
if new_value == "" or new_value is None:
continue
schema_data[key] = new_value
# 检查是否变化
original_value = original_data.get(key)
if new_value != original_value:
# 记录具体差异以便排查
# print(f" [DEBUG] Field '{key}' changed: '{original_value}' -> '{new_value}'")
has_change = True
# 2. 只有存在变化时才返回数据
return schema_data if has_change else {}
def _schema_changed_fields(
self,
schema: Any,
new_data: Dict[str, Any],
origin_data: Dict[str, Any],
) -> List[str]:
schema_fields = get_pydantic_model_fields(schema)
if not schema_fields:
return []
changed: List[str] = []
for field_name in schema_fields.keys():
if new_data.get(field_name) != origin_data.get(field_name):
changed.append(field_name)
return changed
def _build_update_schema_status(
self,
*,
schema_name: str,
schema: Any,
new_data: Dict[str, Any],
origin_data: Dict[str, Any],
will_update: bool,
update_error: Optional[str] = None,
) -> str:
new_error = self._validate_schema(new_data, schema)
origin_error = self._validate_schema(origin_data, schema)
changed_fields = self._schema_changed_fields(schema, new_data, origin_data)
if new_error:
compact = new_error.replace("\n", " ").strip()
if len(compact) > 220:
compact = compact[:220] + "..."
return f"{schema_name}: NEW_INVALID({compact})"
if not changed_fields:
return f"{schema_name}: CONSISTENT"
changed_text = ",".join(changed_fields)
if update_error:
compact = update_error.replace("\n", " ").strip()
if len(compact) > 220:
compact = compact[:220] + "..."
return f"{schema_name}: DIFF[{changed_text}] -> UPDATE_FAILED({compact})"
if will_update:
if origin_error:
compact = origin_error.replace("\n", " ").strip()
if len(compact) > 160:
compact = compact[:160] + "..."
return f"{schema_name}: DIFF[{changed_text}] -> UPDATE_TRIGGERED (origin_invalid={compact})"
return f"{schema_name}: DIFF[{changed_text}] -> UPDATE_TRIGGERED"
if origin_error:
compact = origin_error.replace("\n", " ").strip()
if len(compact) > 160:
compact = compact[:160] + "..."
return f"{schema_name}: DIFF[{changed_text}] -> NOT_TRIGGERED (origin_invalid={compact})"
return f"{schema_name}: DIFF[{changed_text}] -> NOT_TRIGGERED"
async def _execute_with_retry(
self,
operation,
max_retries: int = 3,
retry_delay: float = 1.0
) -> Any:
"""
通用重试逻辑
子类可重写此方法实现自定义重试策略。
默认策略：网络错误/5xx 重试，4xx 不重试。
Args:
operation: 异步操作（协程函数）
max_retries: 最大重试次数
retry_delay: 重试延迟（秒）
Returns:
操作执行结果
Raises:
最后一次重试的异常
"""
import httpx
last_error = None
for attempt in range(max_retries):
try:
return await operation()
except httpx.HTTPStatusError as e:
# 4xx 客户端错误不重试
if 400 <= e.response.status_code < 500:
raise
last_error = e
if attempt < max_retries - 1:
await asyncio.sleep(retry_delay * (attempt + 1))
except (httpx.NetworkError, httpx.TimeoutException) as e:
# 网络错误重试
last_error = e
if attempt < max_retries - 1:
await asyncio.sleep(retry_delay * (attempt + 1))
# 重试次数用尽
if last_error:
raise last_error
else:
raise RuntimeError("Retry failed with unknown error")
# ========== 批量接口（子类必须实现） ==========
@abstractmethod
async def create_all(self, nodes: List['SyncNode']) -> List[TaskResult]:
"""
批量创建节点
Args:
nodes: 需要创建的节点列表（DataSource 已筛选为 CREATE + PENDING）
Returns:
List[TaskResult]，每个结果关联 node_id
"""
pass
@abstractmethod
async def update_all(self, nodes: List['SyncNode']) -> List[TaskResult]:
"""
批量更新节点
Args:
nodes: 需要更新的节点列表（DataSource 已筛选为 UPDATE + PENDING）
Returns:
List[TaskResult]，每个结果关联 node_id
"""
pass
@abstractmethod
async def delete_all(self, nodes: List['SyncNode']) -> List[TaskResult]:
"""
批量删除节点
Args:
nodes: 需要删除的节点列表（DataSource 已筛选为 DELETE + PENDING）
Returns:
List[TaskResult]，每个结果关联 node_id
"""
pass
async def poll_tasks(self, task_ids: List[str]) -> Dict[str, TaskResult]:
"""
批量轮询异步任务状态
Args:
task_ids: 任务 ID 列表（push_id 列表）
Returns:
{task_id: TaskResult} 映射
"""
if not task_ids:
return {}
if not self.api_client:
raise RuntimeError("api_client not injected")
# 批量查询 push_logs
try:
push_logs = await self.api_client.get_push_logs(task_ids)
except Exception as e:
# 查询失败，所有任务标记为失败
return {
task_id: TaskResult.failed(error=f"Poll failed: {str(e)}")
for task_id in task_ids
}
results = {}
for task_id in task_ids:
push_log = push_logs.get(task_id, {})
if 'error' in push_log:
results[task_id] = TaskResult.failed(error=push_log['error'])
elif push_log.get('code') == 200: # 通过code==200判断成功
created_id = self.extract_created_id(push_log)
results[task_id] = TaskResult.success(data_id=created_id)
elif push_log.get('code') and push_log.get('code') != 200: # code存在但不是200表示失败
results[task_id] = TaskResult.failed(error=push_log.get('message', 'Unknown error'))
else:
# 仍在进行中
results[task_id] = TaskResult.in_progress(task_id=task_id)
return results
async def validate(self, data: Dict[str, Any]) -> ValidationResult:
"""
数据质量检查（使用 schema 验证）
Args:
data: 原始数据
Returns:
ValidationResult（是否有效 + 错误列表）
"""
error = self._validate_schema(data, self.schema)
if error:
return ValidationResult(is_valid=False, errors=[error])
return ValidationResult(is_valid=True)
# ========== 抽象方法：子类必须实现 ==========
@abstractmethod
async def load(self) -> List['SyncNode']:
"""
从 API 加载数据并创建节点
Handler 负责：
1. 加载原始数据（可使用 self._collection 查询依赖）
2. 创建 SyncNode（调用 self.create_node）
3. 设置 context（如 project_id, contract_id 等）
Returns:
SyncNode 列表
"""
pass
sync_state_machine/datasource/datasource.py
+962
View File
@@ -0,0 +1,962 @@
"""
DataSource V2 - 状态管理 + Handler 编排层
DataSource 职责：
1. 注册和管理 Handler
2. 按依赖顺序加载数据
3. 执行同步（调用 Handler）
4. 管理节点状态流转（PENDING → IN_PROGRESS → SUCCESS/FAILED）
5. 处理异步任务轮询
6. 统一异常处理
DataSource 不负责：
- ❌ 后端 API 调用（由 Handler 负责）
- ❌ ID 映射（由 Strategy 负责）
- ❌ 业务逻辑（由 Strategy 负责）
"""
from __future__ import annotations
import asyncio
import logging
from typing import Any, List, Dict, Optional, TYPE_CHECKING
from abc import ABC
from pathlib import Path
if TYPE_CHECKING:
from ..common.collection import DataCollection
from .handler import NodeHandler
from ..common.sync_node import SyncNode
from ..common.types import SyncAction
from .task_result import TaskStatus
from ..common.types import SyncStatus
from ..common.type_safety import get_pydantic_model_fields
from .handler import ensure_handler_compat
from .task_result import TaskResult
from ..engine import (
StateMachineConfig,
StateMachineRuntime,
e40_sync_execute,
)
logger = logging.getLogger(__name__)
class BaseDataSource(ABC):
"""
数据源基类 V2（状态管理 + 编排层）
核心流程：
1. 注册 Handler：每个节点类型对应一个 Handler
2. 加载数据：按依赖顺序调用 Handler.load()
3. 执行同步：根据 node.action 调用 Handler 的 create/update/delete
4. 状态管理：根据 TaskResult 更新 node.status
5. 异步轮询：处理 IN_PROGRESS 节点
示例：
# 1. 初始化 DataSource
datasource = MyDataSource()
# 2. 注册 Handler
datasource.register_handler(project_handler)
datasource.register_handler(contract_handler)
# 3. 设置 Collection
datasource.set_collection(collection)
# 4. 加载数据（按顺序）
await datasource.load_all(order=["project", "contract"])
# 5. 执行同步（按顺序）
await datasource.sync_all(order=["project", "contract"])
"""
def __init__(self, poll_max_retries: int = 1, poll_interval: float = 0.5):
self._handlers: Dict[str, "NodeHandler"] = {}
self._collection: Optional["DataCollection"] = None
self._target_project_ids: List[str] = []
# 按节点类型统计信息
# {"company": {"loaded": 10, "synced": 10, "success": 10, "failed": 0}, ...}
self._stats: Dict[str, Dict[str, int]] = {}
# 记录本次 sync 的动作统计（CREATE/UPDATE/DELETE/NONE）
self._action_summary: Dict[str, Dict[str, Dict[str, int]]] = {}
# 记录本次 sync 中的节点动作（用于异步完成后的统计）
self._action_by_node_id: Dict[str, "SyncAction"] = {}
self._poll_max_retries = max(1, poll_max_retries)
self._poll_interval = max(0, poll_interval)
self._sm_runtime: Optional[StateMachineRuntime] = None
def _ensure_sm_runtime(self) -> StateMachineRuntime:
if self._sm_runtime is None:
cfg_path = Path(__file__).resolve().parents[1] / "config" / "node_state_machine.yaml"
self._sm_runtime = StateMachineRuntime(StateMachineConfig.load(cfg_path))
return self._sm_runtime
def set_state_machine_runtime(self, runtime: StateMachineRuntime) -> None:
"""注入统一状态机 runtime（通常由 pipeline 负责）。"""
self._sm_runtime = runtime
def _apply_sync_execute_state(
self,
node: "SyncNode",
*,
handler_status: "TaskStatus",
poll_timeout: bool = False,
detail: str = "",
action_override: Optional["SyncAction"] = None,
result_data_id: Optional[str] = None,
) -> bool:
action_for_event = action_override or node.action
decision = e40_sync_execute(
self._ensure_sm_runtime(),
node=node,
handler_result=handler_status.name,
action=action_for_event.value,
poll_timeout=poll_timeout,
result_data_id=result_data_id,
)
if decision is None:
if detail:
node.append_log(
f"执行状态未变更: action={action_for_event.value}, result={handler_status.name}, detail={detail}"
)
return False
if detail:
node.append_log(
f"执行状态写回: action={action_for_event.value}, result={handler_status.name}, detail={detail}"
)
return True
def _compose_node_error(self, node: "SyncNode", *, source: str, detail: str) -> str:
return (
f"{source} | node={node.node_id} | type={node.node_type} | "
f"action={node.action.value} | status={node.status.value} | detail={detail}"
)
def _on_node_loaded(self, handler: "NodeHandler", node: "SyncNode") -> None:
"""Datasource-specific hook when a loaded node is upserted into collection."""
return
async def _intercept_success_result(
self,
handler: "NodeHandler",
node: "SyncNode",
result: "TaskResult",
async_tasks: Dict[str, str],
) -> bool:
"""Datasource-specific hook to intercept SUCCESS result handling.
Return True if handled/consumed by subclass and no further default SUCCESS flow is needed.
"""
return False
def _on_in_progress_result(self, handler: "NodeHandler", node: "SyncNode", result: "TaskResult") -> None:
"""Datasource-specific hook after an IN_PROGRESS task is queued."""
return
def _on_poll_result(self, handler: "NodeHandler", node: "SyncNode", task_id: str, result: "TaskResult") -> None:
"""Datasource-specific hook after polling result is applied to node."""
return
def _on_failed_result(self, handler: "NodeHandler", node: "SyncNode", result: "TaskResult") -> None:
"""Datasource-specific hook after a FAILED result is applied to node."""
return
async def _handle_success_result(self, node: "SyncNode", result: "TaskResult") -> None:
previous_data_id = node.data_id
ok = self._apply_sync_execute_state(
node,
handler_status=result.status,
detail="远程操作成功完成",
result_data_id=result.data_id,
)
if not ok:
raise RuntimeError(f"[{node.node_type}] SUCCESS result cannot map to state machine: node={node.node_id}")
self._record_action_result(node, result.status)
await self._apply_success_result(node, result.data_id, previous_data_id=previous_data_id)
node.error = None
async def _handle_failed_result(self, node: "SyncNode", result: "TaskResult") -> None:
detail = result.error or "Unknown error"
ok = self._apply_sync_execute_state(
node,
handler_status=result.status,
detail=f"远程操作失败: {detail}",
)
if not ok:
raise RuntimeError(f"[{node.node_type}] FAILED result cannot map to state machine: node={node.node_id}")
self._record_action_result(node, result.status)
node.error = self._compose_node_error(node, source="handler_failed", detail=detail)
async def _handle_skipped_result(self, node: "SyncNode", result: "TaskResult") -> None:
from .task_result import TaskStatus as _TaskStatus
original_action = self._action_by_node_id.get(node.node_id)
if original_action is None or original_action.value != "UPDATE":
raise RuntimeError(
f"[{node.node_type}] SKIPPED is only allowed for UPDATE action: node={node.node_id}, action={original_action.value if original_action is not None else None}"
)
ok = self._apply_sync_execute_state(
node,
handler_status=_TaskStatus.SKIPPED,
detail="远程操作被跳过",
action_override=original_action,
)
if not ok:
raise RuntimeError(f"[{node.node_type}] SKIPPED result cannot map to state machine: node={node.node_id}")
self._record_action_result(node, _TaskStatus.SKIPPED)
if result.sync_log:
node.append_log(f"同步跳过: {result.sync_log}")
node.error = None
async def _handle_in_progress_result(
self,
node: "SyncNode",
result: "TaskResult",
async_tasks: Dict[str, str],
) -> None:
if not result.task_id:
node.error = self._compose_node_error(
node,
source="handler_in_progress_invalid",
detail="missing task_id",
)
node.append_log("异步任务返回无 task_id，已忽略")
return
if not result.node_id:
node.error = self._compose_node_error(
node,
source="handler_in_progress_invalid",
detail="missing node_id",
)
node.append_log("异步任务返回无 node_id，已忽略")
return
async_tasks[result.node_id] = result.task_id
ok = self._apply_sync_execute_state(
node,
handler_status=result.status,
detail=f"异步任务进行中: {result.task_id}",
)
if not ok:
node.append_log(f"异步任务进行中（无状态变更）: {result.task_id}")
def set_polling_config(self, max_retries: int, interval: float = 0.5) -> None:
"""设置异步轮询参数（默认仅轮询一次）"""
self._poll_max_retries = max(1, max_retries)
self._poll_interval = max(0, interval)
def reset_stats(self) -> None:
"""重置统计信息"""
self._stats = {}
self._action_summary = {}
self._action_by_node_id = {}
def get_stats(self) -> Dict[str, Dict[str, int]]:
"""
获取按节点类型的统计信息
Returns:
dict: {"company": {"loaded": 10, "synced": 10, "success": 10, "failed": 0}, ...}
"""
return self._stats.copy()
def get_action_summary(self) -> Dict[str, Dict[str, Dict[str, int]]]:
"""
获取本次 sync 的动作统计（按节点类型）。
Returns:
dict: {
"company": {
"create": {"total": 10, "success": 8, "failed": 2},
"update": {"total": 2, "success": 2, "failed": 0},
"delete": {"total": 0, "success": 0, "failed": 0},
"none": {"total": 5, "success": 0, "failed": 0},
"total": {"total": 17, "success": 10, "failed": 2},
},
...
}
"""
return {k: {kk: vv.copy() for kk, vv in v.items()} for k, v in self._action_summary.items()}
def _init_node_stats(self, node_type: str) -> None:
"""初始化节点类型的统计信息"""
if node_type not in self._stats:
self._stats[node_type] = {
"loaded": 0,
"synced": 0,
"success": 0,
"failed": 0,
}
if node_type not in self._action_summary:
self._action_summary[node_type] = {
"create": {"total": 0, "success": 0, "failed": 0, "skipped": 0},
"update": {"total": 0, "success": 0, "failed": 0, "skipped": 0},
"delete": {"total": 0, "success": 0, "failed": 0, "skipped": 0},
"none": {"total": 0, "success": 0, "failed": 0, "skipped": 0},
"total": {"total": 0, "success": 0, "failed": 0, "skipped": 0},
}
def _set_action_totals(self, node_type: str, create: int, update: int, delete: int, none: int) -> None:
summary = self._action_summary[node_type]
summary["create"]["total"] = create
summary["update"]["total"] = update
summary["delete"]["total"] = delete
summary["none"]["total"] = none
summary["total"]["total"] = create + update + delete + none
def _record_action_result(self, node: "SyncNode", result_status: "TaskStatus") -> None:
from ..common.types import SyncAction
from .task_result import TaskStatus as _TaskStatus
# 情况 A: 原始动作就是 NONE
# 情况 B: Handler 运行时发现没差异，把 action 降级为了 NONE
action = node.action
# 获取最初锁定的动作（用于修正统计列）
original_action = self._action_by_node_id.pop(node.node_id, SyncAction.NONE)
summary = self._action_summary.get(node.node_type)
if summary is None:
return
# 如果当前 action 是 NONE，说明这次“同步”被取消了
if action == SyncAction.NONE:
# 如果原始动作不是 NONE，说明发生了降级动作，我们需要把 total 从对应列挪到 none 列
if original_action != SyncAction.NONE:
orig_key = original_action.value.lower()
if orig_key in summary and summary[orig_key]["total"] > 0:
summary[orig_key]["total"] -= 1
summary["none"]["total"] += 1
return
# 正常记录成功/失败/跳过
key = action.value.lower()
if result_status == _TaskStatus.SUCCESS:
summary[key]["success"] += 1
summary["total"]["success"] += 1
elif result_status == _TaskStatus.FAILED:
summary[key]["failed"] += 1
summary["total"]["failed"] += 1
elif result_status == _TaskStatus.SKIPPED:
summary[key]["skipped"] += 1
summary["total"]["skipped"] += 1
def set_collection(self, collection: "DataCollection") -> None:
"""
设置 Collection（在 load_all/sync_all 前调用）
Args:
collection: 数据集合
"""
self._collection = collection
collection.set_state_machine_runtime(self._ensure_sm_runtime())
# 同时设置到所有已注册的 handler
for handler in self._handlers.values():
handler.set_collection(collection)
# ========== Handler 管理 ==========
def register_handler(self, handler: "NodeHandler") -> None:
"""
注册 Handler
Args:
handler: 节点处理器实例
示例：
datasource.register_handler(ContractNodeHandler(api_client))
"""
compatible_handler = ensure_handler_compat(handler)
self._handlers[compatible_handler.node_type] = compatible_handler
self._apply_target_project_ids_to_handler(compatible_handler)
def set_target_project_ids(self, target_project_ids: Optional[List[str]]) -> None:
"""设置项目过滤列表，并在注册阶段下发给所有 handler。"""
normalized = [str(pid) for pid in (target_project_ids or []) if str(pid)]
self._target_project_ids = normalized
for handler in self._handlers.values():
self._apply_target_project_ids_to_handler(handler)
def _apply_target_project_ids_to_handler(self, handler: "NodeHandler") -> None:
handler.set_target_project_ids(self._target_project_ids)
def get_handler(self, node_type: str) -> "NodeHandler":
"""获取 Handler"""
if node_type not in self._handlers:
raise KeyError(f"Handler for '{node_type}' not registered")
return self._handlers[node_type]
async def _upsert_loaded_nodes(
self,
handler: "NodeHandler",
node_type: str,
nodes: List["SyncNode"]
) -> None:
"""将加载的节点写入 Collection（按 data_id 优先命中已有节点）"""
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() first.")
for node in nodes:
existing_node = None
if node.data_id:
existing_node = self._collection.get_by_data_id(node_type, node.data_id)
load_source = f"{self.__class__.__name__}.load"
if existing_node is not None:
data_payload = node.get_data()
existing_node.depend_ids = list(node.depend_ids or [])
if node.context:
existing_node.context.update(node.context)
existing_node.set_data(data_payload)
existing_node.set_origin_data(data_payload)
if existing_node.context.pop("_loaded_from_persistence", False):
existing_node.append_log(f"加载来源: persistence -> {load_source} 覆盖刷新")
else:
existing_node.append_log(f"加载来源: {load_source} 覆盖刷新")
await self._collection.update(existing_node)
self._on_node_loaded(handler, existing_node)
else:
node.append_log(f"加载来源: {load_source} 新增节点")
await self._collection.add(node)
self._on_node_loaded(handler, node)
# ========== 数据加载 ==========
async def load_all(
self,
order: Optional[List[str]] = None,
data_type: Optional[str] = None
) -> None:
"""
按依赖顺序加载所有数据
注意：调用前需先 set_collection()
Args:
order: 节点类型顺序（如 ["project", "contract", "construction"])
data_type: 仅加载单一类型（与 order 互斥）
工作流程：
1. 按顺序遍历每个节点类型
2. 调用 Handler.load(collection)
3. Handler 可以从 collection 查询依赖节点（如 Contract 查询 Project）
4. 使用 Handler.create_node() 创建 SyncNode
5. 添加到 Collection
示例：
datasource.set_collection(collection)
await datasource.load_all(order=["project", "contract"])
"""
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() first.")
if data_type is not None:
order_to_load = [data_type]
elif order is not None:
order_to_load = order
else:
order_to_load = list(self._handlers.keys())
for node_type in order_to_load:
handler = self.get_handler(node_type)
# 初始化统计
self._init_node_stats(node_type)
try:
# Handler 加载数据并创建节点
nodes = await handler.load()
print(f"Loaded nodes for {node_type}: {len(nodes)}")
# 将节点写入 Collection：优先按 data_id 命中已有节点并更新
await self._upsert_loaded_nodes(handler, node_type, nodes)
# 统计加载的节点数
self._stats[node_type]["loaded"] += len(nodes)
except Exception as exc:
print(f"❌ [{node_type}] 加载失败: {exc}")
continue
# ========== 同步执行 ==========
async def sync_all(
self,
order: Optional[List[str]] = None,
data_type: Optional[str] = None,
poll_async_tasks: bool = True
) -> Dict[str, Dict[str, str]]:
"""
按依赖顺序执行同步
注意：调用前需先 set_collection()
Args:
order: 节点类型顺序
data_type: 仅同步单一类型（与 order 互斥）
工作流程：
1. 按顺序遍历每个节点类型
2. 筛选需要同步的节点（status=PENDING, action!=NONE）
3. 调用 Handler 的 create/update/delete
4. 根据 TaskResult 更新节点状态
5. 处理异步任务轮询
状态流转：
- PENDING + action=CREATE/UPDATE/DELETE → IN_PROGRESS → SUCCESS/FAILED
- PENDING + action=NONE → 保持 PENDING
- DEPENDENCY_ERROR / ABNORMAL / WARNING → SKIPPED
示例：
datasource.set_collection(collection)
await datasource.sync_all(order=["project", "contract"])
"""
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() first.")
if data_type is not None:
order_to_sync = [data_type]
elif order is not None:
order_to_sync = order
else:
order_to_sync = list(self._handlers.keys())
async_tasks_by_type: Dict[str, Dict[str, str]] = {}
for node_type in order_to_sync:
handler = self.get_handler(node_type)
# 批量同步该类型的所有节点
async_tasks = await self._sync_nodes(handler, poll_async_tasks=poll_async_tasks)
if async_tasks:
async_tasks_by_type[node_type] = async_tasks
return async_tasks_by_type
async def _sync_nodes(
self,
handler: "NodeHandler",
poll_async_tasks: bool = True
) -> Dict[str, str]:
"""
同步一批节点（批量接口）
Args:
handler: 节点处理器
流程：
1. DataSource 筛选需要处理的节点
2. 调用 handler.create_all(nodes)
3. 调用 handler.update_all(nodes)
4. 调用 handler.delete_all(nodes)
5. 根据 List[TaskResult] 更新节点状态
6. 收集并轮询异步任务
"""
from ..common.types import SyncAction, SyncStatus
from .task_result import TaskStatus as _TaskStatus
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() first.")
collection = self._collection
# 初始化统计
self._init_node_stats(handler.node_type)
async_tasks: Dict[str, str] = {} # {node_id: task_id}
# DataSource 负责筛选需要创建的节点（排除已失败的节点）
create_nodes = [
n
for n in collection.filter_by_state_ids(
node_type=handler.node_type,
state_ids=["S06"],
)
if n.action == SyncAction.CREATE
]
# DataSource 负责筛选需要更新的节点（排除已失败的节点）
update_nodes = [
n
for n in collection.filter_by_state_ids(
node_type=handler.node_type,
state_ids=["S07"],
)
if n.action == SyncAction.UPDATE
]
# DataSource 负责筛选需要删除的节点（排除已失败的节点）
delete_nodes = [
n
for n in collection.filter_by_state_ids(
node_type=handler.node_type,
state_ids=["S09"],
)
if n.action == SyncAction.DELETE
]
if delete_nodes:
raise NotImplementedError(
f"[{handler.node_type}] delete path is not implemented yet, "
f"but found {len(delete_nodes)} node(s) in S09/DELETE."
)
# 记录本次 sync 的动作统计（在 action 被重置前）
for node in create_nodes:
self._action_by_node_id[node.node_id] = SyncAction.CREATE
for node in update_nodes:
self._action_by_node_id[node.node_id] = SyncAction.UPDATE
for node in delete_nodes:
self._action_by_node_id[node.node_id] = SyncAction.DELETE
none_count = len(
self._collection.filter(
node_type=handler.node_type,
node_filter={"action": SyncAction.NONE}
)
)
self._set_action_totals(
handler.node_type,
create=len(create_nodes),
update=len(update_nodes),
delete=len(delete_nodes),
none=none_count,
)
# 批量创建（传入节点列表）
if create_nodes:
logger.info(f"[{handler.node_type}] create提交开始: nodes={len(create_nodes)}")
try:
create_results = await handler.create_all(create_nodes)
create_success = 0
create_in_progress = 0
create_failed = 0
create_skipped = 0
for result in create_results:
if result.status == _TaskStatus.SUCCESS:
create_success += 1
elif result.status == _TaskStatus.IN_PROGRESS:
create_in_progress += 1
elif result.status == _TaskStatus.FAILED:
create_failed += 1
elif result.status == _TaskStatus.SKIPPED:
create_skipped += 1
await self._handle_task_result(handler, result, async_tasks)
logger.info(
f"[{handler.node_type}] create提交完成: success={create_success}, "
f"in_progress={create_in_progress}, failed={create_failed}, skipped={create_skipped}"
)
except Exception as e:
print(f"❌ [{handler.node_type}] create_all 异常: {e}")
for node in create_nodes:
ok = self._apply_sync_execute_state(
node,
handler_status=_TaskStatus.FAILED,
detail=str(e),
)
if not ok:
raise RuntimeError(f"[{handler.node_type}] create_all exception cannot map to state machine")
# 批量更新（传入节点列表）
if update_nodes:
try:
update_results = await handler.update_all(update_nodes)
for result in update_results:
await self._handle_task_result(handler, result, async_tasks)
except Exception as e:
print(f"❌ [{handler.node_type}] update_all 异常: {e}")
for node in update_nodes:
ok = self._apply_sync_execute_state(
node,
handler_status=_TaskStatus.FAILED,
detail=str(e),
)
if not ok:
raise RuntimeError(f"[{handler.node_type}] update_all exception cannot map to state machine")
# 批量删除（delete stage 当前未实现，检测到 S09 会在上方直接 fail-fast）
# 轮询异步任务
if async_tasks and poll_async_tasks:
logger.info(
f"[{handler.node_type}] create已提交，push_id检查即将开始: pending={len(async_tasks)}, "
f"first_wait={self._poll_interval:.3f}s"
)
await self._poll_async_tasks(handler, async_tasks)
# 统计同步结果
all_nodes = self._collection.filter(node_type=handler.node_type)
synced_count = len([n for n in all_nodes if n.action != SyncAction.NONE])
success_count = len([n for n in all_nodes if n.status == SyncStatus.SUCCESS])
failed_count = len([n for n in all_nodes if n.status == SyncStatus.FAILED])
self._stats[handler.node_type]["synced"] = synced_count
self._stats[handler.node_type]["success"] = success_count
self._stats[handler.node_type]["failed"] = failed_count
return async_tasks
async def poll_async_tasks(self, async_tasks_by_type: Dict[str, Dict[str, str]]) -> None:
"""由 Pipeline 调度的异步任务轮询"""
for node_type, async_tasks in async_tasks_by_type.items():
if not async_tasks:
continue
handler = self.get_handler(node_type)
await self._poll_async_tasks(handler, async_tasks)
async def _handle_task_result(
self,
handler: "NodeHandler",
result: "TaskResult",
async_tasks: Dict[str, str]
) -> None:
"""
处理 Handler 返回的 TaskResult
Args:
collection: Collection 上下文
result: Handler 返回的结果（包含 node_id）
async_tasks: 异步任务字典
"""
from .task_result import TaskStatus as _TaskStatus
if not result.node_id:
# 没有 node_id，无法定位节点
return
if self._collection is None:
# Collection 未设置，无法处理结果
return
# 从 self._collection 中查找节点
node = self._collection.get(result.node_id)
if not node:
return
if result.status == _TaskStatus.SUCCESS:
if await self._intercept_success_result(handler, node, result, async_tasks):
return
if result.status == _TaskStatus.SUCCESS:
await self._handle_success_result(node, result)
return
if result.status == _TaskStatus.FAILED:
self._on_failed_result(handler, node, result)
await self._handle_failed_result(node, result)
return
if result.status == _TaskStatus.SKIPPED:
await self._handle_skipped_result(node, result)
return
if result.status == _TaskStatus.IN_PROGRESS:
self._on_in_progress_result(handler, node, result)
await self._handle_in_progress_result(node, result, async_tasks)
return
def _write_back_data_id_to_data(self, node, data_id: str) -> None:
data = node.get_data()
if data is None:
return
schema = node.get_schema()
candidate_keys = ["id", "_id"]
model_fields = get_pydantic_model_fields(schema)
for key in candidate_keys:
if key in data:
data[key] = data_id
node.set_data(data)
return
if key in model_fields:
data[key] = data_id
node.set_data(data)
return
async def _apply_success_result(self, node, data_id: Optional[str], previous_data_id: str) -> None:
from ..common.types import SyncAction
if data_id and node.action == SyncAction.CREATE:
if node.data_id != data_id:
raise RuntimeError(
f"[{node.node_type}] data_id must be updated in event apply path: expected={data_id}, actual={node.data_id}"
)
self._write_back_data_id_to_data(node, data_id)
if self._collection is not None and data_id != previous_data_id:
await self._collection.update(node)
if node.action == SyncAction.UPDATE:
current_data = node.get_data()
if current_data is not None:
node.set_origin_data(current_data)
if node.action == SyncAction.DELETE:
if self._collection is None:
return
await self._collection.delete(node.node_id)
return
# 成功后 action 是否重置由上层策略决定
async def _poll_async_tasks(
self,
handler: "NodeHandler",
async_tasks: Dict[str, str]
) -> None:
"""
轮询异步任务直到全部完成
Args:
handler: 节点处理器
collection: Collection 上下文
async_tasks: {node_id: task_id} 映射
流程：
1. 批量调用 Handler.poll_tasks()
2. 更新节点状态
3. 如果还有未完成任务，等待后重试
"""
from .task_result import TaskStatus as _TaskStatus
max_retries = self._poll_max_retries
retry_count = 0
poll_interval = self._poll_interval # 轮询间隔（秒）
if self._collection is None:
# Collection 未设置，无法轮询任务
return
if async_tasks and poll_interval > 0:
logger.info(
f"[{handler.node_type}] push_id首次检查前等待 {poll_interval:.3f}s, pending={len(async_tasks)}"
)
await asyncio.sleep(poll_interval)
while async_tasks and retry_count < max_retries:
logger.info(
f"[{handler.node_type}] push_id检查 {retry_count + 1}/{max_retries}, pending={len(async_tasks)}"
)
# 批量查询
task_ids = list(async_tasks.values())
results = await handler.poll_tasks(task_ids)
# 处理结果
completed_nodes = []
for node_id, task_id in list(async_tasks.items()):
if task_id not in results:
continue
result = results[task_id]
node = self._collection.get(node_id)
if not node:
completed_nodes.append(node_id)
continue
if result.status == _TaskStatus.SUCCESS:
self._on_poll_result(handler, node, task_id, result)
previous_data_id = node.data_id
ok = self._apply_sync_execute_state(
node,
handler_status=result.status,
detail=f"异步任务成功完成: {task_id}",
result_data_id=result.data_id,
)
if not ok:
raise RuntimeError(f"[{node.node_type}] async SUCCESS cannot map to state machine: node={node.node_id}")
self._record_action_result(node, result.status)
await self._apply_success_result(node, result.data_id, previous_data_id=previous_data_id)
node.error = None
logger.info(
f"[{handler.node_type}] polling成功: node={node.node_id}, push_id={task_id}, data_id={result.data_id or node.data_id}"
)
completed_nodes.append(node_id)
elif result.status == _TaskStatus.FAILED:
self._on_poll_result(handler, node, task_id, result)
ok = self._apply_sync_execute_state(
node,
handler_status=result.status,
detail=f"异步任务失败: {result.error}",
)
if not ok:
raise RuntimeError(f"[{node.node_type}] async FAILED cannot map to state machine: node={node.node_id}")
self._record_action_result(node, result.status)
node.error = result.error
completed_nodes.append(node_id)
# IN_PROGRESS 继续等待
# 移除已完成的任务
for node_id in completed_nodes:
async_tasks.pop(node_id, None)
# 还有未完成任务，等待后重试
if async_tasks:
retry_count += 1
# 超时未完成的任务标记为失败
if async_tasks:
for node_id in async_tasks:
node = self._collection.get(node_id)
if node:
timeout_msg = f"异步任务超时 (等待 {max_retries * poll_interval} 秒后仍未完成)"
ok = self._apply_sync_execute_state(
node,
handler_status=_TaskStatus.FAILED,
poll_timeout=True,
detail=timeout_msg,
)
if not ok:
raise RuntimeError(f"[{node.node_type}] poll timeout cannot map to state machine: node={node.node_id}")
node.error = f"Async task timeout after {max_retries * poll_interval} seconds"
self._record_action_result(node, _TaskStatus.FAILED)
# ========== 统计信息 ==========
def get_sync_summary(self, collection: "DataCollection") -> Dict[str, Any]:
"""
获取同步统计信息
Returns:
{
"total": 总节点数,
"success": 成功数,
"failed": 失败数,
"in_progress": 进行中,
"skipped": 跳过数,
"by_type": {...} # 按类型统计
}
"""
nodes = list(collection._nodes.values())
summary = {
"total": len(nodes),
"success": sum(1 for n in nodes if n.status == SyncStatus.SUCCESS),
"failed": sum(1 for n in nodes if n.status == SyncStatus.FAILED),
"in_progress": sum(1 for n in nodes if n.status == SyncStatus.IN_PROGRESS),
"skipped": sum(1 for n in nodes if n.status == SyncStatus.SKIPPED),
"by_type": {}
}
# 按类型统计
for node_type in self._handlers.keys():
type_nodes = collection.filter(node_type=node_type)
summary["by_type"][node_type] = {
"total": len(type_nodes),
"success": sum(1 for n in type_nodes if n.status == SyncStatus.SUCCESS),
"failed": sum(1 for n in type_nodes if n.status == SyncStatus.FAILED),
}
return summary
async def close(self) -> None:
"""关闭数据源资源（子类可覆盖以实现特定的清理逻辑）"""
pass # 基类默认空实现
sync_state_machine/datasource/handler.py
+579
View File
@@ -0,0 +1,579 @@
"""
节点处理器 V2 (Node Handler - Pure Execution Layer)
Handler 职责：
1. 执行后端 API 调用（load/create/update/delete）
2. 返回执行结果（TaskResult）
3. 提供异步任务轮询（poll）
4. 提供数据质量检查（validate）
Handler 不负责：
- ❌ 状态管理（由 DataSource 负责）
- ❌ ID 映射（由 Strategy 负责，已保存在 node.data）
- ❌ 依赖关系（由 Strategy 负责）
"""
from __future__ import annotations
from typing import (
TypeVar, Generic, Type, Protocol, Dict, List, Any, Optional,
runtime_checkable, TYPE_CHECKING
)
from abc import ABC, abstractmethod
from pydantic import BaseModel
from .task_result import TaskResult
if TYPE_CHECKING:
from ..common.sync_node import SyncNode
from ..common.collection import DataCollection
from .api.client import ApiClient
T = TypeVar("T", bound=BaseModel)
class ValidationResult:
"""数据质量检查结果"""
def __init__(self, is_valid: bool, errors: Optional[List[str]] = None):
self.is_valid = is_valid
self.errors = errors or []
@runtime_checkable
class NodeHandler(Protocol[T]):
"""
节点处理器协议（纯执行层）
Handler 只负责调用后端 API，不管理状态。
"""
@property
def node_type(self) -> str:
"""节点类型（如 'contract'）"""
...
@property
def node_class(self) -> Type["SyncNode[T]"]:
"""SyncNode 类型"""
...
@property
def schema(self) -> Type[T]:
"""Schema 类型"""
...
def set_collection(self, collection: "DataCollection") -> None:
"""
设置 Collection 引用
Args:
collection: 数据集合
"""
...
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""注入项目过滤列表。默认可忽略。"""
...
def set_api_client(self, client: "ApiClient") -> None:
"""注入 API 客户端。默认可忽略。"""
...
def set_poll_mode(self, mode: str) -> None:
"""设置轮询模式。默认可忽略。"""
...
def get_update_fields(self, *, exclude: frozenset[str] = frozenset({"id"})) -> List[str]:
"""返回 post-check 可比较字段列表。默认空。"""
...
async def load(self) -> List['SyncNode']:
"""
从后端加载全部数据并创建节点
注意：调用前需先 set_collection()
Returns:
SyncNode 列表
示例：
# Project Handler（无依赖）
async def load(self):
response = await self.api.get_projects()
nodes = []
for data in response["data"]:
node = self.create_node(data)
node.context["project_id"] = data["id"]
nodes.append(node)
return nodes
# Contract Handler（依赖 Project）
async def load(self):
# 使用 self._collection 查询依赖
projects = self._collection.filter(node_type="project")
project_ids = [p.data_id for p in projects if p.data_id]
nodes = []
for project_id in project_ids:
response = await self.api.get_contracts(project_id=project_id)
for data in response["data"]:
node = self.create_node(data)
node.context["project_id"] = project_id
node.context["contract_id"] = data["id"]
nodes.append(node)
return nodes
"""
...
async def create_all(
self,
nodes: List["SyncNode"]
) -> List[TaskResult]:
"""
批量创建节点
Args:
nodes: 需要创建的节点列表（DataSource 已筛选为 CREATE + PENDING）
Returns:
List[TaskResult]:
每个 TaskResult 必须包含 node_id，用于定位节点：
- TaskResult(status=SUCCESS, node_id="xxx", data_id="yyy")
- TaskResult(status=FAILED, node_id="xxx", error="...")
- TaskResult(status=IN_PROGRESS, node_id="xxx", task_id="...")
示例：
# JSONL Handler
async def create_all(self, nodes):
results = []
for node in nodes:
try:
new_id = str(uuid4())
data = node.get_data()
# ... 存储逻辑 ...
results.append(TaskResult.success(
node_id=node.node_id,
data_id=new_id
))
except Exception as e:
results.append(TaskResult.failed(
node_id=node.node_id,
error=str(e)
))
return results
# API Handler - 查询依赖
async def create_all(self, nodes):
results = []
for node in nodes:
# 查询依赖数据（如合同创建时查项目）
project_id = node.context.get("project_id")
projects = self._collection.filter(
node_type="project",
node_filter={"data_id": project_id}
)
if not projects:
results.append(TaskResult.failed(
node_id=node.node_id,
error=f"Project {project_id} not found"
))
continue
# 构造请求数据
data = node.get_data()
if not data:
results.append(TaskResult.failed(
node_id=node.node_id,
error="node data is None"
))
continue
contract_data = {
"project_id": projects[0].data_id,
**data
}
# 调用 API
response = await self.api.create_contract(contract_data)
results.append(TaskResult.success(
node_id=node.node_id,
data_id=response["id"]
))
return results
"""
...
async def update_all(
self,
nodes: List["SyncNode"]
) -> List[TaskResult]:
"""
批量更新节点
Args:
nodes: 需要更新的节点列表（DataSource 已筛选为 UPDATE + PENDING）
Returns:
List[TaskResult]（同 create_all）
"""
...
async def delete_all(
self,
nodes: List["SyncNode"]
) -> List[TaskResult]:
"""
批量删除节点
Args:
nodes: 需要删除的节点列表（DataSource 已筛选为 DELETE + PENDING）
Args:
collection: Collection 上下文
Returns:
List[TaskResult]（同 create_all）
"""
...
async def poll_tasks(self, task_ids: List[str]) -> Dict[str, TaskResult]:
"""
批量轮询异步任务状态
Args:
task_ids: 任务 ID 列表
Returns:
{task_id: TaskResult} 字典
- SUCCESS: 任务完成，返回 data_id
- FAILED: 任务失败，返回 error
- IN_PROGRESS: 仍在进行中
示例：
async def poll_tasks(self, task_ids):
response = await self.api.batch_query_tasks(task_ids)
results = {}
for task in response["tasks"]:
if task["status"] == "completed":
results[task["id"]] = TaskResult.success(
data_id=task["result"]["contract_id"]
)
elif task["status"] == "failed":
results[task["id"]] = TaskResult.failed(
error=task["error"]
)
else: # running
results[task["id"]] = TaskResult.in_progress(
task_id=task["id"]
)
return results
"""
...
async def validate(self, data: Dict[str, Any]) -> ValidationResult:
"""
数据质量检查
Args:
data: 原始数据
Returns:
ValidationResult（是否有效 + 错误列表）
示例：
async def validate(self, data):
errors = []
if not data.get("code"):
errors.append("合同编码为空")
if not data.get("project_id"):
errors.append("项目 ID 为空")
return ValidationResult(
is_valid=len(errors) == 0,
errors=errors
)
"""
...
def extract_id(self, data: Dict[str, Any]) -> str:
"""
从原始数据提取 ID
Args:
data: 原始数据
Returns:
业务主键
"""
...
def _create_node(self, data: Dict[str, Any]) -> "SyncNode[T]":
"""
从原始数据创建 SyncNode
Args:
data: 原始数据
Returns:
类型化的 SyncNode
"""
...
class BaseNodeHandler(ABC, Generic[T]):
"""
节点处理器基类
子类需实现：
- load()
- create()
- update()
- delete()（可选）
- poll_tasks()（如果支持异步）
- validate()（可选）
- extract_id()
"""
def __init__(
self,
node_type: str,
node_class: Type["SyncNode[T]"],
schema: Type[T]
):
self._node_type = node_type
self._node_class = node_class
self._schema = schema
self._collection: Optional["DataCollection"] = None
def set_collection(self, collection: "DataCollection") -> None:
"""
设置 Collection 引用
Args:
collection: 数据集合
"""
self._collection = collection
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""默认忽略项目过滤能力。"""
return
def set_api_client(self, client: "ApiClient") -> None:
"""默认忽略 API 客户端注入。"""
return
def set_poll_mode(self, mode: str) -> None:
"""默认忽略轮询模式。"""
return
def get_update_fields(self, *, exclude: frozenset[str] = frozenset({"id"})) -> List[str]:
"""默认无 post-check 字段白名单。"""
return []
@property
def node_type(self) -> str:
return self._node_type
@property
def node_class(self) -> Type["SyncNode[T]"]:
return self._node_class
@property
def schema(self) -> Type[T]:
return self._schema
# ========== 必须实现 ==========
@abstractmethod
async def load(self) -> List['SyncNode']:
"""
从数据源加载数据并创建节点
Handler 负责：
1. 加载原始数据
2. 创建 SyncNode（调用 create_node）
3. 设置 context（如 project_id, contract_id 等）
Returns:
SyncNode 列表
"""
pass
@abstractmethod
async def create_all(
self,
nodes: List["SyncNode"]
) -> List[TaskResult]:
"""批量创建节点"""
pass
@abstractmethod
async def update_all(
self,
nodes: List["SyncNode"]
) -> List[TaskResult]:
"""批量更新节点"""
pass
@abstractmethod
def extract_id(self, data: Dict[str, Any]) -> str:
"""提取 ID"""
pass
# ========== 可选实现 ==========
async def delete_all(
self,
nodes: List["SyncNode"]
) -> List[TaskResult]:
"""批量删除节点（默认不支持）"""
return [
TaskResult.failed(node_id=node.node_id, error="Delete operation not supported")
for node in nodes
]
async def poll_tasks(self, task_ids: List[str]) -> Dict[str, TaskResult]:
"""轮询异步任务（默认不支持）"""
return {
task_id: TaskResult.failed(error="Async tasks not supported")
for task_id in task_ids
}
async def validate(self, data: Dict[str, Any]) -> ValidationResult:
"""数据质量检查（默认不检查）"""
return ValidationResult(is_valid=True)
def create_node(self, data: Dict[str, Any]) -> "SyncNode[T]":
"""创建或复用 SyncNode（默认实现）"""
data_id = self.extract_id(data)
if self._collection is None:
raise RuntimeError("Collection not set. Call set_collection() before create_node().")
if data_id:
existing_node = self._collection.get_by_data_id(self.node_type, data_id)
if existing_node is not None:
existing_node.set_data(data.copy())
existing_node.set_origin_data(data.copy())
return existing_node
node_id = self._collection.generate_node_id()
return self.node_class(
node_id=node_id,
data_id=data_id,
data=data.copy()
)
def _create_node(self, data: Dict[str, Any]) -> "SyncNode[T]":
"""兼容 NodeHandler 协议的默认实现"""
return self.create_node(data)
class CompatibleNodeHandler(Generic[T]):
"""兼容旧式 Handler，实现扩展后的可选接口并委托给原对象。"""
def __init__(self, handler: NodeHandler[T]):
self._handler = handler
@property
def node_type(self) -> str:
return self._handler.node_type
@property
def node_class(self) -> Type["SyncNode[T]"]:
return self._handler.node_class
@property
def schema(self) -> Type[T]:
return self._handler.schema
def set_collection(self, collection: "DataCollection") -> None:
self._handler.set_collection(collection)
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
method = getattr(self._handler, "set_target_project_ids", None)
if callable(method):
method(target_project_ids)
def set_api_client(self, client: "ApiClient") -> None:
method = getattr(self._handler, "set_api_client", None)
if callable(method):
method(client)
def set_poll_mode(self, mode: str) -> None:
method = getattr(self._handler, "set_poll_mode", None)
if callable(method):
method(mode)
def get_update_fields(self, *, exclude: frozenset[str] = frozenset({"id"})) -> List[str]:
method = getattr(self._handler, "get_update_fields", None)
if callable(method):
return method(exclude=exclude)
return []
async def load(self) -> List["SyncNode"]:
return await self._handler.load()
async def create_all(self, nodes: List["SyncNode"]) -> List[TaskResult]:
return await self._handler.create_all(nodes)
async def update_all(self, nodes: List["SyncNode"]) -> List[TaskResult]:
return await self._handler.update_all(nodes)
async def delete_all(self, nodes: List["SyncNode"]) -> List[TaskResult]:
return await self._handler.delete_all(nodes)
async def poll_tasks(self, task_ids: List[str]) -> Dict[str, TaskResult]:
return await self._handler.poll_tasks(task_ids)
async def validate(self, data: Dict[str, Any]) -> ValidationResult:
return await self._handler.validate(data)
def extract_id(self, data: Dict[str, Any]) -> str:
return self._handler.extract_id(data)
def _create_node(self, data: Dict[str, Any]) -> "SyncNode[T]":
return self._handler._create_node(data)
def __getattr__(self, item: str) -> Any:
return getattr(self._handler, item)
def ensure_handler_compat(handler: NodeHandler[T]) -> NodeHandler[T]:
if isinstance(handler, BaseNodeHandler) or isinstance(handler, CompatibleNodeHandler):
return handler
return CompatibleNodeHandler(handler)
class NodeHandlerRegistry:
"""节点处理器注册表"""
_handlers: Dict[str, NodeHandler] = {}
@classmethod
def register(cls, node_type: str, handler: NodeHandler) -> None:
"""注册 handler"""
cls._handlers[node_type] = handler
@classmethod
def get(cls, node_type: str) -> NodeHandler:
"""获取 handler"""
if node_type not in cls._handlers:
raise KeyError(f"Handler for '{node_type}' not registered")
return cls._handlers[node_type]
@classmethod
def has(cls, node_type: str) -> bool:
"""检查是否已注册"""
return node_type in cls._handlers
@classmethod
def clear(cls) -> None:
"""清空注册表"""
cls._handlers.clear()
@classmethod
def list_handlers(cls) -> List[str]:
"""列出所有已注册的 node_type"""
return list(cls._handlers.keys())
sync_state_machine/datasource/jsonl/__init__.py
+13
View File
@@ -0,0 +1,13 @@
"""
JSONL DataSource 模块
提供基于 JSONL 文件的数据源实现，用于测试和本地开发。
"""
from .datasource import JsonlDataSource
from .handler import BaseJsonlHandler
__all__ = [
"JsonlDataSource",
"BaseJsonlHandler",
]
sync_state_machine/datasource/jsonl/datasource.py
+67
View File
@@ -0,0 +1,67 @@
"""
JsonlDataSource - JSONL 文件数据源
职责：
- 提供 JSONL 文件目录管理
- 实现 DataSource 接口（load_all, sync_all）
- 管理内存缓存和文件写入
- 统一异常处理和状态管理
"""
from __future__ import annotations
from pathlib import Path
from typing import Dict, Any, Set
from ..datasource import BaseDataSource
class JsonlDataSource(BaseDataSource):
"""
JSONL 文件数据源
提供文件路径和基础功能，具体的文件查找和读取由各个 Handler 负责。
Args:
dir_path: JSONL 文件目录
read_only: 是否只读模式
示例：
datasource = JsonlDataSource(Path("filtered_datasource/datasource/ecm-2025-12-02"))
datasource.register_handler(ContractJsonlHandler(datasource))
collection = DataCollection()
datasource.set_collection(collection)
await datasource.load_all(order=["project", "contract"])
await datasource.sync_all(order=["project", "contract"])
"""
def __init__(self, dir_path: Path, read_only: bool = False):
super().__init__()
self.dir_path = Path(dir_path)
self.read_only = read_only
# 用于 create/update/delete 的内存缓存
self._data: Dict[str, Dict[str, Dict[str, Any]]] = {} # {node_type: {id: data}}
self._dirty_types: Set[str] = set()
def _on_node_loaded(self, handler, node) -> None:
node.append_log(
f"JSONL链路[load] dir={self.dir_path.as_posix()} node_type={node.node_type} data_id={node.data_id or '<empty>'}"
)
def _on_in_progress_result(self, handler, node, result) -> None:
node.append_log(
f"JSONL链路[submit] action={node.action.value} data_id={node.data_id or '<empty>'}"
)
def _on_poll_result(self, handler, node, task_id: str, result) -> None:
node.append_log(
f"JSONL链路[poll] task_id={task_id} status={result.status.value}"
)
# load_all 和 sync_all 继承自 BaseDataSource
# 保存逻辑在 BaseJsonlHandler 中实现
sync_state_machine/datasource/jsonl/handler.py
+350
View File
@@ -0,0 +1,350 @@
"""
BaseJsonlHandler - JSONL Handler 基类
职责：
- 查找和读取 JSONL 文件
- 实现批量 Handler 接口（load, create_all, update_all, delete_all）
- 使用 datasource 的内存缓存
- 返回 List[TaskResult]（统一状态管理）
- 捕获所有异常并转换为 TaskResult.failed()
"""
from __future__ import annotations
import json
import re
from typing import Dict, List, Any, TYPE_CHECKING, Optional, Set
from uuid import uuid4
from ..handler import BaseNodeHandler
from ..task_result import TaskResult
if TYPE_CHECKING:
from ...common.sync_node import SyncNode
from .datasource import JsonlDataSource
class BaseJsonlHandler(BaseNodeHandler):
"""
JSONL Handler 基类
负责查找和读取对应的 JSONL 文件。
示例：
class ContractJsonlHandler(BaseJsonlHandler):
def __init__(self, datasource: JsonlDataSource):
super().__init__(
datasource=datasource,
node_type="contract",
node_class=ContractSyncNode,
schema=ContractResponse
)
"""
def __init__(
self,
datasource: JsonlDataSource,
node_type: str,
node_class: type,
schema: type
):
super().__init__(node_type, node_class, schema)
self.datasource = datasource
self._target_project_ids: Optional[Set[str]] = None
def set_target_project_ids(self, target_project_ids: List[str]) -> None:
"""注入项目过滤列表，供 load 阶段预过滤使用。"""
if target_project_ids:
self._target_project_ids = {str(pid) for pid in target_project_ids if str(pid)}
else:
self._target_project_ids = None
def _should_skip_item_by_project_filter(self, item: Dict[str, Any], item_id: str) -> bool:
if not self._target_project_ids:
return False
if self.node_type == "project":
return item_id not in self._target_project_ids
project_id = item.get("project_id")
if project_id:
return str(project_id) not in self._target_project_ids
return False
async def load(self) -> List['SyncNode']:
"""
查找并加载对应的 JSONL 文件并创建节点
文件命名规则：{node_type}_N.jsonl (N 为数字)
例如：contract_1.jsonl, contract_2.jsonl
注意：load 操作的异常由 DataSource 的 load_all 捕获处理
"""
nodes = []
if not self.datasource.dir_path.exists():
return nodes
# 查找所有匹配的文件：{node_type}_N.jsonl
pattern = re.compile(rf"^{re.escape(self.node_type)}_\d+\.jsonl$")
for file_path in self.datasource.dir_path.iterdir():
if not pattern.match(file_path.name):
continue
# 读取文件
with open(file_path, "r", encoding="utf-8") as f:
for line_num, line in enumerate(f, 1):
line = line.strip()
if not line:
continue
try:
item = json.loads(line)
item_id = self.extract_id(item)
if self._should_skip_item_by_project_filter(item, item_id):
continue
# 创建节点
node = self.create_node(item)
nodes.append(node)
# 同时加载到缓存
if item_id:
bucket = self.datasource._data.setdefault(self.node_type, {})
bucket[item_id] = item
except json.JSONDecodeError as e:
print(f"Warning: Invalid JSON in {file_path.name}:{line_num}: {e}")
# 继续处理下一行
return nodes
def extract_id(self, data: Dict[str, Any]) -> str:
"""
提取数据 ID
默认实现：优先使用 'id'，其次 '_id'
子类可重写此方法。
Args:
data: 数据字典
Returns:
数据 ID
"""
return str(data.get("id") or data.get("_id") or "")
async def create_all(
self,
nodes: List['SyncNode']
) -> List[TaskResult]:
"""
批量创建节点
Args:
nodes: 需要创建的节点列表（DataSource 已筛选为 CREATE + PENDING）
Returns:
List[TaskResult]，每个结果关联 node_id
"""
results = []
for node in nodes:
try:
# 生成新 ID（如果没有）
if not node.data_id:
new_id = str(uuid4())
else:
new_id = node.data_id
# 获取数据字典
# 使用 exclude_unset=False 以确保测试中的完整性校验（如 amount 字段）能通过
# 因为 JSONL 存储通常需要完整记录
data_dict = node.get_data(exclude_unset=False)
if data_dict is None:
results.append(TaskResult.failed(
node_id=node.node_id,
error="Cannot get data from node"
))
continue
# 设置 ID 和 type
data_dict["id"] = new_id
data_dict["_id"] = new_id
if "type" not in data_dict:
data_dict["type"] = self.node_type
# 存储到缓存
bucket = self.datasource._data.setdefault(self.node_type, {})
bucket[new_id] = data_dict
self.datasource._dirty_types.add(self.node_type)
results.append(TaskResult.success(
node_id=node.node_id,
data_id=new_id
))
except Exception as e:
results.append(TaskResult.failed(
node_id=node.node_id,
error=f"Create failed: {str(e)}"
))
# 自动保存到文件
await self._save_to_file()
return results
async def update_all(
self,
nodes: List['SyncNode']
) -> List[TaskResult]:
"""
批量更新节点
Args:
nodes: 需要更新的节点列表（DataSource 已筛选为 UPDATE + PENDING）
Returns:
List[TaskResult]，每个结果关联 node_id
"""
results = []
for node in nodes:
try:
if not node.data_id:
results.append(TaskResult.failed(
node_id=node.node_id,
error="Cannot update node without data_id"
))
continue
# 获取数据字典
# JSONL 更新通常也是完整记录替换，所以使用 exclude_unset=False
data_dict = node.get_data(exclude_unset=False)
if data_dict is None:
results.append(TaskResult.failed(
node_id=node.node_id,
error="Cannot get data from node"
))
continue
# 检查是否存在
bucket = self.datasource._data.get(self.node_type, {})
existing = bucket.get(node.data_id)
if existing is None:
results.append(TaskResult.failed(
node_id=node.node_id,
error=f"Item {node.data_id} not found"
))
continue
# 合并更新
existing.update(data_dict)
existing["id"] = node.data_id
existing["_id"] = node.data_id
if "type" not in existing:
existing["type"] = self.node_type
# 更新缓存
bucket[node.data_id] = existing
self.datasource._dirty_types.add(self.node_type)
results.append(TaskResult.success(
node_id=node.node_id,
data_id=node.data_id
))
except Exception as e:
results.append(TaskResult.failed(
node_id=node.node_id,
error=f"Update failed: {str(e)}"
))
# 自动保存到文件
await self._save_to_file()
return results
async def _save_to_file(self) -> None:
"""
将缓存数据写回 JSONL 文件
在批量操作（create_all/update_all/delete_all）后自动调用。
"""
if self.datasource.read_only:
return
if self.node_type not in self.datasource._dirty_types:
return
items = self.datasource._data.get(self.node_type, {})
if not items:
# 如果缓存为空，可能是删除了所有数据，仍需写入空文件
pass
try:
# 写入到 {node_type}_1.jsonl
file_path = self.datasource.dir_path / f"{self.node_type}_1.jsonl"
file_path.parent.mkdir(parents=True, exist_ok=True)
with open(file_path, "w", encoding="utf-8") as f:
for item in items.values():
f.write(json.dumps(item, ensure_ascii=False) + "\n")
# 清除该类型的 dirty 标记
self.datasource._dirty_types.discard(self.node_type)
except Exception as e:
print(f"Error saving {self.node_type}: {str(e)}")
# 不抛出异常，继续执行
async def delete_all(
self,
nodes: List['SyncNode']
) -> List[TaskResult]:
"""
批量删除节点
Args:
nodes: 需要删除的节点列表（DataSource 已筛选为 DELETE + PENDING）
Returns:
List[TaskResult]，每个结果关联 node_id
"""
results = []
for node in nodes:
try:
if not node.data_id:
results.append(TaskResult.failed(
node_id=node.node_id,
error="Cannot delete node without data_id"
))
continue
# 删除缓存
bucket = self.datasource._data.get(self.node_type, {})
if node.data_id in bucket:
del bucket[node.data_id]
self.datasource._dirty_types.add(self.node_type)
results.append(TaskResult.success(
node_id=node.node_id
))
else:
results.append(TaskResult.failed(
node_id=node.node_id,
error=f"Item {node.data_id} not found"
))
except Exception as e:
results.append(TaskResult.failed(
node_id=node.node_id,
error=f"Delete failed: {str(e)}"
))
# 自动保存到文件
await self._save_to_file()
return results
sync_state_machine/datasource/task_result.py
+122
View File
@@ -0,0 +1,122 @@
"""
TaskResult - Handler 执行结果
统一的任务执行结果，用于所有 DataSource 的 Handler。
"""
from enum import Enum
from typing import Dict, Any, Optional
class TaskStatus(Enum):
"""任务执行状态"""
SUCCESS = "success" # 同步完成，立即成功
FAILED = "failed" # 同步失败
IN_PROGRESS = "in_progress" # 异步任务进行中
SKIPPED = "skipped" # 跳过（无实际更新或不符合更新条件）
class TaskResult:
"""
Handler 执行结果
用于封装 create/update/delete 操作的返回值。
适用于所有 DataSource（API、JSONL 等）。
支持批量操作时通过 node_id 定位结果。
"""
def __init__(
self,
status: TaskStatus,
node_id: Optional[str] = None,
data_id: Optional[str] = None,
task_id: Optional[str] = None,
push_id: Optional[str] = None,
error: Optional[str] = None,
sync_log: Optional[str] = None,
metadata: Optional[Dict[str, Any]] = None
):
"""
Args:
status: 任务状态（SUCCESS/FAILED/IN_PROGRESS）
node_id: 关联的节点 ID（批量操作时用于定位节点）
data_id: 成功时返回的业务 ID（CREATE 操作必需）
task_id: 异步任务 ID（status=IN_PROGRESS 时必需）
push_id: 推送请求 ID（用于统一 push/poll 跟踪）
error: 失败原因（status=FAILED 时必需）
sync_log: Debug/Info级别日志（SKIPPED时使用）
metadata: 其他元数据
"""
normalized_metadata = dict(metadata or {})
if push_id is None:
raw_push_id = normalized_metadata.get("push_id")
if isinstance(raw_push_id, str) and raw_push_id:
push_id = raw_push_id
elif push_id:
normalized_metadata.setdefault("push_id", push_id)
self.status = status
self.node_id = node_id
self.data_id = data_id
self.task_id = task_id
self.push_id = push_id
self.error = error
self.sync_log = sync_log
self.metadata = normalized_metadata
@classmethod
def success(
cls,
node_id: Optional[str] = None,
data_id: Optional[str] = None,
push_id: Optional[str] = None,
**metadata,
) -> "TaskResult":
"""创建成功结果"""
return cls(TaskStatus.SUCCESS, node_id=node_id, data_id=data_id, push_id=push_id, metadata=metadata)
@classmethod
def failed(
cls,
node_id: Optional[str] = None,
error: str = "",
push_id: Optional[str] = None,
**metadata,
) -> "TaskResult":
"""创建失败结果"""
return cls(TaskStatus.FAILED, node_id=node_id, error=error, push_id=push_id, metadata=metadata)
@classmethod
def skipped(
cls,
node_id: Optional[str] = None,
reason: str = "",
push_id: Optional[str] = None,
**metadata,
) -> "TaskResult":
"""创建跳过结果（无实际更新等）"""
return cls(TaskStatus.SKIPPED, node_id=node_id, sync_log=reason, push_id=push_id, metadata=metadata)
@classmethod
def in_progress(
cls,
node_id: Optional[str] = None,
task_id: str = "",
push_id: Optional[str] = None,
**metadata,
) -> "TaskResult":
"""创建进行中结果"""
return cls(TaskStatus.IN_PROGRESS, node_id=node_id, task_id=task_id, push_id=push_id, metadata=metadata)
def __repr__(self) -> str:
parts = [f"status={self.status.value}"]
if self.node_id:
parts.append(f"node_id={self.node_id}")
if self.data_id:
parts.append(f"data_id={self.data_id}")
if self.task_id:
parts.append(f"task_id={self.task_id}")
if self.push_id:
parts.append(f"push_id={self.push_id}")
if self.error:
parts.append(f"error={self.error!r}")
return f"TaskResult({', '.join(parts)})"