优化加载后删除节点的逻辑;和数据库查询性能
This commit is contained in:
@@ -37,6 +37,10 @@
|
||||
| `ABNORMAL` | 数据损坏或核心绑定异常 | 否 |
|
||||
| `DEPENDENCY_ERROR` | 依赖不可满足 | 否 |
|
||||
|
||||
bootstrap 特殊语义:
|
||||
- 如果节点在持久化恢复后存在绑定,但 bootstrap datasource refresh 未返回最新数据,则节点进入 `S17` 语义并被 `bootstrap_blocked` 隔离。
|
||||
- 如果节点仅残留在 persistence 中、没有绑定,且 datasource refresh 也未返回,则节点按 stale zombie 清理,不再进入 bind。
|
||||
|
||||
## 3. create / update 语义
|
||||
|
||||
### create
|
||||
@@ -64,6 +68,11 @@
|
||||
- 状态落地:`Decision.to_state` 对应 YAML `states[Sxx]`。
|
||||
- 不变量:每次落地后执行 `runtime.check_invariants()`。
|
||||
|
||||
bootstrap 额外输入:
|
||||
- `is_create_zombie`
|
||||
- `bootstrap_source_present`
|
||||
- `has_binding_record`
|
||||
|
||||
## 6. 关键一致性约束
|
||||
|
||||
- `UNCHECKED` 不允许带动作(`INV-1_UNCHECKED_ACTION_NONE`)。
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
- `S12C/S12U/S12D`:按动作拆分的执行失败状态
|
||||
- `S13`:对侧创建等待态(`PEER_CREATING/NONE/PENDING`)
|
||||
- `S14`:UPDATE 运行时降级为 `NONE` 并跳过
|
||||
- `S17`:bootstrap 加载异常隔离(存在绑定,但 datasource refresh 未返回最新节点)
|
||||
|
||||
## 3. 事件到代码映射
|
||||
|
||||
@@ -51,6 +52,7 @@
|
||||
## 5. context 字段(以 YAML 为准)
|
||||
|
||||
### bind 相关
|
||||
- `bootstrap_source_present`
|
||||
- `has_binding_record`
|
||||
- `core_binding_result` / `peer_core_binding_result` (`NORMAL|WARNING|ABNORMAL`)
|
||||
- `data_present`
|
||||
@@ -75,6 +77,7 @@
|
||||
## 6. 语义边界
|
||||
- strategy 负责业务语义计算(如依赖是否满足、自动绑定结果、差异检测)。
|
||||
- 状态机只负责迁移判定与不变量校验,不直接访问数据源。
|
||||
- bootstrap 阶段的数据源刷新由 pipeline 完成,状态机只消费 `bootstrap_source_present / has_binding_record / is_create_zombie` 这些已计算上下文。
|
||||
|
||||
## 7. 不变量
|
||||
- `INV-1_UNCHECKED_ACTION_NONE`
|
||||
|
||||
+20
-7
@@ -13,23 +13,34 @@
|
||||
|
||||
### 2.1 加载时恢复
|
||||
- 在 `sync_state_machine/common/collection.py::load_from_persistence()` 执行。
|
||||
- 语义:仅恢复持久化状态(不在此阶段做状态归并)。
|
||||
- 语义:仅恢复持久化状态,并记录本轮 `sync_log` 的 `persistence` 恢复来源;不在此阶段做状态归并。
|
||||
- 恢复后保留 `data/origin_data`,等待 bootstrap 阶段的 datasource 刷新覆盖最新数据。
|
||||
|
||||
### 2.2 加载后 cleanup
|
||||
- pipeline 直接调用 `run_phase2_cleanup()`。
|
||||
### 2.2 bootstrap datasource refresh
|
||||
- pipeline 在持久化恢复后、`E01` 之前,按已恢复到内存的 `node_type` 做一次 datasource 刷新。
|
||||
- 实际实现:`sync_state_machine/pipeline/full_sync_pipeline.py::_bootstrap_refresh_latest_data()`。
|
||||
- 语义:
|
||||
- 如果 datasource 命中节点,则用最新数据覆盖持久化快照,并清除 `_loaded_from_persistence` 标记。
|
||||
- 如果 datasource 未命中节点,则节点保留持久化快照,供 `E01` 判断它是 stale zombie 还是 bootstrap 异常隔离对象。
|
||||
|
||||
### 2.3 加载后 cleanup
|
||||
- pipeline 在 datasource refresh 完成后调用 `run_phase2_cleanup()`。
|
||||
- 实际实现:`sync_state_machine/sync_system/strategy_ops/reset_ops.py::run_phase2_cleanup()`。
|
||||
- 核心事件:`E01`(`S16 -> S00/S15`)。
|
||||
- `S16` 表示“持久化加载输入态”:本轮从持久化恢复出的节点集合(可包含上轮遗留节点,也可包含上轮已稳定节点)。
|
||||
- `S16` 表示“bootstrap 输入态”:持久化恢复出的节点,叠加 datasource refresh 后仍待 `E01` 判定的节点集合。
|
||||
- 判定:
|
||||
- `is_create_zombie=true` -> `S15`,节点删除(并尝试解绑);
|
||||
- `is_create_zombie=false` -> `S00`,节点归并为 `UNCHECKED/NONE/PENDING` 并清空 `error`。
|
||||
- `bootstrap_source_present=false && has_binding_record=false` -> `S15`,说明该节点仅残留在 persistence 中且未绑定,按 stale zombie 清理;
|
||||
- `bootstrap_source_present=false && has_binding_record=true` -> `S17`,说明该节点存在绑定,但 datasource refresh 未返回,进入 bootstrap 加载异常隔离;
|
||||
- 其余情况 -> `S00`,节点归并为 `UNCHECKED/NONE/PENDING` 并清空 `error`。
|
||||
|
||||
### 2.2.1 `enable_persistence` 与 bootstrap 关系
|
||||
- `enable_persistence=true`:先 `load_from_persistence()`,再对加载出的节点执行 `E01`(`S16` 域内处理)。
|
||||
- `enable_persistence=true`:先 `load_from_persistence()`,再做 bootstrap datasource refresh,最后对输入节点执行 `E01`(`S16` 域内处理)。
|
||||
- `enable_persistence=false`:本轮没有持久化输入,`S16/E01` 不参与;随后 Stage1 新加载节点直接进入本轮 bind/create/update/sync 主流程。
|
||||
|
||||
### 2.3 加载异常隔离(S17)
|
||||
### 2.4 加载异常隔离(S17)
|
||||
- 持久化中若出现非法枚举值(`action/status/binding_status`),节点标记为 `bootstrap_blocked`(`S17` 语义)。
|
||||
- 持久化恢复后若节点存在绑定,但 bootstrap datasource refresh 未返回该节点,也会进入 `S17` 语义并隔离。
|
||||
- 这类节点保留 `error`,用于人工排查。
|
||||
- 这类节点不会进入 `E01`,也不会参与 bind/create/update/sync 自动流程。
|
||||
|
||||
@@ -51,6 +62,8 @@
|
||||
- 有绑定记录:根据本端/对端有效性进入 `S01/S03/S04`。
|
||||
- 无绑定记录:进入 `S02`。
|
||||
|
||||
注意:如果节点在 bootstrap 阶段已被隔离为 `S17` 语义(`bootstrap_blocked=true`),默认不会进入 bind 工作集。
|
||||
|
||||
### 依赖检查
|
||||
- `phase_dependency_check()` -> `check_dependencies_for_nodes()` 内触发 `E15`。
|
||||
- `data_present=true` 且 `dep_satisfied=true` -> 不迁移,节点保持在 `S02`,继续进入自动绑定判定。
|
||||
|
||||
@@ -488,6 +488,7 @@ class DataCollection:
|
||||
context=context_value,
|
||||
)
|
||||
node.context["_loaded_from_persistence"] = True
|
||||
node.append_log("加载来源: persistence 恢复节点")
|
||||
except Exception:
|
||||
import logging
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -623,6 +624,7 @@ class DataCollection:
|
||||
context=context_value,
|
||||
)
|
||||
node.context["_loaded_from_persistence"] = True
|
||||
node.append_log("加载来源: persistence 恢复节点")
|
||||
self._nodes[node.node_id] = node
|
||||
|
||||
if node.data_id and node.data_id != "":
|
||||
|
||||
@@ -112,6 +112,9 @@ class SQLitePersistenceBackend(PersistenceBackend):
|
||||
await self.conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_nodes_collection_scope_data_id ON nodes (collection_id, scope, data_id)"
|
||||
)
|
||||
await self.conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_nodes_scope_type_node_id ON nodes (scope, node_type, node_id)"
|
||||
)
|
||||
|
||||
async def _ensure_bindings_schema(self) -> None:
|
||||
assert self.conn is not None
|
||||
|
||||
@@ -17,6 +17,7 @@ meta:
|
||||
# represented as separate invocations with different inputs.
|
||||
context:
|
||||
is_create_zombie: { type: bool, desc: "初始化时判定为 CREATE 僵尸:action=CREATE 且 (status=FAILED 或 data_id 为空)" }
|
||||
bootstrap_source_present: { type: bool, desc: "bootstrap 阶段 datasource 刷新后,当前节点是否在最新数据源中命中" }
|
||||
|
||||
has_binding_record: { type: bool, desc: "binding_manager 里存在绑定记录 (remote_id/local_id 非空)" }
|
||||
core_binding_result: { type: enum, values: [NORMAL, WARNING, ABNORMAL], desc: "SyncDecisionEngine.determine_core_status() 给本节点的结果(仅在 has_binding_record=true 时有效)" }
|
||||
@@ -304,19 +305,30 @@ states:
|
||||
status: SKIPPED
|
||||
data_id_exist: true
|
||||
|
||||
S17:
|
||||
binding_status: ABNORMAL
|
||||
action: NONE
|
||||
status: FAILED
|
||||
data_id_exist: [false, true]
|
||||
desc: |
|
||||
bootstrap 加载异常隔离态
|
||||
持久化恢复后存在绑定,但 datasource 刷新未命中最新数据
|
||||
binding_status: ABNORMAL
|
||||
action: NONE
|
||||
status: FAILED
|
||||
data_id_exist: false | true
|
||||
|
||||
pseudo_states:
|
||||
S15:
|
||||
desc: "节点被清理删除(zombie cleanup)"
|
||||
S16:
|
||||
desc: "持久化加载输入态:本轮 E01 的输入节点集合(可包含上轮遗留与已稳定节点)"
|
||||
S17:
|
||||
desc: "加载异常隔离态:持久化枚举非法,保留错误供人工处理,不进入自动流程"
|
||||
|
||||
stages:
|
||||
bootstrap:
|
||||
desc: "启动/初始化(仅针对持久化加载出的节点):僵尸清理与归并到 S00"
|
||||
desc: "启动/初始化:持久化恢复后先做 datasource 刷新,再按 E01 清理僵尸或归并/隔离"
|
||||
entry_states: [S16]
|
||||
exit_states: [S00, S15]
|
||||
exit_states: [S00, S15, S17]
|
||||
|
||||
bind:
|
||||
desc: "绑定判定:core matrix +(可选)依赖检查 +(可选)自动绑定;stage 仅约束边界,不要求内部顺序"
|
||||
@@ -346,13 +358,23 @@ stages:
|
||||
transitions:
|
||||
E01:
|
||||
stage: bootstrap
|
||||
desc: "初始化:删CREATE僵尸;其余重置后进入起点"
|
||||
desc: "初始化:先依据 datasource 刷新结果判定 source 是否存在,再做僵尸清理/隔离/归并"
|
||||
from: [S16]
|
||||
outcomes:
|
||||
- when: { is_create_zombie: true }
|
||||
to: S15
|
||||
- when: { is_create_zombie: false }
|
||||
- when: { is_create_zombie: false, bootstrap_source_present: false, has_binding_record: false }
|
||||
to: S15
|
||||
emit:
|
||||
note: "持久化孤儿节点未命中最新数据源,按僵尸清理"
|
||||
- when: { is_create_zombie: false, bootstrap_source_present: false, has_binding_record: true }
|
||||
to: S17
|
||||
emit:
|
||||
note: "存在绑定但最新数据源未返回节点,进入加载异常隔离"
|
||||
- when: { is_create_zombie: false, bootstrap_source_present: true }
|
||||
to: S00
|
||||
emit:
|
||||
note: "bootstrap 已刷新到最新数据,归并到起点"
|
||||
|
||||
# --- Bind phase1 core ---
|
||||
E10:
|
||||
|
||||
@@ -101,7 +101,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
else:
|
||||
results.append(TaskResult.in_progress(node_id=node.node_id, task_id=push_id))
|
||||
except Exception as e:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e), push_id=push_id))
|
||||
|
||||
return results
|
||||
|
||||
@@ -135,6 +135,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
|
||||
node_updated = False
|
||||
error = None
|
||||
push_failure_id = None
|
||||
schema_status_lines: List[str] = []
|
||||
|
||||
# 1. 基础信息更新 (ConstructionUpdate)
|
||||
@@ -186,6 +187,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
except Exception as e:
|
||||
error = str(e)
|
||||
base_update_error = str(e)
|
||||
push_failure_id = push_id
|
||||
else:
|
||||
base_update_error = "show_name changed but task_type != QT"
|
||||
|
||||
@@ -236,6 +238,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
except Exception as e:
|
||||
error = str(e)
|
||||
plan_update_error = str(e)
|
||||
push_failure_id = push_id
|
||||
|
||||
schema_status_lines.append(
|
||||
self.build_update_schema_status(
|
||||
@@ -253,7 +256,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
|
||||
# 汇总结果
|
||||
if error:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=error))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=error, push_id=push_failure_id))
|
||||
elif node_updated:
|
||||
results.append(TaskResult.success(node_id=node.node_id))
|
||||
else:
|
||||
@@ -279,7 +282,7 @@ class ConstructionTaskApiHandler(BaseApiHandler):
|
||||
await api_delete_construction_task(self.api_client, biz_id, push_id)
|
||||
results.append(TaskResult.success(node_id=node.node_id))
|
||||
except Exception as e:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e), push_id=push_id))
|
||||
|
||||
return results
|
||||
|
||||
|
||||
@@ -130,7 +130,7 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
|
||||
else:
|
||||
results.append(TaskResult.in_progress(node_id=node.node_id, task_id=push_id))
|
||||
except Exception as e:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e), push_id=push_id))
|
||||
|
||||
return results
|
||||
|
||||
@@ -222,7 +222,7 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
|
||||
# 整组失败
|
||||
error_msg = str(e)
|
||||
for node, _ in group_items:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=error_msg))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=error_msg, push_id=push_id))
|
||||
|
||||
return results
|
||||
|
||||
@@ -240,7 +240,7 @@ class ConstructionTaskDetailApiHandler(BaseApiHandler):
|
||||
await api_delete_construction_task_detail(self.api_client, biz_id, push_id)
|
||||
results.append(TaskResult.success(node_id=node.node_id))
|
||||
except Exception as e:
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e)))
|
||||
results.append(TaskResult.failed(node_id=node.node_id, error=str(e), push_id=push_id))
|
||||
|
||||
return results
|
||||
|
||||
|
||||
@@ -276,8 +276,14 @@ def e01_bootstrap(
|
||||
*,
|
||||
node: "SyncNode",
|
||||
is_create_zombie: bool,
|
||||
bootstrap_source_present: bool = True,
|
||||
has_binding_record: bool = False,
|
||||
) -> Decision:
|
||||
context = {"is_create_zombie": is_create_zombie}
|
||||
context = {
|
||||
"is_create_zombie": is_create_zombie,
|
||||
"bootstrap_source_present": bootstrap_source_present,
|
||||
"has_binding_record": has_binding_record,
|
||||
}
|
||||
decision = _dispatch_required(
|
||||
runtime,
|
||||
node=node,
|
||||
@@ -289,6 +295,15 @@ def e01_bootstrap(
|
||||
_apply_decision(runtime, node=node, decision=decision, fields={"binding_status", "action", "status"})
|
||||
node.error = None
|
||||
return decision
|
||||
if decision.to_state == "S17":
|
||||
_apply_decision(runtime, node=node, decision=decision, fields={"binding_status", "action", "status"})
|
||||
reason_text = decision_note(decision) or "<no-reason>"
|
||||
node.context["bootstrap_blocked"] = True
|
||||
node.context["bootstrap_block_reason"] = "source_missing_after_persistence_refresh"
|
||||
node.context["bootstrap_block_errors"] = [reason_text]
|
||||
node.context["bootstrap_source_present"] = False
|
||||
node.error = f"BOOTSTRAP_SOURCE_MISSING: {reason_text}"
|
||||
return decision
|
||||
if decision.to_state == "S15":
|
||||
reason_text = decision_note(decision) or "<no-reason>"
|
||||
node.append_log(
|
||||
|
||||
@@ -256,6 +256,12 @@ class FullSyncPipeline:
|
||||
self._prepare_datasources()
|
||||
self._logger.info("✅ Bootstrap: datasource prepared")
|
||||
|
||||
await self._bootstrap_refresh_latest_data()
|
||||
self._logger.info("✅ Bootstrap: datasource refresh completed")
|
||||
|
||||
await self._run_bootstrap_reset_cleanup()
|
||||
self._logger.info("✅ Bootstrap: reset cleanup completed")
|
||||
|
||||
await self._apply_project_scope_cleanup()
|
||||
|
||||
async def phase_bind(self) -> None:
|
||||
@@ -359,7 +365,7 @@ class FullSyncPipeline:
|
||||
await _safe_close("persistence", self.persistence.close())
|
||||
|
||||
async def _load_persistence_state(self) -> None:
|
||||
"""加载持久化数据并清理僵尸节点"""
|
||||
"""加载持久化数据到内存,等待 bootstrap datasource refresh 与 E01 cleanup。"""
|
||||
if not self.enable_persistence:
|
||||
return
|
||||
|
||||
@@ -376,10 +382,50 @@ class FullSyncPipeline:
|
||||
for node_type in binding_node_types:
|
||||
await self.binding_manager.load_from_persistence(node_type)
|
||||
self._logger.info("🔄 Persistence loaded (state preserved; pending E01 normalization)")
|
||||
|
||||
# 加载后 reset 清理:清理 CREATE 失败的僵尸绑定
|
||||
|
||||
def _resolve_bootstrap_refresh_node_types(self) -> List[str]:
|
||||
local_types = {node.node_type for node in self.local_collection.filter()}
|
||||
remote_types = {node.node_type for node in self.remote_collection.filter()}
|
||||
candidate_types = local_types | remote_types
|
||||
if not candidate_types:
|
||||
return []
|
||||
|
||||
ordered: List[str] = []
|
||||
for node_type in [*self.load_order, *self.bootstrap_binding_node_types, *sorted(candidate_types)]:
|
||||
if node_type in candidate_types and node_type not in ordered:
|
||||
ordered.append(node_type)
|
||||
return ordered
|
||||
|
||||
async def _bootstrap_refresh_latest_data(self) -> None:
|
||||
if not self.enable_persistence:
|
||||
return
|
||||
|
||||
node_types = self._resolve_bootstrap_refresh_node_types()
|
||||
if not node_types:
|
||||
self._logger.info("⏭️ Bootstrap datasource refresh skipped: no persisted node types")
|
||||
return
|
||||
|
||||
for node_type in node_types:
|
||||
self._logger.info("🔄 Bootstrap refresh node_type=%s from datasource", node_type)
|
||||
await self._load_node_type_from_datasource(
|
||||
self.local_datasource,
|
||||
datasource_role="local",
|
||||
node_type=node_type,
|
||||
reason="bootstrap persistence refresh",
|
||||
)
|
||||
await self._load_node_type_from_datasource(
|
||||
self.remote_datasource,
|
||||
datasource_role="remote",
|
||||
node_type=node_type,
|
||||
reason="bootstrap persistence refresh",
|
||||
)
|
||||
|
||||
async def _run_bootstrap_reset_cleanup(self) -> None:
|
||||
if not self.enable_persistence:
|
||||
return
|
||||
if not self.strategies:
|
||||
raise RuntimeError("no strategies configured: cannot run bootstrap event E01")
|
||||
|
||||
total_cleaned = await run_phase2_cleanup(
|
||||
node_types=self.node_types,
|
||||
local_collection=self.local_collection,
|
||||
@@ -388,8 +434,7 @@ class FullSyncPipeline:
|
||||
runtime=self.sm_runtime,
|
||||
)
|
||||
if total_cleaned > 0:
|
||||
self._logger.info(f"🧹 Reset cleanup: {total_cleaned} CREATE-failed zombies cleaned")
|
||||
self._logger.info("🔄 Post-load reset cleanup completed")
|
||||
self._logger.info(f"🧹 Reset cleanup: {total_cleaned} bootstrap zombies cleaned")
|
||||
|
||||
async def _apply_preloaded_state(self) -> None:
|
||||
if self._preloaded_local_nodes:
|
||||
|
||||
@@ -30,6 +30,25 @@ async def _resolve_local_id_for_unbind(
|
||||
return None
|
||||
|
||||
|
||||
async def _has_binding_record(
|
||||
*,
|
||||
node_type: str,
|
||||
any_side_node_id: str,
|
||||
binding_manager: "BindingManager",
|
||||
) -> bool:
|
||||
local_id = await _resolve_local_id_for_unbind(
|
||||
node_type=node_type,
|
||||
any_side_node_id=any_side_node_id,
|
||||
binding_manager=binding_manager,
|
||||
)
|
||||
return local_id is not None
|
||||
|
||||
|
||||
def _bootstrap_source_present(node) -> bool:
|
||||
context = node.context if isinstance(node.context, dict) else {}
|
||||
return not bool(context.get("_loaded_from_persistence"))
|
||||
|
||||
|
||||
def _is_create_zombie(node) -> bool:
|
||||
if node.action != SyncAction.CREATE:
|
||||
return False
|
||||
@@ -55,8 +74,27 @@ async def _cleanup_one_side(
|
||||
)
|
||||
continue
|
||||
|
||||
decision = e01_bootstrap(runtime, node=node, is_create_zombie=_is_create_zombie(node))
|
||||
has_binding_record = await _has_binding_record(
|
||||
node_type=node_type,
|
||||
any_side_node_id=node.node_id,
|
||||
binding_manager=binding_manager,
|
||||
)
|
||||
bootstrap_source_present = _bootstrap_source_present(node)
|
||||
|
||||
decision = e01_bootstrap(
|
||||
runtime,
|
||||
node=node,
|
||||
is_create_zombie=_is_create_zombie(node),
|
||||
bootstrap_source_present=bootstrap_source_present,
|
||||
has_binding_record=has_binding_record,
|
||||
)
|
||||
if decision.to_state != "S15":
|
||||
if decision.to_state == "S17":
|
||||
node.context.pop("_loaded_from_persistence", None)
|
||||
logger.warning(
|
||||
f"[{node_type}] Bootstrap isolated node ({label}): "
|
||||
f"node_id={node.node_id}, reason={node.error}"
|
||||
)
|
||||
continue
|
||||
|
||||
local_id_for_unbind = await _resolve_local_id_for_unbind(
|
||||
@@ -74,9 +112,12 @@ async def _cleanup_one_side(
|
||||
else:
|
||||
bind_note = "no_binding"
|
||||
|
||||
reason = "status=FAILED" if node.status == SyncStatus.FAILED else "data_id empty"
|
||||
if _is_create_zombie(node):
|
||||
reason = "status=FAILED" if node.status == SyncStatus.FAILED else "data_id empty"
|
||||
else:
|
||||
reason = "persistence_only_source_missing_without_binding"
|
||||
logger.info(
|
||||
f"[{node_type}] Cleaning up CREATE zombie ({label}): "
|
||||
f"[{node_type}] Cleaning up bootstrap zombie ({label}): "
|
||||
f"node_id={node.node_id}, reason={reason}, {bind_note}"
|
||||
)
|
||||
|
||||
@@ -118,7 +159,7 @@ async def run_phase2_cleanup(
|
||||
total_cleaned += cleaned_count
|
||||
if cleaned_count > 0:
|
||||
logger.warning(
|
||||
f"[{node_type}] Cleaned up CREATE-failed zombies: "
|
||||
f"[{node_type}] Cleaned up bootstrap zombies: "
|
||||
f"local_deleted={len(cleaned_local)}, remote_deleted={len(cleaned_remote)}"
|
||||
)
|
||||
|
||||
|
||||
@@ -128,6 +128,16 @@ async def test_mock_based_full_event_and_state_coverage() -> None:
|
||||
d = e01_bootstrap(runtime, node=node_e01_del, is_create_zombie=True)
|
||||
record(node_e01_del, d)
|
||||
|
||||
node_e01_isolated = _new_node("e01_isolated", action=SyncAction.NONE, status=SyncStatus.PENDING)
|
||||
d = e01_bootstrap(
|
||||
runtime,
|
||||
node=node_e01_isolated,
|
||||
is_create_zombie=False,
|
||||
bootstrap_source_present=False,
|
||||
has_binding_record=True,
|
||||
)
|
||||
record(node_e01_isolated, d)
|
||||
|
||||
# E10 merged outcomes
|
||||
for idx, (has_binding, local_res, peer_res) in enumerate(
|
||||
[
|
||||
@@ -507,7 +517,8 @@ async def test_sync_log_cleared_on_initial_load(tmp_path: Path) -> None:
|
||||
await reloaded.load_from_persistence()
|
||||
loaded_node = reloaded.get("log-node")
|
||||
assert loaded_node is not None
|
||||
assert loaded_node.sync_log is None
|
||||
assert loaded_node.sync_log is not None
|
||||
assert loaded_node.sync_log.count("加载来源: persistence 恢复节点") == 1
|
||||
|
||||
in_memory_after_reload_report = REPORT_DIR / "in_memory_sync_log_after_reload.txt"
|
||||
in_memory_after_reload_report.write_text(
|
||||
@@ -566,6 +577,8 @@ async def test_load_preserves_state_then_e01_normalizes_to_s00(tmp_path: Path) -
|
||||
assert loaded_node.status == SyncStatus.FAILED
|
||||
assert loaded_node.binding_status == BindingStatus.WARNING
|
||||
assert loaded_node.error == "last run failed"
|
||||
assert loaded_node.sync_log is not None
|
||||
assert "加载来源: persistence 恢复节点" in loaded_node.sync_log
|
||||
|
||||
# E01 后归并到 S00,并清空历史 error
|
||||
runtime = StateMachineRuntime(StateMachineConfig.load(CFG_PATH))
|
||||
@@ -579,6 +592,114 @@ async def test_load_preserves_state_then_e01_normalizes_to_s00(tmp_path: Path) -
|
||||
await persistence.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_persistence_only_unbound_node_is_cleaned_after_bootstrap_refresh_miss(tmp_path: Path) -> None:
|
||||
db = tmp_path / "bootstrap_source_miss_unbound.db"
|
||||
|
||||
persistence = SQLitePersistenceBackend(str(db))
|
||||
await persistence.initialize()
|
||||
|
||||
collection = DataCollection("project-bootstrap-unbound", persistence=persistence, auto_persist=False)
|
||||
node = ProjectSyncNode(
|
||||
node_id="stale-unbound-node",
|
||||
data_id="proj-stale-unbound",
|
||||
data={
|
||||
"id": "proj-stale-unbound",
|
||||
"type": "project",
|
||||
"name": "Stale Unbound Project",
|
||||
"code": "P-ST-UNBOUND",
|
||||
"status": "active",
|
||||
"progress_type": "C",
|
||||
"project_type": "onshore_wind",
|
||||
},
|
||||
action=SyncAction.NONE,
|
||||
status=SyncStatus.PENDING,
|
||||
binding_status=BindingStatus.WARNING,
|
||||
)
|
||||
await collection.add(node)
|
||||
await collection.persist()
|
||||
|
||||
reloaded = DataCollection("project-bootstrap-unbound", persistence=persistence, auto_persist=False)
|
||||
await reloaded.load_from_persistence()
|
||||
loaded_node = reloaded.get("stale-unbound-node")
|
||||
assert loaded_node is not None
|
||||
assert loaded_node.context.get("_loaded_from_persistence") is True
|
||||
|
||||
runtime = StateMachineRuntime(StateMachineConfig.load(CFG_PATH))
|
||||
binding_manager = BindingManager(persistence, auto_persist=False)
|
||||
remote_collection = DataCollection("empty-remote-unbound", persistence=persistence, auto_persist=False)
|
||||
cleaned = await run_phase2_cleanup(
|
||||
node_types=["project"],
|
||||
local_collection=reloaded,
|
||||
remote_collection=remote_collection,
|
||||
binding_manager=binding_manager,
|
||||
runtime=runtime,
|
||||
)
|
||||
|
||||
assert cleaned == 1
|
||||
assert reloaded.get("stale-unbound-node") is None
|
||||
|
||||
await persistence.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bound_node_is_isolated_after_bootstrap_refresh_miss(tmp_path: Path) -> None:
|
||||
db = tmp_path / "bootstrap_source_miss_bound.db"
|
||||
|
||||
persistence = SQLitePersistenceBackend(str(db))
|
||||
await persistence.initialize()
|
||||
|
||||
collection = DataCollection("project-bootstrap-bound", persistence=persistence, auto_persist=False)
|
||||
node = ProjectSyncNode(
|
||||
node_id="stale-bound-local-node",
|
||||
data_id="proj-stale-bound",
|
||||
data={
|
||||
"id": "proj-stale-bound",
|
||||
"type": "project",
|
||||
"name": "Stale Bound Project",
|
||||
"code": "P-ST-BOUND",
|
||||
"status": "active",
|
||||
"progress_type": "C",
|
||||
"project_type": "onshore_wind",
|
||||
},
|
||||
action=SyncAction.NONE,
|
||||
status=SyncStatus.PENDING,
|
||||
binding_status=BindingStatus.WARNING,
|
||||
)
|
||||
await collection.add(node)
|
||||
await collection.persist()
|
||||
|
||||
reloaded = DataCollection("project-bootstrap-bound", persistence=persistence, auto_persist=False)
|
||||
await reloaded.load_from_persistence()
|
||||
loaded_node = reloaded.get("stale-bound-local-node")
|
||||
assert loaded_node is not None
|
||||
assert loaded_node.context.get("_loaded_from_persistence") is True
|
||||
|
||||
runtime = StateMachineRuntime(StateMachineConfig.load(CFG_PATH))
|
||||
binding_manager = BindingManager(persistence, auto_persist=False)
|
||||
await binding_manager.bind("project", "stale-bound-local-node", "remote-missing-node")
|
||||
remote_collection = DataCollection("empty-remote-bound", persistence=persistence, auto_persist=False)
|
||||
cleaned = await run_phase2_cleanup(
|
||||
node_types=["project"],
|
||||
local_collection=reloaded,
|
||||
remote_collection=remote_collection,
|
||||
binding_manager=binding_manager,
|
||||
runtime=runtime,
|
||||
)
|
||||
|
||||
isolated = reloaded.get("stale-bound-local-node")
|
||||
assert cleaned == 0
|
||||
assert isolated is not None
|
||||
assert reloaded.is_bootstrap_blocked(isolated)
|
||||
assert isolated.binding_status == BindingStatus.ABNORMAL
|
||||
assert isolated.action == SyncAction.NONE
|
||||
assert isolated.status == SyncStatus.FAILED
|
||||
assert "BOOTSTRAP_SOURCE_MISSING" in (isolated.error or "")
|
||||
assert isolated.context.get("bootstrap_block_reason") == "source_missing_after_persistence_refresh"
|
||||
|
||||
await persistence.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_invalid_persisted_enum_is_bootstrap_blocked_and_skipped_in_reset(tmp_path: Path) -> None:
|
||||
REPORT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@@ -146,3 +146,39 @@ async def test_api_datasource_failed_update_does_not_use_unrelated_op_trace() ->
|
||||
assert updated.sync_log is not None
|
||||
assert "API链路[submit-failed] PUT /material/plan" not in updated.sync_log
|
||||
assert "未找到匹配 biz_id 的请求记录" in updated.sync_log
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_api_datasource_failed_create_uses_push_id_trace_when_data_id_empty() -> None:
|
||||
collection = DataCollection("remote")
|
||||
ds = ApiDataSource(base_url="http://example.com", uid="u", secret="s")
|
||||
ds.set_collection(collection)
|
||||
|
||||
node = _build_node(node_id="n-create-fail", data_id="", name="X")
|
||||
with node.allow_core_state_write("test"):
|
||||
node.binding_status = BindingStatus.NORMAL
|
||||
node.action = SyncAction.CREATE
|
||||
node.status = SyncStatus.IN_PROGRESS
|
||||
await collection.add(node)
|
||||
|
||||
ds.client._record_trace(
|
||||
method="POST",
|
||||
endpoint="/construction/create",
|
||||
url="http://example.com/construction/create",
|
||||
request_params=None,
|
||||
request_payload={"push_id": "push-create-fail", "data": {"task_type": "jxfx"}},
|
||||
response={"code": 400, "message": "施工类型错误"},
|
||||
elapsed_ms=12,
|
||||
)
|
||||
|
||||
await ds._handle_task_result(
|
||||
cast(Any, _Handler()),
|
||||
TaskResult.failed(node_id=node.node_id, error="Create failed: 施工类型错误", push_id="push-create-fail"),
|
||||
{},
|
||||
)
|
||||
|
||||
updated = collection.get(node.node_id)
|
||||
assert updated is not None
|
||||
assert updated.sync_log is not None
|
||||
assert "API链路[submit-failed] POST /construction/create" in updated.sync_log
|
||||
assert "response={\"code\":400,\"message\":\"施工类型错误\"}" in updated.sync_log
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from sync_state_machine.domain.construction.api_handler import ConstructionTaskApiHandler
|
||||
from sync_state_machine.domain.construction.sync_node import ConstructionTaskSyncNode
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_construction_create_failure_preserves_push_id(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
handler = ConstructionTaskApiHandler()
|
||||
handler.api_client = object()
|
||||
|
||||
async def _fake_create(client, data, push_id: str, *, steps=None):
|
||||
del client, data, steps
|
||||
raise Exception(f"API error for {push_id}")
|
||||
|
||||
monkeypatch.setattr(
|
||||
"sync_state_machine.domain.construction.api_handler.api_create_construction_task",
|
||||
_fake_create,
|
||||
)
|
||||
|
||||
node = ConstructionTaskSyncNode(
|
||||
node_id="n1",
|
||||
data_id="",
|
||||
data={
|
||||
"id": "",
|
||||
"project_id": "p1",
|
||||
"contract_id": "c1",
|
||||
"show_name": "架线工程",
|
||||
"construction_section": "SC",
|
||||
"status": -1,
|
||||
"task_type": "jxfx",
|
||||
"plan_start_date": "2024-06-18",
|
||||
"plan_complete_date": "2024-12-05",
|
||||
"actual_complete_date": "2025-12-29",
|
||||
"complete_quantity": 2.0,
|
||||
"complete_rate": 100.0,
|
||||
"contract_quantity": 2.0,
|
||||
"plan_node": [],
|
||||
"statistic_date": "2025-12-29",
|
||||
"is_crucial": False,
|
||||
"delay_days": None,
|
||||
},
|
||||
)
|
||||
node.context["project_id"] = "p1"
|
||||
|
||||
results = await handler.create_all([node])
|
||||
|
||||
assert len(results) == 1
|
||||
result = results[0]
|
||||
assert result.status.value == "failed"
|
||||
assert result.push_id is not None
|
||||
assert result.error == f"API error for {result.push_id}"
|
||||
@@ -52,7 +52,13 @@ def test_e01_bootstrap_accepts_all_combinations(action: SyncAction, status: Sync
|
||||
original_error = node.error
|
||||
|
||||
is_zombie = _is_create_zombie(node)
|
||||
decision = e01_bootstrap(runtime, node=node, is_create_zombie=is_zombie)
|
||||
decision = e01_bootstrap(
|
||||
runtime,
|
||||
node=node,
|
||||
is_create_zombie=is_zombie,
|
||||
bootstrap_source_present=True,
|
||||
has_binding_record=False,
|
||||
)
|
||||
|
||||
if is_zombie:
|
||||
assert decision.to_state == "S15"
|
||||
@@ -66,3 +72,48 @@ def test_e01_bootstrap_accepts_all_combinations(action: SyncAction, status: Sync
|
||||
assert node.action == SyncAction.NONE
|
||||
assert node.status == SyncStatus.PENDING
|
||||
assert node.error is None
|
||||
|
||||
|
||||
def test_e01_bootstrap_cleans_unbound_persistence_only_node_on_source_miss() -> None:
|
||||
runtime = StateMachineRuntime(StateMachineConfig.load(CFG_PATH))
|
||||
node = _new_node(
|
||||
node_id="e01-stale-unbound",
|
||||
action=SyncAction.NONE,
|
||||
status=SyncStatus.PENDING,
|
||||
data_id="id-1",
|
||||
)
|
||||
|
||||
decision = e01_bootstrap(
|
||||
runtime,
|
||||
node=node,
|
||||
is_create_zombie=False,
|
||||
bootstrap_source_present=False,
|
||||
has_binding_record=False,
|
||||
)
|
||||
|
||||
assert decision.to_state == "S15"
|
||||
|
||||
|
||||
def test_e01_bootstrap_isolates_bound_node_on_source_miss() -> None:
|
||||
runtime = StateMachineRuntime(StateMachineConfig.load(CFG_PATH))
|
||||
node = _new_node(
|
||||
node_id="e01-stale-bound",
|
||||
action=SyncAction.NONE,
|
||||
status=SyncStatus.PENDING,
|
||||
data_id="id-1",
|
||||
)
|
||||
|
||||
decision = e01_bootstrap(
|
||||
runtime,
|
||||
node=node,
|
||||
is_create_zombie=False,
|
||||
bootstrap_source_present=False,
|
||||
has_binding_record=True,
|
||||
)
|
||||
|
||||
assert decision.to_state == "S17"
|
||||
assert node.binding_status == BindingStatus.ABNORMAL
|
||||
assert node.action == SyncAction.NONE
|
||||
assert node.status == SyncStatus.FAILED
|
||||
assert node.context["bootstrap_blocked"] is True
|
||||
assert "BOOTSTRAP_SOURCE_MISSING" in (node.error or "")
|
||||
|
||||
@@ -7,7 +7,10 @@ import pytest
|
||||
|
||||
from sync_state_machine.common.binding import BindingManager
|
||||
from sync_state_machine.common.collection import DataCollection
|
||||
from sync_state_machine.common.sqlite_backend import SQLitePersistenceBackend
|
||||
from sync_state_machine.common.types import BindingStatus, SyncAction, SyncStatus
|
||||
from sync_state_machine.datasource.jsonl import JsonlDataSource
|
||||
from sync_state_machine.domain.project.sync_node import ProjectSyncNode
|
||||
import sync_state_machine.pipeline.full_sync_pipeline as full_sync_pipeline_module
|
||||
from sync_state_machine.pipeline.full_sync_pipeline import FullSyncPipeline, NodeTypeLoadError, WriteOutcome
|
||||
|
||||
@@ -30,6 +33,37 @@ class _FakeDataSource:
|
||||
return None
|
||||
|
||||
|
||||
class _BootstrapRefreshDataSource(_FakeDataSource):
|
||||
def __init__(self, *, node_data: dict[str, str]) -> None:
|
||||
self.node_data = dict(node_data)
|
||||
|
||||
async def load_all(self, order=None, data_type=None, raise_on_error=False) -> None:
|
||||
del data_type, raise_on_error
|
||||
if self.collection is None:
|
||||
raise RuntimeError("collection not set")
|
||||
if not order or tuple(order) != ("project",):
|
||||
return
|
||||
|
||||
existing = self.collection.get_by_data_id("project", self.node_data["id"])
|
||||
if existing is None:
|
||||
node = ProjectSyncNode(
|
||||
node_id="bootstrap-refreshed-project",
|
||||
data_id=self.node_data["id"],
|
||||
data=self.node_data,
|
||||
)
|
||||
node.append_log(f"加载来源: {self.__class__.__name__}.load 新增节点")
|
||||
await self.collection.add(node)
|
||||
return
|
||||
|
||||
existing.set_data(self.node_data)
|
||||
existing.set_origin_data(self.node_data)
|
||||
if existing.context.pop("_loaded_from_persistence", False):
|
||||
existing.append_log(f"加载来源: persistence -> {self.__class__.__name__}.load 覆盖刷新")
|
||||
else:
|
||||
existing.append_log(f"加载来源: {self.__class__.__name__}.load 覆盖刷新")
|
||||
await self.collection.update(existing)
|
||||
|
||||
|
||||
class _StubStrategy:
|
||||
def __init__(self, node_type: str):
|
||||
self.node_type = node_type
|
||||
@@ -348,4 +382,73 @@ async def test_run_persists_before_reraising_load_failure(monkeypatch: pytest.Mo
|
||||
await pipeline.run()
|
||||
|
||||
assert calls == ["bootstrap", "sync", "persist"]
|
||||
assert persistence.close_calls == 1
|
||||
assert persistence.close_calls == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_phase_bootstrap_refreshes_persisted_nodes_before_e01_cleanup(tmp_path: Path) -> None:
|
||||
db_path = tmp_path / "bootstrap-refresh.db"
|
||||
persistence = SQLitePersistenceBackend(str(db_path))
|
||||
await persistence.initialize()
|
||||
|
||||
seed_remote = DataCollection("remote", persistence=persistence, auto_persist=False)
|
||||
stale_node = ProjectSyncNode(
|
||||
node_id="persisted-project-node",
|
||||
data_id="proj-bootstrap-1",
|
||||
data={
|
||||
"id": "proj-bootstrap-1",
|
||||
"type": "project",
|
||||
"name": "Persisted Project",
|
||||
"code": "P-BOOT",
|
||||
"status": "active",
|
||||
"progress_type": "C",
|
||||
"project_type": "onshore_wind",
|
||||
},
|
||||
action=SyncAction.NONE,
|
||||
status=SyncStatus.PENDING,
|
||||
binding_status=BindingStatus.WARNING,
|
||||
)
|
||||
await seed_remote.add(stale_node)
|
||||
await seed_remote.persist()
|
||||
|
||||
local_collection = DataCollection("local", persistence=persistence, auto_persist=False)
|
||||
remote_collection = DataCollection("remote", persistence=persistence, auto_persist=False)
|
||||
binding_manager = BindingManager(persistence, auto_persist=False)
|
||||
|
||||
pipeline = FullSyncPipeline(
|
||||
local_datasource=_FakeDataSource(),
|
||||
remote_datasource=_BootstrapRefreshDataSource(
|
||||
node_data={
|
||||
"id": "proj-bootstrap-1",
|
||||
"type": "project",
|
||||
"name": "Refreshed Project",
|
||||
"code": "P-BOOT",
|
||||
"status": "active",
|
||||
"progress_type": "C",
|
||||
"project_type": "onshore_wind",
|
||||
}
|
||||
),
|
||||
strategies=[_StubStrategy("project")],
|
||||
persistence=persistence,
|
||||
local_collection=local_collection,
|
||||
remote_collection=remote_collection,
|
||||
binding_manager=binding_manager,
|
||||
node_types=["project"],
|
||||
load_order=["project"],
|
||||
sync_order=["project"],
|
||||
enable_persistence=True,
|
||||
)
|
||||
|
||||
await pipeline.phase_bootstrap()
|
||||
|
||||
refreshed = remote_collection.get("persisted-project-node")
|
||||
assert refreshed is not None
|
||||
assert refreshed.binding_status == BindingStatus.UNCHECKED
|
||||
assert refreshed.action == SyncAction.NONE
|
||||
assert refreshed.status == SyncStatus.PENDING
|
||||
assert remote_collection.is_bootstrap_blocked(refreshed) is False
|
||||
assert refreshed.sync_log is not None
|
||||
assert "加载来源: persistence 恢复节点" in refreshed.sync_log
|
||||
assert "加载来源: persistence -> _BootstrapRefreshDataSource.load 覆盖刷新" in refreshed.sync_log
|
||||
|
||||
await pipeline.close()
|
||||
@@ -55,4 +55,22 @@ async def test_scope_partitioning_and_copy_between_scopes(tmp_path: Path) -> Non
|
||||
await reloaded_global.load_from_persistence()
|
||||
assert len(reloaded_global.filter(node_type="test_project")) == 1
|
||||
|
||||
await persistence.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sqlite_backend_creates_restore_join_index(tmp_path: Path) -> None:
|
||||
db_path = tmp_path / "scope_partitioning.db"
|
||||
persistence = SQLitePersistenceBackend(str(db_path))
|
||||
await persistence.initialize()
|
||||
|
||||
columns, rows = PersistenceBackend.query_records(
|
||||
backend="sqlite",
|
||||
db_path=str(db_path),
|
||||
sql="PRAGMA index_list('nodes')",
|
||||
)
|
||||
|
||||
assert columns == ["seq", "name", "unique", "origin", "partial"]
|
||||
assert any(row["name"] == "idx_nodes_scope_type_node_id" for row in rows)
|
||||
|
||||
await persistence.close()
|
||||
Reference in New Issue
Block a user