ecm_sync_system/sync_state_machine/datasource

DataSource Layer Implementation

Overview

This document describes the new datasource layer for sync_system_new, which is completely decoupled from the sync strategy and state machine logic.

Architecture

The datasource layer follows a pure execution model:

1. Load: Reads data from backend and creates SyncNode instances
2. Save: Executes actions (CREATE/UPDATE/DELETE) based on node.action
3. Update Status: Sets node.status to SUCCESS or FAILED after execution

Key Principles

• No Strategy Logic: Datasource doesn't make decisions about what to sync
• State-Driven: Only reads node.action and node.status to determine behavior
• No ID Resolution: IDs are already resolved by the strategy layer
• Backend Agnostic: Abstract base class supports multiple backends

Components

1. Base Protocol (base.py)

DataSourceProtocol

Protocol defining the interface for all datasource implementations:

async def load_nodes(
    node_type: str,
    collection: DataCollection,
    **filters: Any
) -> None:
    """Load data from backend and create SyncNodes"""

async def save_nodes(
    node_type: str,
    collection: DataCollection
) -> None:
    """Execute pending actions for nodes"""

BaseDataSource

Abstract base class providing common functionality:

• load_nodes: Creates SyncNodes from raw data
• save_nodes: Executes actions and updates status
• _handle_create/update/delete: Action-specific handlers

Subclasses implement:

• _load_raw_data: Load from backend
• _create_item: Execute CREATE
• _update_item: Execute UPDATE
• _delete_item: Execute DELETE
• _generate_id: Generate new IDs

2. JSONL Implementation (jsonl_datasource.py)

JsonlDataSource

JSONL-based datasource for local testing:

Features:

• Loads .jsonl files into memory
• Generates UUIDs for new items
• Supports filtering during load
• Supports read-only mode
• Persists changes back to files

File Structure:

data/
  ├── contract_1.jsonl
  ├── project_1.jsonl
  └── ...

Usage:

datasource = JsonlDataSource(Path("data/"))

# Load nodes
await datasource.load_nodes("contract", collection, project_id="P1")

# Execute actions
await datasource.save_nodes("contract", collection)

# Persist to disk
await datasource.save_all()

3. Domain Handlers

Domain-specific handlers provide business logic (optional):

Example: domain/contract/jsonl_handler.py

class ContractJsonlHandler:
    @staticmethod
    def validate_create(data: Any) -> None:
        """Validate before create"""
        
    @staticmethod
    def transform_for_create(data: Any) -> Dict[str, Any]:
        """Transform data before create"""

Node Lifecycle

Loading Phase

1. Datasource reads raw data from backend
2. Creates SyncNode for each item:
   • node_id: Unique session ID (e.g., "contract_C1")
   • data_id: Backend ID (e.g., "C1")
   • origin_data: Raw data from backend
   • data: Copy of origin_data (may be modified by strategy)
   • action: NONE (initial)
   • status: PENDING (initial)

Execution Phase

For each node with status == PENDING and action != NONE:

1. CREATE:

   • Generate new ID via _generate_id
   • Call _create_item with new ID and data
   • Set node.data_id to new ID
   • Set node.status to SUCCESS

2. UPDATE:

   • Call _update_item with existing data_id and data
   • Set node.status to SUCCESS

3. DELETE:

   • Call _delete_item with data_id
   • Set node.status to SUCCESS

4. Error Handling:

   • On exception: Set node.status to FAILED
   • Set node.error to exception message

Integration with Sync System

The datasource is used by the pipeline layer:

# 1. Load data
local_ds = JsonlDataSource(Path("local_data/"))
remote_ds = JsonlDataSource(Path("remote_data/"))

local_collection = DataCollection("local")
remote_collection = DataCollection("remote")

await local_ds.load_nodes("contract", local_collection)
await remote_ds.load_nodes("contract", remote_collection)

# 2. Strategy layer determines actions
strategy = ContractSyncStrategy(...)
await strategy.bind(...)  # Sets node.action

# 3. Execute actions
await remote_ds.save_nodes("contract", remote_collection)
await remote_ds.save_all()

Testing

Comprehensive test suite in tests/test_datasource.py:

• Basic loading and node creation
• Filtering during load
• CREATE/UPDATE/DELETE actions
• Error handling and status updates
• Read-only mode
• Multiple node types

Run tests:

python -m pytest sync_system_new/tests/test_datasource.py -v

Future Extensions

API DataSource

For production use, implement ApiDataSource:

class ApiDataSource(BaseDataSource):
    async def _load_raw_data(self, node_type: str, **filters):
        # Call REST API
        
    async def _create_item(self, node_type: str, item_id: str, data):
        # POST request
        
    async def _generate_id(self, node_type: str, data):
        # Return None - server generates ID

Database DataSource

For SQL databases:

class DatabaseDataSource(BaseDataSource):
    async def _load_raw_data(self, node_type: str, **filters):
        # SELECT query
        
    async def _create_item(self, node_type: str, item_id: str, data):
        # INSERT query

Comparison with Legacy System

Legacy System (sync_system/datasource/)

• Tightly coupled with RepositoryProtocol
• Mixed concerns (data access + business logic)
• Schema validation in datasource layer
• Different interface for each entity type

New System (sync_system_new/datasource/)

• Decoupled from strategy and state machine
• Pure data access (business logic in domain handlers)
• Generic interface for all entity types
• Direct integration with SyncNode and DataCollection

Key Differences

Aspect	Legacy	New
Coupling	Tight with sync policies	Decoupled from sync logic
Interface	Repository per entity	Generic load/save
ID Handling	Mixed responsibilities	IDs already resolved
State Management	Mixed with data access	SyncNode-based
Testing	Complex setup	Simple, focused tests

Conclusion

The new datasource layer provides:

1. Clear Separation: Data access is isolated from sync logic
2. Flexibility: Easy to add new backend types
3. Testability: Simple, focused tests
4. Type Safety: Full integration with Pydantic models
5. Maintainability: Single responsibility per component