first commit

2026-03-09 16:31:42 +08:00
commit 4a9c444b10
486 changed files with 52479 additions and 0 deletions
@@ -0,0 +1,252 @@
+# DataSource Layer Implementation
+
+## Overview
+
+This document describes the new datasource layer for `sync_system_new`, which is completely decoupled from the sync strategy and state machine logic.
+
+## Architecture
+
+The datasource layer follows a pure execution model:
+
+1. **Load**: Reads data from backend and creates `SyncNode` instances
+2. **Save**: Executes actions (CREATE/UPDATE/DELETE) based on `node.action`
+3. **Update Status**: Sets `node.status` to SUCCESS or FAILED after execution
+
+### Key Principles
+
+- **No Strategy Logic**: Datasource doesn't make decisions about what to sync
+- **State-Driven**: Only reads `node.action` and `node.status` to determine behavior
+- **No ID Resolution**: IDs are already resolved by the strategy layer
+- **Backend Agnostic**: Abstract base class supports multiple backends
+
+## Components
+
+### 1. Base Protocol (`base.py`)
+
+#### `DataSourceProtocol`
+
+Protocol defining the interface for all datasource implementations:
+
+```python
+async def load_nodes(
+    node_type: str,
+    collection: DataCollection,
+    **filters: Any
+) -> None:
+    """Load data from backend and create SyncNodes"""
+
+async def save_nodes(
+    node_type: str,
+    collection: DataCollection
+) -> None:
+    """Execute pending actions for nodes"""
+```
+
+#### `BaseDataSource`
+
+Abstract base class providing common functionality:
+
+- `load_nodes`: Creates SyncNodes from raw data
+- `save_nodes`: Executes actions and updates status
+- `_handle_create/update/delete`: Action-specific handlers
+
+Subclasses implement:
+- `_load_raw_data`: Load from backend
+- `_create_item`: Execute CREATE
+- `_update_item`: Execute UPDATE
+- `_delete_item`: Execute DELETE
+- `_generate_id`: Generate new IDs
+
+### 2. JSONL Implementation (`jsonl_datasource.py`)
+
+#### `JsonlDataSource`
+
+JSONL-based datasource for local testing:
+
+**Features:**
+- Loads `.jsonl` files into memory
+- Generates UUIDs for new items
+- Supports filtering during load
+- Supports read-only mode
+- Persists changes back to files
+
+**File Structure:**
+```
+data/
+  ├── contract_1.jsonl
+  ├── project_1.jsonl
+  └── ...
+```
+
+**Usage:**
+```python
+datasource = JsonlDataSource(Path("data/"))
+
+# Load nodes
+await datasource.load_nodes("contract", collection, project_id="P1")
+
+# Execute actions
+await datasource.save_nodes("contract", collection)
+
+# Persist to disk
+await datasource.save_all()
+```
+
+### 3. Domain Handlers
+
+Domain-specific handlers provide business logic (optional):
+
+**Example: `domain/contract/jsonl_handler.py`**
+
+```python
+class ContractJsonlHandler:
+    @staticmethod
+    def validate_create(data: Any) -> None:
+        """Validate before create"""
+        
+    @staticmethod
+    def transform_for_create(data: Any) -> Dict[str, Any]:
+        """Transform data before create"""
+```
+
+## Node Lifecycle
+
+### Loading Phase
+
+1. Datasource reads raw data from backend
+2. Creates `SyncNode` for each item:
+   - `node_id`: Unique session ID (e.g., "contract_C1")
+   - `data_id`: Backend ID (e.g., "C1")
+   - `origin_data`: Raw data from backend
+   - `data`: Copy of origin_data (may be modified by strategy)
+   - `action`: NONE (initial)
+   - `status`: PENDING (initial)
+
+### Execution Phase
+
+For each node with `status == PENDING` and `action != NONE`:
+
+1. **CREATE**:
+   - Generate new ID via `_generate_id`
+   - Call `_create_item` with new ID and data
+   - Set `node.data_id` to new ID
+   - Set `node.status` to SUCCESS
+
+2. **UPDATE**:
+   - Call `_update_item` with existing `data_id` and data
+   - Set `node.status` to SUCCESS
+
+3. **DELETE**:
+   - Call `_delete_item` with `data_id`
+   - Set `node.status` to SUCCESS
+
+4. **Error Handling**:
+   - On exception: Set `node.status` to FAILED
+   - Set `node.error` to exception message
+
+## Integration with Sync System
+
+The datasource is used by the pipeline layer:
+
+```python
+# 1. Load data
+local_ds = JsonlDataSource(Path("local_data/"))
+remote_ds = JsonlDataSource(Path("remote_data/"))
+
+local_collection = DataCollection("local")
+remote_collection = DataCollection("remote")
+
+await local_ds.load_nodes("contract", local_collection)
+await remote_ds.load_nodes("contract", remote_collection)
+
+# 2. Strategy layer determines actions
+strategy = ContractSyncStrategy(...)
+await strategy.bind(...)  # Sets node.action
+
+# 3. Execute actions
+await remote_ds.save_nodes("contract", remote_collection)
+await remote_ds.save_all()
+```
+
+## Testing
+
+Comprehensive test suite in `tests/test_datasource.py`:
+
+- Basic loading and node creation
+- Filtering during load
+- CREATE/UPDATE/DELETE actions
+- Error handling and status updates
+- Read-only mode
+- Multiple node types
+
+Run tests:
+```bash
+python -m pytest sync_system_new/tests/test_datasource.py -v
+```
+
+## Future Extensions
+
+### API DataSource
+
+For production use, implement `ApiDataSource`:
+
+```python
+class ApiDataSource(BaseDataSource):
+    async def _load_raw_data(self, node_type: str, **filters):
+        # Call REST API
+        
+    async def _create_item(self, node_type: str, item_id: str, data):
+        # POST request
+        
+    async def _generate_id(self, node_type: str, data):
+        # Return None - server generates ID
+```
+
+### Database DataSource
+
+For SQL databases:
+
+```python
+class DatabaseDataSource(BaseDataSource):
+    async def _load_raw_data(self, node_type: str, **filters):
+        # SELECT query
+        
+    async def _create_item(self, node_type: str, item_id: str, data):
+        # INSERT query
+```
+
+## Comparison with Legacy System
+
+### Legacy System (`sync_system/datasource/`)
+
+- Tightly coupled with RepositoryProtocol
+- Mixed concerns (data access + business logic)
+- Schema validation in datasource layer
+- Different interface for each entity type
+
+### New System (`sync_system_new/datasource/`)
+
+- Decoupled from strategy and state machine
+- Pure data access (business logic in domain handlers)
+- Generic interface for all entity types
+- Direct integration with SyncNode and DataCollection
+
+## Key Differences
+
+| Aspect | Legacy | New |
+|--------|--------|-----|
+| Coupling | Tight with sync policies | Decoupled from sync logic |
+| Interface | Repository per entity | Generic load/save |
+| ID Handling | Mixed responsibilities | IDs already resolved |
+| State Management | Mixed with data access | SyncNode-based |
+| Testing | Complex setup | Simple, focused tests |
+
+## Conclusion
+
+The new datasource layer provides:
+
+1. **Clear Separation**: Data access is isolated from sync logic
+2. **Flexibility**: Easy to add new backend types
+3. **Testability**: Simple, focused tests
+4. **Type Safety**: Full integration with Pydantic models
+5. **Maintainability**: Single responsibility per component