crowd-funder-from-jason/doc/migration-to-wa-sqlite.md

# Migration Guide: Dexie to absurd-sql

## Overview

This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.

**Current Status**: The migration is in **Phase 2** with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress.

## Migration Goals

1. **Data Integrity**
   - Preserve all existing data
   - Maintain data relationships
   - Ensure data consistency

2. **Performance**
   - Improve query performance
   - Reduce storage overhead
   - Optimize for platform-specific features

3. **Security**
   - Maintain or improve encryption
   - Preserve access controls
   - Enhance data protection

4. **User Experience**
   - Zero data loss
   - Minimal downtime
   - Automatic migration where possible

## Migration Fence

The migration is controlled by a **migration fence** that separates legacy Dexie code from the new SQLite implementation. See [Migration Fence Definition](./migration-fence-definition.md) for complete details.

### Key Fence Components

1. **Configuration Control**: `USE_DEXIE_DB = false` (default)
2. **Service Layer**: All database operations go through `PlatformService`
3. **Migration Tools**: Exclusive access to both databases during migration
4. **Code Boundaries**: Clear separation between legacy and new code

## Prerequisites

1. **Backup Requirements**
   ```typescript
   interface MigrationBackup {
     timestamp: number;
     accounts: Account[];
     settings: Setting[];
     contacts: Contact[];
     metadata: {
       version: string;
       platform: string;
       dexieVersion: string;
     };
   }
   ```

2. **Dependencies**
   ```json
   {
     "@jlongster/sql.js": "^1.8.0",
     "absurd-sql": "^1.8.0"
   }
   ```

3. **Storage Requirements**
   - Sufficient IndexedDB quota
   - Available disk space for SQLite
   - Backup storage space

4. **Platform Support**
   - Web: Modern browser with IndexedDB support
   - iOS: iOS 13+ with SQLite support
   - Android: Android 5+ with SQLite support
   - Electron: Latest version with SQLite support

## Current Migration Status

### ✅ Completed
- **SQLite Database Service**: Fully implemented with absurd-sql
- **Platform Service Layer**: Unified database interface
- **Migration Tools**: Data comparison and transfer utilities
- **Settings Migration**: Core user settings transferred
- **Account Migration**: Identity and key management
- **Schema Migration**: Complete table structure migration

### 🔄 In Progress
- **Contact Migration**: User contact data (via import interface)
- **Data Verification**: Comprehensive integrity checks
- **Performance Optimization**: Query optimization and indexing

### 📋 Planned
- **Code Cleanup**: Remove unused Dexie imports
- **Documentation Updates**: Complete migration guides
- **Testing**: Comprehensive migration testing

## Migration Process

### 1. Preparation

```typescript
// src/services/storage/migration/MigrationService.ts
import initSqlJs from '@jlongster/sql.js';
import { SQLiteFS } from 'absurd-sql';
import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend';

class MigrationService {
  private async checkPrerequisites(): Promise<void> {
    // Check IndexedDB availability
    if (!window.indexedDB) {
      throw new StorageError(
        'IndexedDB not available',
        StorageErrorCodes.INITIALIZATION_FAILED
      );
    }

    // Check storage quota
    const quota = await navigator.storage.estimate();
    if (quota.quota && quota.usage && quota.usage > quota.quota * 0.9) {
      throw new StorageError(
        'Insufficient storage space',
        StorageErrorCodes.STORAGE_FULL
      );
    }

    // Check platform support
    const capabilities = await PlatformDetection.getCapabilities();
    if (!capabilities.hasFileSystem) {
      throw new StorageError(
        'Platform does not support required features',
        StorageErrorCodes.INITIALIZATION_FAILED
      );
    }
  }

  private async createBackup(): Promise<MigrationBackup> {
    const dexieDB = new Dexie('TimeSafariDB');
    
    return {
      timestamp: Date.now(),
      accounts: await dexieDB.accounts.toArray(),
      settings: await dexieDB.settings.toArray(),
      contacts: await dexieDB.contacts.toArray(),
      metadata: {
        version: '1.0.0',
        platform: await PlatformDetection.getPlatform(),
        dexieVersion: Dexie.version
      }
    };
  }
}
```

### 2. Data Migration

```typescript
// src/services/storage/migration/DataMigration.ts
class DataMigration {
  async migrateAccounts(): Promise<MigrationResult> {
    const result: MigrationResult = {
      success: true,
      accountsMigrated: 0,
      errors: [],
      warnings: []
    };

    try {
      const dexieAccounts = await this.getDexieAccounts();
      
      for (const account of dexieAccounts) {
        try {
          await this.migrateAccount(account);
          result.accountsMigrated++;
        } catch (error) {
          result.errors.push(`Failed to migrate account ${account.did}: ${error}`);
          result.success = false;
        }
      }
    } catch (error) {
      result.errors.push(`Account migration failed: ${error}`);
      result.success = false;
    }

    return result;
  }

  async migrateSettings(): Promise<MigrationResult> {
    const result: MigrationResult = {
      success: true,
      settingsMigrated: 0,
      errors: [],
      warnings: []
    };

    try {
      const dexieSettings = await this.getDexieSettings();
      
      for (const setting of dexieSettings) {
        try {
          await this.migrateSetting(setting);
          result.settingsMigrated++;
        } catch (error) {
          result.errors.push(`Failed to migrate setting ${setting.id}: ${error}`);
          result.success = false;
        }
      }
    } catch (error) {
      result.errors.push(`Settings migration failed: ${error}`);
      result.success = false;
    }

    return result;
  }

  async migrateContacts(): Promise<MigrationResult> {
    // Contact migration is handled through the contact import interface
    // This provides better user control and validation
    const result: MigrationResult = {
      success: true,
      contactsMigrated: 0,
      errors: [],
      warnings: []
    };

    try {
      const dexieContacts = await this.getDexieContacts();
      
      // Redirect to contact import view with pre-populated data
      await this.redirectToContactImport(dexieContacts);
      
      result.contactsMigrated = dexieContacts.length;
    } catch (error) {
      result.errors.push(`Contact migration failed: ${error}`);
      result.success = false;
    }

    return result;
  }
}
```

### 3. Verification

```typescript
class MigrationVerification {
  async verifyMigration(dexieData: MigrationData): Promise<boolean> {
    // Verify account count
    const accountResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM accounts');
    const accountCount = accountResult[0].values[0][0];
    if (accountCount !== dexieData.accounts.length) {
      return false;
    }

    // Verify settings count
    const settingsResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM settings');
    const settingsCount = settingsResult[0].values[0][0];
    if (settingsCount !== dexieData.settings.length) {
      return false;
    }

    // Verify data integrity
    for (const account of dexieData.accounts) {
      const result = await this.sqliteDB.exec(
        'SELECT * FROM accounts WHERE did = ?',
        [account.did]
      );
      const migratedAccount = result[0]?.values[0];
      if (!migratedAccount ||
          migratedAccount[1] !== account.publicKeyHex) {
        return false;
      }
    }

    return true;
  }
}
```

## Using the Migration Interface

### Accessing Migration Tools

1. Navigate to the **Account** page in the TimeSafari app
2. Scroll down to find the **Database Migration** link
3. Click the link to open the migration interface

### Migration Steps

1. **Compare Databases**
   - Click "Compare Databases" to see differences
   - Review the comparison results
   - Identify data that needs migration

2. **Migrate Settings**
   - Click "Migrate Settings" to transfer user settings
   - Verify settings are correctly transferred
   - Check application functionality

3. **Migrate Contacts**
   - Click "Migrate Contacts" to open contact import
   - Review and confirm contact data
   - Complete the import process

4. **Verify Migration**
   - Run comparison again to verify completion
   - Test application functionality
   - Export backup data if needed

## Error Handling

### Common Issues

1. **Dexie Database Not Enabled**
   - **Error**: "Dexie database is not enabled"
   - **Solution**: Set `USE_DEXIE_DB = true` in `constants/app.ts` temporarily

2. **Database Connection Issues**
   - **Error**: "Failed to retrieve data"
   - **Solution**: Check database initialization and permissions

3. **Migration Failures**
   - **Error**: "Migration failed: [specific error]"
   - **Solution**: Review error details and check data integrity

### Error Recovery

1. **Review** error messages carefully
2. **Check** browser console for additional details
3. **Verify** database connectivity and permissions
4. **Retry** the operation if appropriate
5. **Export** comparison data for manual review if needed

## Best Practices

### Before Migration

1. **Backup** your data if possible
2. **Test** the migration on a small dataset first
3. **Verify** that both databases are accessible
4. **Review** the comparison results before migrating

### During Migration

1. **Don't** interrupt the migration process
2. **Monitor** the progress and error messages
3. **Note** any warnings or skipped records
4. **Export** comparison data for reference

### After Migration

1. **Verify** that data was migrated correctly
2. **Test** the application functionality
3. **Disable** Dexie database (`USE_DEXIE_DB = false`)
4. **Clean up** any temporary files or exports

## Performance Considerations

### 1. Migration Performance
- Use transactions for bulk data transfer
- Implement progress indicators
- Process data in background when possible

### 2. Application Performance
- Optimize SQLite queries
- Maintain proper database indexes
- Use efficient memory management

## Security Considerations

### 1. Data Protection
- Maintain encryption standards across migration
- Preserve user privacy during migration
- Log all migration operations

### 2. Error Handling
- Handle migration failures gracefully
- Provide clear user messaging
- Maintain rollback capabilities

## Testing Strategy

### 1. Migration Testing
```typescript
describe('Database Migration', () => {
  it('should migrate data without loss', async () => {
    // 1. Enable Dexie
    // 2. Create test data
    // 3. Run migration
    // 4. Verify data integrity
    // 5. Disable Dexie
  });
});
```

### 2. Application Testing
```typescript
describe('Feature with Database', () => {
  it('should work with SQLite only', async () => {
    // Test with USE_DEXIE_DB = false
    // Verify all operations use PlatformService
  });
});
```

## Conclusion

The migration from Dexie to absurd-sql provides:
- **Better Performance**: Improved query performance and storage efficiency
- **Cross-Platform Consistency**: Unified database interface across platforms
- **Enhanced Security**: Better encryption and access controls
- **Future-Proof Architecture**: Modern SQLite-based storage system

The migration fence ensures a controlled and safe transition while maintaining data integrity and application stability.