crowd-funder-for-time-pwa/doc/migration-to-wa-sqlite.md

Migration Guide: Dexie to absurd-sql

Overview

This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.

Current Status: The migration is in Phase 2 with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress.

Migration Goals

1. Data Integrity

   • Preserve all existing data
   • Maintain data relationships
   • Ensure data consistency

2. Performance

   • Improve query performance
   • Reduce storage overhead
   • Optimize for platform-specific features

3. Security

   • Maintain or improve encryption
   • Preserve access controls
   • Enhance data protection

4. User Experience

   • Zero data loss
   • Minimal downtime
   • Automatic migration where possible

Migration Fence

The migration is controlled by a migration fence that separates legacy Dexie code from the new SQLite implementation. See Migration Fence Definition for complete details.

Key Fence Components

1. Configuration Control: USE_DEXIE_DB = false (default)
2. Service Layer: All database operations go through PlatformService
3. Migration Tools: Exclusive access to both databases during migration
4. Code Boundaries: Clear separation between legacy and new code

Prerequisites

1. Backup Requirements

   interface MigrationBackup {
     timestamp: number;
     accounts: Account[];
     settings: Setting[];
     contacts: Contact[];
     metadata: {
       version: string;
       platform: string;
       dexieVersion: string;
     };
   }
   

2. Dependencies

   {
     "@jlongster/sql.js": "^1.8.0",
     "absurd-sql": "^1.8.0"
   }
   

3. Storage Requirements

   • Sufficient IndexedDB quota
   • Available disk space for SQLite
   • Backup storage space

4. Platform Support

   • Web: Modern browser with IndexedDB support
   • iOS: iOS 13+ with SQLite support
   • Android: Android 5+ with SQLite support
   • Electron: Latest version with SQLite support

Current Migration Status

✅ Completed

• SQLite Database Service: Fully implemented with absurd-sql
• Platform Service Layer: Unified database interface
• Migration Tools: Data comparison and transfer utilities
• Settings Migration: Core user settings transferred
• Account Migration: Identity and key management
• Schema Migration: Complete table structure migration

🔄 In Progress

• Contact Migration: User contact data (via import interface)
• Data Verification: Comprehensive integrity checks
• Performance Optimization: Query optimization and indexing

📋 Planned

• Code Cleanup: Remove unused Dexie imports
• Documentation Updates: Complete migration guides
• Testing: Comprehensive migration testing

Migration Process

1. Preparation

// src/services/storage/migration/MigrationService.ts
import initSqlJs from '@jlongster/sql.js';
import { SQLiteFS } from 'absurd-sql';
import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend';

class MigrationService {
  private async checkPrerequisites(): Promise<void> {
    // Check IndexedDB availability
    if (!window.indexedDB) {
      throw new StorageError(
        'IndexedDB not available',
        StorageErrorCodes.INITIALIZATION_FAILED
      );
    }

    // Check storage quota
    const quota = await navigator.storage.estimate();
    if (quota.quota && quota.usage && quota.usage > quota.quota * 0.9) {
      throw new StorageError(
        'Insufficient storage space',
        StorageErrorCodes.STORAGE_FULL
      );
    }

    // Check platform support
    const capabilities = await PlatformDetection.getCapabilities();
    if (!capabilities.hasFileSystem) {
      throw new StorageError(
        'Platform does not support required features',
        StorageErrorCodes.INITIALIZATION_FAILED
      );
    }
  }

  private async createBackup(): Promise<MigrationBackup> {
    const dexieDB = new Dexie('TimeSafariDB');
    
    return {
      timestamp: Date.now(),
      accounts: await dexieDB.accounts.toArray(),
      settings: await dexieDB.settings.toArray(),
      contacts: await dexieDB.contacts.toArray(),
      metadata: {
        version: '1.0.0',
        platform: await PlatformDetection.getPlatform(),
        dexieVersion: Dexie.version
      }
    };
  }
}

2. Data Migration

// src/services/storage/migration/DataMigration.ts
class DataMigration {
  async migrateAccounts(): Promise<MigrationResult> {
    const result: MigrationResult = {
      success: true,
      accountsMigrated: 0,
      errors: [],
      warnings: []
    };

    try {
      const dexieAccounts = await this.getDexieAccounts();
      
      for (const account of dexieAccounts) {
        try {
          await this.migrateAccount(account);
          result.accountsMigrated++;
        } catch (error) {
          result.errors.push(`Failed to migrate account ${account.did}: ${error}`);
          result.success = false;
        }
      }
    } catch (error) {
      result.errors.push(`Account migration failed: ${error}`);
      result.success = false;
    }

    return result;
  }

  async migrateSettings(): Promise<MigrationResult> {
    const result: MigrationResult = {
      success: true,
      settingsMigrated: 0,
      errors: [],
      warnings: []
    };

    try {
      const dexieSettings = await this.getDexieSettings();
      
      for (const setting of dexieSettings) {
        try {
          await this.migrateSetting(setting);
          result.settingsMigrated++;
        } catch (error) {
          result.errors.push(`Failed to migrate setting ${setting.id}: ${error}`);
          result.success = false;
        }
      }
    } catch (error) {
      result.errors.push(`Settings migration failed: ${error}`);
      result.success = false;
    }

    return result;
  }

  async migrateContacts(): Promise<MigrationResult> {
    // Contact migration is handled through the contact import interface
    // This provides better user control and validation
    const result: MigrationResult = {
      success: true,
      contactsMigrated: 0,
      errors: [],
      warnings: []
    };

    try {
      const dexieContacts = await this.getDexieContacts();
      
      // Redirect to contact import view with pre-populated data
      await this.redirectToContactImport(dexieContacts);
      
      result.contactsMigrated = dexieContacts.length;
    } catch (error) {
      result.errors.push(`Contact migration failed: ${error}`);
      result.success = false;
    }

    return result;
  }
}

3. Verification

class MigrationVerification {
  async verifyMigration(dexieData: MigrationData): Promise<boolean> {
    // Verify account count
    const accountResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM accounts');
    const accountCount = accountResult[0].values[0][0];
    if (accountCount !== dexieData.accounts.length) {
      return false;
    }

    // Verify settings count
    const settingsResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM settings');
    const settingsCount = settingsResult[0].values[0][0];
    if (settingsCount !== dexieData.settings.length) {
      return false;
    }

    // Verify data integrity
    for (const account of dexieData.accounts) {
      const result = await this.sqliteDB.exec(
        'SELECT * FROM accounts WHERE did = ?',
        [account.did]
      );
      const migratedAccount = result[0]?.values[0];
      if (!migratedAccount ||
          migratedAccount[1] !== account.publicKeyHex) {
        return false;
      }
    }

    return true;
  }
}

Using the Migration Interface

Accessing Migration Tools

1. Navigate to the Account page in the TimeSafari app
2. Scroll down to find the Database Migration link
3. Click the link to open the migration interface

Migration Steps

1. Compare Databases

   • Click "Compare Databases" to see differences
   • Review the comparison results
   • Identify data that needs migration

2. Migrate Settings

   • Click "Migrate Settings" to transfer user settings
   • Verify settings are correctly transferred
   • Check application functionality

3. Migrate Contacts

   • Click "Migrate Contacts" to open contact import
   • Review and confirm contact data
   • Complete the import process

4. Verify Migration

   • Run comparison again to verify completion
   • Test application functionality
   • Export backup data if needed

Error Handling

Common Issues

1. Dexie Database Not Enabled

   • Error: "Dexie database is not enabled"
   • Solution: Set USE_DEXIE_DB = true in constants/app.ts temporarily

2. Database Connection Issues

   • Error: "Failed to retrieve data"
   • Solution: Check database initialization and permissions

3. Migration Failures

   • Error: "Migration failed: [specific error]"
   • Solution: Review error details and check data integrity

Error Recovery

1. Review error messages carefully
2. Check browser console for additional details
3. Verify database connectivity and permissions
4. Retry the operation if appropriate
5. Export comparison data for manual review if needed

Best Practices

Before Migration

1. Backup your data if possible
2. Test the migration on a small dataset first
3. Verify that both databases are accessible
4. Review the comparison results before migrating

During Migration

1. Don't interrupt the migration process
2. Monitor the progress and error messages
3. Note any warnings or skipped records
4. Export comparison data for reference

After Migration

1. Verify that data was migrated correctly
2. Test the application functionality
3. Disable Dexie database (USE_DEXIE_DB = false)
4. Clean up any temporary files or exports

Performance Considerations

1. Migration Performance

• Use transactions for bulk data transfer
• Implement progress indicators
• Process data in background when possible

2. Application Performance

• Optimize SQLite queries
• Maintain proper database indexes
• Use efficient memory management

Security Considerations

1. Data Protection

• Maintain encryption standards across migration
• Preserve user privacy during migration
• Log all migration operations

2. Error Handling

• Handle migration failures gracefully
• Provide clear user messaging
• Maintain rollback capabilities

Testing Strategy

1. Migration Testing

describe('Database Migration', () => {
  it('should migrate data without loss', async () => {
    // 1. Enable Dexie
    // 2. Create test data
    // 3. Run migration
    // 4. Verify data integrity
    // 5. Disable Dexie
  });
});

2. Application Testing

describe('Feature with Database', () => {
  it('should work with SQLite only', async () => {
    // Test with USE_DEXIE_DB = false
    // Verify all operations use PlatformService
  });
});

Conclusion

The migration from Dexie to absurd-sql provides:

• Better Performance: Improved query performance and storage efficiency
• Cross-Platform Consistency: Unified database interface across platforms
• Enhanced Security: Better encryption and access controls
• Future-Proof Architecture: Modern SQLite-based storage system

The migration fence ensures a controlled and safe transition while maintaining data integrity and application stability.