crowd-funder-for-time-pwa/doc/migration-to-wa-sqlite.md

Migration Guide: Dexie to absurd-sql

Overview

This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.

Current Status: The migration is in Phase 2 with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress. ActiveDid migration has been implemented to ensure user identity continuity.

Migration Goals

1. Data Integrity

   • Preserve all existing data
   • Maintain data relationships
   • Ensure data consistency
   • Preserve user's active identity

2. Performance

   • Improve query performance
   • Reduce storage overhead
   • Optimize for platform-specific capabilities

3. User Experience

   • Seamless transition with no data loss
   • Maintain user's active identity and preferences
   • Preserve application state

Migration Architecture

Migration Fence

The migration fence is defined by the USE_DEXIE_DB constant in src/constants/app.ts:

• USE_DEXIE_DB = false (default): Uses SQLite database
• USE_DEXIE_DB = true: Uses Dexie database (for migration purposes)

Migration Order

The migration follows a specific order to maintain data integrity:

1. Accounts (foundational - contains DIDs)
2. Settings (references accountDid, activeDid)
3. ActiveDid (depends on accounts and settings) ⭐ NEW
4. Contacts (independent, but migrated after accounts for consistency)

ActiveDid Migration ⭐ NEW FEATURE

Problem Solved

Previously, the activeDid setting was not migrated from Dexie to SQLite, causing users to lose their active identity after migration.

Solution Implemented

The migration now includes a dedicated step for migrating the activeDid:

1. Detection: Identifies the activeDid from Dexie master settings
2. Validation: Verifies the activeDid exists in SQLite accounts
3. Migration: Updates SQLite master settings with the activeDid
4. Error Handling: Graceful handling of missing accounts

Implementation Details

New Function: migrateActiveDid()

export async function migrateActiveDid(): Promise<MigrationResult> {
  // 1. Get Dexie settings to find the activeDid
  const dexieSettings = await getDexieSettings();
  const masterSettings = dexieSettings.find(setting => !setting.accountDid);
  
  // 2. Verify the activeDid exists in SQLite accounts
  const accountExists = await platformService.dbQuery(
    "SELECT did FROM accounts WHERE did = ?",
    [dexieActiveDid],
  );
  
  // 3. Update SQLite master settings
  await updateDefaultSettings({ activeDid: dexieActiveDid });
}

Enhanced migrateSettings() Function

The settings migration now includes activeDid handling:

• Extracts activeDid from Dexie master settings
• Validates account existence in SQLite
• Updates SQLite master settings with the activeDid

Updated migrateAll() Function

The complete migration now includes a dedicated step for activeDid:

// Step 3: Migrate ActiveDid (depends on accounts and settings)
logger.info("[MigrationService] Step 3: Migrating activeDid...");
const activeDidResult = await migrateActiveDid();

Benefits

• ✅ User Identity Preservation: Users maintain their active identity
• ✅ Seamless Experience: No need to manually select identity after migration
• ✅ Data Consistency: Ensures all identity-related settings are preserved
• ✅ Error Resilience: Graceful handling of edge cases

Migration Process

Phase 1: Preparation ✅

• Enable Dexie database access
• Implement data comparison tools
• Create migration service structure

Phase 2: Core Migration ✅

• Account migration with importFromMnemonic
• Settings migration (excluding activeDid)
• ActiveDid migration ⭐ COMPLETED
• Contact migration framework

Phase 3: Validation and Cleanup 🔄

• Comprehensive data validation
• Performance testing
• User acceptance testing
• Dexie removal

Usage

Manual Migration

import { migrateAll, migrateActiveDid } from '../services/indexedDBMigrationService';

// Complete migration
const result = await migrateAll();

// Or migrate just the activeDid
const activeDidResult = await migrateActiveDid();

Migration Verification

import { compareDatabases } from '../services/indexedDBMigrationService';

const comparison = await compareDatabases();
console.log('Migration differences:', comparison.differences);

Error Handling

ActiveDid Migration Errors

• Missing Account: If the activeDid from Dexie doesn't exist in SQLite accounts
• Database Errors: Connection or query failures
• Settings Update Failures: Issues updating SQLite master settings

Recovery Strategies

1. Automatic Recovery: Migration continues even if activeDid migration fails
2. Manual Recovery: Users can manually select their identity after migration
3. Fallback: System creates new identity if none exists

Security Considerations

Data Protection

• All sensitive data (mnemonics, private keys) are encrypted
• Migration preserves encryption standards
• No plaintext data exposure during migration

Identity Verification

• ActiveDid migration validates account existence
• Prevents setting non-existent identities as active
• Maintains cryptographic integrity

Testing

Migration Testing

# Enable Dexie for testing
# Set USE_DEXIE_DB = true in constants/app.ts

# Run migration
npm run migrate

# Verify results
npm run test:migration

ActiveDid Testing

// Test activeDid migration specifically
const result = await migrateActiveDid();
expect(result.success).toBe(true);
expect(result.warnings).toContain('Successfully migrated activeDid');

Troubleshooting

Common Issues

1. ActiveDid Not Found

   • Ensure accounts were migrated before activeDid migration
   • Check that the Dexie activeDid exists in SQLite accounts

2. Migration Failures

   • Verify Dexie database is accessible
   • Check SQLite database permissions
   • Review migration logs for specific errors

3. Data Inconsistencies

   • Use compareDatabases() to identify differences
   • Re-run migration if necessary
   • Check for duplicate or conflicting records

Debugging

// Enable detailed logging
logger.setLevel('debug');

// Check migration status
const comparison = await compareDatabases();
console.log('Settings differences:', comparison.differences.settings);

Future Enhancements

Planned Improvements

1. Batch Processing: Optimize for large datasets
2. Incremental Migration: Support partial migrations
3. Rollback Capability: Ability to revert migration
4. Progress Tracking: Real-time migration progress

Performance Optimizations

1. Parallel Processing: Migrate independent data concurrently
2. Memory Management: Optimize for large datasets
3. Transaction Batching: Reduce database round trips

Conclusion

The Dexie to SQLite migration provides a robust, secure, and user-friendly transition path. The addition of activeDid migration ensures that users maintain their identity continuity throughout the migration process, significantly improving the user experience.

The migration fence architecture allows for controlled, reversible migration while maintaining application stability and data integrity.