crowd-funder-for-time-pwa/doc/migration-to-wa-sqlite.md

Migration Guide: Dexie to absurd-sql

Overview

This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.

Current Status: The migration is in Phase 2 with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress. ActiveDid migration has been implemented to ensure user identity continuity.

⚠️ UPDATE: The migration fence is now implemented through the PlatformServiceMixin rather than a USE_DEXIE_DB constant. This provides a cleaner, more maintainable approach to database access control.

Migration Goals

1. Data Integrity

   • Preserve all existing data
   • Maintain data relationships
   • Ensure data consistency
   • Preserve user's active identity

2. Performance

   • Improve query performance
   • Reduce storage overhead
   • Optimize for platform-specific capabilities

3. User Experience

   • Seamless transition with no data loss
   • Maintain user's active identity and preferences
   • Preserve application state

Migration Architecture

Migration Fence

The migration fence is now defined by the PlatformServiceMixin in src/utils/PlatformServiceMixin.ts:

• PlatformServiceMixin: Centralized database access with caching and utilities
• Migration Tools: Exclusive interface between legacy and new databases
• Service Layer: All database operations go through PlatformService

Migration Order

The migration follows a specific order to maintain data integrity:

1. Accounts (foundational - contains DIDs)
2. Settings (references accountDid, activeDid)
3. ActiveDid (depends on accounts and settings) ⭐ NEW
4. Contacts (independent, but migrated after accounts for consistency)

ActiveDid Migration ⭐ NEW FEATURE

Problem Solved

Previously, the activeDid setting was not migrated from Dexie to SQLite, causing users to lose their active identity after migration.

Solution Implemented

The migration now includes a dedicated step for migrating the activeDid:

1. Detection: Identifies the activeDid from Dexie master settings
2. Validation: Verifies the activeDid exists in SQLite accounts
3. Migration: Updates SQLite master settings with the activeDid
4. Error Handling: Graceful handling of missing accounts

Implementation Details

New Function: migrateActiveDid()

export async function migrateActiveDid(): Promise<MigrationResult> {
  // 1. Get Dexie settings to find the activeDid
  const dexieSettings = await getDexieSettings();
  const masterSettings = dexieSettings.find(setting => !setting.accountDid);
  
  // 2. Verify the activeDid exists in SQLite accounts
  const accountExists = await platformService.dbQuery(
    "SELECT did FROM accounts WHERE did = ?",
    [dexieActiveDid],
  );
  
  // 3. Update SQLite master settings
  await updateDefaultSettings({ activeDid: dexieActiveDid });
   }

Enhanced migrateSettings() Function

The settings migration now includes activeDid handling:

• Extracts activeDid from Dexie master settings
• Validates account existence in SQLite
• Updates SQLite master settings with the activeDid

Updated migrateAll() Function

The complete migration now includes a dedicated step for activeDid:

// Step 3: Migrate ActiveDid (depends on accounts and settings)
logger.info("[MigrationService] Step 3: Migrating activeDid...");
const activeDidResult = await migrateActiveDid();

Benefits

• ✅ User Identity Preservation: Users maintain their active identity
• ✅ Seamless Experience: No need to manually select identity after migration
• ✅ Data Consistency: Ensures all identity-related settings are preserved
• ✅ Error Resilience: Graceful handling of edge cases

Migration Process

Phase 1: Preparation ✅

• PlatformServiceMixin implementation
• Implement data comparison tools
• Create migration service structure

Phase 2: Core Migration ✅

• Account migration with importFromMnemonic
• Settings migration (excluding activeDid)
• ActiveDid migration ⭐ COMPLETED
• Contact migration framework

Phase 3: Validation and Cleanup 🔄

• Comprehensive data validation
• Performance testing
• User acceptance testing
• Dexie removal

Usage

Manual Migration

import { migrateAll, migrateActiveDid } from '../services/indexedDBMigrationService';

// Complete migration
const result = await migrateAll();

// Or migrate just the activeDid
const activeDidResult = await migrateActiveDid();

Migration Verification

import { compareDatabases } from '../services/indexedDBMigrationService';

const comparison = await compareDatabases();
console.log('Migration differences:', comparison.differences);

PlatformServiceMixin Integration

After migration, use the mixin for all database operations:

// Use mixin methods for database access
const contacts = await this.$contacts();
const settings = await this.$settings();
const result = await this.$db("SELECT * FROM contacts WHERE did = ?", [accountDid]);

Error Handling

ActiveDid Migration Errors

• Missing Account: If the activeDid from Dexie doesn't exist in SQLite accounts
• Database Errors: Connection or query failures
• Settings Update Failures: Issues updating SQLite master settings

Recovery Strategies

1. Automatic Recovery: Migration continues even if activeDid migration fails
2. Manual Recovery: Users can manually select their identity after migration
3. Fallback: System creates new identity if none exists

Security Considerations

Data Protection

• All sensitive data (mnemonics, private keys) are encrypted
• Migration preserves encryption standards
• No plaintext data exposure during migration

Identity Verification

• ActiveDid migration validates account existence
• Prevents setting non-existent identities as active
• Maintains cryptographic integrity

Testing

Migration Testing

# Run migration
npm run migrate

# Verify results
npm run test:migration

ActiveDid Testing

// Test activeDid migration specifically
const result = await migrateActiveDid();
expect(result.success).toBe(true);
expect(result.warnings).toContain('Successfully migrated activeDid');

PlatformServiceMixin Testing

// Test mixin integration
describe('PlatformServiceMixin', () => {
  it('should provide database access methods', async () => {
    const contacts = await this.$contacts();
    const settings = await this.$settings();
    expect(contacts).toBeDefined();
    expect(settings).toBeDefined();
  });
});

Troubleshooting

Common Issues

1. ActiveDid Not Found

   • Ensure accounts were migrated before activeDid migration
   • Check that the Dexie activeDid exists in SQLite accounts

2. Migration Failures

   • Verify Dexie database is accessible
   • Check SQLite database permissions
   • Review migration logs for specific errors

3. Data Inconsistencies

   • Use compareDatabases() to identify differences
   • Re-run migration if necessary
   • Check for duplicate or conflicting records

4. PlatformServiceMixin Issues

   • Ensure mixin is properly imported and used
   • Check that all database operations use mixin methods
   • Verify caching and error handling work correctly

Debugging

// Debug migration process
import { logger } from '../utils/logger';

logger.debug('[Migration] Starting migration process...');
const result = await migrateAll();
logger.debug('[Migration] Migration completed:', result);

Benefits of PlatformServiceMixin Approach

1. Centralized Access: Single point of control for all database operations
2. Caching: Built-in caching for performance optimization
3. Type Safety: Enhanced TypeScript integration
4. Error Handling: Consistent error handling across components
5. Code Reduction: Up to 80% reduction in database boilerplate
6. Maintainability: Single source of truth for database patterns

Migration Status Checklist

✅ Completed

• PlatformServiceMixin implementation
• SQLite database service
• Migration tools
• Settings migration
• Account migration
• ActiveDid migration

🔄 In Progress

• Contact migration
• DatabaseUtil to PlatformServiceMixin migration
• File-by-file migration

❌ Not Started

• Legacy Dexie removal
• Final cleanup and validation

---

Author: Matthew Raymer Created: 2025-07-05 Status: Active Migration Phase Last Updated: 2025-07-05 Note: Migration fence now implemented through PlatformServiceMixin instead of USE_DEXIE_DB constant