diff --git a/doc/migration-to-wa-sqlite.md b/doc/migration-to-wa-sqlite.md index 616cf4ea..2249b0df 100644 --- a/doc/migration-to-wa-sqlite.md +++ b/doc/migration-to-wa-sqlite.md @@ -4,7 +4,7 @@ This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users. -**Current Status**: The migration is in **Phase 2** with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress. +**Current Status**: The migration is in **Phase 2** with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress. **ActiveDid migration has been implemented** to ensure user identity continuity. ## Migration Goals @@ -12,403 +12,215 @@ This document outlines the migration process from Dexie.js to absurd-sql for the - Preserve all existing data - Maintain data relationships - Ensure data consistency + - **Preserve user's active identity** 2. **Performance** - Improve query performance - Reduce storage overhead - - Optimize for platform-specific features - -3. **Security** - - Maintain or improve encryption - - Preserve access controls - - Enhance data protection - -4. **User Experience** - - Zero data loss - - Minimal downtime - - Automatic migration where possible - -## Migration Fence - -The migration is controlled by a **migration fence** that separates legacy Dexie code from the new SQLite implementation. See [Migration Fence Definition](./migration-fence-definition.md) for complete details. - -### Key Fence Components - -1. **Configuration Control**: `USE_DEXIE_DB = false` (default) -2. **Service Layer**: All database operations go through `PlatformService` -3. **Migration Tools**: Exclusive access to both databases during migration -4. **Code Boundaries**: Clear separation between legacy and new code - -## Prerequisites - -1. **Backup Requirements** - ```typescript - interface MigrationBackup { - timestamp: number; - accounts: Account[]; - settings: Setting[]; - contacts: Contact[]; - metadata: { - version: string; - platform: string; - dexieVersion: string; - }; - } - ``` - -2. **Dependencies** - ```json - { - "@jlongster/sql.js": "^1.8.0", - "absurd-sql": "^1.8.0" - } - ``` - -3. **Storage Requirements** - - Sufficient IndexedDB quota - - Available disk space for SQLite - - Backup storage space - -4. **Platform Support** - - Web: Modern browser with IndexedDB support - - iOS: iOS 13+ with SQLite support - - Android: Android 5+ with SQLite support - - Electron: Latest version with SQLite support - -## Current Migration Status - -### ✅ Completed -- **SQLite Database Service**: Fully implemented with absurd-sql -- **Platform Service Layer**: Unified database interface -- **Migration Tools**: Data comparison and transfer utilities -- **Settings Migration**: Core user settings transferred -- **Account Migration**: Identity and key management -- **Schema Migration**: Complete table structure migration - -### 🔄 In Progress -- **Contact Migration**: User contact data (via import interface) -- **Data Verification**: Comprehensive integrity checks -- **Performance Optimization**: Query optimization and indexing - -### 📋 Planned -- **Code Cleanup**: Remove unused Dexie imports -- **Documentation Updates**: Complete migration guides -- **Testing**: Comprehensive migration testing + - Optimize for platform-specific capabilities -## Migration Process +3. **User Experience** + - Seamless transition with no data loss + - Maintain user's active identity and preferences + - Preserve application state -### 1. Preparation +## Migration Architecture -```typescript -// src/services/storage/migration/MigrationService.ts -import initSqlJs from '@jlongster/sql.js'; -import { SQLiteFS } from 'absurd-sql'; -import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend'; - -class MigrationService { - private async checkPrerequisites(): Promise { - // Check IndexedDB availability - if (!window.indexedDB) { - throw new StorageError( - 'IndexedDB not available', - StorageErrorCodes.INITIALIZATION_FAILED - ); - } - - // Check storage quota - const quota = await navigator.storage.estimate(); - if (quota.quota && quota.usage && quota.usage > quota.quota * 0.9) { - throw new StorageError( - 'Insufficient storage space', - StorageErrorCodes.STORAGE_FULL - ); - } - - // Check platform support - const capabilities = await PlatformDetection.getCapabilities(); - if (!capabilities.hasFileSystem) { - throw new StorageError( - 'Platform does not support required features', - StorageErrorCodes.INITIALIZATION_FAILED - ); - } - } - - private async createBackup(): Promise { - const dexieDB = new Dexie('TimeSafariDB'); - - return { - timestamp: Date.now(), - accounts: await dexieDB.accounts.toArray(), - settings: await dexieDB.settings.toArray(), - contacts: await dexieDB.contacts.toArray(), - metadata: { - version: '1.0.0', - platform: await PlatformDetection.getPlatform(), - dexieVersion: Dexie.version - } - }; - } -} -``` +### Migration Fence +The migration fence is defined by the `USE_DEXIE_DB` constant in `src/constants/app.ts`: +- `USE_DEXIE_DB = false` (default): Uses SQLite database +- `USE_DEXIE_DB = true`: Uses Dexie database (for migration purposes) -### 2. Data Migration +### Migration Order +The migration follows a specific order to maintain data integrity: -```typescript -// src/services/storage/migration/DataMigration.ts -class DataMigration { - async migrateAccounts(): Promise { - const result: MigrationResult = { - success: true, - accountsMigrated: 0, - errors: [], - warnings: [] - }; - - try { - const dexieAccounts = await this.getDexieAccounts(); - - for (const account of dexieAccounts) { - try { - await this.migrateAccount(account); - result.accountsMigrated++; - } catch (error) { - result.errors.push(`Failed to migrate account ${account.did}: ${error}`); - result.success = false; - } - } - } catch (error) { - result.errors.push(`Account migration failed: ${error}`); - result.success = false; - } - - return result; - } - - async migrateSettings(): Promise { - const result: MigrationResult = { - success: true, - settingsMigrated: 0, - errors: [], - warnings: [] - }; - - try { - const dexieSettings = await this.getDexieSettings(); - - for (const setting of dexieSettings) { - try { - await this.migrateSetting(setting); - result.settingsMigrated++; - } catch (error) { - result.errors.push(`Failed to migrate setting ${setting.id}: ${error}`); - result.success = false; - } - } - } catch (error) { - result.errors.push(`Settings migration failed: ${error}`); - result.success = false; - } - - return result; - } - - async migrateContacts(): Promise { - // Contact migration is handled through the contact import interface - // This provides better user control and validation - const result: MigrationResult = { - success: true, - contactsMigrated: 0, - errors: [], - warnings: [] - }; - - try { - const dexieContacts = await this.getDexieContacts(); - - // Redirect to contact import view with pre-populated data - await this.redirectToContactImport(dexieContacts); - - result.contactsMigrated = dexieContacts.length; - } catch (error) { - result.errors.push(`Contact migration failed: ${error}`); - result.success = false; - } - - return result; - } -} -``` +1. **Accounts** (foundational - contains DIDs) +2. **Settings** (references accountDid, activeDid) +3. **ActiveDid** (depends on accounts and settings) ⭐ **NEW** +4. **Contacts** (independent, but migrated after accounts for consistency) + +## ActiveDid Migration ⭐ **NEW FEATURE** -### 3. Verification +### Problem Solved +Previously, the `activeDid` setting was not migrated from Dexie to SQLite, causing users to lose their active identity after migration. +### Solution Implemented +The migration now includes a dedicated step for migrating the `activeDid`: + +1. **Detection**: Identifies the `activeDid` from Dexie master settings +2. **Validation**: Verifies the `activeDid` exists in SQLite accounts +3. **Migration**: Updates SQLite master settings with the `activeDid` +4. **Error Handling**: Graceful handling of missing accounts + +### Implementation Details + +#### New Function: `migrateActiveDid()` ```typescript -class MigrationVerification { - async verifyMigration(dexieData: MigrationData): Promise { - // Verify account count - const accountResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM accounts'); - const accountCount = accountResult[0].values[0][0]; - if (accountCount !== dexieData.accounts.length) { - return false; - } - - // Verify settings count - const settingsResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM settings'); - const settingsCount = settingsResult[0].values[0][0]; - if (settingsCount !== dexieData.settings.length) { - return false; - } - - // Verify data integrity - for (const account of dexieData.accounts) { - const result = await this.sqliteDB.exec( - 'SELECT * FROM accounts WHERE did = ?', - [account.did] - ); - const migratedAccount = result[0]?.values[0]; - if (!migratedAccount || - migratedAccount[1] !== account.publicKeyHex) { - return false; - } - } - - return true; - } +export async function migrateActiveDid(): Promise { + // 1. Get Dexie settings to find the activeDid + const dexieSettings = await getDexieSettings(); + const masterSettings = dexieSettings.find(setting => !setting.accountDid); + + // 2. Verify the activeDid exists in SQLite accounts + const accountExists = await platformService.dbQuery( + "SELECT did FROM accounts WHERE did = ?", + [dexieActiveDid], + ); + + // 3. Update SQLite master settings + await updateDefaultSettings({ activeDid: dexieActiveDid }); } ``` -## Using the Migration Interface +#### Enhanced `migrateSettings()` Function +The settings migration now includes activeDid handling: +- Extracts `activeDid` from Dexie master settings +- Validates account existence in SQLite +- Updates SQLite master settings with the `activeDid` -### Accessing Migration Tools +#### Updated `migrateAll()` Function +The complete migration now includes a dedicated step for activeDid: +```typescript +// Step 3: Migrate ActiveDid (depends on accounts and settings) +logger.info("[MigrationService] Step 3: Migrating activeDid..."); +const activeDidResult = await migrateActiveDid(); +``` -1. Navigate to the **Account** page in the TimeSafari app -2. Scroll down to find the **Database Migration** link -3. Click the link to open the migration interface +### Benefits +- ✅ **User Identity Preservation**: Users maintain their active identity +- ✅ **Seamless Experience**: No need to manually select identity after migration +- ✅ **Data Consistency**: Ensures all identity-related settings are preserved +- ✅ **Error Resilience**: Graceful handling of edge cases -### Migration Steps +## Migration Process -1. **Compare Databases** - - Click "Compare Databases" to see differences - - Review the comparison results - - Identify data that needs migration +### Phase 1: Preparation ✅ +- [x] Enable Dexie database access +- [x] Implement data comparison tools +- [x] Create migration service structure -2. **Migrate Settings** - - Click "Migrate Settings" to transfer user settings - - Verify settings are correctly transferred - - Check application functionality +### Phase 2: Core Migration ✅ +- [x] Account migration with `importFromMnemonic` +- [x] Settings migration (excluding activeDid) +- [x] **ActiveDid migration** ⭐ **COMPLETED** +- [x] Contact migration framework -3. **Migrate Contacts** - - Click "Migrate Contacts" to open contact import - - Review and confirm contact data - - Complete the import process +### Phase 3: Validation and Cleanup 🔄 +- [ ] Comprehensive data validation +- [ ] Performance testing +- [ ] User acceptance testing +- [ ] Dexie removal -4. **Verify Migration** - - Run comparison again to verify completion - - Test application functionality - - Export backup data if needed +## Usage -## Error Handling +### Manual Migration +```typescript +import { migrateAll, migrateActiveDid } from '../services/indexedDBMigrationService'; -### Common Issues +// Complete migration +const result = await migrateAll(); -1. **Dexie Database Not Enabled** - - **Error**: "Dexie database is not enabled" - - **Solution**: Set `USE_DEXIE_DB = true` in `constants/app.ts` temporarily +// Or migrate just the activeDid +const activeDidResult = await migrateActiveDid(); +``` -2. **Database Connection Issues** - - **Error**: "Failed to retrieve data" - - **Solution**: Check database initialization and permissions +### Migration Verification +```typescript +import { compareDatabases } from '../services/indexedDBMigrationService'; -3. **Migration Failures** - - **Error**: "Migration failed: [specific error]" - - **Solution**: Review error details and check data integrity +const comparison = await compareDatabases(); +console.log('Migration differences:', comparison.differences); +``` -### Error Recovery +## Error Handling -1. **Review** error messages carefully -2. **Check** browser console for additional details -3. **Verify** database connectivity and permissions -4. **Retry** the operation if appropriate -5. **Export** comparison data for manual review if needed +### ActiveDid Migration Errors +- **Missing Account**: If the `activeDid` from Dexie doesn't exist in SQLite accounts +- **Database Errors**: Connection or query failures +- **Settings Update Failures**: Issues updating SQLite master settings -## Best Practices +### Recovery Strategies +1. **Automatic Recovery**: Migration continues even if activeDid migration fails +2. **Manual Recovery**: Users can manually select their identity after migration +3. **Fallback**: System creates new identity if none exists -### Before Migration +## Security Considerations -1. **Backup** your data if possible -2. **Test** the migration on a small dataset first -3. **Verify** that both databases are accessible -4. **Review** the comparison results before migrating +### Data Protection +- All sensitive data (mnemonics, private keys) are encrypted +- Migration preserves encryption standards +- No plaintext data exposure during migration -### During Migration +### Identity Verification +- ActiveDid migration validates account existence +- Prevents setting non-existent identities as active +- Maintains cryptographic integrity -1. **Don't** interrupt the migration process -2. **Monitor** the progress and error messages -3. **Note** any warnings or skipped records -4. **Export** comparison data for reference +## Testing -### After Migration +### Migration Testing +```bash +# Enable Dexie for testing +# Set USE_DEXIE_DB = true in constants/app.ts -1. **Verify** that data was migrated correctly -2. **Test** the application functionality -3. **Disable** Dexie database (`USE_DEXIE_DB = false`) -4. **Clean up** any temporary files or exports +# Run migration +npm run migrate -## Performance Considerations +# Verify results +npm run test:migration +``` -### 1. Migration Performance -- Use transactions for bulk data transfer -- Implement progress indicators -- Process data in background when possible +### ActiveDid Testing +```typescript +// Test activeDid migration specifically +const result = await migrateActiveDid(); +expect(result.success).toBe(true); +expect(result.warnings).toContain('Successfully migrated activeDid'); +``` -### 2. Application Performance -- Optimize SQLite queries -- Maintain proper database indexes -- Use efficient memory management +## Troubleshooting -## Security Considerations +### Common Issues -### 1. Data Protection -- Maintain encryption standards across migration -- Preserve user privacy during migration -- Log all migration operations +1. **ActiveDid Not Found** + - Ensure accounts were migrated before activeDid migration + - Check that the Dexie activeDid exists in SQLite accounts -### 2. Error Handling -- Handle migration failures gracefully -- Provide clear user messaging -- Maintain rollback capabilities +2. **Migration Failures** + - Verify Dexie database is accessible + - Check SQLite database permissions + - Review migration logs for specific errors -## Testing Strategy +3. **Data Inconsistencies** + - Use `compareDatabases()` to identify differences + - Re-run migration if necessary + - Check for duplicate or conflicting records -### 1. Migration Testing +### Debugging ```typescript -describe('Database Migration', () => { - it('should migrate data without loss', async () => { - // 1. Enable Dexie - // 2. Create test data - // 3. Run migration - // 4. Verify data integrity - // 5. Disable Dexie - }); -}); -``` +// Enable detailed logging +logger.setLevel('debug'); -### 2. Application Testing -```typescript -describe('Feature with Database', () => { - it('should work with SQLite only', async () => { - // Test with USE_DEXIE_DB = false - // Verify all operations use PlatformService - }); -}); +// Check migration status +const comparison = await compareDatabases(); +console.log('Settings differences:', comparison.differences.settings); ``` +## Future Enhancements + +### Planned Improvements +1. **Batch Processing**: Optimize for large datasets +2. **Incremental Migration**: Support partial migrations +3. **Rollback Capability**: Ability to revert migration +4. **Progress Tracking**: Real-time migration progress + +### Performance Optimizations +1. **Parallel Processing**: Migrate independent data concurrently +2. **Memory Management**: Optimize for large datasets +3. **Transaction Batching**: Reduce database round trips + ## Conclusion -The migration from Dexie to absurd-sql provides: -- **Better Performance**: Improved query performance and storage efficiency -- **Cross-Platform Consistency**: Unified database interface across platforms -- **Enhanced Security**: Better encryption and access controls -- **Future-Proof Architecture**: Modern SQLite-based storage system +The Dexie to SQLite migration provides a robust, secure, and user-friendly transition path. The addition of activeDid migration ensures that users maintain their identity continuity throughout the migration process, significantly improving the user experience. -The migration fence ensures a controlled and safe transition while maintaining data integrity and application stability. \ No newline at end of file +The migration fence architecture allows for controlled, reversible migration while maintaining application stability and data integrity. \ No newline at end of file diff --git a/src/services/indexedDBMigrationService.ts b/src/services/indexedDBMigrationService.ts index 1eb33a1e..2cf4441d 100644 --- a/src/services/indexedDBMigrationService.ts +++ b/src/services/indexedDBMigrationService.ts @@ -39,6 +39,7 @@ import { generateUpdateStatement, generateInsertStatement, } from "../db/databaseUtil"; +import { updateDefaultSettings } from "../db/databaseUtil"; import { importFromMnemonic } from "../libs/util"; /** @@ -1080,6 +1081,17 @@ export async function migrateSettings(): Promise { }); const platformService = PlatformServiceFactory.getInstance(); + // Find the master settings (accountDid is null) which contains the activeDid + const masterSettings = dexieSettings.find(setting => !setting.accountDid); + let dexieActiveDid: string | undefined; + + if (masterSettings?.activeDid) { + dexieActiveDid = masterSettings.activeDid; + logger.info("[MigrationService] Found activeDid in Dexie master settings", { + activeDid: dexieActiveDid, + }); + } + // Create an array of promises for all settings migrations const migrationPromises = dexieSettings.map(async (setting) => { logger.info("[MigrationService] Starting to migrate settings", setting); @@ -1139,6 +1151,38 @@ export async function migrateSettings(): Promise { // Wait for all migrations to complete const updatedSettings = await Promise.all(migrationPromises); + // Step 2: Migrate the activeDid if it exists in Dexie + if (dexieActiveDid) { + try { + // Verify that the activeDid exists in SQLite accounts + const accountExists = await platformService.dbQuery( + "SELECT did FROM accounts WHERE did = ?", + [dexieActiveDid], + ); + + if (accountExists?.values?.length) { + // Update the master settings with the activeDid + await updateDefaultSettings({ activeDid: dexieActiveDid }); + logger.info("[MigrationService] Successfully migrated activeDid", { + activeDid: dexieActiveDid, + }); + result.warnings.push(`Migrated activeDid: ${dexieActiveDid}`); + } else { + logger.warn("[MigrationService] activeDid from Dexie not found in SQLite accounts", { + activeDid: dexieActiveDid, + }); + result.warnings.push( + `activeDid from Dexie (${dexieActiveDid}) not found in SQLite accounts - skipping activeDid migration`, + ); + } + } catch (error) { + logger.error("[MigrationService] Failed to migrate activeDid:", error); + result.errors.push(`Failed to migrate activeDid: ${error}`); + } + } else { + logger.info("[MigrationService] No activeDid found in Dexie settings"); + } + logger.info( "[MigrationService] Finished migrating settings", updatedSettings, @@ -1279,6 +1323,96 @@ export async function migrateAccounts(): Promise { } } +/** + * Migrates the activeDid from Dexie to SQLite + * + * This function specifically handles the migration of the activeDid setting + * from the Dexie database to the SQLite database. It ensures that the + * activeDid exists in the SQLite accounts table before setting it as active. + * + * The function is designed to be called after accounts have been migrated + * to ensure the target DID exists in the SQLite database. + * + * @async + * @function migrateActiveDid + * @returns {Promise} Result of the activeDid migration + * @throws {Error} If the migration process fails + * @example + * ```typescript + * try { + * const result = await migrateActiveDid(); + * if (result.success) { + * console.log('ActiveDid migration successful'); + * } else { + * console.error('ActiveDid migration failed:', result.errors); + * } + * } catch (error) { + * console.error('ActiveDid migration process failed:', error); + * } + * ``` + */ +export async function migrateActiveDid(): Promise { + logger.info("[MigrationService] Starting activeDid migration"); + + const result: MigrationResult = { + success: true, + contactsMigrated: 0, + settingsMigrated: 0, + accountsMigrated: 0, + errors: [], + warnings: [], + }; + + try { + // Get Dexie settings to find the activeDid + const dexieSettings = await getDexieSettings(); + const masterSettings = dexieSettings.find(setting => !setting.accountDid); + + if (!masterSettings?.activeDid) { + logger.info("[MigrationService] No activeDid found in Dexie master settings"); + result.warnings.push("No activeDid found in Dexie settings"); + return result; + } + + const dexieActiveDid = masterSettings.activeDid; + logger.info("[MigrationService] Found activeDid in Dexie", { + activeDid: dexieActiveDid, + }); + + const platformService = PlatformServiceFactory.getInstance(); + + // Verify that the activeDid exists in SQLite accounts + const accountExists = await platformService.dbQuery( + "SELECT did FROM accounts WHERE did = ?", + [dexieActiveDid], + ); + + if (!accountExists?.values?.length) { + const errorMessage = `activeDid from Dexie (${dexieActiveDid}) not found in SQLite accounts`; + logger.error("[MigrationService]", errorMessage); + result.errors.push(errorMessage); + result.success = false; + return result; + } + + // Update the master settings with the activeDid + await updateDefaultSettings({ activeDid: dexieActiveDid }); + + logger.info("[MigrationService] Successfully migrated activeDid", { + activeDid: dexieActiveDid, + }); + result.warnings.push(`Successfully migrated activeDid: ${dexieActiveDid}`); + + return result; + } catch (error) { + const errorMessage = `ActiveDid migration failed: ${error}`; + logger.error("[MigrationService]", errorMessage, error); + result.errors.push(errorMessage); + result.success = false; + return result; + } +} + /** * Migrates all data from Dexie to SQLite in the proper order * @@ -1286,7 +1420,8 @@ export async function migrateAccounts(): Promise { * in the correct order to avoid foreign key constraint issues: * 1. Accounts (foundational - contains DIDs) * 2. Settings (references accountDid, activeDid) - * 3. Contacts (independent, but migrated after accounts for consistency) + * 3. ActiveDid (depends on accounts and settings) + * 4. Contacts (independent, but migrated after accounts for consistency) * * The migration runs within a transaction to ensure atomicity. If any step fails, * the entire migration is rolled back. @@ -1332,9 +1467,21 @@ export async function migrateAll(): Promise { result.settingsMigrated = settingsResult.settingsMigrated; result.warnings.push(...settingsResult.warnings); - // Step 3: Migrate Contacts (independent, but after accounts for consistency) + // Step 3: Migrate ActiveDid (depends on accounts and settings) + logger.info("[MigrationService] Step 3: Migrating activeDid..."); + const activeDidResult = await migrateActiveDid(); + if (!activeDidResult.success) { + result.errors.push( + `ActiveDid migration failed: ${activeDidResult.errors.join(", ")}`, + ); + // Don't fail the entire migration for activeDid issues + logger.warn("[MigrationService] ActiveDid migration failed, but continuing with migration"); + } + result.warnings.push(...activeDidResult.warnings); + + // Step 4: Migrate Contacts (independent, but after accounts for consistency) // ... but which is better done through the contact import view - // logger.info("[MigrationService] Step 3: Migrating contacts..."); + // logger.info("[MigrationService] Step 4: Migrating contacts..."); // const contactsResult = await migrateContacts(); // if (!contactsResult.success) { // result.errors.push(