diff --git a/doc/activeDid-migration-plan.md b/doc/activeDid-migration-plan.md index 9329a6f51a..00803923d1 100644 --- a/doc/activeDid-migration-plan.md +++ b/doc/activeDid-migration-plan.md @@ -1,8 +1,8 @@ # ActiveDid Migration Plan - Separate Table Architecture **Author**: Matthew Raymer -**Date**: 2025-08-29T08:03Z -**Status**: 🎯 **PLANNING** - Active migration planning phase +**Date**: 2025-08-29T15:00Z +**Status**: 🎯 **IMPLEMENTATION READY** - Streamlined for existing migration service ## Objective @@ -12,15 +12,14 @@ and separate identity selection from user preferences. ## Result -This document serves as the comprehensive planning and implementation -guide for the ActiveDid migration with enhanced data integrity and -rollback capabilities. +This document serves as the focused implementation guide for the ActiveDid +migration, integrated with the existing IndexedDB migration service. ## Use/Run Reference this document during implementation to ensure all migration -steps are followed correctly and all stakeholders are aligned on the -approach. +steps are followed correctly. **Critical**: This plan integrates with the +existing `indexedDBMigrationService.ts` and maintains backward compatibility. ## Context & Scope @@ -28,9 +27,7 @@ approach. - Database schema modification for active_identity table with proper constraints - Migration of existing activeDid data with validation - Updates to PlatformServiceMixin API layer - - Type definition updates - - Testing across all platforms - - Comprehensive rollback procedures + - Integration with existing IndexedDB migration service - **Out of scope**: - Changes to user interface for identity selection - Modifications to identity creation logic @@ -41,51 +38,47 @@ approach. - **OS/Runtime**: All platforms (Web, Electron, iOS, Android) - **Versions/Builds**: Current development branch, SQLite database -- **Services/Endpoints**: Local database, PlatformServiceMixin +- **Services/Endpoints**: Local database, PlatformServiceMixin, indexedDBMigrationService - **Auth mode**: Existing authentication system unchanged ## Architecture / Process Overview -The migration follows a phased approach to minimize risk and ensure -data integrity with enhanced validation and rollback capabilities: +The migration integrates with the existing IndexedDB migration service: ```mermaid flowchart TD A[Current State
activeDid in settings] --> B[Phase 1: Schema Creation
Add active_identity table with constraints] B --> C[Phase 2: Data Migration
Copy activeDid data with validation] C --> D[Phase 3: API Updates
Update PlatformServiceMixin methods] - D --> E[Phase 4: Cleanup
Remove activeDid from settings] - E --> F[Final State
Separate active_identity table] + D --> E[Final State
Separate active_identity table + dual-write] - G[Enhanced Rollback Plan
Schema and data rollback] --> H[Data Validation
Verify integrity at each step] - H --> I[Platform Testing
Test all platforms] - I --> J[Production Deployment
Gradual rollout with monitoring] - - K[Foreign Key Constraints
Prevent future corruption] --> L[Performance Optimization
Proper indexing] - L --> M[Error Recovery
Graceful failure handling] + F[Existing IndexedDB Migration] --> G[Enhanced with active_identity support] + G --> H[Maintains backward compatibility] + H --> I[Preserves existing migration paths] ``` -## Interfaces & Contracts +## Current Codebase Assessment -### Database Schema Changes +### ✅ What's Already Implemented -| Table | Current Schema | New Schema | Migration Required | -|-------|----------------|------------|-------------------| -| `settings` | `activeDid TEXT` | Field removed | Yes - data migration | -| `active_identity` | Does not exist | New table with `activeDid TEXT` + constraints | Yes - table creation | +- **Database Schema**: `activeDid` field exists in `settings` table (`src/db-sql/migration.ts:67`) +- **Constants**: `MASTER_SETTINGS_KEY = "1"` is properly defined (`src/db/tables/settings.ts:88`) +- **Types**: Settings type includes `activeDid?: string` (`src/db/tables/settings.ts:25`) +- **Migration Infrastructure**: SQLite migration system exists (`src/db-sql/migration.ts:31`) +- **IndexedDB Migration Service**: Complete service exists (`src/services/indexedDBMigrationService.ts`) +- **PlatformServiceMixin**: Basic structure exists with `$updateActiveDid()` method -### Enhanced API Contract Changes +### ❌ What Needs Implementation -| Method | Current Behavior | New Behavior | Breaking Change | -|---------|------------------|--------------|-----------------| -| `$accountSettings()` | Returns settings with activeDid | Returns settings with activeDid from new table | No - backward compatible | -| `$saveSettings()` | Updates settings.activeDid | Updates active_identity.activeDid | Yes - requires updates | -| `$updateActiveDid()` | Updates internal tracking | Updates active_identity table | Yes - requires updates | -| `$getActiveIdentity()` | Does not exist | New method for active identity management | No - new functionality | +- **Missing Table**: `active_identity` table doesn't exist in current schema +- **Missing API Methods**: Core PlatformServiceMixin methods need implementation +- **Missing Types**: `ActiveIdentity` interface needs creation -## Repro: End-to-End Procedure +## Required Changes -### Phase 1: Enhanced Schema Creation via migration.ts +### **1. Database Schema via migration.ts** + +Add migration 003 to existing MIGRATIONS array: ```typescript // Add to MIGRATIONS array in src/db-sql/migration.ts @@ -110,88 +103,54 @@ flowchart TD }, ``` -### Phase 2: Enhanced Data Migration with Validation +### **2. Type Definitions** + +Create ActiveIdentity interface in `src/db/tables/settings.ts`: ```typescript -// Enhanced migration function with comprehensive validation -async function migrateActiveDidToSeparateTable(): Promise { - const result: MigrationResult = { - success: false, - errors: [], - warnings: [], - dataMigrated: 0 - }; - - try { - // 1. Get current activeDid from settings (legacy approach) - const currentSettings = await retrieveSettingsForDefaultAccount(); - const activeDid = currentSettings.activeDid; - - if (!activeDid) { - result.warnings.push("No activeDid found in current settings"); - return result; - } - - // 2. Validate activeDid exists in accounts table - const accountExists = await dbQuery( - "SELECT did FROM accounts WHERE did = ?", - [activeDid] - ); - - if (!accountExists?.values?.length) { - result.errors.push(`ActiveDid ${activeDid} not found in accounts table - data corruption detected`); - return result; - } - - // 3. Update active_identity table (new system) - await dbExec( - "UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1", - [activeDid] - ); - - // 4. Ensure legacy settings.activeDid stays in sync (backward compatibility) - // This maintains compatibility with IndexedDB migration service - await dbExec( - "UPDATE settings SET activeDid = ? WHERE id = ?", - [activeDid, MASTER_SETTINGS_KEY] - ); - - dataMigrated = 1; - result.warnings.push(`Successfully migrated activeDid: ${activeDid}`); - - } catch (error) { - result.errors.push(`Migration failed: ${error}`); - logger.error("[ActiveDid Migration] Critical error during migration:", error); - } - - return result; +// Add to src/db/tables/settings.ts +export interface ActiveIdentity { + id: number; + activeDid: string; + lastUpdated: string; } ``` -### Phase 3: Focused API Updates with Dual-Write Pattern +### **3. PlatformServiceMixin Methods** + +Implement required methods in `src/utils/PlatformServiceMixin.ts`: ```typescript -// Updated PlatformServiceMixin method - maintains backward compatibility +// Add to PlatformServiceMixin methods section +async $getActiveIdentity(): Promise { + try { + const result = await this.$dbQuery( + "SELECT id, activeDid, lastUpdated FROM active_identity WHERE id = 1" + ); + + if (result?.values?.length) { + const [id, activeDid, lastUpdated] = result.values[0]; + return { id: id as number, activeDid: activeDid as string, lastUpdated: lastUpdated as string }; + } + + // Return default if no record exists + return { id: 1, activeDid: '', lastUpdated: new Date().toISOString() }; + } catch (error) { + logger.error("[PlatformServiceMixin] Error getting active identity:", error); + throw error; + } +} + async $accountSettings(did?: string, defaults: Settings = {}): Promise { try { - // Get settings without activeDid (unchanged logic) - const settings = await this._getSettingsWithoutActiveDid(); + // Get settings without activeDid + const settings = await this.$getSettings(MASTER_SETTINGS_KEY, defaults); if (!settings) { return defaults; } - // Get activeDid from new table (new logic) - const activeIdentity = await this._getActiveIdentity(); - - // Return combined result (maintains backward compatibility) - return { ...settings, activeDid: activeIdentity.activeDid }; - } catch (error) { - logger.error("[Settings Trace] ❌ Error in $accountSettings:", error); - return defaults; - } -} - + // Get activeDid from new table> // Enhanced update activeDid method with dual-write pattern async $updateActiveDid(newDid: string | null): Promise { try { @@ -200,7 +159,7 @@ async $updateActiveDid(newDid: string | null): Promise { await this.$dbExec( "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1" ); - + // Keep legacy field in sync (backward compatibility) await this.$dbExec( "UPDATE settings SET activeDid = '' WHERE id = ?", @@ -223,7 +182,7 @@ async $updateActiveDid(newDid: string | null): Promise { "UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1", [newDid] ); - + // Keep legacy field in sync (backward compatibility) await this.$dbExec( "UPDATE settings SET activeDid = ? WHERE id = ?", @@ -231,7 +190,7 @@ async $updateActiveDid(newDid: string | null): Promise { ); } - // Update internal tracking + // Update internal tracking (existing functionality) await this._updateInternalActiveDid(newDid); return true; } catch (error) { @@ -241,286 +200,14 @@ async $updateActiveDid(newDid: string | null): Promise { } ``` -### **Master Settings Functions Implementation Strategy** +### **4. Integration with Existing IndexedDB Migration Service** -#### **1. Update `retrieveSettingsForDefaultAccount()`** +The existing `indexedDBMigrationService.ts` already handles activeDid migration +from Dexie to SQLite. This plan adds the separate table architecture while +maintaining compatibility. -```typescript -// Enhanced implementation with active_identity table integration -export async function retrieveSettingsForDefaultAccount(): Promise { - const platform = PlatformServiceFactory.getInstance(); - - // Get settings without activeDid - const sql = "SELECT id, accountDid, apiServer, filterFeedByNearby, filterFeedByVisible, " + - "finishedOnboarding, firstName, hideRegisterPromptOnNewContact, isRegistered, " + - "lastName, lastAckedOfferToUserJwtId, lastAckedOfferToUserProjectsJwtId, " + - "lastNotifiedClaimId, lastViewedClaimId, notifyingNewActivityTime, " + - "notifyingReminderMessage, notifyingReminderTime, partnerApiServer, " + - "passkeyExpirationMinutes, profileImageUrl, searchBoxes, showContactGivesInline, " + - "showGeneralAdvanced, showShortcutBvc, vapid, warnIfProdServer, warnIfTestServer, " + - "webPushServer FROM settings WHERE id = ?"; - - const result = await platform.dbQuery(sql, [MASTER_SETTINGS_KEY]); - - if (!result) { - return DEFAULT_SETTINGS; - } else { - const settings = mapColumnsToValues(result.columns, result.values)[0] as Settings; - - // Handle JSON parsing - if (settings.searchBoxes) { - settings.searchBoxes = JSON.parse(settings.searchBoxes); - } - - // Get activeDid from separate table - const activeIdentityResult = await platform.dbQuery( - "SELECT activeDid FROM active_identity WHERE id = 1" - ); - - if (activeIdentityResult?.values?.length) { - const activeDid = activeIdentityResult.values[0][0] as string; - if (activeDid) { - // Validate activeDid exists in accounts - const accountExists = await platform.dbQuery( - "SELECT did FROM accounts WHERE did = ?", - [activeDid] - ); - - if (accountExists?.values?.length) { - settings.activeDid = activeDid; - } else { - logger.warn(`[databaseUtil] ActiveDid ${activeDid} not found in accounts, clearing`); - // Clear corrupted activeDid - await platform.dbExec( - "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1" - ); - } - } - } - - return settings; - } -} -``` - -#### **2. Update `$getMergedSettings()` Method** - -```typescript -// Enhanced implementation with active_identity table integration -async $getMergedSettings(defaultKey: string, accountDid?: string, defaultFallback: Settings = {}): Promise { - try { - // Get default settings (now without activeDid) - const defaultSettings = await this.$getSettings(defaultKey, defaultFallback); - - // If no account DID, return defaults with activeDid from separate table - if (!accountDid) { - if (defaultSettings) { - // Get activeDid from separate table - const activeIdentityResult = await this.$dbQuery( - "SELECT activeDid FROM active_identity WHERE id = 1" - ); - - if (activeIdentityResult?.values?.length) { - const activeDid = activeIdentityResult.values[0][0] as string; - if (activeDid) { - // Validate activeDid exists in accounts - const accountExists = await this.$dbQuery( - "SELECT did FROM accounts WHERE did = ?", - [activeDid] - ); - - if (accountExists?.values?.length) { - defaultSettings.activeDid = activeDid; - } else { - logger.warn(`[Settings Trace] ActiveDid ${activeDid} not found in accounts, clearing`); - // Clear corrupted activeDid - await this.$dbExec( - "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1" - ); - } - } - } - } - return defaultSettings || defaultFallback; - } - - // ... rest of existing implementation for account-specific settings - } catch (error) { - logger.error(`[Settings Trace] ❌ Failed to get merged settings:`, { defaultKey, accountDid, error }); - return defaultFallback; - } -} -``` - -## What Works (Evidence) - -- ✅ **Current activeDid storage** in settings table - - **Time**: 2025-08-29T08:03Z - - **Evidence**: `src/db-sql/migration.ts:67` - activeDid field exists in initial migration - - **Verify at**: Current database schema and Settings type definition - -- ✅ **PlatformServiceMixin integration** with activeDid - - **Time**: 2025-08-29T08:03Z - - **Evidence**: `src/utils/PlatformServiceMixin.ts:108` - activeDid tracking - - **Verify at**: Component usage across all platforms - -- ✅ **Database migration infrastructure** exists - - **Time**: 2025-08-29T08:03Z - - **Evidence**: `src/db-sql/migration.ts:31` - migration system in place - - **Verify at**: Existing migration scripts and database versioning - -## What Doesn't (Evidence & Hypotheses) - -- ❌ **No separate active_identity table** exists - - **Time**: 2025-08-29T08:03Z - - **Evidence**: Database schema only shows settings table - - **Hypothesis**: Table needs to be created as part of migration - - **Next probe**: Add migration to existing MIGRATIONS array - -- ❌ **Data corruption issues** with orphaned activeDid references - - **Time**: 2025-08-29T08:03Z - - **Evidence**: `IdentitySwitcherView.vue:175` - `hasCorruptedIdentity` detection - - **Hypothesis**: Current schema allows activeDid to point to non-existent accounts - - **Next probe**: Implement foreign key constraints in new table - -## Risks, Limits, Assumptions - -- **Data Loss Risk**: Migration failure could lose activeDid values -- **Breaking Changes**: API updates required in PlatformServiceMixin -- **Rollback Complexity**: Schema changes make rollback difficult -- **Testing Overhead**: All platforms must be tested with new structure -- **Performance Impact**: Additional table join for activeDid retrieval -- **Migration Timing**: Must be coordinated with other database changes -- **Data Corruption**: Current system has documented corruption issues -- **Foreign Key Constraints**: New constraints may prevent some operations - -## Enhanced Rollback Strategy - -### **Schema Rollback** -```sql --- If migration fails, restore original schema -DROP TABLE IF EXISTS active_identity; - --- Restore activeDid field to settings table if needed -ALTER TABLE settings ADD COLUMN activeDid TEXT; -``` - -### **Data Rollback** -```typescript -// Rollback function to restore activeDid to settings table -async function rollbackActiveDidMigration(): Promise { - try { - // Get activeDid from active_identity table - const activeIdentityResult = await dbQuery( - "SELECT activeDid FROM active_identity WHERE id = 1" - ); - - if (activeIdentityResult?.values?.length) { - const activeDid = activeIdentityResult.values[0][0] as string; - - // Restore to settings table - await dbExec( - "UPDATE settings SET activeDid = ? WHERE id = ?", - [activeDid, MASTER_SETTINGS_KEY] - ); - - return true; - } - - return false; - } catch (error) { - logger.error("[Rollback] Failed to restore activeDid:", error); - return false; - } -} -``` - -### **Rollback Triggers** -- Migration validation fails -- Data integrity checks fail -- Performance regression detected -- User reports data loss -- Cross-platform inconsistencies found - -## Next Steps - -| Owner | Task | Exit Criteria | Target Date (UTC) | -|-------|------|---------------|-------------------| -| Development Team | Add migration to existing MIGRATIONS array | Migration script integrated with existing system | 2025-08-30 | -| Development Team | Update type definitions | Settings type updated, ActiveIdentity type created | 2025-08-30 | -| Development Team | Update PlatformServiceMixin | Core methods updated and tested | 2025-08-31 | -| Development Team | Implement foreign key constraints | Schema validation prevents corruption | 2025-08-31 | -| QA Team | Platform testing | All platforms tested and verified | 2025-09-01 | -| Development Team | Deploy migration | Production deployment successful | 2025-09-02 | - -## References - -- [Database Migration Guide](./database-migration-guide.md) -- [Dexie to SQLite Mapping](./dexie-to-sqlite-mapping.md) -- [PlatformServiceMixin Documentation](./component-communication-guide.md) -- [Migration Templates](./migration-templates/) - -## Competence Hooks - -- *Why this works*: Separates concerns between identity selection and - user preferences, prevents data corruption with foreign key constraints, - centralizes identity management through API layer -- *Common pitfalls*: Forgetting to implement foreign key constraints, not - testing rollback scenarios, missing data validation during migration, - over-engineering component updates when API layer handles everything -- *Next skill unlock*: Advanced database schema design with constraints, - migration planning with rollback strategies -- *Teach-back*: Explain the four-phase migration approach and why each - phase is necessary, especially the foreign key constraints - -## Collaboration Hooks - -- **Sign-off checklist**: - - [ ] Migration script integrated with existing MIGRATIONS array - - [ ] Foreign key constraints implemented and tested - - [ ] PlatformServiceMixin updated and tested - - [ ] Rollback procedures validated - - [ ] Performance impact assessed - - [ ] All stakeholders approve deployment timeline - -## Assumptions & Limits - -- Current activeDid values are valid and should be preserved -- All platforms can handle the additional database table -- Migration can be completed without user downtime -- Rollback to previous schema is acceptable if needed -- Performance impact of additional table join is minimal -- Foreign key constraints will prevent future corruption -- API layer updates will handle component compatibility - -## What Needs to Change - -### **1. Database Schema via migration.ts** -- Add migration to existing MIGRATIONS array in `src/db-sql/migration.ts` -- Create `active_identity` table with foreign key constraints -- Add performance indexes -- **Keep `activeDid` field in `settings` table temporarily** for backward compatibility -- **Preserve `MASTER_SETTINGS_KEY = "1"`** for legacy migration support - -### **2. PlatformServiceMixin Methods** -- `$accountSettings()` - integrate with new table while maintaining backward compatibility -- `$saveSettings()` - handle activeDid in new table, sync with legacy field -- `$updateActiveDid()` - validate and update new table, sync with legacy field -- `$getActiveIdentity()` - new method for identity management - -### **3. Master Settings Functions** -- `retrieveSettingsForDefaultAccount()` - integrate with new table while preserving legacy support -- `$getMergedSettings()` - integrate with new table while preserving legacy support - -### **4. Type Definitions** -- **Keep `activeDid` in Settings type temporarily** for backward compatibility -- Create ActiveIdentity type for new table -- Update related interfaces - -### **5. Legacy Compatibility** -- **Preserve `MASTER_SETTINGS_KEY = "1"`** for IndexedDB migration service -- **Maintain dual-write pattern** during transition period -- **Ensure legacy clients can still migrate** from Dexie to SQLite +**No changes needed** to the existing migration service - it will continue to +work with the dual-write pattern. ## What Doesn't Need to Change @@ -529,48 +216,91 @@ async function rollbackActiveDidMigration(): Promise { - **User interface** - No changes to identity selection UI - **Authentication flow** - Existing system unchanged - **Component logic** - All activeDid handling through API methods -- **Migration system** - Use existing migration.ts approach, not separate files -- **IndexedDB migration service** - Must continue working for legacy clients +- **Migration system** - Use existing migration.ts approach +- **IndexedDB migration service** - Continues working unchanged +- **Existing database operations** - All current queries continue working -## Enhanced Architecture: Dual-Write Pattern +## Implementation Steps -### **Phase 1: Add New Table (Current)** -```typescript -// Create active_identity table -// Keep existing settings.activeDid for backward compatibility -// Use dual-write pattern during transition +### **Step 1: Add Migration** + +- Add migration 003 to `MIGRATIONS` array in `src/db-sql/migration.ts` +- Deploy migration to create `active_identity` table + +### **Step 2: Implement API Methods** + +- Create `ActiveIdentity` interface in `src/db/tables/settings.ts` +- Implement `$getActiveIdentity()` method in PlatformServiceMixin +- Implement `$accountSettings()` method in PlatformServiceMixin +- Enhance `$updateActiveDid()` method with dual-write pattern + +### **Step 3: Test Integration** + +- Test new methods with existing components +- Verify dual-write pattern works correctly +- Validate backward compatibility + +## Backward Compatibility + +### **Critical Requirements** + +- **IndexedDB Migration Service**: Must continue working unchanged +- **MASTER_SETTINGS_KEY = "1"**: Must be preserved for legacy support +- **Dual-Write Pattern**: Ensures both old and new systems stay in sync +- **Existing Queries**: All current database operations continue working + +### **Migration Strategy** + +- **Phase 1**: Add new table alongside existing system +- **Phase 2**: Use dual-write pattern during transition +- **Phase 3**: Future cleanup (not in current scope) + +## Rollback Strategy + +If migration fails, the existing `activeDid` field in settings table remains functional: + +```sql +-- Rollback: Remove new table +DROP TABLE IF EXISTS active_identity; ``` -### **Phase 2: Dual-Write Pattern** -```typescript -// When updating activeDid: -// 1. Update active_identity table (new system) -// 2. Update settings.activeDid (legacy compatibility) -// 3. Ensure both stay in sync -``` +No data loss risk - the legacy field continues working unchanged. -### **Phase 3: Future Cleanup (Not in Current Scope)** -```typescript -// Eventually: -// 1. Remove activeDid from settings table -// 2. Deprecate MASTER_SETTINGS_KEY -// 3. Use pure accountDid IS NULL pattern -// 4. Update IndexedDB migration service -``` +## Success Criteria -## Backward Compatibility Requirements +- [ ] `active_identity` table created with proper constraints +- [ ] All new PlatformServiceMixin methods implemented and tested +- [ ] Dual-write pattern working correctly +- [ ] Existing IndexedDB migration service continues working +- [ ] No breaking changes to existing functionality +- [ ] All platforms tested and verified -### **Critical: IndexedDB Migration Service** -- **Must continue working** for users migrating from Dexie -- **Must recognize `id = "1"`** as master settings -- **Must preserve existing migration paths** +## Risks & Mitigation -### **Important: Legacy Database Operations** -- **Must continue working** for existing SQLite databases -- **Must handle both old and new patterns** -- **Must not break existing queries** +### **Low Risk: Migration Failure** -### **Desired: Cleaner Architecture** -- **New operations** use `accountDid IS NULL` pattern -- **Legacy operations** continue using `MASTER_SETTINGS_KEY` -- **Gradual migration** toward cleaner patterns \ No newline at end of file +- **Mitigation**: Rollback removes new table, legacy system continues working +- **Impact**: No data loss, no service interruption + +### **Low Risk: API Changes** + +- **Mitigation**: Dual-write pattern maintains backward compatibility +- **Impact**: Existing components continue working unchanged + +### **Low Risk: Performance Impact** + +- **Mitigation**: Proper indexing and minimal additional queries +- **Impact**: Negligible performance change + +## Summary + +This migration plan: + +1. **Adds new architecture** without breaking existing functionality +2. **Integrates seamlessly** with existing IndexedDB migration service +3. **Maintains full backward compatibility** during transition +4. **Requires minimal changes** to existing codebase +5. **Provides clear rollback path** if issues arise + +The plan focuses only on necessary changes while preserving all existing +functionality and migration paths.