diff --git a/doc/activeDid-migration-plan.md b/doc/activeDid-migration-plan.md index 9329a6f5..00803923 100644 --- a/doc/activeDid-migration-plan.md +++ b/doc/activeDid-migration-plan.md @@ -1,8 +1,8 @@ # ActiveDid Migration Plan - Separate Table Architecture **Author**: Matthew Raymer -**Date**: 2025-08-29T08:03Z -**Status**: 🎯 **PLANNING** - Active migration planning phase +**Date**: 2025-08-29T15:00Z +**Status**: 🎯 **IMPLEMENTATION READY** - Streamlined for existing migration service ## Objective @@ -12,15 +12,14 @@ and separate identity selection from user preferences. ## Result -This document serves as the comprehensive planning and implementation -guide for the ActiveDid migration with enhanced data integrity and -rollback capabilities. +This document serves as the focused implementation guide for the ActiveDid +migration, integrated with the existing IndexedDB migration service. ## Use/Run Reference this document during implementation to ensure all migration -steps are followed correctly and all stakeholders are aligned on the -approach. +steps are followed correctly. **Critical**: This plan integrates with the +existing `indexedDBMigrationService.ts` and maintains backward compatibility. ## Context & Scope @@ -28,9 +27,7 @@ approach. - Database schema modification for active_identity table with proper constraints - Migration of existing activeDid data with validation - Updates to PlatformServiceMixin API layer - - Type definition updates - - Testing across all platforms - - Comprehensive rollback procedures + - Integration with existing IndexedDB migration service - **Out of scope**: - Changes to user interface for identity selection - Modifications to identity creation logic @@ -41,51 +38,47 @@ approach. - **OS/Runtime**: All platforms (Web, Electron, iOS, Android) - **Versions/Builds**: Current development branch, SQLite database -- **Services/Endpoints**: Local database, PlatformServiceMixin +- **Services/Endpoints**: Local database, PlatformServiceMixin, indexedDBMigrationService - **Auth mode**: Existing authentication system unchanged ## Architecture / Process Overview -The migration follows a phased approach to minimize risk and ensure -data integrity with enhanced validation and rollback capabilities: +The migration integrates with the existing IndexedDB migration service: ```mermaid flowchart TD A[Current State
activeDid in settings] --> B[Phase 1: Schema Creation
Add active_identity table with constraints] B --> C[Phase 2: Data Migration
Copy activeDid data with validation] C --> D[Phase 3: API Updates
Update PlatformServiceMixin methods] - D --> E[Phase 4: Cleanup
Remove activeDid from settings] - E --> F[Final State
Separate active_identity table] - - G[Enhanced Rollback Plan
Schema and data rollback] --> H[Data Validation
Verify integrity at each step] - H --> I[Platform Testing
Test all platforms] - I --> J[Production Deployment
Gradual rollout with monitoring] - - K[Foreign Key Constraints
Prevent future corruption] --> L[Performance Optimization
Proper indexing] - L --> M[Error Recovery
Graceful failure handling] + D --> E[Final State
Separate active_identity table + dual-write] + + F[Existing IndexedDB Migration] --> G[Enhanced with active_identity support] + G --> H[Maintains backward compatibility] + H --> I[Preserves existing migration paths] ``` -## Interfaces & Contracts +## Current Codebase Assessment + +### ✅ What's Already Implemented -### Database Schema Changes +- **Database Schema**: `activeDid` field exists in `settings` table (`src/db-sql/migration.ts:67`) +- **Constants**: `MASTER_SETTINGS_KEY = "1"` is properly defined (`src/db/tables/settings.ts:88`) +- **Types**: Settings type includes `activeDid?: string` (`src/db/tables/settings.ts:25`) +- **Migration Infrastructure**: SQLite migration system exists (`src/db-sql/migration.ts:31`) +- **IndexedDB Migration Service**: Complete service exists (`src/services/indexedDBMigrationService.ts`) +- **PlatformServiceMixin**: Basic structure exists with `$updateActiveDid()` method -| Table | Current Schema | New Schema | Migration Required | -|-------|----------------|------------|-------------------| -| `settings` | `activeDid TEXT` | Field removed | Yes - data migration | -| `active_identity` | Does not exist | New table with `activeDid TEXT` + constraints | Yes - table creation | +### ❌ What Needs Implementation -### Enhanced API Contract Changes +- **Missing Table**: `active_identity` table doesn't exist in current schema +- **Missing API Methods**: Core PlatformServiceMixin methods need implementation +- **Missing Types**: `ActiveIdentity` interface needs creation -| Method | Current Behavior | New Behavior | Breaking Change | -|---------|------------------|--------------|-----------------| -| `$accountSettings()` | Returns settings with activeDid | Returns settings with activeDid from new table | No - backward compatible | -| `$saveSettings()` | Updates settings.activeDid | Updates active_identity.activeDid | Yes - requires updates | -| `$updateActiveDid()` | Updates internal tracking | Updates active_identity table | Yes - requires updates | -| `$getActiveIdentity()` | Does not exist | New method for active identity management | No - new functionality | +## Required Changes -## Repro: End-to-End Procedure +### **1. Database Schema via migration.ts** -### Phase 1: Enhanced Schema Creation via migration.ts +Add migration 003 to existing MIGRATIONS array: ```typescript // Add to MIGRATIONS array in src/db-sql/migration.ts @@ -110,88 +103,54 @@ flowchart TD }, ``` -### Phase 2: Enhanced Data Migration with Validation +### **2. Type Definitions** + +Create ActiveIdentity interface in `src/db/tables/settings.ts`: ```typescript -// Enhanced migration function with comprehensive validation -async function migrateActiveDidToSeparateTable(): Promise { - const result: MigrationResult = { - success: false, - errors: [], - warnings: [], - dataMigrated: 0 - }; - - try { - // 1. Get current activeDid from settings (legacy approach) - const currentSettings = await retrieveSettingsForDefaultAccount(); - const activeDid = currentSettings.activeDid; +// Add to src/db/tables/settings.ts +export interface ActiveIdentity { + id: number; + activeDid: string; + lastUpdated: string; +} +``` - if (!activeDid) { - result.warnings.push("No activeDid found in current settings"); - return result; - } +### **3. PlatformServiceMixin Methods** - // 2. Validate activeDid exists in accounts table - const accountExists = await dbQuery( - "SELECT did FROM accounts WHERE did = ?", - [activeDid] - ); +Implement required methods in `src/utils/PlatformServiceMixin.ts`: - if (!accountExists?.values?.length) { - result.errors.push(`ActiveDid ${activeDid} not found in accounts table - data corruption detected`); - return result; - } - - // 3. Update active_identity table (new system) - await dbExec( - "UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1", - [activeDid] +```typescript +// Add to PlatformServiceMixin methods section +async $getActiveIdentity(): Promise { + try { + const result = await this.$dbQuery( + "SELECT id, activeDid, lastUpdated FROM active_identity WHERE id = 1" ); - // 4. Ensure legacy settings.activeDid stays in sync (backward compatibility) - // This maintains compatibility with IndexedDB migration service - await dbExec( - "UPDATE settings SET activeDid = ? WHERE id = ?", - [activeDid, MASTER_SETTINGS_KEY] - ); + if (result?.values?.length) { + const [id, activeDid, lastUpdated] = result.values[0]; + return { id: id as number, activeDid: activeDid as string, lastUpdated: lastUpdated as string }; + } - dataMigrated = 1; - result.warnings.push(`Successfully migrated activeDid: ${activeDid}`); - + // Return default if no record exists + return { id: 1, activeDid: '', lastUpdated: new Date().toISOString() }; } catch (error) { - result.errors.push(`Migration failed: ${error}`); - logger.error("[ActiveDid Migration] Critical error during migration:", error); + logger.error("[PlatformServiceMixin] Error getting active identity:", error); + throw error; } - - return result; } -``` -### Phase 3: Focused API Updates with Dual-Write Pattern - -```typescript -// Updated PlatformServiceMixin method - maintains backward compatibility async $accountSettings(did?: string, defaults: Settings = {}): Promise { try { - // Get settings without activeDid (unchanged logic) - const settings = await this._getSettingsWithoutActiveDid(); + // Get settings without activeDid + const settings = await this.$getSettings(MASTER_SETTINGS_KEY, defaults); if (!settings) { return defaults; } - // Get activeDid from new table (new logic) - const activeIdentity = await this._getActiveIdentity(); - - // Return combined result (maintains backward compatibility) - return { ...settings, activeDid: activeIdentity.activeDid }; - } catch (error) { - logger.error("[Settings Trace] ❌ Error in $accountSettings:", error); - return defaults; - } -} - + // Get activeDid from new table> // Enhanced update activeDid method with dual-write pattern async $updateActiveDid(newDid: string | null): Promise { try { @@ -200,7 +159,7 @@ async $updateActiveDid(newDid: string | null): Promise { await this.$dbExec( "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1" ); - + // Keep legacy field in sync (backward compatibility) await this.$dbExec( "UPDATE settings SET activeDid = '' WHERE id = ?", @@ -223,7 +182,7 @@ async $updateActiveDid(newDid: string | null): Promise { "UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1", [newDid] ); - + // Keep legacy field in sync (backward compatibility) await this.$dbExec( "UPDATE settings SET activeDid = ? WHERE id = ?", @@ -231,7 +190,7 @@ async $updateActiveDid(newDid: string | null): Promise { ); } - // Update internal tracking + // Update internal tracking (existing functionality) await this._updateInternalActiveDid(newDid); return true; } catch (error) { @@ -241,336 +200,107 @@ async $updateActiveDid(newDid: string | null): Promise { } ``` -### **Master Settings Functions Implementation Strategy** +### **4. Integration with Existing IndexedDB Migration Service** -#### **1. Update `retrieveSettingsForDefaultAccount()`** +The existing `indexedDBMigrationService.ts` already handles activeDid migration +from Dexie to SQLite. This plan adds the separate table architecture while +maintaining compatibility. -```typescript -// Enhanced implementation with active_identity table integration -export async function retrieveSettingsForDefaultAccount(): Promise { - const platform = PlatformServiceFactory.getInstance(); - - // Get settings without activeDid - const sql = "SELECT id, accountDid, apiServer, filterFeedByNearby, filterFeedByVisible, " + - "finishedOnboarding, firstName, hideRegisterPromptOnNewContact, isRegistered, " + - "lastName, lastAckedOfferToUserJwtId, lastAckedOfferToUserProjectsJwtId, " + - "lastNotifiedClaimId, lastViewedClaimId, notifyingNewActivityTime, " + - "notifyingReminderMessage, notifyingReminderTime, partnerApiServer, " + - "passkeyExpirationMinutes, profileImageUrl, searchBoxes, showContactGivesInline, " + - "showGeneralAdvanced, showShortcutBvc, vapid, warnIfProdServer, warnIfTestServer, " + - "webPushServer FROM settings WHERE id = ?"; - - const result = await platform.dbQuery(sql, [MASTER_SETTINGS_KEY]); - - if (!result) { - return DEFAULT_SETTINGS; - } else { - const settings = mapColumnsToValues(result.columns, result.values)[0] as Settings; - - // Handle JSON parsing - if (settings.searchBoxes) { - settings.searchBoxes = JSON.parse(settings.searchBoxes); - } +**No changes needed** to the existing migration service - it will continue to +work with the dual-write pattern. - // Get activeDid from separate table - const activeIdentityResult = await platform.dbQuery( - "SELECT activeDid FROM active_identity WHERE id = 1" - ); +## What Doesn't Need to Change - if (activeIdentityResult?.values?.length) { - const activeDid = activeIdentityResult.values[0][0] as string; - if (activeDid) { - // Validate activeDid exists in accounts - const accountExists = await platform.dbQuery( - "SELECT did FROM accounts WHERE did = ?", - [activeDid] - ); - - if (accountExists?.values?.length) { - settings.activeDid = activeDid; - } else { - logger.warn(`[databaseUtil] ActiveDid ${activeDid} not found in accounts, clearing`); - // Clear corrupted activeDid - await platform.dbExec( - "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1" - ); - } - } - } +- **All Vue components** - API layer handles migration transparently +- **Platform services** - Use PlatformServiceMixin, no direct access +- **User interface** - No changes to identity selection UI +- **Authentication flow** - Existing system unchanged +- **Component logic** - All activeDid handling through API methods +- **Migration system** - Use existing migration.ts approach +- **IndexedDB migration service** - Continues working unchanged +- **Existing database operations** - All current queries continue working - return settings; - } -} -``` +## Implementation Steps -#### **2. Update `$getMergedSettings()` Method** +### **Step 1: Add Migration** -```typescript -// Enhanced implementation with active_identity table integration -async $getMergedSettings(defaultKey: string, accountDid?: string, defaultFallback: Settings = {}): Promise { - try { - // Get default settings (now without activeDid) - const defaultSettings = await this.$getSettings(defaultKey, defaultFallback); - - // If no account DID, return defaults with activeDid from separate table - if (!accountDid) { - if (defaultSettings) { - // Get activeDid from separate table - const activeIdentityResult = await this.$dbQuery( - "SELECT activeDid FROM active_identity WHERE id = 1" - ); - - if (activeIdentityResult?.values?.length) { - const activeDid = activeIdentityResult.values[0][0] as string; - if (activeDid) { - // Validate activeDid exists in accounts - const accountExists = await this.$dbQuery( - "SELECT did FROM accounts WHERE did = ?", - [activeDid] - ); - - if (accountExists?.values?.length) { - defaultSettings.activeDid = activeDid; - } else { - logger.warn(`[Settings Trace] ActiveDid ${activeDid} not found in accounts, clearing`); - // Clear corrupted activeDid - await this.$dbExec( - "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1" - ); - } - } - } - } - return defaultSettings || defaultFallback; - } +- Add migration 003 to `MIGRATIONS` array in `src/db-sql/migration.ts` +- Deploy migration to create `active_identity` table - // ... rest of existing implementation for account-specific settings - } catch (error) { - logger.error(`[Settings Trace] ❌ Failed to get merged settings:`, { defaultKey, accountDid, error }); - return defaultFallback; - } -} -``` +### **Step 2: Implement API Methods** -## What Works (Evidence) +- Create `ActiveIdentity` interface in `src/db/tables/settings.ts` +- Implement `$getActiveIdentity()` method in PlatformServiceMixin +- Implement `$accountSettings()` method in PlatformServiceMixin +- Enhance `$updateActiveDid()` method with dual-write pattern -- ✅ **Current activeDid storage** in settings table - - **Time**: 2025-08-29T08:03Z - - **Evidence**: `src/db-sql/migration.ts:67` - activeDid field exists in initial migration - - **Verify at**: Current database schema and Settings type definition +### **Step 3: Test Integration** -- ✅ **PlatformServiceMixin integration** with activeDid - - **Time**: 2025-08-29T08:03Z - - **Evidence**: `src/utils/PlatformServiceMixin.ts:108` - activeDid tracking - - **Verify at**: Component usage across all platforms +- Test new methods with existing components +- Verify dual-write pattern works correctly +- Validate backward compatibility -- ✅ **Database migration infrastructure** exists - - **Time**: 2025-08-29T08:03Z - - **Evidence**: `src/db-sql/migration.ts:31` - migration system in place - - **Verify at**: Existing migration scripts and database versioning +## Backward Compatibility -## What Doesn't (Evidence & Hypotheses) +### **Critical Requirements** -- ❌ **No separate active_identity table** exists - - **Time**: 2025-08-29T08:03Z - - **Evidence**: Database schema only shows settings table - - **Hypothesis**: Table needs to be created as part of migration - - **Next probe**: Add migration to existing MIGRATIONS array +- **IndexedDB Migration Service**: Must continue working unchanged +- **MASTER_SETTINGS_KEY = "1"**: Must be preserved for legacy support +- **Dual-Write Pattern**: Ensures both old and new systems stay in sync +- **Existing Queries**: All current database operations continue working -- ❌ **Data corruption issues** with orphaned activeDid references - - **Time**: 2025-08-29T08:03Z - - **Evidence**: `IdentitySwitcherView.vue:175` - `hasCorruptedIdentity` detection - - **Hypothesis**: Current schema allows activeDid to point to non-existent accounts - - **Next probe**: Implement foreign key constraints in new table +### **Migration Strategy** -## Risks, Limits, Assumptions +- **Phase 1**: Add new table alongside existing system +- **Phase 2**: Use dual-write pattern during transition +- **Phase 3**: Future cleanup (not in current scope) -- **Data Loss Risk**: Migration failure could lose activeDid values -- **Breaking Changes**: API updates required in PlatformServiceMixin -- **Rollback Complexity**: Schema changes make rollback difficult -- **Testing Overhead**: All platforms must be tested with new structure -- **Performance Impact**: Additional table join for activeDid retrieval -- **Migration Timing**: Must be coordinated with other database changes -- **Data Corruption**: Current system has documented corruption issues -- **Foreign Key Constraints**: New constraints may prevent some operations +## Rollback Strategy -## Enhanced Rollback Strategy +If migration fails, the existing `activeDid` field in settings table remains functional: -### **Schema Rollback** ```sql --- If migration fails, restore original schema +-- Rollback: Remove new table DROP TABLE IF EXISTS active_identity; - --- Restore activeDid field to settings table if needed -ALTER TABLE settings ADD COLUMN activeDid TEXT; ``` -### **Data Rollback** -```typescript -// Rollback function to restore activeDid to settings table -async function rollbackActiveDidMigration(): Promise { - try { - // Get activeDid from active_identity table - const activeIdentityResult = await dbQuery( - "SELECT activeDid FROM active_identity WHERE id = 1" - ); - - if (activeIdentityResult?.values?.length) { - const activeDid = activeIdentityResult.values[0][0] as string; - - // Restore to settings table - await dbExec( - "UPDATE settings SET activeDid = ? WHERE id = ?", - [activeDid, MASTER_SETTINGS_KEY] - ); - - return true; - } - - return false; - } catch (error) { - logger.error("[Rollback] Failed to restore activeDid:", error); - return false; - } -} -``` +No data loss risk - the legacy field continues working unchanged. -### **Rollback Triggers** -- Migration validation fails -- Data integrity checks fail -- Performance regression detected -- User reports data loss -- Cross-platform inconsistencies found - -## Next Steps - -| Owner | Task | Exit Criteria | Target Date (UTC) | -|-------|------|---------------|-------------------| -| Development Team | Add migration to existing MIGRATIONS array | Migration script integrated with existing system | 2025-08-30 | -| Development Team | Update type definitions | Settings type updated, ActiveIdentity type created | 2025-08-30 | -| Development Team | Update PlatformServiceMixin | Core methods updated and tested | 2025-08-31 | -| Development Team | Implement foreign key constraints | Schema validation prevents corruption | 2025-08-31 | -| QA Team | Platform testing | All platforms tested and verified | 2025-09-01 | -| Development Team | Deploy migration | Production deployment successful | 2025-09-02 | - -## References - -- [Database Migration Guide](./database-migration-guide.md) -- [Dexie to SQLite Mapping](./dexie-to-sqlite-mapping.md) -- [PlatformServiceMixin Documentation](./component-communication-guide.md) -- [Migration Templates](./migration-templates/) - -## Competence Hooks - -- *Why this works*: Separates concerns between identity selection and - user preferences, prevents data corruption with foreign key constraints, - centralizes identity management through API layer -- *Common pitfalls*: Forgetting to implement foreign key constraints, not - testing rollback scenarios, missing data validation during migration, - over-engineering component updates when API layer handles everything -- *Next skill unlock*: Advanced database schema design with constraints, - migration planning with rollback strategies -- *Teach-back*: Explain the four-phase migration approach and why each - phase is necessary, especially the foreign key constraints - -## Collaboration Hooks - -- **Sign-off checklist**: - - [ ] Migration script integrated with existing MIGRATIONS array - - [ ] Foreign key constraints implemented and tested - - [ ] PlatformServiceMixin updated and tested - - [ ] Rollback procedures validated - - [ ] Performance impact assessed - - [ ] All stakeholders approve deployment timeline - -## Assumptions & Limits - -- Current activeDid values are valid and should be preserved -- All platforms can handle the additional database table -- Migration can be completed without user downtime -- Rollback to previous schema is acceptable if needed -- Performance impact of additional table join is minimal -- Foreign key constraints will prevent future corruption -- API layer updates will handle component compatibility - -## What Needs to Change +## Success Criteria -### **1. Database Schema via migration.ts** -- Add migration to existing MIGRATIONS array in `src/db-sql/migration.ts` -- Create `active_identity` table with foreign key constraints -- Add performance indexes -- **Keep `activeDid` field in `settings` table temporarily** for backward compatibility -- **Preserve `MASTER_SETTINGS_KEY = "1"`** for legacy migration support - -### **2. PlatformServiceMixin Methods** -- `$accountSettings()` - integrate with new table while maintaining backward compatibility -- `$saveSettings()` - handle activeDid in new table, sync with legacy field -- `$updateActiveDid()` - validate and update new table, sync with legacy field -- `$getActiveIdentity()` - new method for identity management - -### **3. Master Settings Functions** -- `retrieveSettingsForDefaultAccount()` - integrate with new table while preserving legacy support -- `$getMergedSettings()` - integrate with new table while preserving legacy support - -### **4. Type Definitions** -- **Keep `activeDid` in Settings type temporarily** for backward compatibility -- Create ActiveIdentity type for new table -- Update related interfaces - -### **5. Legacy Compatibility** -- **Preserve `MASTER_SETTINGS_KEY = "1"`** for IndexedDB migration service -- **Maintain dual-write pattern** during transition period -- **Ensure legacy clients can still migrate** from Dexie to SQLite +- [ ] `active_identity` table created with proper constraints +- [ ] All new PlatformServiceMixin methods implemented and tested +- [ ] Dual-write pattern working correctly +- [ ] Existing IndexedDB migration service continues working +- [ ] No breaking changes to existing functionality +- [ ] All platforms tested and verified -## What Doesn't Need to Change +## Risks & Mitigation -- **All Vue components** - API layer handles migration transparently -- **Platform services** - Use PlatformServiceMixin, no direct access -- **User interface** - No changes to identity selection UI -- **Authentication flow** - Existing system unchanged -- **Component logic** - All activeDid handling through API methods -- **Migration system** - Use existing migration.ts approach, not separate files -- **IndexedDB migration service** - Must continue working for legacy clients +### **Low Risk: Migration Failure** -## Enhanced Architecture: Dual-Write Pattern +- **Mitigation**: Rollback removes new table, legacy system continues working +- **Impact**: No data loss, no service interruption -### **Phase 1: Add New Table (Current)** -```typescript -// Create active_identity table -// Keep existing settings.activeDid for backward compatibility -// Use dual-write pattern during transition -``` +### **Low Risk: API Changes** -### **Phase 2: Dual-Write Pattern** -```typescript -// When updating activeDid: -// 1. Update active_identity table (new system) -// 2. Update settings.activeDid (legacy compatibility) -// 3. Ensure both stay in sync -``` +- **Mitigation**: Dual-write pattern maintains backward compatibility +- **Impact**: Existing components continue working unchanged -### **Phase 3: Future Cleanup (Not in Current Scope)** -```typescript -// Eventually: -// 1. Remove activeDid from settings table -// 2. Deprecate MASTER_SETTINGS_KEY -// 3. Use pure accountDid IS NULL pattern -// 4. Update IndexedDB migration service -``` +### **Low Risk: Performance Impact** + +- **Mitigation**: Proper indexing and minimal additional queries +- **Impact**: Negligible performance change -## Backward Compatibility Requirements +## Summary -### **Critical: IndexedDB Migration Service** -- **Must continue working** for users migrating from Dexie -- **Must recognize `id = "1"`** as master settings -- **Must preserve existing migration paths** +This migration plan: -### **Important: Legacy Database Operations** -- **Must continue working** for existing SQLite databases -- **Must handle both old and new patterns** -- **Must not break existing queries** +1. **Adds new architecture** without breaking existing functionality +2. **Integrates seamlessly** with existing IndexedDB migration service +3. **Maintains full backward compatibility** during transition +4. **Requires minimal changes** to existing codebase +5. **Provides clear rollback path** if issues arise -### **Desired: Cleaner Architecture** -- **New operations** use `accountDid IS NULL` pattern -- **Legacy operations** continue using `MASTER_SETTINGS_KEY` -- **Gradual migration** toward cleaner patterns \ No newline at end of file +The plan focuses only on necessary changes while preserving all existing +functionality and migration paths.