crowd-funder-for-time-pwa/doc/activeDid-migration-plan.md

ActiveDid Migration Plan - Separate Table Architecture

Author: Matthew Raymer Date: 2025-08-29T08:03Z Status: 🎯 PLANNING - Active migration planning phase

Objective

Move the activeDid field from the settings table to a dedicated active_identity table to improve database architecture, prevent data corruption, and separate identity selection from user preferences.

Result

This document serves as the comprehensive planning and implementation guide for the ActiveDid migration with enhanced data integrity and rollback capabilities.

Use/Run

Reference this document during implementation to ensure all migration steps are followed correctly and all stakeholders are aligned on the approach.

Context & Scope

• In scope:
  • Database schema modification for active_identity table with proper constraints
  • Migration of existing activeDid data with validation
  • Updates to PlatformServiceMixin API layer
  • Type definition updates
  • Testing across all platforms
  • Comprehensive rollback procedures
• Out of scope:
  • Changes to user interface for identity selection
  • Modifications to identity creation logic
  • Changes to authentication flow
  • Updates to individual components (handled by API layer)

Environment & Preconditions

• OS/Runtime: All platforms (Web, Electron, iOS, Android)
• Versions/Builds: Current development branch, SQLite database
• Services/Endpoints: Local database, PlatformServiceMixin
• Auth mode: Existing authentication system unchanged

Architecture / Process Overview

The migration follows a phased approach to minimize risk and ensure data integrity with enhanced validation and rollback capabilities:

flowchart TD
    A[Current State<br/>activeDid in settings] --> B[Phase 1: Schema Creation<br/>Add active_identity table with constraints]
    B --> C[Phase 2: Data Migration<br/>Copy activeDid data with validation]
    C --> D[Phase 3: API Updates<br/>Update PlatformServiceMixin methods]
    D --> E[Phase 4: Cleanup<br/>Remove activeDid from settings]
    E --> F[Final State<br/>Separate active_identity table]

    G[Enhanced Rollback Plan<br/>Schema and data rollback] --> H[Data Validation<br/>Verify integrity at each step]
    H --> I[Platform Testing<br/>Test all platforms]
    I --> J[Production Deployment<br/>Gradual rollout with monitoring]
    
    K[Foreign Key Constraints<br/>Prevent future corruption] --> L[Performance Optimization<br/>Proper indexing]
    L --> M[Error Recovery<br/>Graceful failure handling]

Interfaces & Contracts

Database Schema Changes

Table	Current Schema	New Schema	Migration Required
settings	activeDid TEXT	Field removed	Yes - data migration
active_identity	Does not exist	New table with activeDid TEXT + constraints	Yes - table creation

Enhanced API Contract Changes

Method	Current Behavior	New Behavior	Breaking Change
$accountSettings()	Returns settings with activeDid	Returns settings with activeDid from new table	No - backward compatible
$saveSettings()	Updates settings.activeDid	Updates active_identity.activeDid	Yes - requires updates
$updateActiveDid()	Updates internal tracking	Updates active_identity table	Yes - requires updates
$getActiveIdentity()	Does not exist	New method for active identity management	No - new functionality

Repro: End-to-End Procedure

Phase 1: Enhanced Schema Creation via migration.ts

// Add to MIGRATIONS array in src/db-sql/migration.ts
{
  name: "003_active_did_separate_table",
  sql: `
    -- Create new active_identity table with proper constraints
    CREATE TABLE IF NOT EXISTS active_identity (
      id INTEGER PRIMARY KEY CHECK (id = 1),
      activeDid TEXT NOT NULL,
      lastUpdated TEXT NOT NULL DEFAULT (datetime('now')),
      FOREIGN KEY (activeDid) REFERENCES accounts(did) ON DELETE CASCADE
    );

    -- Add performance indexes
    CREATE INDEX IF NOT EXISTS idx_active_identity_activeDid ON active_identity(activeDid);
    CREATE UNIQUE INDEX IF NOT EXISTS idx_active_identity_single_record ON active_identity(id);

    -- Insert default record (will be updated during migration)
    INSERT OR IGNORE INTO active_identity (id, activeDid, lastUpdated) VALUES (1, '', datetime('now'));
  `,
},

Phase 2: Enhanced Data Migration with Validation

// Enhanced migration function with comprehensive validation
async function migrateActiveDidToSeparateTable(): Promise<MigrationResult> {
  const result: MigrationResult = { 
    success: false, 
    errors: [], 
    warnings: [],
    dataMigrated: 0
  };
  
  try {
    // 1. Get current activeDid from settings (legacy approach)
    const currentSettings = await retrieveSettingsForDefaultAccount();
    const activeDid = currentSettings.activeDid;

    if (!activeDid) {
      result.warnings.push("No activeDid found in current settings");
      return result;
    }

    // 2. Validate activeDid exists in accounts table
    const accountExists = await dbQuery(
      "SELECT did FROM accounts WHERE did = ?",
      [activeDid]
    );

    if (!accountExists?.values?.length) {
      result.errors.push(`ActiveDid ${activeDid} not found in accounts table - data corruption detected`);
      return result;
    }

    // 3. Update active_identity table (new system)
    await dbExec(
      "UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1",
      [activeDid]
    );

    // 4. Ensure legacy settings.activeDid stays in sync (backward compatibility)
    // This maintains compatibility with IndexedDB migration service
    await dbExec(
      "UPDATE settings SET activeDid = ? WHERE id = ?",
      [activeDid, MASTER_SETTINGS_KEY]
    );

    dataMigrated = 1;
    result.warnings.push(`Successfully migrated activeDid: ${activeDid}`);
    
  } catch (error) {
    result.errors.push(`Migration failed: ${error}`);
    logger.error("[ActiveDid Migration] Critical error during migration:", error);
  }

  return result;
}

Phase 3: Focused API Updates with Dual-Write Pattern

// Updated PlatformServiceMixin method - maintains backward compatibility
async $accountSettings(did?: string, defaults: Settings = {}): Promise<Settings> {
  try {
    // Get settings without activeDid (unchanged logic)
    const settings = await this._getSettingsWithoutActiveDid();

    if (!settings) {
      return defaults;
    }

    // Get activeDid from new table (new logic)
    const activeIdentity = await this._getActiveIdentity();

    // Return combined result (maintains backward compatibility)
    return { ...settings, activeDid: activeIdentity.activeDid };
  } catch (error) {
    logger.error("[Settings Trace] ❌ Error in $accountSettings:", error);
    return defaults;
  }
}

// Enhanced update activeDid method with dual-write pattern
async $updateActiveDid(newDid: string | null): Promise<boolean> {
  try {
    if (newDid === null) {
      // Clear active identity in both tables
      await this.$dbExec(
        "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1"
      );
      
      // Keep legacy field in sync (backward compatibility)
      await this.$dbExec(
        "UPDATE settings SET activeDid = '' WHERE id = ?",
        [MASTER_SETTINGS_KEY]
      );
    } else {
      // Validate DID exists before setting
      const accountExists = await this.$dbQuery(
        "SELECT did FROM accounts WHERE did = ?",
        [newDid]
      );

      if (!accountExists?.values?.length) {
        logger.error(`[PlatformServiceMixin] Cannot set activeDid to non-existent DID: ${newDid}`);
        return false;
      }

      // Update active identity in new table
      await this.$dbExec(
        "UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1",
        [newDid]
      );
      
      // Keep legacy field in sync (backward compatibility)
      await this.$dbExec(
        "UPDATE settings SET activeDid = ? WHERE id = ?",
        [newDid, MASTER_SETTINGS_KEY]
      );
    }

    // Update internal tracking
    await this._updateInternalActiveDid(newDid);
    return true;
  } catch (error) {
    logger.error("[PlatformServiceMixin] Error updating activeDid:", error);
    return false;
  }
}

Master Settings Functions Implementation Strategy

1. Update retrieveSettingsForDefaultAccount()

// Enhanced implementation with active_identity table integration
export async function retrieveSettingsForDefaultAccount(): Promise<Settings> {
  const platform = PlatformServiceFactory.getInstance();

  // Get settings without activeDid
  const sql = "SELECT id, accountDid, apiServer, filterFeedByNearby, filterFeedByVisible, " +
              "finishedOnboarding, firstName, hideRegisterPromptOnNewContact, isRegistered, " +
              "lastName, lastAckedOfferToUserJwtId, lastAckedOfferToUserProjectsJwtId, " +
              "lastNotifiedClaimId, lastViewedClaimId, notifyingNewActivityTime, " +
              "notifyingReminderMessage, notifyingReminderTime, partnerApiServer, " +
              "passkeyExpirationMinutes, profileImageUrl, searchBoxes, showContactGivesInline, " +
              "showGeneralAdvanced, showShortcutBvc, vapid, warnIfProdServer, warnIfTestServer, " +
              "webPushServer FROM settings WHERE id = ?";

  const result = await platform.dbQuery(sql, [MASTER_SETTINGS_KEY]);

  if (!result) {
    return DEFAULT_SETTINGS;
  } else {
    const settings = mapColumnsToValues(result.columns, result.values)[0] as Settings;

    // Handle JSON parsing
    if (settings.searchBoxes) {
      settings.searchBoxes = JSON.parse(settings.searchBoxes);
    }

    // Get activeDid from separate table
    const activeIdentityResult = await platform.dbQuery(
      "SELECT activeDid FROM active_identity WHERE id = 1"
    );

    if (activeIdentityResult?.values?.length) {
      const activeDid = activeIdentityResult.values[0][0] as string;
      if (activeDid) {
        // Validate activeDid exists in accounts
        const accountExists = await platform.dbQuery(
          "SELECT did FROM accounts WHERE did = ?",
          [activeDid]
        );
        
        if (accountExists?.values?.length) {
          settings.activeDid = activeDid;
        } else {
          logger.warn(`[databaseUtil] ActiveDid ${activeDid} not found in accounts, clearing`);
          // Clear corrupted activeDid
          await platform.dbExec(
            "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1"
          );
        }
      }
    }

    return settings;
  }
}

2. Update $getMergedSettings() Method

// Enhanced implementation with active_identity table integration
async $getMergedSettings(defaultKey: string, accountDid?: string, defaultFallback: Settings = {}): Promise<Settings> {
  try {
    // Get default settings (now without activeDid)
    const defaultSettings = await this.$getSettings(defaultKey, defaultFallback);

    // If no account DID, return defaults with activeDid from separate table
    if (!accountDid) {
      if (defaultSettings) {
        // Get activeDid from separate table
        const activeIdentityResult = await this.$dbQuery(
          "SELECT activeDid FROM active_identity WHERE id = 1"
        );

        if (activeIdentityResult?.values?.length) {
          const activeDid = activeIdentityResult.values[0][0] as string;
          if (activeDid) {
            // Validate activeDid exists in accounts
            const accountExists = await this.$dbQuery(
              "SELECT did FROM accounts WHERE did = ?",
              [activeDid]
            );
            
            if (accountExists?.values?.length) {
              defaultSettings.activeDid = activeDid;
            } else {
              logger.warn(`[Settings Trace] ActiveDid ${activeDid} not found in accounts, clearing`);
              // Clear corrupted activeDid
              await this.$dbExec(
                "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1"
              );
            }
          }
        }
      }
      return defaultSettings || defaultFallback;
    }

    // ... rest of existing implementation for account-specific settings
  } catch (error) {
    logger.error(`[Settings Trace] ❌ Failed to get merged settings:`, { defaultKey, accountDid, error });
    return defaultFallback;
  }
}

What Works (Evidence)

• ✅ Current activeDid storage in settings table

  • Time: 2025-08-29T08:03Z
  • Evidence: src/db-sql/migration.ts:67 - activeDid field exists in initial migration
  • Verify at: Current database schema and Settings type definition

• ✅ PlatformServiceMixin integration with activeDid

  • Time: 2025-08-29T08:03Z
  • Evidence: src/utils/PlatformServiceMixin.ts:108 - activeDid tracking
  • Verify at: Component usage across all platforms

• ✅ Database migration infrastructure exists

  • Time: 2025-08-29T08:03Z
  • Evidence: src/db-sql/migration.ts:31 - migration system in place
  • Verify at: Existing migration scripts and database versioning

What Doesn't (Evidence & Hypotheses)

• ❌ No separate active_identity table exists

  • Time: 2025-08-29T08:03Z
  • Evidence: Database schema only shows settings table
  • Hypothesis: Table needs to be created as part of migration
  • Next probe: Add migration to existing MIGRATIONS array

• ❌ Data corruption issues with orphaned activeDid references

  • Time: 2025-08-29T08:03Z
  • Evidence: IdentitySwitcherView.vue:175 - hasCorruptedIdentity detection
  • Hypothesis: Current schema allows activeDid to point to non-existent accounts
  • Next probe: Implement foreign key constraints in new table

Risks, Limits, Assumptions

• Data Loss Risk: Migration failure could lose activeDid values
• Breaking Changes: API updates required in PlatformServiceMixin
• Rollback Complexity: Schema changes make rollback difficult
• Testing Overhead: All platforms must be tested with new structure
• Performance Impact: Additional table join for activeDid retrieval
• Migration Timing: Must be coordinated with other database changes
• Data Corruption: Current system has documented corruption issues
• Foreign Key Constraints: New constraints may prevent some operations

Enhanced Rollback Strategy

Schema Rollback

-- If migration fails, restore original schema
DROP TABLE IF EXISTS active_identity;

-- Restore activeDid field to settings table if needed
ALTER TABLE settings ADD COLUMN activeDid TEXT;

Data Rollback

// Rollback function to restore activeDid to settings table
async function rollbackActiveDidMigration(): Promise<boolean> {
  try {
    // Get activeDid from active_identity table
    const activeIdentityResult = await dbQuery(
      "SELECT activeDid FROM active_identity WHERE id = 1"
    );
    
    if (activeIdentityResult?.values?.length) {
      const activeDid = activeIdentityResult.values[0][0] as string;
      
      // Restore to settings table
      await dbExec(
        "UPDATE settings SET activeDid = ? WHERE id = ?",
        [activeDid, MASTER_SETTINGS_KEY]
      );
      
      return true;
    }
    
    return false;
  } catch (error) {
    logger.error("[Rollback] Failed to restore activeDid:", error);
    return false;
  }
}

Rollback Triggers

• Migration validation fails
• Data integrity checks fail
• Performance regression detected
• User reports data loss
• Cross-platform inconsistencies found

Next Steps

Owner	Task	Exit Criteria	Target Date (UTC)
Development Team	Add migration to existing MIGRATIONS array	Migration script integrated with existing system	2025-08-30
Development Team	Update type definitions	Settings type updated, ActiveIdentity type created	2025-08-30
Development Team	Update PlatformServiceMixin	Core methods updated and tested	2025-08-31
Development Team	Implement foreign key constraints	Schema validation prevents corruption	2025-08-31
QA Team	Platform testing	All platforms tested and verified	2025-09-01
Development Team	Deploy migration	Production deployment successful	2025-09-02

References

• Database Migration Guide
• Dexie to SQLite Mapping
• PlatformServiceMixin Documentation
• Migration Templates

Competence Hooks

• Why this works: Separates concerns between identity selection and user preferences, prevents data corruption with foreign key constraints, centralizes identity management through API layer
• Common pitfalls: Forgetting to implement foreign key constraints, not testing rollback scenarios, missing data validation during migration, over-engineering component updates when API layer handles everything
• Next skill unlock: Advanced database schema design with constraints, migration planning with rollback strategies
• Teach-back: Explain the four-phase migration approach and why each phase is necessary, especially the foreign key constraints

Collaboration Hooks

• Sign-off checklist:
  • Migration script integrated with existing MIGRATIONS array
  • Foreign key constraints implemented and tested
  • PlatformServiceMixin updated and tested
  • Rollback procedures validated
  • Performance impact assessed
  • All stakeholders approve deployment timeline

Assumptions & Limits

• Current activeDid values are valid and should be preserved
• All platforms can handle the additional database table
• Migration can be completed without user downtime
• Rollback to previous schema is acceptable if needed
• Performance impact of additional table join is minimal
• Foreign key constraints will prevent future corruption
• API layer updates will handle component compatibility

What Needs to Change

1. Database Schema via migration.ts

• Add migration to existing MIGRATIONS array in src/db-sql/migration.ts
• Create active_identity table with foreign key constraints
• Add performance indexes
• Keep activeDid field in settings table temporarily for backward compatibility
• Preserve MASTER_SETTINGS_KEY = "1" for legacy migration support

2. PlatformServiceMixin Methods

• $accountSettings() - integrate with new table while maintaining backward compatibility
• $saveSettings() - handle activeDid in new table, sync with legacy field
• $updateActiveDid() - validate and update new table, sync with legacy field
• $getActiveIdentity() - new method for identity management

3. Master Settings Functions

• retrieveSettingsForDefaultAccount() - integrate with new table while preserving legacy support
• $getMergedSettings() - integrate with new table while preserving legacy support

4. Type Definitions

• Keep activeDid in Settings type temporarily for backward compatibility
• Create ActiveIdentity type for new table
• Update related interfaces

5. Legacy Compatibility

• Preserve MASTER_SETTINGS_KEY = "1" for IndexedDB migration service
• Maintain dual-write pattern during transition period
• Ensure legacy clients can still migrate from Dexie to SQLite

What Doesn't Need to Change

• All Vue components - API layer handles migration transparently
• Platform services - Use PlatformServiceMixin, no direct access
• User interface - No changes to identity selection UI
• Authentication flow - Existing system unchanged
• Component logic - All activeDid handling through API methods
• Migration system - Use existing migration.ts approach, not separate files
• IndexedDB migration service - Must continue working for legacy clients

Enhanced Architecture: Dual-Write Pattern

Phase 1: Add New Table (Current)

// Create active_identity table
// Keep existing settings.activeDid for backward compatibility
// Use dual-write pattern during transition

Phase 2: Dual-Write Pattern

// When updating activeDid:
// 1. Update active_identity table (new system)
// 2. Update settings.activeDid (legacy compatibility)
// 3. Ensure both stay in sync

Phase 3: Future Cleanup (Not in Current Scope)

// Eventually:
// 1. Remove activeDid from settings table
// 2. Deprecate MASTER_SETTINGS_KEY
// 3. Use pure accountDid IS NULL pattern
// 4. Update IndexedDB migration service

Backward Compatibility Requirements

Critical: IndexedDB Migration Service

• Must continue working for users migrating from Dexie
• Must recognize id = "1" as master settings
• Must preserve existing migration paths

Important: Legacy Database Operations

• Must continue working for existing SQLite databases
• Must handle both old and new patterns
• Must not break existing queries

Desired: Cleaner Architecture

• New operations use accountDid IS NULL pattern
• Legacy operations continue using MASTER_SETTINGS_KEY
• Gradual migration toward cleaner patterns