crowd-funder-for-time-pwa/doc/activeDid-migration-plan.md

ActiveDid Migration Plan - Separate Table Architecture

Author: Matthew Raymer Date: 2025-08-29T07:24Z Status: 🎯 PLANNING - Active migration planning phase

Objective

Move the activeDid field from the settings table to a dedicated active_identity table to improve database architecture, prevent data corruption, and separate identity selection from user preferences.

Result

This document serves as the comprehensive planning and implementation guide for the ActiveDid migration with enhanced data integrity and rollback capabilities.

Use/Run

Reference this document during implementation to ensure all migration steps are followed correctly and all stakeholders are aligned on the approach.

Context & Scope

• In scope:
  • Database schema modification for active_identity table with proper constraints
  • Migration of existing activeDid data with validation
  • Updates to PlatformServiceMixin API layer
  • Type definition updates
  • Testing across all platforms
  • Comprehensive rollback procedures
• Out of scope:
  • Changes to user interface for identity selection
  • Modifications to identity creation logic
  • Changes to authentication flow
  • Updates to individual components (handled by API layer)

Environment & Preconditions

• OS/Runtime: All platforms (Web, Electron, iOS, Android)
• Versions/Builds: Current development branch, SQLite database
• Services/Endpoints: Local database, PlatformServiceMixin
• Auth mode: Existing authentication system unchanged

Architecture / Process Overview

The migration follows a phased approach to minimize risk and ensure data integrity with enhanced validation and rollback capabilities:

flowchart TD
    A[Current State<br/>activeDid in settings] --> B[Phase 1: Schema Creation<br/>Add active_identity table with constraints]
    B --> C[Phase 2: Data Migration<br/>Copy activeDid data with validation]
    C --> D[Phase 3: API Updates<br/>Update PlatformServiceMixin methods]
    D --> E[Phase 4: Cleanup<br/>Remove activeDid from settings]
    E --> F[Final State<br/>Separate active_identity table]

    G[Enhanced Rollback Plan<br/>Schema and data rollback] --> H[Data Validation<br/>Verify integrity at each step]
    H --> I[Platform Testing<br/>Test all platforms]
    I --> J[Production Deployment<br/>Gradual rollout with monitoring]
    
    K[Foreign Key Constraints<br/>Prevent future corruption] --> L[Performance Optimization<br/>Proper indexing]
    L --> M[Error Recovery<br/>Graceful failure handling]

Interfaces & Contracts

Database Schema Changes

Table	Current Schema	New Schema	Migration Required
settings	activeDid TEXT	Field removed	Yes - data migration
active_identity	Does not exist	New table with activeDid TEXT + constraints	Yes - table creation

Enhanced API Contract Changes

Method	Current Behavior	New Behavior	Breaking Change
$accountSettings()	Returns settings with activeDid	Returns settings with activeDid from new table	No - backward compatible
$saveSettings()	Updates settings.activeDid	Updates active_identity.activeDid	Yes - requires updates
$updateActiveDid()	Updates internal tracking	Updates active_identity table	Yes - requires updates
$getActiveIdentity()	Does not exist	New method for active identity management	No - new functionality

Repro: End-to-End Procedure

Phase 1: Enhanced Schema Creation

-- Create new active_identity table with proper constraints
CREATE TABLE active_identity (
  id INTEGER PRIMARY KEY CHECK (id = 1),
  activeDid TEXT NOT NULL,
  lastUpdated TEXT NOT NULL DEFAULT (datetime('now')),
  FOREIGN KEY (activeDid) REFERENCES accounts(did) ON DELETE CASCADE
);

-- Add performance indexes
CREATE INDEX IF NOT EXISTS idx_active_identity_activeDid ON active_identity(activeDid);
CREATE UNIQUE INDEX IF NOT EXISTS idx_active_identity_single_record ON active_identity(id);

-- Insert default record (will be updated during migration)
INSERT INTO active_identity (id, activeDid, lastUpdated) VALUES (1, '', datetime('now'));

Phase 2: Enhanced Data Migration with Validation

// Enhanced migration script with comprehensive validation
async function migrateActiveDidToSeparateTable(): Promise<MigrationResult> {
  const result: MigrationResult = { 
    success: false, 
    errors: [], 
    warnings: [],
    dataMigrated: 0
  };
  
  try {
    // 1. Get current activeDid from settings
    const currentSettings = await retrieveSettingsForDefaultAccount();
    const activeDid = currentSettings.activeDid;

    if (!activeDid) {
      result.warnings.push("No activeDid found in current settings");
      return result;
    }

    // 2. Validate activeDid exists in accounts table
    const accountExists = await dbQuery(
      "SELECT did FROM accounts WHERE did = ?",
      [activeDid]
    );

    if (!accountExists?.values?.length) {
      result.errors.push(`ActiveDid ${activeDid} not found in accounts table - data corruption detected`);
      return result;
    }

    // 3. Check if active_identity table already has data
    const existingActiveIdentity = await dbQuery(
      "SELECT activeDid FROM active_identity WHERE id = 1"
    );

    if (existingActiveIdentity?.values?.length) {
      // Update existing record
      await dbExec(
        "UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1",
        [activeDid]
      );
    } else {
      // Insert new record
      await dbExec(
        "INSERT INTO active_identity (id, activeDid, lastUpdated) VALUES (1, ?, datetime('now'))",
        [activeDid]
      );
    }

    result.success = true;
    result.dataMigrated = 1;
    result.warnings.push(`Successfully migrated activeDid: ${activeDid}`);
    
  } catch (error) {
    result.errors.push(`Migration failed: ${error}`);
    logger.error("[ActiveDid Migration] Critical error during migration:", error);
  }

  return result;
}

// Migration result interface
interface MigrationResult {
  success: boolean;
  errors: string[];
  warnings: string[];
  dataMigrated: number;
}

Phase 3: Focused API Updates

// Updated PlatformServiceMixin method - maintains backward compatibility
async $accountSettings(did?: string, defaults: Settings = {}): Promise<Settings> {
  try {
    // Get settings without activeDid (unchanged logic)
    const settings = await this._getSettingsWithoutActiveDid();

    if (!settings) {
      return defaults;
    }

    // Get activeDid from new table (new logic)
    const activeIdentity = await this._getActiveIdentity();

    // Return combined result (maintains backward compatibility)
    return { ...settings, activeDid: activeIdentity.activeDid };
  } catch (error) {
    logger.error("[Settings Trace] ❌ Error in $accountSettings:", error);
    return defaults;
  }
}

// New method for active identity management
async $getActiveIdentity(): Promise<{ activeDid: string | null }> {
  try {
    const result = await this.$dbQuery(
      "SELECT activeDid FROM active_identity WHERE id = 1"
    );
    
    if (!result?.values?.length) {
      return { activeDid: null };
    }
    
    return { activeDid: result.values[0][0] as string };
  } catch (error) {
    logger.error("[Settings Trace] ❌ Failed to get active identity:", error);
    return { activeDid: null };
  }
}

// Enhanced method to get settings without activeDid
async _getSettingsWithoutActiveDid(): Promise<Settings> {
  const result = await this.$dbQuery(
    "SELECT id, accountDid, apiServer, filterFeedByNearby, filterFeedByVisible, " +
    "finishedOnboarding, firstName, hideRegisterPromptOnNewContact, isRegistered, " +
    "lastName, lastAckedOfferToUserJwtId, lastAckedOfferToUserProjectsJwtId, " +
    "lastNotifiedClaimId, lastViewedClaimId, notifyingNewActivityTime, " +
    "notifyingReminderMessage, notifyingReminderTime, partnerApiServer, " +
    "passkeyExpirationMinutes, profileImageUrl, searchBoxes, showContactGivesInline, " +
    "showGeneralAdvanced, showShortcutBvc, vapid, warnIfProdServer, warnIfTestServer, " +
    "webPushServer FROM settings WHERE id = ?",
    [MASTER_SETTINGS_KEY]
  );

  if (!result?.values?.length) {
    return DEFAULT_SETTINGS;
  }

  return this._mapColumnsToValues(result.columns, result.values)[0] as Settings;
}

// Enhanced save settings method
async $saveSettings(changes: Partial<Settings>): Promise<boolean> {
  try {
    // Remove fields that shouldn't be updated
    const { accountDid, id, activeDid, ...safeChanges } = changes;
    
    if (Object.keys(safeChanges).length > 0) {
      // Convert settings for database storage
      const convertedChanges = this._convertSettingsForStorage(safeChanges);
      const setParts: string[] = [];
      const params: unknown[] = [];

      Object.entries(convertedChanges).forEach(([key, value]) => {
        if (value !== undefined) {
          setParts.push(`${key} = ?`);
          params.push(value);
        }
      });

      if (setParts.length > 0) {
        params.push(MASTER_SETTINGS_KEY);
        await this.$dbExec(
          `UPDATE settings SET ${setParts.join(", ")} WHERE id = ?`,
          params,
        );
      }
    }

    // Handle activeDid separately in new table
    if (changes.activeDid !== undefined) {
      await this.$updateActiveDid(changes.activeDid);
    }

    return true;
  } catch (error) {
    logger.error("[PlatformServiceMixin] Error saving settings:", error);
    return false;
  }
}

// Enhanced update activeDid method
async $updateActiveDid(newDid: string | null): Promise<boolean> {
  try {
    if (newDid === null) {
      // Clear active identity
      await this.$dbExec(
        "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1"
      );
    } else {
      // Validate DID exists before setting
      const accountExists = await this.$dbQuery(
        "SELECT did FROM accounts WHERE did = ?",
        [newDid]
      );

      if (!accountExists?.values?.length) {
        logger.error(`[PlatformServiceMixin] Cannot set activeDid to non-existent DID: ${newDid}`);
        return false;
      }

      // Update active identity
      await this.$dbExec(
        "UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1",
        [newDid]
      );
    }

    // Update internal tracking
    await this._updateInternalActiveDid(newDid);
    return true;
  } catch (error) {
    logger.error("[PlatformServiceMixin] Error updating activeDid:", error);
    return false;
  }
}

Master Settings Functions Implementation Strategy

1. Update retrieveSettingsForDefaultAccount()

// Enhanced implementation with active_identity table integration
export async function retrieveSettingsForDefaultAccount(): Promise<Settings> {
  const platform = PlatformServiceFactory.getInstance();

  // Get settings without activeDid
  const sql = "SELECT id, accountDid, apiServer, filterFeedByNearby, filterFeedByVisible, " +
              "finishedOnboarding, firstName, hideRegisterPromptOnNewContact, isRegistered, " +
              "lastName, lastAckedOfferToUserJwtId, lastAckedOfferToUserProjectsJwtId, " +
              "lastNotifiedClaimId, lastViewedClaimId, notifyingNewActivityTime, " +
              "notifyingReminderMessage, notifyingReminderTime, partnerApiServer, " +
              "passkeyExpirationMinutes, profileImageUrl, searchBoxes, showContactGivesInline, " +
              "showGeneralAdvanced, showShortcutBvc, vapid, warnIfProdServer, warnIfTestServer, " +
              "webPushServer FROM settings WHERE id = ?";

  const result = await platform.dbQuery(sql, [MASTER_SETTINGS_KEY]);

  if (!result) {
    return DEFAULT_SETTINGS;
  } else {
    const settings = mapColumnsToValues(result.columns, result.values)[0] as Settings;

    // Handle JSON parsing
    if (settings.searchBoxes) {
      settings.searchBoxes = JSON.parse(settings.searchBoxes);
    }

    // Get activeDid from separate table
    const activeIdentityResult = await platform.dbQuery(
      "SELECT activeDid FROM active_identity WHERE id = 1"
    );

    if (activeIdentityResult?.values?.length) {
      const activeDid = activeIdentityResult.values[0][0] as string;
      if (activeDid) {
        // Validate activeDid exists in accounts
        const accountExists = await platform.dbQuery(
          "SELECT did FROM accounts WHERE did = ?",
          [activeDid]
        );
        
        if (accountExists?.values?.length) {
          settings.activeDid = activeDid;
        } else {
          logger.warn(`[databaseUtil] ActiveDid ${activeDid} not found in accounts, clearing`);
          // Clear corrupted activeDid
          await platform.dbExec(
            "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1"
          );
        }
      }
    }

    return settings;
  }
}

2. Update $getMergedSettings() Method

// Enhanced implementation with active_identity table integration
async $getMergedSettings(defaultKey: string, accountDid?: string, defaultFallback: Settings = {}): Promise<Settings> {
  try {
    // Get default settings (now without activeDid)
    const defaultSettings = await this.$getSettings(defaultKey, defaultFallback);

    // If no account DID, return defaults with activeDid from separate table
    if (!accountDid) {
      if (defaultSettings) {
        // Get activeDid from separate table
        const activeIdentityResult = await this.$dbQuery(
          "SELECT activeDid FROM active_identity WHERE id = 1"
        );

        if (activeIdentityResult?.values?.length) {
          const activeDid = activeIdentityResult.values[0][0] as string;
          if (activeDid) {
            // Validate activeDid exists in accounts
            const accountExists = await this.$dbQuery(
              "SELECT did FROM accounts WHERE did = ?",
              [activeDid]
            );
            
            if (accountExists?.values?.length) {
              defaultSettings.activeDid = activeDid;
            } else {
              logger.warn(`[Settings Trace] ActiveDid ${activeDid} not found in accounts, clearing`);
              // Clear corrupted activeDid
              await this.$dbExec(
                "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1"
              );
            }
          }
        }
      }
      return defaultSettings || defaultFallback;
    }

    // ... rest of existing implementation for account-specific settings
  } catch (error) {
    logger.error(`[Settings Trace] ❌ Failed to get merged settings:`, { defaultKey, accountDid, error });
    return defaultFallback;
  }
}

What Works (Evidence)

• ✅ Current activeDid storage in settings table

  • Time: 2025-08-29T07:24Z
  • Evidence: src/db/tables/settings.ts:25 - activeDid field exists
  • Verify at: Current database schema and Settings type definition

• ✅ PlatformServiceMixin integration with activeDid

  • Time: 2025-08-29T07:24Z
  • Evidence: src/utils/PlatformServiceMixin.ts:108 - activeDid tracking
  • Verify at: Component usage across all platforms

• ✅ Database migration infrastructure exists

  • Time: 2025-08-29T07:24Z
  • Evidence: src/db-sql/migration.ts:31 - migration system in place
  • Verify at: Existing migration scripts and database versioning

What Doesn't (Evidence & Hypotheses)

• ❌ No separate active_identity table exists

  • Time: 2025-08-29T07:24Z
  • Evidence: Database schema only shows settings table
  • Hypothesis: Table needs to be created as part of migration
  • Next probe: Create migration script for new table

• ❌ Data corruption issues with orphaned activeDid references

  • Time: 2025-08-29T07:24Z
  • Evidence: IdentitySwitcherView.vue:175 - hasCorruptedIdentity detection
  • Hypothesis: Current schema allows activeDid to point to non-existent accounts
  • Next probe: Implement foreign key constraints in new table

Risks, Limits, Assumptions

• Data Loss Risk: Migration failure could lose activeDid values
• Breaking Changes: API updates required in PlatformServiceMixin
• Rollback Complexity: Schema changes make rollback difficult
• Testing Overhead: All platforms must be tested with new structure
• Performance Impact: Additional table join for activeDid retrieval
• Migration Timing: Must be coordinated with other database changes
• Data Corruption: Current system has documented corruption issues
• Foreign Key Constraints: New constraints may prevent some operations

Enhanced Rollback Strategy

Schema Rollback

-- If migration fails, restore original schema
DROP TABLE IF EXISTS active_identity;

-- Restore activeDid field to settings table if needed
ALTER TABLE settings ADD COLUMN activeDid TEXT;

Data Rollback

// Rollback function to restore activeDid to settings table
async function rollbackActiveDidMigration(): Promise<boolean> {
  try {
    // Get activeDid from active_identity table
    const activeIdentityResult = await dbQuery(
      "SELECT activeDid FROM active_identity WHERE id = 1"
    );
    
    if (activeIdentityResult?.values?.length) {
      const activeDid = activeIdentityResult.values[0][0] as string;
      
      // Restore to settings table
      await dbExec(
        "UPDATE settings SET activeDid = ? WHERE id = ?",
        [activeDid, MASTER_SETTINGS_KEY]
      );
      
      return true;
    }
    
    return false;
  } catch (error) {
    logger.error("[Rollback] Failed to restore activeDid:", error);
    return false;
  }
}

Rollback Triggers

• Migration validation fails
• Data integrity checks fail
• Performance regression detected
• User reports data loss
• Cross-platform inconsistencies found

Next Steps

Owner	Task	Exit Criteria	Target Date (UTC)
Development Team	Create enhanced migration script	Migration script with validation and rollback	2025-08-30
Development Team	Update type definitions	Settings type updated, ActiveIdentity type created	2025-08-30
Development Team	Update PlatformServiceMixin	Core methods updated and tested	2025-08-31
Development Team	Implement foreign key constraints	Schema validation prevents corruption	2025-08-31
QA Team	Platform testing	All platforms tested and verified	2025-09-01
Development Team	Deploy migration	Production deployment successful	2025-09-02

References

• Database Migration Guide
• Dexie to SQLite Mapping
• PlatformServiceMixin Documentation
• Migration Templates

Competence Hooks

• Why this works: Separates concerns between identity selection and user preferences, prevents data corruption with foreign key constraints, centralizes identity management through API layer
• Common pitfalls: Forgetting to implement foreign key constraints, not testing rollback scenarios, missing data validation during migration, over-engineering component updates when API layer handles everything
• Next skill unlock: Advanced database schema design with constraints, migration planning with rollback strategies
• Teach-back: Explain the four-phase migration approach and why each phase is necessary, especially the foreign key constraints

Collaboration Hooks

• Sign-off checklist:
  • Migration script tested on development database
  • Foreign key constraints implemented and tested
  • PlatformServiceMixin updated and tested
  • Rollback procedures validated
  • Performance impact assessed
  • All stakeholders approve deployment timeline

Assumptions & Limits

• Current activeDid values are valid and should be preserved
• All platforms can handle the additional database table
• Migration can be completed without user downtime
• Rollback to previous schema is acceptable if needed
• Performance impact of additional table join is minimal
• Foreign key constraints will prevent future corruption
• API layer updates will handle component compatibility

What Needs to Change

1. Database Schema

• Create active_identity table with foreign key constraints
• Add performance indexes
• Remove activeDid field from settings table

2. PlatformServiceMixin Methods

• $accountSettings() - integrate with new table
• $saveSettings() - handle activeDid in new table
• $updateActiveDid() - validate and update new table
• $getActiveIdentity() - new method for identity management

3. Master Settings Functions

• retrieveSettingsForDefaultAccount() - integrate with new table
• $getMergedSettings() - integrate with new table

4. Migration Scripts

• Create migration script with validation
• Implement rollback procedures
• Add data corruption detection

5. Type Definitions

• Update Settings type to remove activeDid
• Create ActiveIdentity type for new table
• Update related interfaces

What Doesn't Need to Change

• All Vue components - API layer handles migration transparently
• Platform services - Use PlatformServiceMixin, no direct access
• User interface - No changes to identity selection UI
• Authentication flow - Existing system unchanged
• Component logic - All activeDid handling through API methods