# ActiveDid Migration Plan - Separate Table Architecture

**Author**: Matthew Raymer
**Date**: 2025-08-29T08:03Z
**Status**: 🎯 **PLANNING** - Active migration planning phase

## Objective

Move the `activeDid` field from the `settings` table to a dedicated
`active_identity` table to improve database architecture, prevent data corruption,
and separate identity selection from user preferences.

## Result

This document serves as the comprehensive planning and implementation
guide for the ActiveDid migration with enhanced data integrity and
rollback capabilities.

## Use/Run

Reference this document during implementation to ensure all migration
steps are followed correctly and all stakeholders are aligned on the
approach.

## Context & Scope

- **In scope**:
  - Database schema modification for active_identity table with proper constraints
  - Migration of existing activeDid data with validation
  - Updates to PlatformServiceMixin API layer
  - Type definition updates
  - Testing across all platforms
  - Comprehensive rollback procedures
- **Out of scope**:
  - Changes to user interface for identity selection
  - Modifications to identity creation logic
  - Changes to authentication flow
  - Updates to individual components (handled by API layer)

## Environment & Preconditions

- **OS/Runtime**: All platforms (Web, Electron, iOS, Android)
- **Versions/Builds**: Current development branch, SQLite database
- **Services/Endpoints**: Local database, PlatformServiceMixin
- **Auth mode**: Existing authentication system unchanged

## Architecture / Process Overview

The migration follows a phased approach to minimize risk and ensure
data integrity with enhanced validation and rollback capabilities:

```mermaid
flowchart TD
    A[Current State<br/>activeDid in settings] --> B[Phase 1: Schema Creation<br/>Add active_identity table with constraints]
    B --> C[Phase 2: Data Migration<br/>Copy activeDid data with validation]
    C --> D[Phase 3: API Updates<br/>Update PlatformServiceMixin methods]
    D --> E[Phase 4: Cleanup<br/>Remove activeDid from settings]
    E --> F[Final State<br/>Separate active_identity table]

    G[Enhanced Rollback Plan<br/>Schema and data rollback] --> H[Data Validation<br/>Verify integrity at each step]
    H --> I[Platform Testing<br/>Test all platforms]
    I --> J[Production Deployment<br/>Gradual rollout with monitoring]
    
    K[Foreign Key Constraints<br/>Prevent future corruption] --> L[Performance Optimization<br/>Proper indexing]
    L --> M[Error Recovery<br/>Graceful failure handling]
```

## Interfaces & Contracts

### Database Schema Changes

| Table | Current Schema | New Schema | Migration Required |
|-------|----------------|------------|-------------------|
| `settings` | `activeDid TEXT` | Field removed | Yes - data migration |
| `active_identity` | Does not exist | New table with `activeDid TEXT` + constraints | Yes - table creation |

### Enhanced API Contract Changes

| Method | Current Behavior | New Behavior | Breaking Change |
|---------|------------------|--------------|-----------------|
| `$accountSettings()` | Returns settings with activeDid | Returns settings with activeDid from new table | No - backward compatible |
| `$saveSettings()` | Updates settings.activeDid | Updates active_identity.activeDid | Yes - requires updates |
| `$updateActiveDid()` | Updates internal tracking | Updates active_identity table | Yes - requires updates |
| `$getActiveIdentity()` | Does not exist | New method for active identity management | No - new functionality |

## Repro: End-to-End Procedure

### Phase 1: Enhanced Schema Creation via migration.ts

```typescript
// Add to MIGRATIONS array in src/db-sql/migration.ts
{
  name: "003_active_did_separate_table",
  sql: `
    -- Create new active_identity table with proper constraints
    CREATE TABLE IF NOT EXISTS active_identity (
      id INTEGER PRIMARY KEY CHECK (id = 1),
      activeDid TEXT NOT NULL,
      lastUpdated TEXT NOT NULL DEFAULT (datetime('now')),
      FOREIGN KEY (activeDid) REFERENCES accounts(did) ON DELETE CASCADE
    );

    -- Add performance indexes
    CREATE INDEX IF NOT EXISTS idx_active_identity_activeDid ON active_identity(activeDid);
    CREATE UNIQUE INDEX IF NOT EXISTS idx_active_identity_single_record ON active_identity(id);

    -- Insert default record (will be updated during migration)
    INSERT OR IGNORE INTO active_identity (id, activeDid, lastUpdated) VALUES (1, '', datetime('now'));
  `,
},
```

### Phase 2: Enhanced Data Migration with Validation

```typescript
// Enhanced migration function with comprehensive validation
async function migrateActiveDidToSeparateTable(): Promise<MigrationResult> {
  const result: MigrationResult = { 
    success: false, 
    errors: [], 
    warnings: [],
    dataMigrated: 0
  };
  
  try {
    // 1. Get current activeDid from settings (legacy approach)
    const currentSettings = await retrieveSettingsForDefaultAccount();
    const activeDid = currentSettings.activeDid;

    if (!activeDid) {
      result.warnings.push("No activeDid found in current settings");
      return result;
    }

    // 2. Validate activeDid exists in accounts table
    const accountExists = await dbQuery(
      "SELECT did FROM accounts WHERE did = ?",
      [activeDid]
    );

    if (!accountExists?.values?.length) {
      result.errors.push(`ActiveDid ${activeDid} not found in accounts table - data corruption detected`);
      return result;
    }

    // 3. Update active_identity table (new system)
    await dbExec(
      "UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1",
      [activeDid]
    );

    // 4. Ensure legacy settings.activeDid stays in sync (backward compatibility)
    // This maintains compatibility with IndexedDB migration service
    await dbExec(
      "UPDATE settings SET activeDid = ? WHERE id = ?",
      [activeDid, MASTER_SETTINGS_KEY]
    );

    dataMigrated = 1;
    result.warnings.push(`Successfully migrated activeDid: ${activeDid}`);
    
  } catch (error) {
    result.errors.push(`Migration failed: ${error}`);
    logger.error("[ActiveDid Migration] Critical error during migration:", error);
  }

  return result;
}
```

### Phase 3: Focused API Updates with Dual-Write Pattern

```typescript
// Updated PlatformServiceMixin method - maintains backward compatibility
async $accountSettings(did?: string, defaults: Settings = {}): Promise<Settings> {
  try {
    // Get settings without activeDid (unchanged logic)
    const settings = await this._getSettingsWithoutActiveDid();

    if (!settings) {
      return defaults;
    }

    // Get activeDid from new table (new logic)
    const activeIdentity = await this._getActiveIdentity();

    // Return combined result (maintains backward compatibility)
    return { ...settings, activeDid: activeIdentity.activeDid };
  } catch (error) {
    logger.error("[Settings Trace] ❌ Error in $accountSettings:", error);
    return defaults;
  }
}

// Enhanced update activeDid method with dual-write pattern
async $updateActiveDid(newDid: string | null): Promise<boolean> {
  try {
    if (newDid === null) {
      // Clear active identity in both tables
      await this.$dbExec(
        "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1"
      );
      
      // Keep legacy field in sync (backward compatibility)
      await this.$dbExec(
        "UPDATE settings SET activeDid = '' WHERE id = ?",
        [MASTER_SETTINGS_KEY]
      );
    } else {
      // Validate DID exists before setting
      const accountExists = await this.$dbQuery(
        "SELECT did FROM accounts WHERE did = ?",
        [newDid]
      );

      if (!accountExists?.values?.length) {
        logger.error(`[PlatformServiceMixin] Cannot set activeDid to non-existent DID: ${newDid}`);
        return false;
      }

      // Update active identity in new table
      await this.$dbExec(
        "UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1",
        [newDid]
      );
      
      // Keep legacy field in sync (backward compatibility)
      await this.$dbExec(
        "UPDATE settings SET activeDid = ? WHERE id = ?",
        [newDid, MASTER_SETTINGS_KEY]
      );
    }

    // Update internal tracking
    await this._updateInternalActiveDid(newDid);
    return true;
  } catch (error) {
    logger.error("[PlatformServiceMixin] Error updating activeDid:", error);
    return false;
  }
}
```

### **Master Settings Functions Implementation Strategy**

#### **1. Update `retrieveSettingsForDefaultAccount()`**

```typescript
// Enhanced implementation with active_identity table integration
export async function retrieveSettingsForDefaultAccount(): Promise<Settings> {
  const platform = PlatformServiceFactory.getInstance();

  // Get settings without activeDid
  const sql = "SELECT id, accountDid, apiServer, filterFeedByNearby, filterFeedByVisible, " +
              "finishedOnboarding, firstName, hideRegisterPromptOnNewContact, isRegistered, " +
              "lastName, lastAckedOfferToUserJwtId, lastAckedOfferToUserProjectsJwtId, " +
              "lastNotifiedClaimId, lastViewedClaimId, notifyingNewActivityTime, " +
              "notifyingReminderMessage, notifyingReminderTime, partnerApiServer, " +
              "passkeyExpirationMinutes, profileImageUrl, searchBoxes, showContactGivesInline, " +
              "showGeneralAdvanced, showShortcutBvc, vapid, warnIfProdServer, warnIfTestServer, " +
              "webPushServer FROM settings WHERE id = ?";

  const result = await platform.dbQuery(sql, [MASTER_SETTINGS_KEY]);

  if (!result) {
    return DEFAULT_SETTINGS;
  } else {
    const settings = mapColumnsToValues(result.columns, result.values)[0] as Settings;

    // Handle JSON parsing
    if (settings.searchBoxes) {
      settings.searchBoxes = JSON.parse(settings.searchBoxes);
    }

    // Get activeDid from separate table
    const activeIdentityResult = await platform.dbQuery(
      "SELECT activeDid FROM active_identity WHERE id = 1"
    );

    if (activeIdentityResult?.values?.length) {
      const activeDid = activeIdentityResult.values[0][0] as string;
      if (activeDid) {
        // Validate activeDid exists in accounts
        const accountExists = await platform.dbQuery(
          "SELECT did FROM accounts WHERE did = ?",
          [activeDid]
        );
        
        if (accountExists?.values?.length) {
          settings.activeDid = activeDid;
        } else {
          logger.warn(`[databaseUtil] ActiveDid ${activeDid} not found in accounts, clearing`);
          // Clear corrupted activeDid
          await platform.dbExec(
            "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1"
          );
        }
      }
    }

    return settings;
  }
}
```

#### **2. Update `$getMergedSettings()` Method**

```typescript
// Enhanced implementation with active_identity table integration
async $getMergedSettings(defaultKey: string, accountDid?: string, defaultFallback: Settings = {}): Promise<Settings> {
  try {
    // Get default settings (now without activeDid)
    const defaultSettings = await this.$getSettings(defaultKey, defaultFallback);

    // If no account DID, return defaults with activeDid from separate table
    if (!accountDid) {
      if (defaultSettings) {
        // Get activeDid from separate table
        const activeIdentityResult = await this.$dbQuery(
          "SELECT activeDid FROM active_identity WHERE id = 1"
        );

        if (activeIdentityResult?.values?.length) {
          const activeDid = activeIdentityResult.values[0][0] as string;
          if (activeDid) {
            // Validate activeDid exists in accounts
            const accountExists = await this.$dbQuery(
              "SELECT did FROM accounts WHERE did = ?",
              [activeDid]
            );
            
            if (accountExists?.values?.length) {
              defaultSettings.activeDid = activeDid;
            } else {
              logger.warn(`[Settings Trace] ActiveDid ${activeDid} not found in accounts, clearing`);
              // Clear corrupted activeDid
              await this.$dbExec(
                "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1"
              );
            }
          }
        }
      }
      return defaultSettings || defaultFallback;
    }

    // ... rest of existing implementation for account-specific settings
  } catch (error) {
    logger.error(`[Settings Trace] ❌ Failed to get merged settings:`, { defaultKey, accountDid, error });
    return defaultFallback;
  }
}
```

## What Works (Evidence)

- ✅ **Current activeDid storage** in settings table
  - **Time**: 2025-08-29T08:03Z
  - **Evidence**: `src/db-sql/migration.ts:67` - activeDid field exists in initial migration
  - **Verify at**: Current database schema and Settings type definition

- ✅ **PlatformServiceMixin integration** with activeDid
  - **Time**: 2025-08-29T08:03Z
  - **Evidence**: `src/utils/PlatformServiceMixin.ts:108` - activeDid tracking
  - **Verify at**: Component usage across all platforms

- ✅ **Database migration infrastructure** exists
  - **Time**: 2025-08-29T08:03Z
  - **Evidence**: `src/db-sql/migration.ts:31` - migration system in place
  - **Verify at**: Existing migration scripts and database versioning

## What Doesn't (Evidence & Hypotheses)

- ❌ **No separate active_identity table** exists
  - **Time**: 2025-08-29T08:03Z
  - **Evidence**: Database schema only shows settings table
  - **Hypothesis**: Table needs to be created as part of migration
  - **Next probe**: Add migration to existing MIGRATIONS array

- ❌ **Data corruption issues** with orphaned activeDid references
  - **Time**: 2025-08-29T08:03Z
  - **Evidence**: `IdentitySwitcherView.vue:175` - `hasCorruptedIdentity` detection
  - **Hypothesis**: Current schema allows activeDid to point to non-existent accounts
  - **Next probe**: Implement foreign key constraints in new table

## Risks, Limits, Assumptions

- **Data Loss Risk**: Migration failure could lose activeDid values
- **Breaking Changes**: API updates required in PlatformServiceMixin
- **Rollback Complexity**: Schema changes make rollback difficult
- **Testing Overhead**: All platforms must be tested with new structure
- **Performance Impact**: Additional table join for activeDid retrieval
- **Migration Timing**: Must be coordinated with other database changes
- **Data Corruption**: Current system has documented corruption issues
- **Foreign Key Constraints**: New constraints may prevent some operations

## Enhanced Rollback Strategy

### **Schema Rollback**
```sql
-- If migration fails, restore original schema
DROP TABLE IF EXISTS active_identity;

-- Restore activeDid field to settings table if needed
ALTER TABLE settings ADD COLUMN activeDid TEXT;
```

### **Data Rollback**
```typescript
// Rollback function to restore activeDid to settings table
async function rollbackActiveDidMigration(): Promise<boolean> {
  try {
    // Get activeDid from active_identity table
    const activeIdentityResult = await dbQuery(
      "SELECT activeDid FROM active_identity WHERE id = 1"
    );
    
    if (activeIdentityResult?.values?.length) {
      const activeDid = activeIdentityResult.values[0][0] as string;
      
      // Restore to settings table
      await dbExec(
        "UPDATE settings SET activeDid = ? WHERE id = ?",
        [activeDid, MASTER_SETTINGS_KEY]
      );
      
      return true;
    }
    
    return false;
  } catch (error) {
    logger.error("[Rollback] Failed to restore activeDid:", error);
    return false;
  }
}
```

### **Rollback Triggers**
- Migration validation fails
- Data integrity checks fail
- Performance regression detected
- User reports data loss
- Cross-platform inconsistencies found

## Next Steps

| Owner | Task | Exit Criteria | Target Date (UTC) |
|-------|------|---------------|-------------------|
| Development Team | Add migration to existing MIGRATIONS array | Migration script integrated with existing system | 2025-08-30 |
| Development Team | Update type definitions | Settings type updated, ActiveIdentity type created | 2025-08-30 |
| Development Team | Update PlatformServiceMixin | Core methods updated and tested | 2025-08-31 |
| Development Team | Implement foreign key constraints | Schema validation prevents corruption | 2025-08-31 |
| QA Team | Platform testing | All platforms tested and verified | 2025-09-01 |
| Development Team | Deploy migration | Production deployment successful | 2025-09-02 |

## References

- [Database Migration Guide](./database-migration-guide.md)
- [Dexie to SQLite Mapping](./dexie-to-sqlite-mapping.md)
- [PlatformServiceMixin Documentation](./component-communication-guide.md)
- [Migration Templates](./migration-templates/)

## Competence Hooks

- *Why this works*: Separates concerns between identity selection and
  user preferences, prevents data corruption with foreign key constraints,
  centralizes identity management through API layer
- *Common pitfalls*: Forgetting to implement foreign key constraints, not
  testing rollback scenarios, missing data validation during migration,
  over-engineering component updates when API layer handles everything
- *Next skill unlock*: Advanced database schema design with constraints,
  migration planning with rollback strategies
- *Teach-back*: Explain the four-phase migration approach and why each
  phase is necessary, especially the foreign key constraints

## Collaboration Hooks

- **Sign-off checklist**:
  - [ ] Migration script integrated with existing MIGRATIONS array
  - [ ] Foreign key constraints implemented and tested
  - [ ] PlatformServiceMixin updated and tested
  - [ ] Rollback procedures validated
  - [ ] Performance impact assessed
  - [ ] All stakeholders approve deployment timeline

## Assumptions & Limits

- Current activeDid values are valid and should be preserved
- All platforms can handle the additional database table
- Migration can be completed without user downtime
- Rollback to previous schema is acceptable if needed
- Performance impact of additional table join is minimal
- Foreign key constraints will prevent future corruption
- API layer updates will handle component compatibility

## What Needs to Change

### **1. Database Schema via migration.ts**
- Add migration to existing MIGRATIONS array in `src/db-sql/migration.ts`
- Create `active_identity` table with foreign key constraints
- Add performance indexes
- **Keep `activeDid` field in `settings` table temporarily** for backward compatibility
- **Preserve `MASTER_SETTINGS_KEY = "1"`** for legacy migration support

### **2. PlatformServiceMixin Methods**
- `$accountSettings()` - integrate with new table while maintaining backward compatibility
- `$saveSettings()` - handle activeDid in new table, sync with legacy field
- `$updateActiveDid()` - validate and update new table, sync with legacy field
- `$getActiveIdentity()` - new method for identity management

### **3. Master Settings Functions**
- `retrieveSettingsForDefaultAccount()` - integrate with new table while preserving legacy support
- `$getMergedSettings()` - integrate with new table while preserving legacy support

### **4. Type Definitions**
- **Keep `activeDid` in Settings type temporarily** for backward compatibility
- Create ActiveIdentity type for new table
- Update related interfaces

### **5. Legacy Compatibility**
- **Preserve `MASTER_SETTINGS_KEY = "1"`** for IndexedDB migration service
- **Maintain dual-write pattern** during transition period
- **Ensure legacy clients can still migrate** from Dexie to SQLite

## What Doesn't Need to Change

- **All Vue components** - API layer handles migration transparently
- **Platform services** - Use PlatformServiceMixin, no direct access
- **User interface** - No changes to identity selection UI
- **Authentication flow** - Existing system unchanged
- **Component logic** - All activeDid handling through API methods
- **Migration system** - Use existing migration.ts approach, not separate files
- **IndexedDB migration service** - Must continue working for legacy clients

## Enhanced Architecture: Dual-Write Pattern

### **Phase 1: Add New Table (Current)**
```typescript
// Create active_identity table
// Keep existing settings.activeDid for backward compatibility
// Use dual-write pattern during transition
```

### **Phase 2: Dual-Write Pattern**
```typescript
// When updating activeDid:
// 1. Update active_identity table (new system)
// 2. Update settings.activeDid (legacy compatibility)
// 3. Ensure both stay in sync
```

### **Phase 3: Future Cleanup (Not in Current Scope)**
```typescript
// Eventually:
// 1. Remove activeDid from settings table
// 2. Deprecate MASTER_SETTINGS_KEY
// 3. Use pure accountDid IS NULL pattern
// 4. Update IndexedDB migration service
```

## Backward Compatibility Requirements

### **Critical: IndexedDB Migration Service**
- **Must continue working** for users migrating from Dexie
- **Must recognize `id = "1"`** as master settings
- **Must preserve existing migration paths**

### **Important: Legacy Database Operations**
- **Must continue working** for existing SQLite databases
- **Must handle both old and new patterns**
- **Must not break existing queries**

### **Desired: Cleaner Architecture**
- **New operations** use `accountDid IS NULL` pattern
- **Legacy operations** continue using `MASTER_SETTINGS_KEY`
- **Gradual migration** toward cleaner patterns