crowd-funder-from-jason/doc/activeDid-migration-plan.md

# ActiveDid Migration Plan - Separate Table Architecture

**Author**: Matthew Raymer
**Date**: 2025-08-29T07:24Z
**Status**: 🎯 **PLANNING** - Active migration planning phase

## Objective

Move the `activeDid` field from the `settings` table to a dedicated
`active_identity` table to improve database architecture, prevent data corruption,
and separate identity selection from user preferences.

## Result

This document serves as the comprehensive planning and implementation
guide for the ActiveDid migration with enhanced data integrity and
rollback capabilities.

## Use/Run

Reference this document during implementation to ensure all migration
steps are followed correctly and all stakeholders are aligned on the
approach.

## Context & Scope

- **In scope**:
  - Database schema modification for active_identity table with proper constraints
  - Migration of existing activeDid data with validation
  - Updates to PlatformServiceMixin API layer
  - Type definition updates
  - Testing across all platforms
  - Comprehensive rollback procedures
- **Out of scope**:
  - Changes to user interface for identity selection
  - Modifications to identity creation logic
  - Changes to authentication flow
  - Updates to individual components (handled by API layer)

## Environment & Preconditions

- **OS/Runtime**: All platforms (Web, Electron, iOS, Android)
- **Versions/Builds**: Current development branch, SQLite database
- **Services/Endpoints**: Local database, PlatformServiceMixin
- **Auth mode**: Existing authentication system unchanged

## Architecture / Process Overview

The migration follows a phased approach to minimize risk and ensure
data integrity with enhanced validation and rollback capabilities:

```mermaid
flowchart TD
    A[Current State<br/>activeDid in settings] --> B[Phase 1: Schema Creation<br/>Add active_identity table with constraints]
    B --> C[Phase 2: Data Migration<br/>Copy activeDid data with validation]
    C --> D[Phase 3: API Updates<br/>Update PlatformServiceMixin methods]
    D --> E[Phase 4: Cleanup<br/>Remove activeDid from settings]
    E --> F[Final State<br/>Separate active_identity table]

    G[Enhanced Rollback Plan<br/>Schema and data rollback] --> H[Data Validation<br/>Verify integrity at each step]
    H --> I[Platform Testing<br/>Test all platforms]
    I --> J[Production Deployment<br/>Gradual rollout with monitoring]
    
    K[Foreign Key Constraints<br/>Prevent future corruption] --> L[Performance Optimization<br/>Proper indexing]
    L --> M[Error Recovery<br/>Graceful failure handling]
```

## Interfaces & Contracts

### Database Schema Changes

| Table | Current Schema | New Schema | Migration Required |
|-------|----------------|------------|-------------------|
| `settings` | `activeDid TEXT` | Field removed | Yes - data migration |
| `active_identity` | Does not exist | New table with `activeDid TEXT` + constraints | Yes - table creation |

### Enhanced API Contract Changes

| Method | Current Behavior | New Behavior | Breaking Change |
|---------|------------------|--------------|-----------------|
| `$accountSettings()` | Returns settings with activeDid | Returns settings with activeDid from new table | No - backward compatible |
| `$saveSettings()` | Updates settings.activeDid | Updates active_identity.activeDid | Yes - requires updates |
| `$updateActiveDid()` | Updates internal tracking | Updates active_identity table | Yes - requires updates |
| `$getActiveIdentity()` | Does not exist | New method for active identity management | No - new functionality |

## Repro: End-to-End Procedure

### Phase 1: Enhanced Schema Creation

```sql
-- Create new active_identity table with proper constraints
CREATE TABLE active_identity (
  id INTEGER PRIMARY KEY CHECK (id = 1),
  activeDid TEXT NOT NULL,
  lastUpdated TEXT NOT NULL DEFAULT (datetime('now')),
  FOREIGN KEY (activeDid) REFERENCES accounts(did) ON DELETE CASCADE
);

-- Add performance indexes
CREATE INDEX IF NOT EXISTS idx_active_identity_activeDid ON active_identity(activeDid);
CREATE UNIQUE INDEX IF NOT EXISTS idx_active_identity_single_record ON active_identity(id);

-- Insert default record (will be updated during migration)
INSERT INTO active_identity (id, activeDid, lastUpdated) VALUES (1, '', datetime('now'));
```

### Phase 2: Enhanced Data Migration with Validation

```typescript
// Enhanced migration script with comprehensive validation
async function migrateActiveDidToSeparateTable(): Promise<MigrationResult> {
  const result: MigrationResult = { 
    success: false, 
    errors: [], 
    warnings: [],
    dataMigrated: 0
  };
  
  try {
    // 1. Get current activeDid from settings
    const currentSettings = await retrieveSettingsForDefaultAccount();
    const activeDid = currentSettings.activeDid;

    if (!activeDid) {
      result.warnings.push("No activeDid found in current settings");
      return result;
    }

    // 2. Validate activeDid exists in accounts table
    const accountExists = await dbQuery(
      "SELECT did FROM accounts WHERE did = ?",
      [activeDid]
    );

    if (!accountExists?.values?.length) {
      result.errors.push(`ActiveDid ${activeDid} not found in accounts table - data corruption detected`);
      return result;
    }

    // 3. Check if active_identity table already has data
    const existingActiveIdentity = await dbQuery(
      "SELECT activeDid FROM active_identity WHERE id = 1"
    );

    if (existingActiveIdentity?.values?.length) {
      // Update existing record
      await dbExec(
        "UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1",
        [activeDid]
      );
    } else {
      // Insert new record
      await dbExec(
        "INSERT INTO active_identity (id, activeDid, lastUpdated) VALUES (1, ?, datetime('now'))",
        [activeDid]
      );
    }

    result.success = true;
    result.dataMigrated = 1;
    result.warnings.push(`Successfully migrated activeDid: ${activeDid}`);
    
  } catch (error) {
    result.errors.push(`Migration failed: ${error}`);
    logger.error("[ActiveDid Migration] Critical error during migration:", error);
  }

  return result;
}

// Migration result interface
interface MigrationResult {
  success: boolean;
  errors: string[];
  warnings: string[];
  dataMigrated: number;
}
```

### Phase 3: Focused API Updates

```typescript
// Updated PlatformServiceMixin method - maintains backward compatibility
async $accountSettings(did?: string, defaults: Settings = {}): Promise<Settings> {
  try {
    // Get settings without activeDid (unchanged logic)
    const settings = await this._getSettingsWithoutActiveDid();

    if (!settings) {
      return defaults;
    }

    // Get activeDid from new table (new logic)
    const activeIdentity = await this._getActiveIdentity();

    // Return combined result (maintains backward compatibility)
    return { ...settings, activeDid: activeIdentity.activeDid };
  } catch (error) {
    logger.error("[Settings Trace] ❌ Error in $accountSettings:", error);
    return defaults;
  }
}

// New method for active identity management
async $getActiveIdentity(): Promise<{ activeDid: string | null }> {
  try {
    const result = await this.$dbQuery(
      "SELECT activeDid FROM active_identity WHERE id = 1"
    );
    
    if (!result?.values?.length) {
      return { activeDid: null };
    }
    
    return { activeDid: result.values[0][0] as string };
  } catch (error) {
    logger.error("[Settings Trace] ❌ Failed to get active identity:", error);
    return { activeDid: null };
  }
}

// Enhanced method to get settings without activeDid
async _getSettingsWithoutActiveDid(): Promise<Settings> {
  const result = await this.$dbQuery(
    "SELECT id, accountDid, apiServer, filterFeedByNearby, filterFeedByVisible, " +
    "finishedOnboarding, firstName, hideRegisterPromptOnNewContact, isRegistered, " +
    "lastName, lastAckedOfferToUserJwtId, lastAckedOfferToUserProjectsJwtId, " +
    "lastNotifiedClaimId, lastViewedClaimId, notifyingNewActivityTime, " +
    "notifyingReminderMessage, notifyingReminderTime, partnerApiServer, " +
    "passkeyExpirationMinutes, profileImageUrl, searchBoxes, showContactGivesInline, " +
    "showGeneralAdvanced, showShortcutBvc, vapid, warnIfProdServer, warnIfTestServer, " +
    "webPushServer FROM settings WHERE id = ?",
    [MASTER_SETTINGS_KEY]
  );

  if (!result?.values?.length) {
    return DEFAULT_SETTINGS;
  }

  return this._mapColumnsToValues(result.columns, result.values)[0] as Settings;
}

// Enhanced save settings method
async $saveSettings(changes: Partial<Settings>): Promise<boolean> {
  try {
    // Remove fields that shouldn't be updated
    const { accountDid, id, activeDid, ...safeChanges } = changes;
    
    if (Object.keys(safeChanges).length > 0) {
      // Convert settings for database storage
      const convertedChanges = this._convertSettingsForStorage(safeChanges);
      const setParts: string[] = [];
      const params: unknown[] = [];

      Object.entries(convertedChanges).forEach(([key, value]) => {
        if (value !== undefined) {
          setParts.push(`${key} = ?`);
          params.push(value);
        }
      });

      if (setParts.length > 0) {
        params.push(MASTER_SETTINGS_KEY);
        await this.$dbExec(
          `UPDATE settings SET ${setParts.join(", ")} WHERE id = ?`,
          params,
        );
      }
    }

    // Handle activeDid separately in new table
    if (changes.activeDid !== undefined) {
      await this.$updateActiveDid(changes.activeDid);
    }

    return true;
  } catch (error) {
    logger.error("[PlatformServiceMixin] Error saving settings:", error);
    return false;
  }
}

// Enhanced update activeDid method
async $updateActiveDid(newDid: string | null): Promise<boolean> {
  try {
    if (newDid === null) {
      // Clear active identity
      await this.$dbExec(
        "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1"
      );
    } else {
      // Validate DID exists before setting
      const accountExists = await this.$dbQuery(
        "SELECT did FROM accounts WHERE did = ?",
        [newDid]
      );

      if (!accountExists?.values?.length) {
        logger.error(`[PlatformServiceMixin] Cannot set activeDid to non-existent DID: ${newDid}`);
        return false;
      }

      // Update active identity
      await this.$dbExec(
        "UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1",
        [newDid]
      );
    }

    // Update internal tracking
    await this._updateInternalActiveDid(newDid);
    return true;
  } catch (error) {
    logger.error("[PlatformServiceMixin] Error updating activeDid:", error);
    return false;
  }
}
```

### **Master Settings Functions Implementation Strategy**

#### **1. Update `retrieveSettingsForDefaultAccount()`**

```typescript
// Enhanced implementation with active_identity table integration
export async function retrieveSettingsForDefaultAccount(): Promise<Settings> {
  const platform = PlatformServiceFactory.getInstance();

  // Get settings without activeDid
  const sql = "SELECT id, accountDid, apiServer, filterFeedByNearby, filterFeedByVisible, " +
              "finishedOnboarding, firstName, hideRegisterPromptOnNewContact, isRegistered, " +
              "lastName, lastAckedOfferToUserJwtId, lastAckedOfferToUserProjectsJwtId, " +
              "lastNotifiedClaimId, lastViewedClaimId, notifyingNewActivityTime, " +
              "notifyingReminderMessage, notifyingReminderTime, partnerApiServer, " +
              "passkeyExpirationMinutes, profileImageUrl, searchBoxes, showContactGivesInline, " +
              "showGeneralAdvanced, showShortcutBvc, vapid, warnIfProdServer, warnIfTestServer, " +
              "webPushServer FROM settings WHERE id = ?";

  const result = await platform.dbQuery(sql, [MASTER_SETTINGS_KEY]);

  if (!result) {
    return DEFAULT_SETTINGS;
  } else {
    const settings = mapColumnsToValues(result.columns, result.values)[0] as Settings;

    // Handle JSON parsing
    if (settings.searchBoxes) {
      settings.searchBoxes = JSON.parse(settings.searchBoxes);
    }

    // Get activeDid from separate table
    const activeIdentityResult = await platform.dbQuery(
      "SELECT activeDid FROM active_identity WHERE id = 1"
    );

    if (activeIdentityResult?.values?.length) {
      const activeDid = activeIdentityResult.values[0][0] as string;
      if (activeDid) {
        // Validate activeDid exists in accounts
        const accountExists = await platform.dbQuery(
          "SELECT did FROM accounts WHERE did = ?",
          [activeDid]
        );
        
        if (accountExists?.values?.length) {
          settings.activeDid = activeDid;
        } else {
          logger.warn(`[databaseUtil] ActiveDid ${activeDid} not found in accounts, clearing`);
          // Clear corrupted activeDid
          await platform.dbExec(
            "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1"
          );
        }
      }
    }

    return settings;
  }
}
```

#### **2. Update `$getMergedSettings()` Method**

```typescript
// Enhanced implementation with active_identity table integration
async $getMergedSettings(defaultKey: string, accountDid?: string, defaultFallback: Settings = {}): Promise<Settings> {
  try {
    // Get default settings (now without activeDid)
    const defaultSettings = await this.$getSettings(defaultKey, defaultFallback);

    // If no account DID, return defaults with activeDid from separate table
    if (!accountDid) {
      if (defaultSettings) {
        // Get activeDid from separate table
        const activeIdentityResult = await this.$dbQuery(
          "SELECT activeDid FROM active_identity WHERE id = 1"
        );

        if (activeIdentityResult?.values?.length) {
          const activeDid = activeIdentityResult.values[0][0] as string;
          if (activeDid) {
            // Validate activeDid exists in accounts
            const accountExists = await this.$dbQuery(
              "SELECT did FROM accounts WHERE did = ?",
              [activeDid]
            );
            
            if (accountExists?.values?.length) {
              defaultSettings.activeDid = activeDid;
            } else {
              logger.warn(`[Settings Trace] ActiveDid ${activeDid} not found in accounts, clearing`);
              // Clear corrupted activeDid
              await this.$dbExec(
                "UPDATE active_identity SET activeDid = '', lastUpdated = datetime('now') WHERE id = 1"
              );
            }
          }
        }
      }
      return defaultSettings || defaultFallback;
    }

    // ... rest of existing implementation for account-specific settings
  } catch (error) {
    logger.error(`[Settings Trace] ❌ Failed to get merged settings:`, { defaultKey, accountDid, error });
    return defaultFallback;
  }
}
```

## What Works (Evidence)

- ✅ **Current activeDid storage** in settings table
  - **Time**: 2025-08-29T07:24Z
  - **Evidence**: `src/db/tables/settings.ts:25` - activeDid field exists
  - **Verify at**: Current database schema and Settings type definition

- ✅ **PlatformServiceMixin integration** with activeDid
  - **Time**: 2025-08-29T07:24Z
  - **Evidence**: `src/utils/PlatformServiceMixin.ts:108` - activeDid tracking
  - **Verify at**: Component usage across all platforms

- ✅ **Database migration infrastructure** exists
  - **Time**: 2025-08-29T07:24Z
  - **Evidence**: `src/db-sql/migration.ts:31` - migration system in place
  - **Verify at**: Existing migration scripts and database versioning

## What Doesn't (Evidence & Hypotheses)

- ❌ **No separate active_identity table** exists
  - **Time**: 2025-08-29T07:24Z
  - **Evidence**: Database schema only shows settings table
  - **Hypothesis**: Table needs to be created as part of migration
  - **Next probe**: Create migration script for new table

- ❌ **Data corruption issues** with orphaned activeDid references
  - **Time**: 2025-08-29T07:24Z
  - **Evidence**: `IdentitySwitcherView.vue:175` - `hasCorruptedIdentity` detection
  - **Hypothesis**: Current schema allows activeDid to point to non-existent accounts
  - **Next probe**: Implement foreign key constraints in new table

## Risks, Limits, Assumptions

- **Data Loss Risk**: Migration failure could lose activeDid values
- **Breaking Changes**: API updates required in PlatformServiceMixin
- **Rollback Complexity**: Schema changes make rollback difficult
- **Testing Overhead**: All platforms must be tested with new structure
- **Performance Impact**: Additional table join for activeDid retrieval
- **Migration Timing**: Must be coordinated with other database changes
- **Data Corruption**: Current system has documented corruption issues
- **Foreign Key Constraints**: New constraints may prevent some operations

## Enhanced Rollback Strategy

### **Schema Rollback**
```sql
-- If migration fails, restore original schema
DROP TABLE IF EXISTS active_identity;

-- Restore activeDid field to settings table if needed
ALTER TABLE settings ADD COLUMN activeDid TEXT;
```

### **Data Rollback**
```typescript
// Rollback function to restore activeDid to settings table
async function rollbackActiveDidMigration(): Promise<boolean> {
  try {
    // Get activeDid from active_identity table
    const activeIdentityResult = await dbQuery(
      "SELECT activeDid FROM active_identity WHERE id = 1"
    );
    
    if (activeIdentityResult?.values?.length) {
      const activeDid = activeIdentityResult.values[0][0] as string;
      
      // Restore to settings table
      await dbExec(
        "UPDATE settings SET activeDid = ? WHERE id = ?",
        [activeDid, MASTER_SETTINGS_KEY]
      );
      
      return true;
    }
    
    return false;
  } catch (error) {
    logger.error("[Rollback] Failed to restore activeDid:", error);
    return false;
  }
}
```

### **Rollback Triggers**
- Migration validation fails
- Data integrity checks fail
- Performance regression detected
- User reports data loss
- Cross-platform inconsistencies found

## Next Steps

| Owner | Task | Exit Criteria | Target Date (UTC) |
|-------|------|---------------|-------------------|
| Development Team | Create enhanced migration script | Migration script with validation and rollback | 2025-08-30 |
| Development Team | Update type definitions | Settings type updated, ActiveIdentity type created | 2025-08-30 |
| Development Team | Update PlatformServiceMixin | Core methods updated and tested | 2025-08-31 |
| Development Team | Implement foreign key constraints | Schema validation prevents corruption | 2025-08-31 |
| QA Team | Platform testing | All platforms tested and verified | 2025-09-01 |
| Development Team | Deploy migration | Production deployment successful | 2025-09-02 |

## References

- [Database Migration Guide](./database-migration-guide.md)
- [Dexie to SQLite Mapping](./dexie-to-sqlite-mapping.md)
- [PlatformServiceMixin Documentation](./component-communication-guide.md)
- [Migration Templates](./migration-templates/)

## Competence Hooks

- *Why this works*: Separates concerns between identity selection and
  user preferences, prevents data corruption with foreign key constraints,
  centralizes identity management through API layer
- *Common pitfalls*: Forgetting to implement foreign key constraints, not
  testing rollback scenarios, missing data validation during migration,
  over-engineering component updates when API layer handles everything
- *Next skill unlock*: Advanced database schema design with constraints,
  migration planning with rollback strategies
- *Teach-back*: Explain the four-phase migration approach and why each
  phase is necessary, especially the foreign key constraints

## Collaboration Hooks

- **Sign-off checklist**:
  - [ ] Migration script tested on development database
  - [ ] Foreign key constraints implemented and tested
  - [ ] PlatformServiceMixin updated and tested
  - [ ] Rollback procedures validated
  - [ ] Performance impact assessed
  - [ ] All stakeholders approve deployment timeline

## Assumptions & Limits

- Current activeDid values are valid and should be preserved
- All platforms can handle the additional database table
- Migration can be completed without user downtime
- Rollback to previous schema is acceptable if needed
- Performance impact of additional table join is minimal
- Foreign key constraints will prevent future corruption
- API layer updates will handle component compatibility

## What Needs to Change

### **1. Database Schema**
- Create `active_identity` table with foreign key constraints
- Add performance indexes
- Remove `activeDid` field from `settings` table

### **2. PlatformServiceMixin Methods**
- `$accountSettings()` - integrate with new table
- `$saveSettings()` - handle activeDid in new table
- `$updateActiveDid()` - validate and update new table
- `$getActiveIdentity()` - new method for identity management

### **3. Master Settings Functions**
- `retrieveSettingsForDefaultAccount()` - integrate with new table
- `$getMergedSettings()` - integrate with new table

### **4. Migration Scripts**
- Create migration script with validation
- Implement rollback procedures
- Add data corruption detection

### **5. Type Definitions**
- Update Settings type to remove activeDid
- Create ActiveIdentity type for new table
- Update related interfaces

## What Doesn't Need to Change

- **All Vue components** - API layer handles migration transparently
- **Platform services** - Use PlatformServiceMixin, no direct access
- **User interface** - No changes to identity selection UI
- **Authentication flow** - Existing system unchanged
- **Component logic** - All activeDid handling through API methods