crowd-funder-from-jason/doc/active-identity-upgrade-plan.md

# Active Identity Upgrade Plan

**Author**: Matthew Raymer  
**Date**: 2025-09-11  
**Status**: 🎯 **PLANNING** - Database migration and active identity system upgrade

## Overview

Comprehensive upgrade to the active identity system, addressing architectural issues and implementing enhanced database constraints. Includes database migration enhancements and settings table cleanup based on team feedback.

## Implementation Status

**✅ COMPLETED**: Migration structure updated according to team member feedback

### Implemented Changes

1. **✅ Migration 003**: `003_add_hasBackedUpSeed_to_settings` - Adds `hasBackedUpSeed` column to settings (assumes master deployment)
2. **✅ Migration 004**: `004_active_identity_and_seed_backup` - Creates `active_identity` table with data migration
3. **✅ Migration Service**: Updated validation and schema detection logic for new migration structure
4. **✅ TypeScript**: Fixed type compatibility issues

### Migration Structure Now Follows Team Guidance

- **Migration 003**: `003_add_hasBackedUpSeed_to_settings` (assumes master code deployed)
- **Migration 004**: `004_active_identity_and_seed_backup` (creates active_identity table)
- **All migrations are additional** - no editing of previous migrations
- **Data migration logic** preserves existing `activeDid` from settings
- **iOS/Android compatibility** confirmed with SQLCipher 4.9.0 (SQLite 3.44.2)

## Educational Context

### Why This Upgrade Matters

The active identity system is **critical infrastructure** affecting every user interaction:

1. **Data Integrity**: Current `ON DELETE SET NULL` allows accidental deletion of active accounts
2. **Manual Maintenance**: Timestamps require manual updates, creating inconsistency opportunities
3. **Architectural Clarity**: Separating active identity from user settings improves maintainability

### What This Upgrade Achieves

- **Prevents Data Loss**: `ON DELETE RESTRICT` prevents accidental account deletion
- **Automatic Consistency**: Database triggers ensure timestamps are always current
- **Cleaner Architecture**: Complete separation of identity management from user preferences
- **Better Performance**: Optimized indexes for faster account selection

## Current State Analysis

### Existing Migration Structure

- **Migration 003**: `003_add_hasBackedUpSeed_to_settings` - Adds `hasBackedUpSeed` column to settings (already deployed in master)
- **Migration 004**: `004_active_identity_and_seed_backup` - Creates `active_identity` table with data migration
- **Foreign Key**: `ON DELETE SET NULL` constraint
- **Data Migration**: Copies existing `activeDid` from settings to `active_identity` table
- **Bootstrapping**: Auto-selects first account if `activeDid` is null/empty

**Important**: All migrations are **additional** - no editing of previous migrations since master code has been deployed.

### Current Schema (Migration 004) - IMPLEMENTED

```sql
-- Migration 004: active_identity_and_seed_backup
-- Assumes master code deployed with migration 003

PRAGMA foreign_keys = ON;
CREATE UNIQUE INDEX IF NOT EXISTS idx_accounts_did_unique ON accounts(did);

CREATE TABLE IF NOT EXISTS active_identity (
  id INTEGER PRIMARY KEY CHECK (id = 1),
  activeDid TEXT DEFAULT NULL,
  lastUpdated TEXT NOT NULL DEFAULT (datetime('now')),
  FOREIGN KEY (activeDid) REFERENCES accounts(did) ON DELETE SET NULL
);

CREATE UNIQUE INDEX IF NOT EXISTS idx_active_identity_single_record ON active_identity(id);

-- Seed singleton row
INSERT INTO active_identity (id, activeDid, lastUpdated)
SELECT 1, NULL, datetime('now')
WHERE NOT EXISTS (SELECT 1 FROM active_identity WHERE id = 1);

-- MIGRATE EXISTING DATA: Copy activeDid from settings
UPDATE active_identity
SET activeDid = (SELECT activeDid FROM settings WHERE id = 1),
    lastUpdated = datetime('now')
WHERE id = 1
AND EXISTS (SELECT 1 FROM settings WHERE id = 1 AND activeDid IS NOT NULL AND activeDid != '');
```

## Current Implementation Details

### PlatformServiceMixin.ts Implementation

The current `$getActiveIdentity()` method in `src/utils/PlatformServiceMixin.ts`:

```typescript
async $getActiveIdentity(): Promise<{ activeDid: string }> {
  try {
    const result = await this.$dbQuery("SELECT activeDid FROM active_identity WHERE id = 1");
    
    if (!result?.values?.length) {
      logger.warn("[PlatformServiceMixin] Active identity table is empty - migration issue");
      return { activeDid: "" };
    }

    const activeDid = result.values[0][0] as string | null;

    // Handle null activeDid - auto-select first account
    if (activeDid === null) {
      const firstAccount = await this.$dbQuery("SELECT did FROM accounts ORDER BY dateCreated, did LIMIT 1");
      if (firstAccount?.values?.length) {
        const firstAccountDid = firstAccount.values[0][0] as string;
        await this.$dbExec("UPDATE active_identity SET activeDid = ?, lastUpdated = datetime('now') WHERE id = 1", [firstAccountDid]);
        return { activeDid: firstAccountDid };
      }
      logger.warn("[PlatformServiceMixin] No accounts available for auto-selection");
      return { activeDid: "" };
    }

    // Validate activeDid exists in accounts
    const accountExists = await this.$dbQuery("SELECT did FROM accounts WHERE did = ?", [activeDid]);
    if (accountExists?.values?.length) {
      return { activeDid };
    }

    // Clear corrupted activeDid and return empty
    logger.warn("[PlatformServiceMixin] Active identity not found in accounts, clearing");
    await this.$dbExec("UPDATE active_identity SET activeDid = NULL, lastUpdated = datetime('now') WHERE id = 1");
    return { activeDid: "" };
  } catch (error) {
    logger.error("[PlatformServiceMixin] Error getting active identity:", error);
    return { activeDid: "" };
  }
}
```

### Key Implementation Notes

1. **Null Handling**: Auto-selects first account when `activeDid` is null
2. **Corruption Detection**: Clears invalid `activeDid` values
3. **Manual Timestamps**: Updates `lastUpdated` manually in code
4. **Error Handling**: Returns empty string on any error with appropriate logging

## Proposed Changes Impact

### 1. Foreign Key Constraint Change
**Current**: `ON DELETE SET NULL` → **Proposed**: `ON DELETE RESTRICT`
- **Data Safety**: Prevents accidental deletion of active account
- **New Migration**: Add migration 005 to update constraint

### 2. Automatic Timestamp Updates
**Current**: Manual `lastUpdated` updates → **Proposed**: Database trigger
- **Code Simplification**: Remove manual timestamp updates from `PlatformServiceMixin`
- **Consistency**: Ensures `lastUpdated` is always current

### 3. Enhanced Indexing
**Current**: Single unique index on `id` → **Proposed**: Additional index on `accounts(dateCreated, did)`
- **Performance Improvement**: Faster account selection queries
- **Minimal Risk**: Additive change only

## Implementation Strategy

### Add Migration 005

Since the `active_identity` table already exists and is working, we can add a new migration to enhance it:

```sql
{
  name: "005_active_identity_enhancements",
  sql: `
    PRAGMA foreign_keys = ON;
    
    -- Recreate table with ON DELETE RESTRICT constraint
    CREATE TABLE active_identity_new (
      id INTEGER PRIMARY KEY CHECK (id = 1),
      activeDid TEXT REFERENCES accounts(did) ON DELETE RESTRICT,
      lastUpdated TEXT NOT NULL DEFAULT (datetime('now'))
    );
    
    -- Copy existing data
    INSERT INTO active_identity_new (id, activeDid, lastUpdated)
    SELECT id, activeDid, lastUpdated FROM active_identity;
    
    -- Replace old table
    DROP TABLE active_identity;
    ALTER TABLE active_identity_new RENAME TO active_identity;
    
    -- Add performance indexes
    CREATE INDEX IF NOT EXISTS idx_accounts_pick ON accounts(dateCreated, did);
    
    -- Create automatic timestamp trigger
    CREATE TRIGGER IF NOT EXISTS trg_active_identity_touch
    AFTER UPDATE ON active_identity
    FOR EACH ROW
    BEGIN
      UPDATE active_identity
        SET lastUpdated = datetime('now')
        WHERE id = 1;
    END;
  `
}
```

## Migration Service Updates Required

### Enhanced Validation Logic

**File**: `src/services/migrationService.ts`

**Migration 004 validation**:
- **Table existence**: `SELECT name FROM sqlite_master WHERE type='table' AND name='active_identity'`
- **Column structure**: `SELECT id, activeDid, lastUpdated FROM active_identity LIMIT 1`
- **Schema detection**: Uses `isSchemaAlreadyPresent()` to check if migration was already applied

**Migration 005 validation**:
- **Trigger existence**: `trg_active_identity_touch`
- **Performance index**: `idx_accounts_pick`
- **Foreign key constraint**: `ON DELETE RESTRICT`
- **Table recreation**: Verify table was successfully recreated

### Enhanced Schema Detection

**Migration 004 verification**:
- **Table structure**: Checks `active_identity` table exists and has correct columns
- **Data integrity**: Validates that the table can be queried successfully
- **Migration tracking**: Uses `isSchemaAlreadyPresent()` to avoid re-applying migrations

**Migration 005 verification**:
- **Table structure**: Enhanced constraints with `ON DELETE RESTRICT`
- **Trigger presence**: Automatic timestamp updates
- **Index presence**: Performance optimization
- **Data integrity**: Existing data was preserved during table recreation

## Risk Assessment

### Low Risk Changes
- **Performance Index**: Additive only, no data changes
- **Trigger Creation**: Additive only, improves consistency
- **New Migration**: Clean implementation, no modification of existing migrations

### Medium Risk Changes
- **Foreign Key Change**: `ON DELETE RESTRICT` is more restrictive
- **Table Recreation**: Requires careful data preservation
- **Validation Updates**: Need to test enhanced validation logic

### Mitigation Strategies
1. **Comprehensive Testing**: Test migration on various database states
2. **Data Preservation**: Verify existing data is copied correctly
3. **Clean Implementation**: New migration with all enhancements
4. **Validation Coverage**: Enhanced validation ensures correctness
5. **Rollback Plan**: Can drop new table and restore original if needed

## Implementation Timeline

### Phase 1: Migration Enhancement
- [ ] Add migration 005 with enhanced constraints
- [ ] Add enhanced validation logic
- [ ] Add enhanced schema detection logic
- [ ] Test migration on clean database

### Phase 2: Testing
- [ ] Test migration on existing databases
- [ ] Validate foreign key constraints work correctly
- [ ] Test trigger functionality
- [ ] Test performance improvements
- [ ] Verify data preservation during table recreation

### Phase 3: Deployment
- [ ] Deploy enhanced migration to development
- [ ] Monitor migration success rates
- [ ] Deploy to production
- [ ] Monitor for any issues

### Phase 4: Settings Table Cleanup
- [ ] Create migration 006 to clean up settings table
- [ ] Remove orphaned settings records (accountDid is null)
- [ ] Clear any remaining activeDid values in settings
- [ ] Consider removing activeDid column entirely (future task)

## Settings Table Cleanup Strategy

### Current State Analysis
The settings table currently contains:
- **Legacy activeDid column**: Still present from original design
- **Orphaned records**: Settings with `accountDid = null` that may be obsolete
- **Redundant data**: Some settings may have been copied unnecessarily

Based on team feedback, the cleanup should include:

1. **Remove orphaned settings records**:
   ```sql
   DELETE FROM settings WHERE accountDid IS NULL;
   ```

2. **Clear any remaining activeDid values**:
   ```sql
   UPDATE settings SET activeDid = NULL;
   ```

3. **Future consideration**: Remove the activeDid column entirely from settings table

### Migration 006: Settings Cleanup

```sql
{
  name: "006_settings_cleanup",
  sql: `
    -- Remove orphaned settings records (accountDid is null)
    DELETE FROM settings WHERE accountDid IS NULL;
    
    -- Clear any remaining activeDid values in settings
    UPDATE settings SET activeDid = NULL;
    
    -- Optional: Consider removing the activeDid column entirely
    -- ALTER TABLE settings DROP COLUMN activeDid;
  `
}
```

### Benefits of Settings Cleanup
- **Reduced confusion**: Eliminates dual-purpose columns
- **Cleaner architecture**: Settings table focuses only on user preferences
- **Reduced storage**: Removes unnecessary data
- **Clearer separation**: Active identity vs. user settings are distinct concerns

### Risk Assessment: LOW
- **Data safety**: Only removes orphaned/obsolete records
- **Backward compatibility**: Maintains existing column structure
- **Rollback**: Easy to restore if needed
- **Testing**: Can be validated with existing data

## Code Changes Required

### Files to Modify
1. **`src/db-sql/migration.ts`** - Add migration 005 with enhanced constraints
2. **`src/db-sql/migration.ts`** - Add migration 006 for settings cleanup
3. **`src/services/migrationService.ts`** - Add enhanced validation and detection logic
4. **`src/utils/PlatformServiceMixin.ts`** - Remove manual timestamp updates

### Estimated Impact
- **Migration File**: ~25 lines added (migration 005) + ~15 lines added (migration 006)
- **Migration Service**: ~50 lines added (enhanced validation)
- **PlatformServiceMixin**: ~20 lines removed (manual timestamps)
- **Total**: ~90 lines changed

## Conclusion

**✅ IMPLEMENTATION COMPLETE**: The active identity upgrade plan has been successfully applied to the current project.

### Successfully Implemented

**✅ Migration Structure Updated**:
- **Migration 003**: `003_add_hasBackedUpSeed_to_settings` (assumes master deployment)
- **Migration 004**: `004_active_identity_and_seed_backup` (creates active_identity table)
- **All migrations are additional** - follows team member feedback exactly

**✅ Technical Implementation**:
- **Data Migration**: Preserves existing `activeDid` from settings table
- **Foreign Key Constraints**: `ON DELETE SET NULL` for data safety
- **iOS/Android Compatibility**: Confirmed with SQLCipher 4.9.0 (SQLite 3.44.2)
- **Migration Service**: Updated validation and schema detection logic

**✅ Code Quality**:
- **TypeScript**: All type errors resolved
- **Linting**: No linting errors
- **Team Guidance**: Follows "additional migrations only" requirement

### Next Steps (Future Enhancements)

The foundation is now in place for future enhancements:

1. **Migration 005**: `005_active_identity_enhancements` (ON DELETE RESTRICT, triggers, indexes)
2. **Migration 006**: `006_settings_cleanup` (remove orphaned settings, clear legacy activeDid)
3. **Code Simplification**: Remove manual timestamp updates from PlatformServiceMixin

### Current Status

**Migration 004 is ready for deployment** and will:
- ✅ Create `active_identity` table with proper constraints
- ✅ Migrate existing `activeDid` data from settings
- ✅ Work identically on iOS and Android
- ✅ Follow team member feedback for additional migrations only

**Key Point**: All migrations are **additional** - no editing of previous migrations since master code has been deployed. This ensures compatibility and proper testing.

---

**Status**: Ready for team review and implementation approval  
**Last Updated**: 2025-09-11  
**Next Review**: After team feedback and approval