crowd-funder-from-jason/doc/migration-to-wa-sqlite.md

# Migration Guide: Dexie to absurd-sql

## Overview

This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.

**Current Status**: The migration is in **Phase 2** with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress. **ActiveDid migration has been implemented** to ensure user identity continuity.

## Migration Goals

1. **Data Integrity**
   - Preserve all existing data
   - Maintain data relationships
   - Ensure data consistency
   - **Preserve user's active identity**

2. **Performance**
   - Improve query performance
   - Reduce storage overhead
   - Optimize for platform-specific capabilities

3. **User Experience**
   - Seamless transition with no data loss
   - Maintain user's active identity and preferences
   - Preserve application state

## Migration Architecture

### Migration Fence
The migration fence is defined by the `USE_DEXIE_DB` constant in `src/constants/app.ts`:
- `USE_DEXIE_DB = false` (default): Uses SQLite database
- `USE_DEXIE_DB = true`: Uses Dexie database (for migration purposes)

### Migration Order
The migration follows a specific order to maintain data integrity:

1. **Accounts** (foundational - contains DIDs)
2. **Settings** (references accountDid, activeDid)
3. **ActiveDid** (depends on accounts and settings) ⭐ **NEW**
4. **Contacts** (independent, but migrated after accounts for consistency)

## ActiveDid Migration ⭐ **NEW FEATURE**

### Problem Solved
Previously, the `activeDid` setting was not migrated from Dexie to SQLite, causing users to lose their active identity after migration.

### Solution Implemented
The migration now includes a dedicated step for migrating the `activeDid`:

1. **Detection**: Identifies the `activeDid` from Dexie master settings
2. **Validation**: Verifies the `activeDid` exists in SQLite accounts
3. **Migration**: Updates SQLite master settings with the `activeDid`
4. **Error Handling**: Graceful handling of missing accounts

### Implementation Details

#### New Function: `migrateActiveDid()`
```typescript
export async function migrateActiveDid(): Promise<MigrationResult> {
  // 1. Get Dexie settings to find the activeDid
  const dexieSettings = await getDexieSettings();
  const masterSettings = dexieSettings.find(setting => !setting.accountDid);
  
  // 2. Verify the activeDid exists in SQLite accounts
  const accountExists = await platformService.dbQuery(
    "SELECT did FROM accounts WHERE did = ?",
    [dexieActiveDid],
  );
  
  // 3. Update SQLite master settings
  await updateDefaultSettings({ activeDid: dexieActiveDid });
   }
   ```

#### Enhanced `migrateSettings()` Function
The settings migration now includes activeDid handling:
- Extracts `activeDid` from Dexie master settings
- Validates account existence in SQLite
- Updates SQLite master settings with the `activeDid`

#### Updated `migrateAll()` Function
The complete migration now includes a dedicated step for activeDid:
```typescript
// Step 3: Migrate ActiveDid (depends on accounts and settings)
logger.info("[MigrationService] Step 3: Migrating activeDid...");
const activeDidResult = await migrateActiveDid();
```

### Benefits
- ✅ **User Identity Preservation**: Users maintain their active identity
- ✅ **Seamless Experience**: No need to manually select identity after migration
- ✅ **Data Consistency**: Ensures all identity-related settings are preserved
- ✅ **Error Resilience**: Graceful handling of edge cases

## Migration Process

### Phase 1: Preparation ✅
- [x] Enable Dexie database access
- [x] Implement data comparison tools
- [x] Create migration service structure

### Phase 2: Core Migration ✅
- [x] Account migration with `importFromMnemonic`
- [x] Settings migration (excluding activeDid)
- [x] **ActiveDid migration** ⭐ **COMPLETED**
- [x] Contact migration framework

### Phase 3: Validation and Cleanup 🔄
- [ ] Comprehensive data validation
- [ ] Performance testing
- [ ] User acceptance testing
- [ ] Dexie removal

## Usage

### Manual Migration
```typescript
import { migrateAll, migrateActiveDid } from '../services/indexedDBMigrationService';

// Complete migration
const result = await migrateAll();

// Or migrate just the activeDid
const activeDidResult = await migrateActiveDid();
```

### Migration Verification
```typescript
import { compareDatabases } from '../services/indexedDBMigrationService';

const comparison = await compareDatabases();
console.log('Migration differences:', comparison.differences);
```

## Error Handling

### ActiveDid Migration Errors
- **Missing Account**: If the `activeDid` from Dexie doesn't exist in SQLite accounts
- **Database Errors**: Connection or query failures
- **Settings Update Failures**: Issues updating SQLite master settings

### Recovery Strategies
1. **Automatic Recovery**: Migration continues even if activeDid migration fails
2. **Manual Recovery**: Users can manually select their identity after migration
3. **Fallback**: System creates new identity if none exists

## Security Considerations

### Data Protection
- All sensitive data (mnemonics, private keys) are encrypted
- Migration preserves encryption standards
- No plaintext data exposure during migration

### Identity Verification
- ActiveDid migration validates account existence
- Prevents setting non-existent identities as active
- Maintains cryptographic integrity

## Testing

### Migration Testing
```bash
# Enable Dexie for testing
# Set USE_DEXIE_DB = true in constants/app.ts

# Run migration
npm run migrate

# Verify results
npm run test:migration
```

### ActiveDid Testing
   ```typescript
// Test activeDid migration specifically
const result = await migrateActiveDid();
expect(result.success).toBe(true);
expect(result.warnings).toContain('Successfully migrated activeDid');
```

## Troubleshooting

### Common Issues

1. **ActiveDid Not Found**
   - Ensure accounts were migrated before activeDid migration
   - Check that the Dexie activeDid exists in SQLite accounts

2. **Migration Failures**
   - Verify Dexie database is accessible
   - Check SQLite database permissions
   - Review migration logs for specific errors

3. **Data Inconsistencies**
   - Use `compareDatabases()` to identify differences
   - Re-run migration if necessary
   - Check for duplicate or conflicting records

### Debugging
```typescript
// Enable detailed logging
logger.setLevel('debug');

// Check migration status
const comparison = await compareDatabases();
console.log('Settings differences:', comparison.differences.settings);
   ```

## Future Enhancements

### Planned Improvements
1. **Batch Processing**: Optimize for large datasets
2. **Incremental Migration**: Support partial migrations
3. **Rollback Capability**: Ability to revert migration
4. **Progress Tracking**: Real-time migration progress

### Performance Optimizations
1. **Parallel Processing**: Migrate independent data concurrently
2. **Memory Management**: Optimize for large datasets
3. **Transaction Batching**: Reduce database round trips

## Conclusion

The Dexie to SQLite migration provides a robust, secure, and user-friendly transition path. The addition of activeDid migration ensures that users maintain their identity continuity throughout the migration process, significantly improving the user experience.

The migration fence architecture allows for controlled, reversible migration while maintaining application stability and data integrity.