forked from trent_larson/crowd-funder-for-time-pwa
feat: Implement activeDid migration from Dexie to SQLite
- Add migrateActiveDid() function for dedicated activeDid migration - Enhance migrateSettings() to handle activeDid extraction and validation - Update migrateAll() to include activeDid migration step - Add comprehensive error handling and validation - Update migration documentation with activeDid migration details - Ensure user identity continuity during migration process Files changed: - src/services/indexedDBMigrationService.ts (153 lines added) - doc/migration-to-wa-sqlite.md (documentation updated) Migration order: Accounts -> Settings -> ActiveDid -> Contacts
This commit is contained in:
@@ -4,7 +4,7 @@
|
|||||||
|
|
||||||
This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.
|
This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.
|
||||||
|
|
||||||
**Current Status**: The migration is in **Phase 2** with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress.
|
**Current Status**: The migration is in **Phase 2** with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress. **ActiveDid migration has been implemented** to ensure user identity continuity.
|
||||||
|
|
||||||
## Migration Goals
|
## Migration Goals
|
||||||
|
|
||||||
@@ -12,403 +12,215 @@ This document outlines the migration process from Dexie.js to absurd-sql for the
|
|||||||
- Preserve all existing data
|
- Preserve all existing data
|
||||||
- Maintain data relationships
|
- Maintain data relationships
|
||||||
- Ensure data consistency
|
- Ensure data consistency
|
||||||
|
- **Preserve user's active identity**
|
||||||
|
|
||||||
2. **Performance**
|
2. **Performance**
|
||||||
- Improve query performance
|
- Improve query performance
|
||||||
- Reduce storage overhead
|
- Reduce storage overhead
|
||||||
- Optimize for platform-specific features
|
- Optimize for platform-specific capabilities
|
||||||
|
|
||||||
3. **Security**
|
3. **User Experience**
|
||||||
- Maintain or improve encryption
|
- Seamless transition with no data loss
|
||||||
- Preserve access controls
|
- Maintain user's active identity and preferences
|
||||||
- Enhance data protection
|
- Preserve application state
|
||||||
|
|
||||||
4. **User Experience**
|
## Migration Architecture
|
||||||
- Zero data loss
|
|
||||||
- Minimal downtime
|
|
||||||
- Automatic migration where possible
|
|
||||||
|
|
||||||
## Migration Fence
|
### Migration Fence
|
||||||
|
The migration fence is defined by the `USE_DEXIE_DB` constant in `src/constants/app.ts`:
|
||||||
|
- `USE_DEXIE_DB = false` (default): Uses SQLite database
|
||||||
|
- `USE_DEXIE_DB = true`: Uses Dexie database (for migration purposes)
|
||||||
|
|
||||||
The migration is controlled by a **migration fence** that separates legacy Dexie code from the new SQLite implementation. See [Migration Fence Definition](./migration-fence-definition.md) for complete details.
|
### Migration Order
|
||||||
|
The migration follows a specific order to maintain data integrity:
|
||||||
|
|
||||||
### Key Fence Components
|
1. **Accounts** (foundational - contains DIDs)
|
||||||
|
2. **Settings** (references accountDid, activeDid)
|
||||||
|
3. **ActiveDid** (depends on accounts and settings) ⭐ **NEW**
|
||||||
|
4. **Contacts** (independent, but migrated after accounts for consistency)
|
||||||
|
|
||||||
1. **Configuration Control**: `USE_DEXIE_DB = false` (default)
|
## ActiveDid Migration ⭐ **NEW FEATURE**
|
||||||
2. **Service Layer**: All database operations go through `PlatformService`
|
|
||||||
3. **Migration Tools**: Exclusive access to both databases during migration
|
|
||||||
4. **Code Boundaries**: Clear separation between legacy and new code
|
|
||||||
|
|
||||||
## Prerequisites
|
### Problem Solved
|
||||||
|
Previously, the `activeDid` setting was not migrated from Dexie to SQLite, causing users to lose their active identity after migration.
|
||||||
|
|
||||||
1. **Backup Requirements**
|
### Solution Implemented
|
||||||
```typescript
|
The migration now includes a dedicated step for migrating the `activeDid`:
|
||||||
interface MigrationBackup {
|
|
||||||
timestamp: number;
|
|
||||||
accounts: Account[];
|
|
||||||
settings: Setting[];
|
|
||||||
contacts: Contact[];
|
|
||||||
metadata: {
|
|
||||||
version: string;
|
|
||||||
platform: string;
|
|
||||||
dexieVersion: string;
|
|
||||||
};
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Dependencies**
|
1. **Detection**: Identifies the `activeDid` from Dexie master settings
|
||||||
```json
|
2. **Validation**: Verifies the `activeDid` exists in SQLite accounts
|
||||||
{
|
3. **Migration**: Updates SQLite master settings with the `activeDid`
|
||||||
"@jlongster/sql.js": "^1.8.0",
|
4. **Error Handling**: Graceful handling of missing accounts
|
||||||
"absurd-sql": "^1.8.0"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Storage Requirements**
|
### Implementation Details
|
||||||
- Sufficient IndexedDB quota
|
|
||||||
- Available disk space for SQLite
|
|
||||||
- Backup storage space
|
|
||||||
|
|
||||||
4. **Platform Support**
|
#### New Function: `migrateActiveDid()`
|
||||||
- Web: Modern browser with IndexedDB support
|
```typescript
|
||||||
- iOS: iOS 13+ with SQLite support
|
export async function migrateActiveDid(): Promise<MigrationResult> {
|
||||||
- Android: Android 5+ with SQLite support
|
// 1. Get Dexie settings to find the activeDid
|
||||||
- Electron: Latest version with SQLite support
|
const dexieSettings = await getDexieSettings();
|
||||||
|
const masterSettings = dexieSettings.find(setting => !setting.accountDid);
|
||||||
|
|
||||||
|
// 2. Verify the activeDid exists in SQLite accounts
|
||||||
|
const accountExists = await platformService.dbQuery(
|
||||||
|
"SELECT did FROM accounts WHERE did = ?",
|
||||||
|
[dexieActiveDid],
|
||||||
|
);
|
||||||
|
|
||||||
|
// 3. Update SQLite master settings
|
||||||
|
await updateDefaultSettings({ activeDid: dexieActiveDid });
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
## Current Migration Status
|
#### Enhanced `migrateSettings()` Function
|
||||||
|
The settings migration now includes activeDid handling:
|
||||||
|
- Extracts `activeDid` from Dexie master settings
|
||||||
|
- Validates account existence in SQLite
|
||||||
|
- Updates SQLite master settings with the `activeDid`
|
||||||
|
|
||||||
### ✅ Completed
|
#### Updated `migrateAll()` Function
|
||||||
- **SQLite Database Service**: Fully implemented with absurd-sql
|
The complete migration now includes a dedicated step for activeDid:
|
||||||
- **Platform Service Layer**: Unified database interface
|
```typescript
|
||||||
- **Migration Tools**: Data comparison and transfer utilities
|
// Step 3: Migrate ActiveDid (depends on accounts and settings)
|
||||||
- **Settings Migration**: Core user settings transferred
|
logger.info("[MigrationService] Step 3: Migrating activeDid...");
|
||||||
- **Account Migration**: Identity and key management
|
const activeDidResult = await migrateActiveDid();
|
||||||
- **Schema Migration**: Complete table structure migration
|
```
|
||||||
|
|
||||||
### 🔄 In Progress
|
### Benefits
|
||||||
- **Contact Migration**: User contact data (via import interface)
|
- ✅ **User Identity Preservation**: Users maintain their active identity
|
||||||
- **Data Verification**: Comprehensive integrity checks
|
- ✅ **Seamless Experience**: No need to manually select identity after migration
|
||||||
- **Performance Optimization**: Query optimization and indexing
|
- ✅ **Data Consistency**: Ensures all identity-related settings are preserved
|
||||||
|
- ✅ **Error Resilience**: Graceful handling of edge cases
|
||||||
### 📋 Planned
|
|
||||||
- **Code Cleanup**: Remove unused Dexie imports
|
|
||||||
- **Documentation Updates**: Complete migration guides
|
|
||||||
- **Testing**: Comprehensive migration testing
|
|
||||||
|
|
||||||
## Migration Process
|
## Migration Process
|
||||||
|
|
||||||
### 1. Preparation
|
### Phase 1: Preparation ✅
|
||||||
|
- [x] Enable Dexie database access
|
||||||
|
- [x] Implement data comparison tools
|
||||||
|
- [x] Create migration service structure
|
||||||
|
|
||||||
|
### Phase 2: Core Migration ✅
|
||||||
|
- [x] Account migration with `importFromMnemonic`
|
||||||
|
- [x] Settings migration (excluding activeDid)
|
||||||
|
- [x] **ActiveDid migration** ⭐ **COMPLETED**
|
||||||
|
- [x] Contact migration framework
|
||||||
|
|
||||||
|
### Phase 3: Validation and Cleanup 🔄
|
||||||
|
- [ ] Comprehensive data validation
|
||||||
|
- [ ] Performance testing
|
||||||
|
- [ ] User acceptance testing
|
||||||
|
- [ ] Dexie removal
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### Manual Migration
|
||||||
```typescript
|
```typescript
|
||||||
// src/services/storage/migration/MigrationService.ts
|
import { migrateAll, migrateActiveDid } from '../services/indexedDBMigrationService';
|
||||||
import initSqlJs from '@jlongster/sql.js';
|
|
||||||
import { SQLiteFS } from 'absurd-sql';
|
|
||||||
import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend';
|
|
||||||
|
|
||||||
class MigrationService {
|
// Complete migration
|
||||||
private async checkPrerequisites(): Promise<void> {
|
const result = await migrateAll();
|
||||||
// Check IndexedDB availability
|
|
||||||
if (!window.indexedDB) {
|
|
||||||
throw new StorageError(
|
|
||||||
'IndexedDB not available',
|
|
||||||
StorageErrorCodes.INITIALIZATION_FAILED
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check storage quota
|
// Or migrate just the activeDid
|
||||||
const quota = await navigator.storage.estimate();
|
const activeDidResult = await migrateActiveDid();
|
||||||
if (quota.quota && quota.usage && quota.usage > quota.quota * 0.9) {
|
|
||||||
throw new StorageError(
|
|
||||||
'Insufficient storage space',
|
|
||||||
StorageErrorCodes.STORAGE_FULL
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check platform support
|
|
||||||
const capabilities = await PlatformDetection.getCapabilities();
|
|
||||||
if (!capabilities.hasFileSystem) {
|
|
||||||
throw new StorageError(
|
|
||||||
'Platform does not support required features',
|
|
||||||
StorageErrorCodes.INITIALIZATION_FAILED
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private async createBackup(): Promise<MigrationBackup> {
|
|
||||||
const dexieDB = new Dexie('TimeSafariDB');
|
|
||||||
|
|
||||||
return {
|
|
||||||
timestamp: Date.now(),
|
|
||||||
accounts: await dexieDB.accounts.toArray(),
|
|
||||||
settings: await dexieDB.settings.toArray(),
|
|
||||||
contacts: await dexieDB.contacts.toArray(),
|
|
||||||
metadata: {
|
|
||||||
version: '1.0.0',
|
|
||||||
platform: await PlatformDetection.getPlatform(),
|
|
||||||
dexieVersion: Dexie.version
|
|
||||||
}
|
|
||||||
};
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. Data Migration
|
### Migration Verification
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// src/services/storage/migration/DataMigration.ts
|
import { compareDatabases } from '../services/indexedDBMigrationService';
|
||||||
class DataMigration {
|
|
||||||
async migrateAccounts(): Promise<MigrationResult> {
|
|
||||||
const result: MigrationResult = {
|
|
||||||
success: true,
|
|
||||||
accountsMigrated: 0,
|
|
||||||
errors: [],
|
|
||||||
warnings: []
|
|
||||||
};
|
|
||||||
|
|
||||||
try {
|
const comparison = await compareDatabases();
|
||||||
const dexieAccounts = await this.getDexieAccounts();
|
console.log('Migration differences:', comparison.differences);
|
||||||
|
|
||||||
for (const account of dexieAccounts) {
|
|
||||||
try {
|
|
||||||
await this.migrateAccount(account);
|
|
||||||
result.accountsMigrated++;
|
|
||||||
} catch (error) {
|
|
||||||
result.errors.push(`Failed to migrate account ${account.did}: ${error}`);
|
|
||||||
result.success = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} catch (error) {
|
|
||||||
result.errors.push(`Account migration failed: ${error}`);
|
|
||||||
result.success = false;
|
|
||||||
}
|
|
||||||
|
|
||||||
return result;
|
|
||||||
}
|
|
||||||
|
|
||||||
async migrateSettings(): Promise<MigrationResult> {
|
|
||||||
const result: MigrationResult = {
|
|
||||||
success: true,
|
|
||||||
settingsMigrated: 0,
|
|
||||||
errors: [],
|
|
||||||
warnings: []
|
|
||||||
};
|
|
||||||
|
|
||||||
try {
|
|
||||||
const dexieSettings = await this.getDexieSettings();
|
|
||||||
|
|
||||||
for (const setting of dexieSettings) {
|
|
||||||
try {
|
|
||||||
await this.migrateSetting(setting);
|
|
||||||
result.settingsMigrated++;
|
|
||||||
} catch (error) {
|
|
||||||
result.errors.push(`Failed to migrate setting ${setting.id}: ${error}`);
|
|
||||||
result.success = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} catch (error) {
|
|
||||||
result.errors.push(`Settings migration failed: ${error}`);
|
|
||||||
result.success = false;
|
|
||||||
}
|
|
||||||
|
|
||||||
return result;
|
|
||||||
}
|
|
||||||
|
|
||||||
async migrateContacts(): Promise<MigrationResult> {
|
|
||||||
// Contact migration is handled through the contact import interface
|
|
||||||
// This provides better user control and validation
|
|
||||||
const result: MigrationResult = {
|
|
||||||
success: true,
|
|
||||||
contactsMigrated: 0,
|
|
||||||
errors: [],
|
|
||||||
warnings: []
|
|
||||||
};
|
|
||||||
|
|
||||||
try {
|
|
||||||
const dexieContacts = await this.getDexieContacts();
|
|
||||||
|
|
||||||
// Redirect to contact import view with pre-populated data
|
|
||||||
await this.redirectToContactImport(dexieContacts);
|
|
||||||
|
|
||||||
result.contactsMigrated = dexieContacts.length;
|
|
||||||
} catch (error) {
|
|
||||||
result.errors.push(`Contact migration failed: ${error}`);
|
|
||||||
result.success = false;
|
|
||||||
}
|
|
||||||
|
|
||||||
return result;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Verification
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
class MigrationVerification {
|
|
||||||
async verifyMigration(dexieData: MigrationData): Promise<boolean> {
|
|
||||||
// Verify account count
|
|
||||||
const accountResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM accounts');
|
|
||||||
const accountCount = accountResult[0].values[0][0];
|
|
||||||
if (accountCount !== dexieData.accounts.length) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Verify settings count
|
|
||||||
const settingsResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM settings');
|
|
||||||
const settingsCount = settingsResult[0].values[0][0];
|
|
||||||
if (settingsCount !== dexieData.settings.length) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Verify data integrity
|
|
||||||
for (const account of dexieData.accounts) {
|
|
||||||
const result = await this.sqliteDB.exec(
|
|
||||||
'SELECT * FROM accounts WHERE did = ?',
|
|
||||||
[account.did]
|
|
||||||
);
|
|
||||||
const migratedAccount = result[0]?.values[0];
|
|
||||||
if (!migratedAccount ||
|
|
||||||
migratedAccount[1] !== account.publicKeyHex) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Using the Migration Interface
|
|
||||||
|
|
||||||
### Accessing Migration Tools
|
|
||||||
|
|
||||||
1. Navigate to the **Account** page in the TimeSafari app
|
|
||||||
2. Scroll down to find the **Database Migration** link
|
|
||||||
3. Click the link to open the migration interface
|
|
||||||
|
|
||||||
### Migration Steps
|
|
||||||
|
|
||||||
1. **Compare Databases**
|
|
||||||
- Click "Compare Databases" to see differences
|
|
||||||
- Review the comparison results
|
|
||||||
- Identify data that needs migration
|
|
||||||
|
|
||||||
2. **Migrate Settings**
|
|
||||||
- Click "Migrate Settings" to transfer user settings
|
|
||||||
- Verify settings are correctly transferred
|
|
||||||
- Check application functionality
|
|
||||||
|
|
||||||
3. **Migrate Contacts**
|
|
||||||
- Click "Migrate Contacts" to open contact import
|
|
||||||
- Review and confirm contact data
|
|
||||||
- Complete the import process
|
|
||||||
|
|
||||||
4. **Verify Migration**
|
|
||||||
- Run comparison again to verify completion
|
|
||||||
- Test application functionality
|
|
||||||
- Export backup data if needed
|
|
||||||
|
|
||||||
## Error Handling
|
## Error Handling
|
||||||
|
|
||||||
### Common Issues
|
### ActiveDid Migration Errors
|
||||||
|
- **Missing Account**: If the `activeDid` from Dexie doesn't exist in SQLite accounts
|
||||||
|
- **Database Errors**: Connection or query failures
|
||||||
|
- **Settings Update Failures**: Issues updating SQLite master settings
|
||||||
|
|
||||||
1. **Dexie Database Not Enabled**
|
### Recovery Strategies
|
||||||
- **Error**: "Dexie database is not enabled"
|
1. **Automatic Recovery**: Migration continues even if activeDid migration fails
|
||||||
- **Solution**: Set `USE_DEXIE_DB = true` in `constants/app.ts` temporarily
|
2. **Manual Recovery**: Users can manually select their identity after migration
|
||||||
|
3. **Fallback**: System creates new identity if none exists
|
||||||
2. **Database Connection Issues**
|
|
||||||
- **Error**: "Failed to retrieve data"
|
|
||||||
- **Solution**: Check database initialization and permissions
|
|
||||||
|
|
||||||
3. **Migration Failures**
|
|
||||||
- **Error**: "Migration failed: [specific error]"
|
|
||||||
- **Solution**: Review error details and check data integrity
|
|
||||||
|
|
||||||
### Error Recovery
|
|
||||||
|
|
||||||
1. **Review** error messages carefully
|
|
||||||
2. **Check** browser console for additional details
|
|
||||||
3. **Verify** database connectivity and permissions
|
|
||||||
4. **Retry** the operation if appropriate
|
|
||||||
5. **Export** comparison data for manual review if needed
|
|
||||||
|
|
||||||
## Best Practices
|
|
||||||
|
|
||||||
### Before Migration
|
|
||||||
|
|
||||||
1. **Backup** your data if possible
|
|
||||||
2. **Test** the migration on a small dataset first
|
|
||||||
3. **Verify** that both databases are accessible
|
|
||||||
4. **Review** the comparison results before migrating
|
|
||||||
|
|
||||||
### During Migration
|
|
||||||
|
|
||||||
1. **Don't** interrupt the migration process
|
|
||||||
2. **Monitor** the progress and error messages
|
|
||||||
3. **Note** any warnings or skipped records
|
|
||||||
4. **Export** comparison data for reference
|
|
||||||
|
|
||||||
### After Migration
|
|
||||||
|
|
||||||
1. **Verify** that data was migrated correctly
|
|
||||||
2. **Test** the application functionality
|
|
||||||
3. **Disable** Dexie database (`USE_DEXIE_DB = false`)
|
|
||||||
4. **Clean up** any temporary files or exports
|
|
||||||
|
|
||||||
## Performance Considerations
|
|
||||||
|
|
||||||
### 1. Migration Performance
|
|
||||||
- Use transactions for bulk data transfer
|
|
||||||
- Implement progress indicators
|
|
||||||
- Process data in background when possible
|
|
||||||
|
|
||||||
### 2. Application Performance
|
|
||||||
- Optimize SQLite queries
|
|
||||||
- Maintain proper database indexes
|
|
||||||
- Use efficient memory management
|
|
||||||
|
|
||||||
## Security Considerations
|
## Security Considerations
|
||||||
|
|
||||||
### 1. Data Protection
|
### Data Protection
|
||||||
- Maintain encryption standards across migration
|
- All sensitive data (mnemonics, private keys) are encrypted
|
||||||
- Preserve user privacy during migration
|
- Migration preserves encryption standards
|
||||||
- Log all migration operations
|
- No plaintext data exposure during migration
|
||||||
|
|
||||||
### 2. Error Handling
|
### Identity Verification
|
||||||
- Handle migration failures gracefully
|
- ActiveDid migration validates account existence
|
||||||
- Provide clear user messaging
|
- Prevents setting non-existent identities as active
|
||||||
- Maintain rollback capabilities
|
- Maintains cryptographic integrity
|
||||||
|
|
||||||
## Testing Strategy
|
## Testing
|
||||||
|
|
||||||
### 1. Migration Testing
|
### Migration Testing
|
||||||
```typescript
|
```bash
|
||||||
describe('Database Migration', () => {
|
# Enable Dexie for testing
|
||||||
it('should migrate data without loss', async () => {
|
# Set USE_DEXIE_DB = true in constants/app.ts
|
||||||
// 1. Enable Dexie
|
|
||||||
// 2. Create test data
|
# Run migration
|
||||||
// 3. Run migration
|
npm run migrate
|
||||||
// 4. Verify data integrity
|
|
||||||
// 5. Disable Dexie
|
# Verify results
|
||||||
});
|
npm run test:migration
|
||||||
});
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. Application Testing
|
### ActiveDid Testing
|
||||||
```typescript
|
```typescript
|
||||||
describe('Feature with Database', () => {
|
// Test activeDid migration specifically
|
||||||
it('should work with SQLite only', async () => {
|
const result = await migrateActiveDid();
|
||||||
// Test with USE_DEXIE_DB = false
|
expect(result.success).toBe(true);
|
||||||
// Verify all operations use PlatformService
|
expect(result.warnings).toContain('Successfully migrated activeDid');
|
||||||
});
|
|
||||||
});
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
1. **ActiveDid Not Found**
|
||||||
|
- Ensure accounts were migrated before activeDid migration
|
||||||
|
- Check that the Dexie activeDid exists in SQLite accounts
|
||||||
|
|
||||||
|
2. **Migration Failures**
|
||||||
|
- Verify Dexie database is accessible
|
||||||
|
- Check SQLite database permissions
|
||||||
|
- Review migration logs for specific errors
|
||||||
|
|
||||||
|
3. **Data Inconsistencies**
|
||||||
|
- Use `compareDatabases()` to identify differences
|
||||||
|
- Re-run migration if necessary
|
||||||
|
- Check for duplicate or conflicting records
|
||||||
|
|
||||||
|
### Debugging
|
||||||
|
```typescript
|
||||||
|
// Enable detailed logging
|
||||||
|
logger.setLevel('debug');
|
||||||
|
|
||||||
|
// Check migration status
|
||||||
|
const comparison = await compareDatabases();
|
||||||
|
console.log('Settings differences:', comparison.differences.settings);
|
||||||
|
```
|
||||||
|
|
||||||
|
## Future Enhancements
|
||||||
|
|
||||||
|
### Planned Improvements
|
||||||
|
1. **Batch Processing**: Optimize for large datasets
|
||||||
|
2. **Incremental Migration**: Support partial migrations
|
||||||
|
3. **Rollback Capability**: Ability to revert migration
|
||||||
|
4. **Progress Tracking**: Real-time migration progress
|
||||||
|
|
||||||
|
### Performance Optimizations
|
||||||
|
1. **Parallel Processing**: Migrate independent data concurrently
|
||||||
|
2. **Memory Management**: Optimize for large datasets
|
||||||
|
3. **Transaction Batching**: Reduce database round trips
|
||||||
|
|
||||||
## Conclusion
|
## Conclusion
|
||||||
|
|
||||||
The migration from Dexie to absurd-sql provides:
|
The Dexie to SQLite migration provides a robust, secure, and user-friendly transition path. The addition of activeDid migration ensures that users maintain their identity continuity throughout the migration process, significantly improving the user experience.
|
||||||
- **Better Performance**: Improved query performance and storage efficiency
|
|
||||||
- **Cross-Platform Consistency**: Unified database interface across platforms
|
|
||||||
- **Enhanced Security**: Better encryption and access controls
|
|
||||||
- **Future-Proof Architecture**: Modern SQLite-based storage system
|
|
||||||
|
|
||||||
The migration fence ensures a controlled and safe transition while maintaining data integrity and application stability.
|
The migration fence architecture allows for controlled, reversible migration while maintaining application stability and data integrity.
|
||||||
@@ -39,6 +39,7 @@ import {
|
|||||||
generateUpdateStatement,
|
generateUpdateStatement,
|
||||||
generateInsertStatement,
|
generateInsertStatement,
|
||||||
} from "../db/databaseUtil";
|
} from "../db/databaseUtil";
|
||||||
|
import { updateDefaultSettings } from "../db/databaseUtil";
|
||||||
import { importFromMnemonic } from "../libs/util";
|
import { importFromMnemonic } from "../libs/util";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -1080,6 +1081,17 @@ export async function migrateSettings(): Promise<MigrationResult> {
|
|||||||
});
|
});
|
||||||
const platformService = PlatformServiceFactory.getInstance();
|
const platformService = PlatformServiceFactory.getInstance();
|
||||||
|
|
||||||
|
// Find the master settings (accountDid is null) which contains the activeDid
|
||||||
|
const masterSettings = dexieSettings.find(setting => !setting.accountDid);
|
||||||
|
let dexieActiveDid: string | undefined;
|
||||||
|
|
||||||
|
if (masterSettings?.activeDid) {
|
||||||
|
dexieActiveDid = masterSettings.activeDid;
|
||||||
|
logger.info("[MigrationService] Found activeDid in Dexie master settings", {
|
||||||
|
activeDid: dexieActiveDid,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
// Create an array of promises for all settings migrations
|
// Create an array of promises for all settings migrations
|
||||||
const migrationPromises = dexieSettings.map(async (setting) => {
|
const migrationPromises = dexieSettings.map(async (setting) => {
|
||||||
logger.info("[MigrationService] Starting to migrate settings", setting);
|
logger.info("[MigrationService] Starting to migrate settings", setting);
|
||||||
@@ -1139,6 +1151,38 @@ export async function migrateSettings(): Promise<MigrationResult> {
|
|||||||
// Wait for all migrations to complete
|
// Wait for all migrations to complete
|
||||||
const updatedSettings = await Promise.all(migrationPromises);
|
const updatedSettings = await Promise.all(migrationPromises);
|
||||||
|
|
||||||
|
// Step 2: Migrate the activeDid if it exists in Dexie
|
||||||
|
if (dexieActiveDid) {
|
||||||
|
try {
|
||||||
|
// Verify that the activeDid exists in SQLite accounts
|
||||||
|
const accountExists = await platformService.dbQuery(
|
||||||
|
"SELECT did FROM accounts WHERE did = ?",
|
||||||
|
[dexieActiveDid],
|
||||||
|
);
|
||||||
|
|
||||||
|
if (accountExists?.values?.length) {
|
||||||
|
// Update the master settings with the activeDid
|
||||||
|
await updateDefaultSettings({ activeDid: dexieActiveDid });
|
||||||
|
logger.info("[MigrationService] Successfully migrated activeDid", {
|
||||||
|
activeDid: dexieActiveDid,
|
||||||
|
});
|
||||||
|
result.warnings.push(`Migrated activeDid: ${dexieActiveDid}`);
|
||||||
|
} else {
|
||||||
|
logger.warn("[MigrationService] activeDid from Dexie not found in SQLite accounts", {
|
||||||
|
activeDid: dexieActiveDid,
|
||||||
|
});
|
||||||
|
result.warnings.push(
|
||||||
|
`activeDid from Dexie (${dexieActiveDid}) not found in SQLite accounts - skipping activeDid migration`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("[MigrationService] Failed to migrate activeDid:", error);
|
||||||
|
result.errors.push(`Failed to migrate activeDid: ${error}`);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
logger.info("[MigrationService] No activeDid found in Dexie settings");
|
||||||
|
}
|
||||||
|
|
||||||
logger.info(
|
logger.info(
|
||||||
"[MigrationService] Finished migrating settings",
|
"[MigrationService] Finished migrating settings",
|
||||||
updatedSettings,
|
updatedSettings,
|
||||||
@@ -1279,6 +1323,96 @@ export async function migrateAccounts(): Promise<MigrationResult> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Migrates the activeDid from Dexie to SQLite
|
||||||
|
*
|
||||||
|
* This function specifically handles the migration of the activeDid setting
|
||||||
|
* from the Dexie database to the SQLite database. It ensures that the
|
||||||
|
* activeDid exists in the SQLite accounts table before setting it as active.
|
||||||
|
*
|
||||||
|
* The function is designed to be called after accounts have been migrated
|
||||||
|
* to ensure the target DID exists in the SQLite database.
|
||||||
|
*
|
||||||
|
* @async
|
||||||
|
* @function migrateActiveDid
|
||||||
|
* @returns {Promise<MigrationResult>} Result of the activeDid migration
|
||||||
|
* @throws {Error} If the migration process fails
|
||||||
|
* @example
|
||||||
|
* ```typescript
|
||||||
|
* try {
|
||||||
|
* const result = await migrateActiveDid();
|
||||||
|
* if (result.success) {
|
||||||
|
* console.log('ActiveDid migration successful');
|
||||||
|
* } else {
|
||||||
|
* console.error('ActiveDid migration failed:', result.errors);
|
||||||
|
* }
|
||||||
|
* } catch (error) {
|
||||||
|
* console.error('ActiveDid migration process failed:', error);
|
||||||
|
* }
|
||||||
|
* ```
|
||||||
|
*/
|
||||||
|
export async function migrateActiveDid(): Promise<MigrationResult> {
|
||||||
|
logger.info("[MigrationService] Starting activeDid migration");
|
||||||
|
|
||||||
|
const result: MigrationResult = {
|
||||||
|
success: true,
|
||||||
|
contactsMigrated: 0,
|
||||||
|
settingsMigrated: 0,
|
||||||
|
accountsMigrated: 0,
|
||||||
|
errors: [],
|
||||||
|
warnings: [],
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Get Dexie settings to find the activeDid
|
||||||
|
const dexieSettings = await getDexieSettings();
|
||||||
|
const masterSettings = dexieSettings.find(setting => !setting.accountDid);
|
||||||
|
|
||||||
|
if (!masterSettings?.activeDid) {
|
||||||
|
logger.info("[MigrationService] No activeDid found in Dexie master settings");
|
||||||
|
result.warnings.push("No activeDid found in Dexie settings");
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
const dexieActiveDid = masterSettings.activeDid;
|
||||||
|
logger.info("[MigrationService] Found activeDid in Dexie", {
|
||||||
|
activeDid: dexieActiveDid,
|
||||||
|
});
|
||||||
|
|
||||||
|
const platformService = PlatformServiceFactory.getInstance();
|
||||||
|
|
||||||
|
// Verify that the activeDid exists in SQLite accounts
|
||||||
|
const accountExists = await platformService.dbQuery(
|
||||||
|
"SELECT did FROM accounts WHERE did = ?",
|
||||||
|
[dexieActiveDid],
|
||||||
|
);
|
||||||
|
|
||||||
|
if (!accountExists?.values?.length) {
|
||||||
|
const errorMessage = `activeDid from Dexie (${dexieActiveDid}) not found in SQLite accounts`;
|
||||||
|
logger.error("[MigrationService]", errorMessage);
|
||||||
|
result.errors.push(errorMessage);
|
||||||
|
result.success = false;
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Update the master settings with the activeDid
|
||||||
|
await updateDefaultSettings({ activeDid: dexieActiveDid });
|
||||||
|
|
||||||
|
logger.info("[MigrationService] Successfully migrated activeDid", {
|
||||||
|
activeDid: dexieActiveDid,
|
||||||
|
});
|
||||||
|
result.warnings.push(`Successfully migrated activeDid: ${dexieActiveDid}`);
|
||||||
|
|
||||||
|
return result;
|
||||||
|
} catch (error) {
|
||||||
|
const errorMessage = `ActiveDid migration failed: ${error}`;
|
||||||
|
logger.error("[MigrationService]", errorMessage, error);
|
||||||
|
result.errors.push(errorMessage);
|
||||||
|
result.success = false;
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Migrates all data from Dexie to SQLite in the proper order
|
* Migrates all data from Dexie to SQLite in the proper order
|
||||||
*
|
*
|
||||||
@@ -1286,7 +1420,8 @@ export async function migrateAccounts(): Promise<MigrationResult> {
|
|||||||
* in the correct order to avoid foreign key constraint issues:
|
* in the correct order to avoid foreign key constraint issues:
|
||||||
* 1. Accounts (foundational - contains DIDs)
|
* 1. Accounts (foundational - contains DIDs)
|
||||||
* 2. Settings (references accountDid, activeDid)
|
* 2. Settings (references accountDid, activeDid)
|
||||||
* 3. Contacts (independent, but migrated after accounts for consistency)
|
* 3. ActiveDid (depends on accounts and settings)
|
||||||
|
* 4. Contacts (independent, but migrated after accounts for consistency)
|
||||||
*
|
*
|
||||||
* The migration runs within a transaction to ensure atomicity. If any step fails,
|
* The migration runs within a transaction to ensure atomicity. If any step fails,
|
||||||
* the entire migration is rolled back.
|
* the entire migration is rolled back.
|
||||||
@@ -1332,9 +1467,21 @@ export async function migrateAll(): Promise<MigrationResult> {
|
|||||||
result.settingsMigrated = settingsResult.settingsMigrated;
|
result.settingsMigrated = settingsResult.settingsMigrated;
|
||||||
result.warnings.push(...settingsResult.warnings);
|
result.warnings.push(...settingsResult.warnings);
|
||||||
|
|
||||||
// Step 3: Migrate Contacts (independent, but after accounts for consistency)
|
// Step 3: Migrate ActiveDid (depends on accounts and settings)
|
||||||
|
logger.info("[MigrationService] Step 3: Migrating activeDid...");
|
||||||
|
const activeDidResult = await migrateActiveDid();
|
||||||
|
if (!activeDidResult.success) {
|
||||||
|
result.errors.push(
|
||||||
|
`ActiveDid migration failed: ${activeDidResult.errors.join(", ")}`,
|
||||||
|
);
|
||||||
|
// Don't fail the entire migration for activeDid issues
|
||||||
|
logger.warn("[MigrationService] ActiveDid migration failed, but continuing with migration");
|
||||||
|
}
|
||||||
|
result.warnings.push(...activeDidResult.warnings);
|
||||||
|
|
||||||
|
// Step 4: Migrate Contacts (independent, but after accounts for consistency)
|
||||||
// ... but which is better done through the contact import view
|
// ... but which is better done through the contact import view
|
||||||
// logger.info("[MigrationService] Step 3: Migrating contacts...");
|
// logger.info("[MigrationService] Step 4: Migrating contacts...");
|
||||||
// const contactsResult = await migrateContacts();
|
// const contactsResult = await migrateContacts();
|
||||||
// if (!contactsResult.success) {
|
// if (!contactsResult.success) {
|
||||||
// result.errors.push(
|
// result.errors.push(
|
||||||
|
|||||||
Reference in New Issue
Block a user