jsnbuchanan/crowd-funder-for-time-pwa
1
0
Fork 1
forked from trent_larson/crowd-funder-for-time-pwa
Code Issues Pull Requests Projects Releases Wiki Activity

feat: Implement activeDid migration from Dexie to SQLite

Browse Source 
- Add migrateActiveDid() function for dedicated activeDid migration
- Enhance migrateSettings() to handle activeDid extraction and validation
- Update migrateAll() to include activeDid migration step
- Add comprehensive error handling and validation
- Update migration documentation with activeDid migration details
- Ensure user identity continuity during migration process

Files changed:
- src/services/indexedDBMigrationService.ts (153 lines added)
- doc/migration-to-wa-sqlite.md (documentation updated)

Migration order: Accounts -> Settings -> ActiveDid -> Contacts
This commit is contained in:
Matthew Raymer
2025-06-20 04:48:51 +00:00
parent b89f3310ef
commit 7978133e7f
2 changed files with 315 additions and 356 deletions
Download Patch File Download Diff File Expand all files Collapse all files

518
doc/migration-to-wa-sqlite.md
View File

@@ -4,7 +4,7 @@
This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.
**Current Status**: The migration is in **Phase 2** with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress.
**Current Status**: The migration is in **Phase 2** with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress. **ActiveDid migration has been implemented** to ensure user identity continuity.
## Migration Goals
@@ -12,403 +12,215 @@ This document outlines the migration process from Dexie.js to absurd-sql for the
- Preserve all existing data
- Maintain data relationships
- Ensure data consistency
- **Preserve user's active identity**
2. **Performance**
- Improve query performance
- Reduce storage overhead
- Optimize for platform-specific features
- Optimize for platform-specific capabilities
3. **Security**
- Maintain or improve encryption
- Preserve access controls
- Enhance data protection
3. **User Experience**
- Seamless transition with no data loss
- Maintain user's active identity and preferences
- Preserve application state
4. **User Experience**
- Zero data loss
- Minimal downtime
- Automatic migration where possible
## Migration Architecture
## Migration Fence
### Migration Fence
The migration fence is defined by the `USE_DEXIE_DB` constant in `src/constants/app.ts`:
- `USE_DEXIE_DB = false` (default): Uses SQLite database
- `USE_DEXIE_DB = true`: Uses Dexie database (for migration purposes)
The migration is controlled by a **migration fence** that separates legacy Dexie code from the new SQLite implementation. See [Migration Fence Definition](./migration-fence-definition.md) for complete details.
### Migration Order
The migration follows a specific order to maintain data integrity:
### Key Fence Components
1. **Accounts** (foundational - contains DIDs)
2. **Settings** (references accountDid, activeDid)
3. **ActiveDid** (depends on accounts and settings) ⭐ **NEW**
4. **Contacts** (independent, but migrated after accounts for consistency)
1. **Configuration Control**: `USE_DEXIE_DB = false` (default)
2. **Service Layer**: All database operations go through `PlatformService`
3. **Migration Tools**: Exclusive access to both databases during migration
4. **Code Boundaries**: Clear separation between legacy and new code
## ActiveDid Migration ⭐ **NEW FEATURE**
## Prerequisites
### Problem Solved
Previously, the `activeDid` setting was not migrated from Dexie to SQLite, causing users to lose their active identity after migration.
1. **Backup Requirements**
```typescript
interface MigrationBackup {
timestamp: number;
accounts: Account[];
settings: Setting[];
contacts: Contact[];
metadata: {
version: string;
platform: string;
dexieVersion: string;
};
}
```
### Solution Implemented
The migration now includes a dedicated step for migrating the `activeDid`:
2. **Dependencies**
```json
{
"@jlongster/sql.js": "^1.8.0",
"absurd-sql": "^1.8.0"
}
```
1. **Detection**: Identifies the `activeDid` from Dexie master settings
2. **Validation**: Verifies the `activeDid` exists in SQLite accounts
3. **Migration**: Updates SQLite master settings with the `activeDid`
4. **Error Handling**: Graceful handling of missing accounts
3. **Storage Requirements**
- Sufficient IndexedDB quota
- Available disk space for SQLite
- Backup storage space
### Implementation Details
4. **Platform Support**
- Web: Modern browser with IndexedDB support
- iOS: iOS 13+ with SQLite support
- Android: Android 5+ with SQLite support
- Electron: Latest version with SQLite support
#### New Function: `migrateActiveDid()`
```typescript
export async function migrateActiveDid(): Promise<MigrationResult> {
// 1. Get Dexie settings to find the activeDid
const dexieSettings = await getDexieSettings();
const masterSettings = dexieSettings.find(setting => !setting.accountDid);
// 2. Verify the activeDid exists in SQLite accounts
const accountExists = await platformService.dbQuery(
"SELECT did FROM accounts WHERE did = ?",
[dexieActiveDid],
);
// 3. Update SQLite master settings
await updateDefaultSettings({ activeDid: dexieActiveDid });
}
```
## Current Migration Status
#### Enhanced `migrateSettings()` Function
The settings migration now includes activeDid handling:
- Extracts `activeDid` from Dexie master settings
- Validates account existence in SQLite
- Updates SQLite master settings with the `activeDid`
### ✅ Completed
- **SQLite Database Service**: Fully implemented with absurd-sql
- **Platform Service Layer**: Unified database interface
- **Migration Tools**: Data comparison and transfer utilities
- **Settings Migration**: Core user settings transferred
- **Account Migration**: Identity and key management
- **Schema Migration**: Complete table structure migration
#### Updated `migrateAll()` Function
The complete migration now includes a dedicated step for activeDid:
```typescript
// Step 3: Migrate ActiveDid (depends on accounts and settings)
logger.info("[MigrationService] Step 3: Migrating activeDid...");
const activeDidResult = await migrateActiveDid();
```
### 🔄 In Progress
- **Contact Migration**: User contact data (via import interface)
- **Data Verification**: Comprehensive integrity checks
- **Performance Optimization**: Query optimization and indexing
### 📋 Planned
- **Code Cleanup**: Remove unused Dexie imports
- **Documentation Updates**: Complete migration guides
- **Testing**: Comprehensive migration testing
### Benefits
- ✅ **User Identity Preservation**: Users maintain their active identity
- ✅ **Seamless Experience**: No need to manually select identity after migration
- ✅ **Data Consistency**: Ensures all identity-related settings are preserved
- ✅ **Error Resilience**: Graceful handling of edge cases
## Migration Process
### 1. Preparation
### Phase 1: Preparation ✅
- [x] Enable Dexie database access
- [x] Implement data comparison tools
- [x] Create migration service structure
### Phase 2: Core Migration ✅
- [x] Account migration with `importFromMnemonic`
- [x] Settings migration (excluding activeDid)
- [x] **ActiveDid migration** ⭐ **COMPLETED**
- [x] Contact migration framework
### Phase 3: Validation and Cleanup 🔄
- [ ] Comprehensive data validation
- [ ] Performance testing
- [ ] User acceptance testing
- [ ] Dexie removal
## Usage
### Manual Migration
```typescript
// src/services/storage/migration/MigrationService.ts
import initSqlJs from '@jlongster/sql.js';
import { SQLiteFS } from 'absurd-sql';
import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend';
import { migrateAll, migrateActiveDid } from '../services/indexedDBMigrationService';
class MigrationService {
private async checkPrerequisites(): Promise<void> {
// Check IndexedDB availability
if (!window.indexedDB) {
throw new StorageError(
'IndexedDB not available',
StorageErrorCodes.INITIALIZATION_FAILED
);
}
// Complete migration
const result = await migrateAll();
// Check storage quota
const quota = await navigator.storage.estimate();
if (quota.quota && quota.usage && quota.usage > quota.quota * 0.9) {
throw new StorageError(
'Insufficient storage space',
StorageErrorCodes.STORAGE_FULL
);
}
// Check platform support
const capabilities = await PlatformDetection.getCapabilities();
if (!capabilities.hasFileSystem) {
throw new StorageError(
'Platform does not support required features',
StorageErrorCodes.INITIALIZATION_FAILED
);
}
}
private async createBackup(): Promise<MigrationBackup> {
const dexieDB = new Dexie('TimeSafariDB');
return {
timestamp: Date.now(),
accounts: await dexieDB.accounts.toArray(),
settings: await dexieDB.settings.toArray(),
contacts: await dexieDB.contacts.toArray(),
metadata: {
version: '1.0.0',
platform: await PlatformDetection.getPlatform(),
dexieVersion: Dexie.version
}
};
}
}
// Or migrate just the activeDid
const activeDidResult = await migrateActiveDid();
```
### 2. Data Migration
### Migration Verification
```typescript
// src/services/storage/migration/DataMigration.ts
class DataMigration {
async migrateAccounts(): Promise<MigrationResult> {
const result: MigrationResult = {
success: true,
accountsMigrated: 0,
errors: [],
warnings: []
};
import { compareDatabases } from '../services/indexedDBMigrationService';
try {
const dexieAccounts = await this.getDexieAccounts();
for (const account of dexieAccounts) {
try {
await this.migrateAccount(account);
result.accountsMigrated++;
} catch (error) {
result.errors.push(`Failed to migrate account ${account.did}: ${error}`);
result.success = false;
}
}
} catch (error) {
result.errors.push(`Account migration failed: ${error}`);
result.success = false;
}
return result;
}
async migrateSettings(): Promise<MigrationResult> {
const result: MigrationResult = {
success: true,
settingsMigrated: 0,
errors: [],
warnings: []
};
try {
const dexieSettings = await this.getDexieSettings();
for (const setting of dexieSettings) {
try {
await this.migrateSetting(setting);
result.settingsMigrated++;
} catch (error) {
result.errors.push(`Failed to migrate setting ${setting.id}: ${error}`);
result.success = false;
}
}
} catch (error) {
result.errors.push(`Settings migration failed: ${error}`);
result.success = false;
}
return result;
}
async migrateContacts(): Promise<MigrationResult> {
// Contact migration is handled through the contact import interface
// This provides better user control and validation
const result: MigrationResult = {
success: true,
contactsMigrated: 0,
errors: [],
warnings: []
};
try {
const dexieContacts = await this.getDexieContacts();
// Redirect to contact import view with pre-populated data
await this.redirectToContactImport(dexieContacts);
result.contactsMigrated = dexieContacts.length;
} catch (error) {
result.errors.push(`Contact migration failed: ${error}`);
result.success = false;
}
return result;
}
}
const comparison = await compareDatabases();
console.log('Migration differences:', comparison.differences);
```
### 3. Verification
```typescript
class MigrationVerification {
async verifyMigration(dexieData: MigrationData): Promise<boolean> {
// Verify account count
const accountResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM accounts');
const accountCount = accountResult[0].values[0][0];
if (accountCount !== dexieData.accounts.length) {
return false;
}
// Verify settings count
const settingsResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM settings');
const settingsCount = settingsResult[0].values[0][0];
if (settingsCount !== dexieData.settings.length) {
return false;
}
// Verify data integrity
for (const account of dexieData.accounts) {
const result = await this.sqliteDB.exec(
'SELECT * FROM accounts WHERE did = ?',
[account.did]
);
const migratedAccount = result[0]?.values[0];
if (!migratedAccount ||
migratedAccount[1] !== account.publicKeyHex) {
return false;
}
}
return true;
}
}
```
## Using the Migration Interface
### Accessing Migration Tools
1. Navigate to the **Account** page in the TimeSafari app
2. Scroll down to find the **Database Migration** link
3. Click the link to open the migration interface
### Migration Steps
1. **Compare Databases**
- Click "Compare Databases" to see differences
- Review the comparison results
- Identify data that needs migration
2. **Migrate Settings**
- Click "Migrate Settings" to transfer user settings
- Verify settings are correctly transferred
- Check application functionality
3. **Migrate Contacts**
- Click "Migrate Contacts" to open contact import
- Review and confirm contact data
- Complete the import process
4. **Verify Migration**
- Run comparison again to verify completion
- Test application functionality
- Export backup data if needed
## Error Handling
### Common Issues
### ActiveDid Migration Errors
- **Missing Account**: If the `activeDid` from Dexie doesn't exist in SQLite accounts
- **Database Errors**: Connection or query failures
- **Settings Update Failures**: Issues updating SQLite master settings
1. **Dexie Database Not Enabled**
- **Error**: "Dexie database is not enabled"
- **Solution**: Set `USE_DEXIE_DB = true` in `constants/app.ts` temporarily
2. **Database Connection Issues**
- **Error**: "Failed to retrieve data"
- **Solution**: Check database initialization and permissions
3. **Migration Failures**
- **Error**: "Migration failed: [specific error]"
- **Solution**: Review error details and check data integrity
### Error Recovery
1. **Review** error messages carefully
2. **Check** browser console for additional details
3. **Verify** database connectivity and permissions
4. **Retry** the operation if appropriate
5. **Export** comparison data for manual review if needed
## Best Practices
### Before Migration
1. **Backup** your data if possible
2. **Test** the migration on a small dataset first
3. **Verify** that both databases are accessible
4. **Review** the comparison results before migrating
### During Migration
1. **Don't** interrupt the migration process
2. **Monitor** the progress and error messages
3. **Note** any warnings or skipped records
4. **Export** comparison data for reference
### After Migration
1. **Verify** that data was migrated correctly
2. **Test** the application functionality
3. **Disable** Dexie database (`USE_DEXIE_DB = false`)
4. **Clean up** any temporary files or exports
## Performance Considerations
### 1. Migration Performance
- Use transactions for bulk data transfer
- Implement progress indicators
- Process data in background when possible
### 2. Application Performance
- Optimize SQLite queries
- Maintain proper database indexes
- Use efficient memory management
### Recovery Strategies
1. **Automatic Recovery**: Migration continues even if activeDid migration fails
2. **Manual Recovery**: Users can manually select their identity after migration
3. **Fallback**: System creates new identity if none exists
## Security Considerations
### 1. Data Protection
- Maintain encryption standards across migration
- Preserve user privacy during migration
- Log all migration operations
### Data Protection
- All sensitive data (mnemonics, private keys) are encrypted
- Migration preserves encryption standards
- No plaintext data exposure during migration
### 2. Error Handling
- Handle migration failures gracefully
- Provide clear user messaging
- Maintain rollback capabilities
### Identity Verification
- ActiveDid migration validates account existence
- Prevents setting non-existent identities as active
- Maintains cryptographic integrity
## Testing Strategy
## Testing
### 1. Migration Testing
```typescript
describe('Database Migration', () => {
it('should migrate data without loss', async () => {
// 1. Enable Dexie
// 2. Create test data
// 3. Run migration
// 4. Verify data integrity
// 5. Disable Dexie
});
});
### Migration Testing
```bash
# Enable Dexie for testing
# Set USE_DEXIE_DB = true in constants/app.ts
# Run migration
npm run migrate
# Verify results
npm run test:migration
```
### 2. Application Testing
### ActiveDid Testing
```typescript
describe('Feature with Database', () => {
it('should work with SQLite only', async () => {
// Test with USE_DEXIE_DB = false
// Verify all operations use PlatformService
});
});
// Test activeDid migration specifically
const result = await migrateActiveDid();
expect(result.success).toBe(true);
expect(result.warnings).toContain('Successfully migrated activeDid');
```
## Troubleshooting
### Common Issues
1. **ActiveDid Not Found**
- Ensure accounts were migrated before activeDid migration
- Check that the Dexie activeDid exists in SQLite accounts
2. **Migration Failures**
- Verify Dexie database is accessible
- Check SQLite database permissions
- Review migration logs for specific errors
3. **Data Inconsistencies**
- Use `compareDatabases()` to identify differences
- Re-run migration if necessary
- Check for duplicate or conflicting records
### Debugging
```typescript
// Enable detailed logging
logger.setLevel('debug');
// Check migration status
const comparison = await compareDatabases();
console.log('Settings differences:', comparison.differences.settings);
```
## Future Enhancements
### Planned Improvements
1. **Batch Processing**: Optimize for large datasets
2. **Incremental Migration**: Support partial migrations
3. **Rollback Capability**: Ability to revert migration
4. **Progress Tracking**: Real-time migration progress
### Performance Optimizations
1. **Parallel Processing**: Migrate independent data concurrently
2. **Memory Management**: Optimize for large datasets
3. **Transaction Batching**: Reduce database round trips
## Conclusion
The migration from Dexie to absurd-sql provides:
- **Better Performance**: Improved query performance and storage efficiency
- **Cross-Platform Consistency**: Unified database interface across platforms
- **Enhanced Security**: Better encryption and access controls
- **Future-Proof Architecture**: Modern SQLite-based storage system
The Dexie to SQLite migration provides a robust, secure, and user-friendly transition path. The addition of activeDid migration ensures that users maintain their identity continuity throughout the migration process, significantly improving the user experience.
The migration fence ensures a controlled and safe transition while maintaining data integrity and application stability.
The migration fence architecture allows for controlled, reversible migration while maintaining application stability and data integrity.

153
src/services/indexedDBMigrationService.ts
View File

@@ -39,6 +39,7 @@ import {
generateUpdateStatement,
generateInsertStatement,
} from "../db/databaseUtil";
import { updateDefaultSettings } from "../db/databaseUtil";
import { importFromMnemonic } from "../libs/util";
/**
@@ -1080,6 +1081,17 @@ export async function migrateSettings(): Promise<MigrationResult> {
});
const platformService = PlatformServiceFactory.getInstance();
// Find the master settings (accountDid is null) which contains the activeDid
const masterSettings = dexieSettings.find(setting => !setting.accountDid);
let dexieActiveDid: string | undefined;
if (masterSettings?.activeDid) {
dexieActiveDid = masterSettings.activeDid;
logger.info("[MigrationService] Found activeDid in Dexie master settings", {
activeDid: dexieActiveDid,
});
}
// Create an array of promises for all settings migrations
const migrationPromises = dexieSettings.map(async (setting) => {
logger.info("[MigrationService] Starting to migrate settings", setting);
@@ -1139,6 +1151,38 @@ export async function migrateSettings(): Promise<MigrationResult> {
// Wait for all migrations to complete
const updatedSettings = await Promise.all(migrationPromises);
// Step 2: Migrate the activeDid if it exists in Dexie
if (dexieActiveDid) {
try {
// Verify that the activeDid exists in SQLite accounts
const accountExists = await platformService.dbQuery(
"SELECT did FROM accounts WHERE did = ?",
[dexieActiveDid],
);
if (accountExists?.values?.length) {
// Update the master settings with the activeDid
await updateDefaultSettings({ activeDid: dexieActiveDid });
logger.info("[MigrationService] Successfully migrated activeDid", {
activeDid: dexieActiveDid,
});
result.warnings.push(`Migrated activeDid: ${dexieActiveDid}`);
} else {
logger.warn("[MigrationService] activeDid from Dexie not found in SQLite accounts", {
activeDid: dexieActiveDid,
});
result.warnings.push(
`activeDid from Dexie (${dexieActiveDid}) not found in SQLite accounts - skipping activeDid migration`,
);
}
} catch (error) {
logger.error("[MigrationService] Failed to migrate activeDid:", error);
result.errors.push(`Failed to migrate activeDid: ${error}`);
}
} else {
logger.info("[MigrationService] No activeDid found in Dexie settings");
}
logger.info(
"[MigrationService] Finished migrating settings",
updatedSettings,
@@ -1279,6 +1323,96 @@ export async function migrateAccounts(): Promise<MigrationResult> {
}
}
/**
* Migrates the activeDid from Dexie to SQLite
*
* This function specifically handles the migration of the activeDid setting
* from the Dexie database to the SQLite database. It ensures that the
* activeDid exists in the SQLite accounts table before setting it as active.
*
* The function is designed to be called after accounts have been migrated
* to ensure the target DID exists in the SQLite database.
*
* @async
* @function migrateActiveDid
* @returns {Promise<MigrationResult>} Result of the activeDid migration
* @throws {Error} If the migration process fails
* @example
* ```typescript
* try {
* const result = await migrateActiveDid();
* if (result.success) {
* console.log('ActiveDid migration successful');
* } else {
* console.error('ActiveDid migration failed:', result.errors);
* }
* } catch (error) {
* console.error('ActiveDid migration process failed:', error);
* }
* ```
*/
export async function migrateActiveDid(): Promise<MigrationResult> {
logger.info("[MigrationService] Starting activeDid migration");
const result: MigrationResult = {
success: true,
contactsMigrated: 0,
settingsMigrated: 0,
accountsMigrated: 0,
errors: [],
warnings: [],
};
try {
// Get Dexie settings to find the activeDid
const dexieSettings = await getDexieSettings();
const masterSettings = dexieSettings.find(setting => !setting.accountDid);
if (!masterSettings?.activeDid) {
logger.info("[MigrationService] No activeDid found in Dexie master settings");
result.warnings.push("No activeDid found in Dexie settings");
return result;
}
const dexieActiveDid = masterSettings.activeDid;
logger.info("[MigrationService] Found activeDid in Dexie", {
activeDid: dexieActiveDid,
});
const platformService = PlatformServiceFactory.getInstance();
// Verify that the activeDid exists in SQLite accounts
const accountExists = await platformService.dbQuery(
"SELECT did FROM accounts WHERE did = ?",
[dexieActiveDid],
);
if (!accountExists?.values?.length) {
const errorMessage = `activeDid from Dexie (${dexieActiveDid}) not found in SQLite accounts`;
logger.error("[MigrationService]", errorMessage);
result.errors.push(errorMessage);
result.success = false;
return result;
}
// Update the master settings with the activeDid
await updateDefaultSettings({ activeDid: dexieActiveDid });
logger.info("[MigrationService] Successfully migrated activeDid", {
activeDid: dexieActiveDid,
});
result.warnings.push(`Successfully migrated activeDid: ${dexieActiveDid}`);
return result;
} catch (error) {
const errorMessage = `ActiveDid migration failed: ${error}`;
logger.error("[MigrationService]", errorMessage, error);
result.errors.push(errorMessage);
result.success = false;
return result;
}
}
/**
* Migrates all data from Dexie to SQLite in the proper order
*
@@ -1286,7 +1420,8 @@ export async function migrateAccounts(): Promise<MigrationResult> {
* in the correct order to avoid foreign key constraint issues:
* 1. Accounts (foundational - contains DIDs)
* 2. Settings (references accountDid, activeDid)
* 3. Contacts (independent, but migrated after accounts for consistency)
* 3. ActiveDid (depends on accounts and settings)
* 4. Contacts (independent, but migrated after accounts for consistency)
*
* The migration runs within a transaction to ensure atomicity. If any step fails,
* the entire migration is rolled back.
@@ -1332,9 +1467,21 @@ export async function migrateAll(): Promise<MigrationResult> {
result.settingsMigrated = settingsResult.settingsMigrated;
result.warnings.push(...settingsResult.warnings);
// Step 3: Migrate Contacts (independent, but after accounts for consistency)
// Step 3: Migrate ActiveDid (depends on accounts and settings)
logger.info("[MigrationService] Step 3: Migrating activeDid...");
const activeDidResult = await migrateActiveDid();
if (!activeDidResult.success) {
result.errors.push(
`ActiveDid migration failed: ${activeDidResult.errors.join(", ")}`,
);
// Don't fail the entire migration for activeDid issues
logger.warn("[MigrationService] ActiveDid migration failed, but continuing with migration");
}
result.warnings.push(...activeDidResult.warnings);
// Step 4: Migrate Contacts (independent, but after accounts for consistency)
// ... but which is better done through the contact import view
// logger.info("[MigrationService] Step 3: Migrating contacts...");
// logger.info("[MigrationService] Step 4: Migrating contacts...");
// const contactsResult = await migrateContacts();
// if (!contactsResult.success) {
// result.errors.push(