docs: Update migration documentation with fence definition and security checklist - Add comprehensive migration fence definition with clear boundaries - Update migration guide to reflect current Phase 2 status - Create security audit checklist for migration process - Update README with migration status and architecture details - Document migration fence enforcement and guidelines - Add security considerations and compliance requirements

This commit is contained in:
Matthew Raymer
2025-06-20 03:30:43 +00:00
parent 78d27eecfb
commit b89f3310ef
4 changed files with 918 additions and 455 deletions

View File

@@ -4,6 +4,8 @@
This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.
**Current Status**: The migration is in **Phase 2** with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress.
## Migration Goals
1. **Data Integrity**
@@ -26,6 +28,17 @@ This document outlines the migration process from Dexie.js to absurd-sql for the
- Minimal downtime
- Automatic migration where possible
## Migration Fence
The migration is controlled by a **migration fence** that separates legacy Dexie code from the new SQLite implementation. See [Migration Fence Definition](./migration-fence-definition.md) for complete details.
### Key Fence Components
1. **Configuration Control**: `USE_DEXIE_DB = false` (default)
2. **Service Layer**: All database operations go through `PlatformService`
3. **Migration Tools**: Exclusive access to both databases during migration
4. **Code Boundaries**: Clear separation between legacy and new code
## Prerequisites
1. **Backup Requirements**
@@ -62,6 +75,26 @@ This document outlines the migration process from Dexie.js to absurd-sql for the
- Android: Android 5+ with SQLite support
- Electron: Latest version with SQLite support
## Current Migration Status
### ✅ Completed
- **SQLite Database Service**: Fully implemented with absurd-sql
- **Platform Service Layer**: Unified database interface
- **Migration Tools**: Data comparison and transfer utilities
- **Settings Migration**: Core user settings transferred
- **Account Migration**: Identity and key management
- **Schema Migration**: Complete table structure migration
### 🔄 In Progress
- **Contact Migration**: User contact data (via import interface)
- **Data Verification**: Comprehensive integrity checks
- **Performance Optimization**: Query optimization and indexing
### 📋 Planned
- **Code Cleanup**: Remove unused Dexie imports
- **Documentation Updates**: Complete migration guides
- **Testing**: Comprehensive migration testing
## Migration Process
### 1. Preparation
@@ -72,70 +105,7 @@ import initSqlJs from '@jlongster/sql.js';
import { SQLiteFS } from 'absurd-sql';
import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend';
export class MigrationService {
private static instance: MigrationService;
private backup: MigrationBackup | null = null;
private sql: any = null;
private db: any = null;
async prepare(): Promise<void> {
try {
// 1. Check prerequisites
await this.checkPrerequisites();
// 2. Create backup
this.backup = await this.createBackup();
// 3. Verify backup integrity
await this.verifyBackup();
// 4. Initialize absurd-sql
await this.initializeAbsurdSql();
} catch (error) {
throw new StorageError(
'Migration preparation failed',
StorageErrorCodes.MIGRATION_FAILED,
error
);
}
}
private async initializeAbsurdSql(): Promise<void> {
// Initialize SQL.js
this.sql = await initSqlJs({
locateFile: (file: string) => {
return new URL(`/node_modules/@jlongster/sql.js/dist/${file}`, import.meta.url).href;
}
});
// Setup SQLiteFS with IndexedDB backend
const sqlFS = new SQLiteFS(this.sql.FS, new IndexedDBBackend());
this.sql.register_for_idb(sqlFS);
// Create and mount filesystem
this.sql.FS.mkdir('/sql');
this.sql.FS.mount(sqlFS, {}, '/sql');
// Open database
const path = '/sql/db.sqlite';
if (typeof SharedArrayBuffer === 'undefined') {
let stream = this.sql.FS.open(path, 'a+');
await stream.node.contents.readIfFallback();
this.sql.FS.close(stream);
}
this.db = new this.sql.Database(path, { filename: true });
if (!this.db) {
throw new StorageError(
'Database initialization failed',
StorageErrorCodes.INITIALIZATION_FAILED
);
}
// Configure database
await this.db.exec(`PRAGMA journal_mode=MEMORY;`);
}
class MigrationService {
private async checkPrerequisites(): Promise<void> {
// Check IndexedDB availability
if (!window.indexedDB) {
@@ -186,428 +156,259 @@ export class MigrationService {
```typescript
// src/services/storage/migration/DataMigration.ts
export class DataMigration {
async migrate(backup: MigrationBackup): Promise<void> {
try {
// 1. Create new database schema
await this.createSchema();
// 2. Migrate accounts
await this.migrateAccounts(backup.accounts);
// 3. Migrate settings
await this.migrateSettings(backup.settings);
// 4. Migrate contacts
await this.migrateContacts(backup.contacts);
// 5. Verify migration
await this.verifyMigration(backup);
} catch (error) {
// 6. Handle failure
await this.handleMigrationFailure(error, backup);
}
}
class DataMigration {
async migrateAccounts(): Promise<MigrationResult> {
const result: MigrationResult = {
success: true,
accountsMigrated: 0,
errors: [],
warnings: []
};
private async migrateAccounts(accounts: Account[]): Promise<void> {
// Use transaction for atomicity
await this.db.exec('BEGIN TRANSACTION;');
try {
for (const account of accounts) {
await this.db.run(`
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
VALUES (?, ?, ?, ?)
`, [
account.did,
account.publicKeyHex,
account.createdAt,
account.updatedAt
]);
const dexieAccounts = await this.getDexieAccounts();
for (const account of dexieAccounts) {
try {
await this.migrateAccount(account);
result.accountsMigrated++;
} catch (error) {
result.errors.push(`Failed to migrate account ${account.did}: ${error}`);
result.success = false;
}
}
await this.db.exec('COMMIT;');
} catch (error) {
await this.db.exec('ROLLBACK;');
throw error;
result.errors.push(`Account migration failed: ${error}`);
result.success = false;
}
return result;
}
private async verifyMigration(backup: MigrationBackup): Promise<void> {
async migrateSettings(): Promise<MigrationResult> {
const result: MigrationResult = {
success: true,
settingsMigrated: 0,
errors: [],
warnings: []
};
try {
const dexieSettings = await this.getDexieSettings();
for (const setting of dexieSettings) {
try {
await this.migrateSetting(setting);
result.settingsMigrated++;
} catch (error) {
result.errors.push(`Failed to migrate setting ${setting.id}: ${error}`);
result.success = false;
}
}
} catch (error) {
result.errors.push(`Settings migration failed: ${error}`);
result.success = false;
}
return result;
}
async migrateContacts(): Promise<MigrationResult> {
// Contact migration is handled through the contact import interface
// This provides better user control and validation
const result: MigrationResult = {
success: true,
contactsMigrated: 0,
errors: [],
warnings: []
};
try {
const dexieContacts = await this.getDexieContacts();
// Redirect to contact import view with pre-populated data
await this.redirectToContactImport(dexieContacts);
result.contactsMigrated = dexieContacts.length;
} catch (error) {
result.errors.push(`Contact migration failed: ${error}`);
result.success = false;
}
return result;
}
}
```
### 3. Verification
```typescript
class MigrationVerification {
async verifyMigration(dexieData: MigrationData): Promise<boolean> {
// Verify account count
const result = await this.db.exec('SELECT COUNT(*) as count FROM accounts');
const accountCount = result[0].values[0][0];
if (accountCount !== backup.accounts.length) {
throw new StorageError(
'Account count mismatch',
StorageErrorCodes.VERIFICATION_FAILED
);
const accountResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM accounts');
const accountCount = accountResult[0].values[0][0];
if (accountCount !== dexieData.accounts.length) {
return false;
}
// Verify settings count
const settingsResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM settings');
const settingsCount = settingsResult[0].values[0][0];
if (settingsCount !== dexieData.settings.length) {
return false;
}
// Verify data integrity
await this.verifyDataIntegrity(backup);
}
}
```
### 3. Rollback Strategy
```typescript
// src/services/storage/migration/RollbackService.ts
export class RollbackService {
async rollback(backup: MigrationBackup): Promise<void> {
try {
// 1. Stop all database operations
await this.stopDatabaseOperations();
// 2. Restore from backup
await this.restoreFromBackup(backup);
// 3. Verify restoration
await this.verifyRestoration(backup);
// 4. Clean up absurd-sql
await this.cleanupAbsurdSql();
} catch (error) {
throw new StorageError(
'Rollback failed',
StorageErrorCodes.ROLLBACK_FAILED,
error
for (const account of dexieData.accounts) {
const result = await this.sqliteDB.exec(
'SELECT * FROM accounts WHERE did = ?',
[account.did]
);
const migratedAccount = result[0]?.values[0];
if (!migratedAccount ||
migratedAccount[1] !== account.publicKeyHex) {
return false;
}
}
}
private async restoreFromBackup(backup: MigrationBackup): Promise<void> {
const dexieDB = new Dexie('TimeSafariDB');
// Restore accounts
await dexieDB.accounts.bulkPut(backup.accounts);
// Restore settings
await dexieDB.settings.bulkPut(backup.settings);
// Restore contacts
await dexieDB.contacts.bulkPut(backup.contacts);
return true;
}
}
```
## Migration UI
## Using the Migration Interface
```vue
<!-- src/components/MigrationProgress.vue -->
<template>
<div class="migration-progress">
<h2>Database Migration</h2>
<div class="progress-container">
<div class="progress-bar" :style="{ width: `${progress}%` }" />
<div class="progress-text">{{ progress }}%</div>
</div>
<div class="status-message">{{ statusMessage }}</div>
<div v-if="error" class="error-message">
{{ error }}
<button @click="retryMigration">Retry</button>
</div>
</div>
</template>
### Accessing Migration Tools
<script setup lang="ts">
import { ref, onMounted } from 'vue';
import { MigrationService } from '@/services/storage/migration/MigrationService';
1. Navigate to the **Account** page in the TimeSafari app
2. Scroll down to find the **Database Migration** link
3. Click the link to open the migration interface
const progress = ref(0);
const statusMessage = ref('Preparing migration...');
const error = ref<string | null>(null);
### Migration Steps
const migrationService = MigrationService.getInstance();
1. **Compare Databases**
- Click "Compare Databases" to see differences
- Review the comparison results
- Identify data that needs migration
async function startMigration() {
try {
// 1. Preparation
statusMessage.value = 'Creating backup...';
await migrationService.prepare();
progress.value = 20;
// 2. Data migration
statusMessage.value = 'Migrating data...';
await migrationService.migrate();
progress.value = 80;
// 3. Verification
statusMessage.value = 'Verifying migration...';
await migrationService.verify();
progress.value = 100;
statusMessage.value = 'Migration completed successfully!';
} catch (err) {
error.value = err instanceof Error ? err.message : 'Migration failed';
statusMessage.value = 'Migration failed';
}
}
2. **Migrate Settings**
- Click "Migrate Settings" to transfer user settings
- Verify settings are correctly transferred
- Check application functionality
async function retryMigration() {
error.value = null;
progress.value = 0;
await startMigration();
}
3. **Migrate Contacts**
- Click "Migrate Contacts" to open contact import
- Review and confirm contact data
- Complete the import process
onMounted(() => {
startMigration();
});
</script>
4. **Verify Migration**
- Run comparison again to verify completion
- Test application functionality
- Export backup data if needed
<style scoped>
.migration-progress {
padding: 2rem;
max-width: 600px;
margin: 0 auto;
}
## Error Handling
.progress-container {
position: relative;
height: 20px;
background: #eee;
border-radius: 10px;
overflow: hidden;
margin: 1rem 0;
}
### Common Issues
.progress-bar {
position: absolute;
height: 100%;
background: #4CAF50;
transition: width 0.3s ease;
}
1. **Dexie Database Not Enabled**
- **Error**: "Dexie database is not enabled"
- **Solution**: Set `USE_DEXIE_DB = true` in `constants/app.ts` temporarily
.progress-text {
position: absolute;
width: 100%;
text-align: center;
line-height: 20px;
color: #000;
}
2. **Database Connection Issues**
- **Error**: "Failed to retrieve data"
- **Solution**: Check database initialization and permissions
.status-message {
text-align: center;
margin: 1rem 0;
}
3. **Migration Failures**
- **Error**: "Migration failed: [specific error]"
- **Solution**: Review error details and check data integrity
.error-message {
color: #f44336;
text-align: center;
margin: 1rem 0;
}
### Error Recovery
button {
margin-top: 1rem;
padding: 0.5rem 1rem;
background: #2196F3;
color: white;
border: none;
border-radius: 4px;
cursor: pointer;
}
1. **Review** error messages carefully
2. **Check** browser console for additional details
3. **Verify** database connectivity and permissions
4. **Retry** the operation if appropriate
5. **Export** comparison data for manual review if needed
button:hover {
background: #1976D2;
}
</style>
```
## Best Practices
### Before Migration
1. **Backup** your data if possible
2. **Test** the migration on a small dataset first
3. **Verify** that both databases are accessible
4. **Review** the comparison results before migrating
### During Migration
1. **Don't** interrupt the migration process
2. **Monitor** the progress and error messages
3. **Note** any warnings or skipped records
4. **Export** comparison data for reference
### After Migration
1. **Verify** that data was migrated correctly
2. **Test** the application functionality
3. **Disable** Dexie database (`USE_DEXIE_DB = false`)
4. **Clean up** any temporary files or exports
## Performance Considerations
### 1. Migration Performance
- Use transactions for bulk data transfer
- Implement progress indicators
- Process data in background when possible
### 2. Application Performance
- Optimize SQLite queries
- Maintain proper database indexes
- Use efficient memory management
## Security Considerations
### 1. Data Protection
- Maintain encryption standards across migration
- Preserve user privacy during migration
- Log all migration operations
### 2. Error Handling
- Handle migration failures gracefully
- Provide clear user messaging
- Maintain rollback capabilities
## Testing Strategy
1. **Unit Tests**
```typescript
// src/services/storage/migration/__tests__/MigrationService.spec.ts
describe('MigrationService', () => {
it('should initialize absurd-sql correctly', async () => {
const service = MigrationService.getInstance();
await service.initializeAbsurdSql();
expect(service.isInitialized()).toBe(true);
expect(service.getDatabase()).toBeDefined();
});
### 1. Migration Testing
```typescript
describe('Database Migration', () => {
it('should migrate data without loss', async () => {
// 1. Enable Dexie
// 2. Create test data
// 3. Run migration
// 4. Verify data integrity
// 5. Disable Dexie
});
});
```
it('should create valid backup', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
expect(backup).toBeDefined();
expect(backup.accounts).toBeInstanceOf(Array);
expect(backup.settings).toBeInstanceOf(Array);
expect(backup.contacts).toBeInstanceOf(Array);
});
### 2. Application Testing
```typescript
describe('Feature with Database', () => {
it('should work with SQLite only', async () => {
// Test with USE_DEXIE_DB = false
// Verify all operations use PlatformService
});
});
```
it('should migrate data correctly', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
await service.migrate(backup);
// Verify migration
const accounts = await service.getMigratedAccounts();
expect(accounts).toHaveLength(backup.accounts.length);
});
## Conclusion
it('should handle rollback correctly', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
// Simulate failed migration
await service.migrate(backup);
await service.simulateFailure();
// Perform rollback
await service.rollback(backup);
// Verify rollback
const accounts = await service.getOriginalAccounts();
expect(accounts).toHaveLength(backup.accounts.length);
});
});
```
The migration from Dexie to absurd-sql provides:
- **Better Performance**: Improved query performance and storage efficiency
- **Cross-Platform Consistency**: Unified database interface across platforms
- **Enhanced Security**: Better encryption and access controls
- **Future-Proof Architecture**: Modern SQLite-based storage system
2. **Integration Tests**
```typescript
// src/services/storage/migration/__tests__/integration/Migration.spec.ts
describe('Migration Integration', () => {
it('should handle concurrent access during migration', async () => {
const service = MigrationService.getInstance();
// Start migration
const migrationPromise = service.migrate();
// Simulate concurrent access
const accessPromises = Array(5).fill(null).map(() =>
service.getAccount('did:test:123')
);
// Wait for all operations
const [migrationResult, ...accessResults] = await Promise.allSettled([
migrationPromise,
...accessPromises
]);
// Verify results
expect(migrationResult.status).toBe('fulfilled');
expect(accessResults.some(r => r.status === 'rejected')).toBe(true);
});
it('should maintain data integrity during platform transition', async () => {
const service = MigrationService.getInstance();
// Simulate platform change
await service.simulatePlatformChange();
// Verify data
const accounts = await service.getAllAccounts();
const settings = await service.getAllSettings();
const contacts = await service.getAllContacts();
expect(accounts).toBeDefined();
expect(settings).toBeDefined();
expect(contacts).toBeDefined();
});
});
```
## Success Criteria
1. **Data Integrity**
- [ ] All accounts migrated successfully
- [ ] All settings preserved
- [ ] All contacts transferred
- [ ] No data corruption
2. **Performance**
- [ ] Migration completes within acceptable time
- [ ] No significant performance degradation
- [ ] Efficient storage usage
- [ ] Smooth user experience
3. **Security**
- [ ] Encrypted data remains secure
- [ ] Access controls maintained
- [ ] No sensitive data exposure
- [ ] Secure backup process
4. **User Experience**
- [ ] Clear migration progress
- [ ] Informative error messages
- [ ] Automatic recovery from failures
- [ ] No data loss
## Rollback Plan
1. **Automatic Rollback**
- Triggered by migration failure
- Restores from verified backup
- Maintains data consistency
- Logs rollback reason
2. **Manual Rollback**
- Available through settings
- Requires user confirmation
- Preserves backup data
- Provides rollback status
3. **Emergency Recovery**
- Manual backup restoration
- Database repair tools
- Data recovery procedures
- Support contact information
## Post-Migration
1. **Verification**
- Data integrity checks
- Performance monitoring
- Error rate tracking
- User feedback collection
2. **Cleanup**
- Remove old database
- Clear migration artifacts
- Update application state
- Archive backup data
3. **Monitoring**
- Track migration success rate
- Monitor performance metrics
- Collect error reports
- Gather user feedback
## Support
For assistance with migration:
1. Check the troubleshooting guide
2. Review error logs
3. Contact support team
4. Submit issue report
## Timeline
1. **Preparation Phase** (1 week)
- Backup system implementation
- Migration service development
- Testing framework setup
2. **Testing Phase** (2 weeks)
- Unit testing
- Integration testing
- Performance testing
- Security testing
3. **Deployment Phase** (1 week)
- Staged rollout
- Monitoring
- Support preparation
- Documentation updates
4. **Post-Deployment** (2 weeks)
- Monitoring
- Bug fixes
- Performance optimization
- User feedback collection
The migration fence ensures a controlled and safe transition while maintaining data integrity and application stability.