Browse Source

docs: Update migration documentation with fence definition and security checklist - Add comprehensive migration fence definition with clear boundaries - Update migration guide to reflect current Phase 2 status - Create security audit checklist for migration process - Update README with migration status and architecture details - Document migration fence enforcement and guidelines - Add security considerations and compliance requirements

migrate-dexie-to-sqlite
Matthew Raymer 6 days ago
parent
commit
d1f61e3530
  1. 47
      README.md
  2. 272
      doc/migration-fence-definition.md
  3. 355
      doc/migration-security-checklist.md
  4. 687
      doc/migration-to-wa-sqlite.md

47
README.md

@ -3,6 +3,32 @@
[Time Safari](https://timesafari.org/) allows people to ease into collaboration: start with expressions of gratitude [Time Safari](https://timesafari.org/) allows people to ease into collaboration: start with expressions of gratitude
and expand to crowd-fund with time & money, then record and see the impact of contributions. and expand to crowd-fund with time & money, then record and see the impact of contributions.
## Database Migration Status
**Current Status**: The application is undergoing a migration from Dexie (IndexedDB) to SQLite using absurd-sql. This migration is in **Phase 2** with a well-defined migration fence in place.
### Migration Progress
- ✅ **SQLite Database Service**: Fully implemented with absurd-sql
- ✅ **Platform Service Layer**: Unified database interface across platforms
- ✅ **Settings Migration**: Core user settings transferred
- ✅ **Account Migration**: Identity and key management
- 🔄 **Contact Migration**: User contact data (via import interface)
- 📋 **Code Cleanup**: Remove unused Dexie imports
### Migration Fence
The migration is controlled by a **migration fence** that separates legacy Dexie code from the new SQLite implementation. See [Migration Fence Definition](doc/migration-fence-definition.md) for complete details.
**Key Points**:
- Legacy Dexie database is disabled by default (`USE_DEXIE_DB = false`)
- All database operations go through `PlatformService`
- Migration tools provide controlled access to both databases
- Clear separation between legacy and new code
### Migration Documentation
- [Migration Guide](doc/migration-to-wa-sqlite.md) - Complete migration process
- [Migration Fence Definition](doc/migration-fence-definition.md) - Fence boundaries and rules
- [Database Migration Guide](doc/database-migration-guide.md) - User-facing migration tools
## Roadmap ## Roadmap
See [project.task.yaml](project.task.yaml) for current priorities. See [project.task.yaml](project.task.yaml) for current priorities.
@ -21,16 +47,10 @@ npm run dev
See [BUILDING.md](BUILDING.md) for more details. See [BUILDING.md](BUILDING.md) for more details.
## Tests ## Tests
See [TESTING.md](test-playwright/TESTING.md) for detailed test instructions. See [TESTING.md](test-playwright/TESTING.md) for detailed test instructions.
## Icons ## Icons
Application icons are in the `assets` directory, processed by the `capacitor-assets` command. Application icons are in the `assets` directory, processed by the `capacitor-assets` command.
@ -66,6 +86,21 @@ Key principles:
- Common interfaces are shared through `common.ts` - Common interfaces are shared through `common.ts`
- Type definitions are generated from Zod schemas where possible - Type definitions are generated from Zod schemas where possible
### Database Architecture
The application uses a platform-agnostic database layer:
* `src/services/PlatformService.ts` - Database interface definition
* `src/services/PlatformServiceFactory.ts` - Platform-specific service factory
* `src/services/AbsurdSqlDatabaseService.ts` - SQLite implementation
* `src/db/` - Legacy Dexie database (migration in progress)
**Development Guidelines**:
- Always use `PlatformService` for database operations
- Never import Dexie directly in application code
- Test with `USE_DEXIE_DB = false` for new features
- Use migration tools for data transfer between systems
### Kudos ### Kudos
Gifts make the world go 'round! Gifts make the world go 'round!

272
doc/migration-fence-definition.md

@ -0,0 +1,272 @@
# Migration Fence Definition: Dexie to SQLite
## Overview
This document defines the **migration fence** - the boundary between the legacy Dexie (IndexedDB) storage system and the new SQLite-based storage system in TimeSafari. The fence ensures controlled migration while maintaining data integrity and application stability.
## Current Migration Status
### ✅ Completed Components
- **SQLite Database Service**: Fully implemented with absurd-sql
- **Platform Service Layer**: Unified database interface across platforms
- **Migration Tools**: Data comparison and transfer utilities
- **Schema Migration**: Complete table structure migration
- **Data Export/Import**: Backup and restore functionality
### 🔄 Active Migration Components
- **Settings Migration**: Core user settings transferred
- **Account Migration**: Identity and key management
- **Contact Migration**: User contact data (via import interface)
### ❌ Legacy Components (Fence Boundary)
- **Dexie Database**: Legacy IndexedDB storage (disabled by default)
- **Dexie-Specific Code**: Direct database access patterns
- **Legacy Migration Paths**: Old data transfer methods
## Migration Fence Definition
### 1. Configuration Boundary
```typescript
// src/constants/app.ts
export const USE_DEXIE_DB = false; // FENCE: Controls legacy database access
```
**Fence Rule**: When `USE_DEXIE_DB = false`:
- All new data operations use SQLite
- Legacy Dexie database is not initialized
- Migration tools are the only path to legacy data
**Fence Rule**: When `USE_DEXIE_DB = true`:
- Legacy database is available for migration
- Dual-write operations may be enabled
- Migration tools can access both databases
### 2. Service Layer Boundary
```typescript
// src/services/PlatformServiceFactory.ts
export class PlatformServiceFactory {
public static getInstance(): PlatformService {
// FENCE: All database operations go through platform service
// No direct Dexie access outside migration tools
}
}
```
**Fence Rule**: All database operations must use:
- `PlatformService.dbQuery()` for read operations
- `PlatformService.dbExec()` for write operations
- No direct `db.` or `accountsDBPromise` access in application code
### 3. Data Access Patterns
#### ✅ Allowed (Inside Fence)
```typescript
// Use platform service for all database operations
const platformService = PlatformServiceFactory.getInstance();
const contacts = await platformService.dbQuery(
"SELECT * FROM contacts WHERE did = ?",
[accountDid]
);
```
#### ❌ Forbidden (Outside Fence)
```typescript
// Direct Dexie access (legacy pattern)
const contacts = await db.contacts.where('did').equals(accountDid).toArray();
// Direct database reference
const result = await accountsDBPromise;
```
### 4. Migration Tool Boundary
```typescript
// src/services/indexedDBMigrationService.ts
// FENCE: Only migration tools can access both databases
export async function compareDatabases(): Promise<DataComparison> {
// This is the ONLY place where both databases are accessed
}
```
**Fence Rule**: Migration tools are the exclusive interface between:
- Legacy Dexie database
- New SQLite database
- Data comparison and transfer operations
## Migration Fence Guidelines
### 1. Code Development Rules
#### New Feature Development
- **Always** use `PlatformService` for database operations
- **Never** import or reference Dexie directly
- **Always** test with `USE_DEXIE_DB = false`
#### Legacy Code Maintenance
- **Only** modify Dexie code for migration purposes
- **Always** add migration tests for schema changes
- **Never** add new Dexie-specific features
### 2. Data Integrity Rules
#### Migration Safety
- **Always** create backups before migration
- **Always** verify data integrity after migration
- **Never** delete legacy data until verified
#### Rollback Strategy
- **Always** maintain ability to rollback to Dexie
- **Always** preserve migration logs
- **Never** assume migration is irreversible
### 3. Testing Requirements
#### Migration Testing
```typescript
// Required test pattern for migration
describe('Database Migration', () => {
it('should migrate data without loss', async () => {
// 1. Enable Dexie
// 2. Create test data
// 3. Run migration
// 4. Verify data integrity
// 5. Disable Dexie
});
});
```
#### Application Testing
```typescript
// Required test pattern for application features
describe('Feature with Database', () => {
it('should work with SQLite only', async () => {
// Test with USE_DEXIE_DB = false
// Verify all operations use PlatformService
});
});
```
## Migration Fence Enforcement
### 1. Static Analysis
#### ESLint Rules
```json
{
"rules": {
"no-restricted-imports": [
"error",
{
"patterns": [
{
"group": ["../db/index"],
"message": "Use PlatformService instead of direct Dexie access"
}
]
}
]
}
}
```
#### TypeScript Rules
```json
{
"compilerOptions": {
"strict": true,
"noImplicitAny": true
}
}
```
### 2. Runtime Checks
#### Development Mode Validation
```typescript
// Development-only fence validation
if (import.meta.env.DEV && USE_DEXIE_DB) {
console.warn('⚠️ Dexie is enabled - migration mode active');
}
```
#### Production Safety
```typescript
// Production fence enforcement
if (import.meta.env.PROD && USE_DEXIE_DB) {
throw new Error('Dexie cannot be enabled in production');
}
```
## Migration Fence Timeline
### Phase 1: Fence Establishment ✅
- [x] Define migration fence boundaries
- [x] Implement PlatformService layer
- [x] Create migration tools
- [x] Set `USE_DEXIE_DB = false` by default
### Phase 2: Data Migration 🔄
- [x] Migrate core settings
- [x] Migrate account data
- [ ] Complete contact migration
- [ ] Verify all data integrity
### Phase 3: Code Cleanup 📋
- [ ] Remove unused Dexie imports
- [ ] Clean up legacy database code
- [ ] Update all documentation
- [ ] Remove migration tools
### Phase 4: Fence Removal 🎯
- [ ] Remove `USE_DEXIE_DB` constant
- [ ] Remove Dexie dependencies
- [ ] Remove migration service
- [ ] Finalize SQLite-only architecture
## Security Considerations
### 1. Data Protection
- **Encryption**: Maintain encryption standards across migration
- **Access Control**: Preserve user privacy during migration
- **Audit Trail**: Log all migration operations
### 2. Error Handling
- **Graceful Degradation**: Handle migration failures gracefully
- **User Communication**: Clear messaging about migration status
- **Recovery Options**: Provide rollback mechanisms
## Performance Considerations
### 1. Migration Performance
- **Batch Operations**: Use transactions for bulk data transfer
- **Progress Indicators**: Show migration progress to users
- **Background Processing**: Non-blocking migration operations
### 2. Application Performance
- **Query Optimization**: Optimize SQLite queries for performance
- **Indexing Strategy**: Maintain proper database indexes
- **Memory Management**: Efficient memory usage during migration
## Documentation Requirements
### 1. Code Documentation
- **Migration Fence Comments**: Document fence boundaries in code
- **API Documentation**: Update all database API documentation
- **Migration Guides**: Comprehensive migration documentation
### 2. User Documentation
- **Migration Instructions**: Clear user migration steps
- **Troubleshooting**: Common migration issues and solutions
- **Rollback Instructions**: How to revert if needed
## Conclusion
The migration fence provides a controlled boundary between legacy and new database systems, ensuring:
- **Data Integrity**: No data loss during migration
- **Application Stability**: Consistent behavior across platforms
- **Development Clarity**: Clear guidelines for code development
- **Migration Safety**: Controlled and reversible migration process
This fence will remain in place until all data is successfully migrated and verified, at which point the legacy system can be safely removed.

355
doc/migration-security-checklist.md

@ -0,0 +1,355 @@
# Database Migration Security Audit Checklist
## Overview
This document provides a comprehensive security audit checklist for the Dexie to SQLite migration in TimeSafari. The checklist ensures that data protection, privacy, and security are maintained throughout the migration process.
## Pre-Migration Security Assessment
### 1. Data Classification and Sensitivity
- [ ] **Data Inventory**
- [ ] Identify all sensitive data types (DIDs, private keys, personal information)
- [ ] Document data retention requirements
- [ ] Map data relationships and dependencies
- [ ] Assess data sensitivity levels (public, internal, confidential, restricted)
- [ ] **Encryption Assessment**
- [ ] Verify current encryption methods for sensitive data
- [ ] Document encryption keys and their management
- [ ] Assess encryption strength and compliance
- [ ] Plan encryption migration strategy
### 2. Access Control Review
- [ ] **User Access Rights**
- [ ] Audit current user permissions and roles
- [ ] Document access control mechanisms
- [ ] Verify principle of least privilege
- [ ] Plan access control migration
- [ ] **System Access**
- [ ] Review database access patterns
- [ ] Document authentication mechanisms
- [ ] Assess session management
- [ ] Plan authentication migration
### 3. Compliance Requirements
- [ ] **Regulatory Compliance**
- [ ] Identify applicable regulations (GDPR, CCPA, etc.)
- [ ] Document data processing requirements
- [ ] Assess privacy impact
- [ ] Plan compliance verification
- [ ] **Industry Standards**
- [ ] Review security standards compliance
- [ ] Document security controls
- [ ] Assess audit requirements
- [ ] Plan standards compliance
## Migration Security Controls
### 1. Data Protection During Migration
- [ ] **Encryption in Transit**
- [ ] Verify all data transfers are encrypted
- [ ] Use secure communication protocols (TLS 1.3+)
- [ ] Implement secure API endpoints
- [ ] Monitor encryption status
- [ ] **Encryption at Rest**
- [ ] Maintain encryption for stored data
- [ ] Verify encryption key management
- [ ] Test encryption/decryption processes
- [ ] Document encryption procedures
### 2. Access Control During Migration
- [ ] **Authentication**
- [ ] Maintain user authentication during migration
- [ ] Verify session management
- [ ] Implement secure token handling
- [ ] Monitor authentication events
- [ ] **Authorization**
- [ ] Preserve user permissions during migration
- [ ] Verify role-based access control
- [ ] Implement audit logging
- [ ] Monitor access patterns
### 3. Data Integrity
- [ ] **Data Validation**
- [ ] Implement input validation for all data
- [ ] Verify data format consistency
- [ ] Test data transformation processes
- [ ] Document validation rules
- [ ] **Data Verification**
- [ ] Implement checksums for data integrity
- [ ] Verify data completeness after migration
- [ ] Test data consistency checks
- [ ] Document verification procedures
## Migration Process Security
### 1. Backup Security
- [ ] **Backup Creation**
- [ ] Create encrypted backups before migration
- [ ] Verify backup integrity
- [ ] Store backups securely
- [ ] Test backup restoration
- [ ] **Backup Access**
- [ ] Limit backup access to authorized personnel
- [ ] Implement backup access logging
- [ ] Verify backup encryption
- [ ] Document backup procedures
### 2. Migration Tool Security
- [ ] **Tool Authentication**
- [ ] Implement secure authentication for migration tools
- [ ] Verify tool access controls
- [ ] Monitor tool usage
- [ ] Document tool security
- [ ] **Tool Validation**
- [ ] Verify migration tool integrity
- [ ] Test tool security features
- [ ] Validate tool outputs
- [ ] Document tool validation
### 3. Error Handling
- [ ] **Error Security**
- [ ] Implement secure error handling
- [ ] Avoid information disclosure in errors
- [ ] Log security-relevant errors
- [ ] Document error procedures
- [ ] **Recovery Security**
- [ ] Implement secure recovery procedures
- [ ] Verify recovery data protection
- [ ] Test recovery processes
- [ ] Document recovery security
## Post-Migration Security
### 1. Data Verification
- [ ] **Data Completeness**
- [ ] Verify all data was migrated successfully
- [ ] Check for data corruption
- [ ] Validate data relationships
- [ ] Document verification results
- [ ] **Data Accuracy**
- [ ] Verify data accuracy after migration
- [ ] Test data consistency
- [ ] Validate data integrity
- [ ] Document accuracy checks
### 2. Access Control Verification
- [ ] **User Access**
- [ ] Verify user access rights after migration
- [ ] Test authentication mechanisms
- [ ] Validate authorization rules
- [ ] Document access verification
- [ ] **System Access**
- [ ] Verify system access controls
- [ ] Test API security
- [ ] Validate session management
- [ ] Document system security
### 3. Security Testing
- [ ] **Penetration Testing**
- [ ] Conduct security penetration testing
- [ ] Test for common vulnerabilities
- [ ] Verify security controls
- [ ] Document test results
- [ ] **Vulnerability Assessment**
- [ ] Scan for security vulnerabilities
- [ ] Assess security posture
- [ ] Identify security gaps
- [ ] Document assessment results
## Monitoring and Logging
### 1. Security Monitoring
- [ ] **Access Monitoring**
- [ ] Monitor database access patterns
- [ ] Track user authentication events
- [ ] Monitor system access
- [ ] Document monitoring procedures
- [ ] **Data Monitoring**
- [ ] Monitor data access patterns
- [ ] Track data modification events
- [ ] Monitor data integrity
- [ ] Document data monitoring
### 2. Security Logging
- [ ] **Audit Logging**
- [ ] Implement comprehensive audit logging
- [ ] Log all security-relevant events
- [ ] Secure log storage and access
- [ ] Document logging procedures
- [ ] **Log Analysis**
- [ ] Implement log analysis tools
- [ ] Monitor for security incidents
- [ ] Analyze security trends
- [ ] Document analysis procedures
## Incident Response
### 1. Security Incident Planning
- [ ] **Incident Response Plan**
- [ ] Develop security incident response plan
- [ ] Define incident response procedures
- [ ] Train incident response team
- [ ] Document response procedures
- [ ] **Incident Detection**
- [ ] Implement incident detection mechanisms
- [ ] Monitor for security incidents
- [ ] Establish incident reporting procedures
- [ ] Document detection procedures
### 2. Recovery Procedures
- [ ] **Data Recovery**
- [ ] Develop data recovery procedures
- [ ] Test recovery processes
- [ ] Verify recovery data integrity
- [ ] Document recovery procedures
- [ ] **System Recovery**
- [ ] Develop system recovery procedures
- [ ] Test system recovery
- [ ] Verify system security after recovery
- [ ] Document recovery procedures
## Compliance Verification
### 1. Regulatory Compliance
- [ ] **Privacy Compliance**
- [ ] Verify GDPR compliance
- [ ] Check CCPA compliance
- [ ] Assess other privacy regulations
- [ ] Document compliance status
- [ ] **Security Compliance**
- [ ] Verify security standard compliance
- [ ] Check industry requirements
- [ ] Assess security certifications
- [ ] Document compliance status
### 2. Audit Requirements
- [ ] **Audit Trail**
- [ ] Maintain comprehensive audit trail
- [ ] Verify audit log integrity
- [ ] Test audit log accessibility
- [ ] Document audit procedures
- [ ] **Audit Reporting**
- [ ] Generate audit reports
- [ ] Verify report accuracy
- [ ] Distribute reports securely
- [ ] Document reporting procedures
## Documentation and Training
### 1. Security Documentation
- [ ] **Security Procedures**
- [ ] Document security procedures
- [ ] Update security policies
- [ ] Create security guidelines
- [ ] Maintain documentation
- [ ] **Security Training**
- [ ] Develop security training materials
- [ ] Train staff on security procedures
- [ ] Verify training effectiveness
- [ ] Document training procedures
### 2. Ongoing Security
- [ ] **Security Maintenance**
- [ ] Establish security maintenance procedures
- [ ] Schedule security updates
- [ ] Monitor security trends
- [ ] Document maintenance procedures
- [ ] **Security Review**
- [ ] Conduct regular security reviews
- [ ] Update security controls
- [ ] Assess security effectiveness
- [ ] Document review procedures
## Risk Assessment
### 1. Risk Identification
- [ ] **Security Risks**
- [ ] Identify potential security risks
- [ ] Assess risk likelihood and impact
- [ ] Prioritize security risks
- [ ] Document risk assessment
- [ ] **Mitigation Strategies**
- [ ] Develop risk mitigation strategies
- [ ] Implement risk controls
- [ ] Monitor risk status
- [ ] Document mitigation procedures
### 2. Risk Monitoring
- [ ] **Risk Tracking**
- [ ] Track identified risks
- [ ] Monitor risk status
- [ ] Update risk assessments
- [ ] Document risk tracking
- [ ] **Risk Reporting**
- [ ] Generate risk reports
- [ ] Distribute risk information
- [ ] Update risk documentation
- [ ] Document reporting procedures
## Conclusion
This security audit checklist ensures that the database migration maintains the highest standards of data protection, privacy, and security. Regular review and updates of this checklist are essential to maintain security throughout the migration process and beyond.
### Security Checklist Summary
- [ ] **Pre-Migration Assessment**: Complete
- [ ] **Migration Controls**: Complete
- [ ] **Process Security**: Complete
- [ ] **Post-Migration Verification**: Complete
- [ ] **Monitoring and Logging**: Complete
- [ ] **Incident Response**: Complete
- [ ] **Compliance Verification**: Complete
- [ ] **Documentation and Training**: Complete
- [ ] **Risk Assessment**: Complete
**Overall Security Status**: [ ] Secure [ ] Needs Attention [ ] Critical Issues
**Next Review Date**: _______________
**Reviewed By**: _______________
**Approved By**: _______________

687
doc/migration-to-wa-sqlite.md

@ -4,6 +4,8 @@
This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users. This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.
**Current Status**: The migration is in **Phase 2** with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress.
## Migration Goals ## Migration Goals
1. **Data Integrity** 1. **Data Integrity**
@ -26,6 +28,17 @@ This document outlines the migration process from Dexie.js to absurd-sql for the
- Minimal downtime - Minimal downtime
- Automatic migration where possible - Automatic migration where possible
## Migration Fence
The migration is controlled by a **migration fence** that separates legacy Dexie code from the new SQLite implementation. See [Migration Fence Definition](./migration-fence-definition.md) for complete details.
### Key Fence Components
1. **Configuration Control**: `USE_DEXIE_DB = false` (default)
2. **Service Layer**: All database operations go through `PlatformService`
3. **Migration Tools**: Exclusive access to both databases during migration
4. **Code Boundaries**: Clear separation between legacy and new code
## Prerequisites ## Prerequisites
1. **Backup Requirements** 1. **Backup Requirements**
@ -62,6 +75,26 @@ This document outlines the migration process from Dexie.js to absurd-sql for the
- Android: Android 5+ with SQLite support - Android: Android 5+ with SQLite support
- Electron: Latest version with SQLite support - Electron: Latest version with SQLite support
## Current Migration Status
### ✅ Completed
- **SQLite Database Service**: Fully implemented with absurd-sql
- **Platform Service Layer**: Unified database interface
- **Migration Tools**: Data comparison and transfer utilities
- **Settings Migration**: Core user settings transferred
- **Account Migration**: Identity and key management
- **Schema Migration**: Complete table structure migration
### 🔄 In Progress
- **Contact Migration**: User contact data (via import interface)
- **Data Verification**: Comprehensive integrity checks
- **Performance Optimization**: Query optimization and indexing
### 📋 Planned
- **Code Cleanup**: Remove unused Dexie imports
- **Documentation Updates**: Complete migration guides
- **Testing**: Comprehensive migration testing
## Migration Process ## Migration Process
### 1. Preparation ### 1. Preparation
@ -72,70 +105,7 @@ import initSqlJs from '@jlongster/sql.js';
import { SQLiteFS } from 'absurd-sql'; import { SQLiteFS } from 'absurd-sql';
import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend'; import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend';
export class MigrationService { class MigrationService {
private static instance: MigrationService;
private backup: MigrationBackup | null = null;
private sql: any = null;
private db: any = null;
async prepare(): Promise<void> {
try {
// 1. Check prerequisites
await this.checkPrerequisites();
// 2. Create backup
this.backup = await this.createBackup();
// 3. Verify backup integrity
await this.verifyBackup();
// 4. Initialize absurd-sql
await this.initializeAbsurdSql();
} catch (error) {
throw new StorageError(
'Migration preparation failed',
StorageErrorCodes.MIGRATION_FAILED,
error
);
}
}
private async initializeAbsurdSql(): Promise<void> {
// Initialize SQL.js
this.sql = await initSqlJs({
locateFile: (file: string) => {
return new URL(`/node_modules/@jlongster/sql.js/dist/${file}`, import.meta.url).href;
}
});
// Setup SQLiteFS with IndexedDB backend
const sqlFS = new SQLiteFS(this.sql.FS, new IndexedDBBackend());
this.sql.register_for_idb(sqlFS);
// Create and mount filesystem
this.sql.FS.mkdir('/sql');
this.sql.FS.mount(sqlFS, {}, '/sql');
// Open database
const path = '/sql/db.sqlite';
if (typeof SharedArrayBuffer === 'undefined') {
let stream = this.sql.FS.open(path, 'a+');
await stream.node.contents.readIfFallback();
this.sql.FS.close(stream);
}
this.db = new this.sql.Database(path, { filename: true });
if (!this.db) {
throw new StorageError(
'Database initialization failed',
StorageErrorCodes.INITIALIZATION_FAILED
);
}
// Configure database
await this.db.exec(`PRAGMA journal_mode=MEMORY;`);
}
private async checkPrerequisites(): Promise<void> { private async checkPrerequisites(): Promise<void> {
// Check IndexedDB availability // Check IndexedDB availability
if (!window.indexedDB) { if (!window.indexedDB) {
@ -186,428 +156,259 @@ export class MigrationService {
```typescript ```typescript
// src/services/storage/migration/DataMigration.ts // src/services/storage/migration/DataMigration.ts
export class DataMigration { class DataMigration {
async migrate(backup: MigrationBackup): Promise<void> { async migrateAccounts(): Promise<MigrationResult> {
const result: MigrationResult = {
success: true,
accountsMigrated: 0,
errors: [],
warnings: []
};
try { try {
// 1. Create new database schema const dexieAccounts = await this.getDexieAccounts();
await this.createSchema();
// 2. Migrate accounts
await this.migrateAccounts(backup.accounts);
// 3. Migrate settings
await this.migrateSettings(backup.settings);
// 4. Migrate contacts for (const account of dexieAccounts) {
await this.migrateContacts(backup.contacts); try {
await this.migrateAccount(account);
// 5. Verify migration result.accountsMigrated++;
await this.verifyMigration(backup); } catch (error) {
result.errors.push(`Failed to migrate account ${account.did}: ${error}`);
result.success = false;
}
}
} catch (error) { } catch (error) {
// 6. Handle failure result.errors.push(`Account migration failed: ${error}`);
await this.handleMigrationFailure(error, backup); result.success = false;
} }
return result;
} }
private async migrateAccounts(accounts: Account[]): Promise<void> { async migrateSettings(): Promise<MigrationResult> {
// Use transaction for atomicity const result: MigrationResult = {
await this.db.exec('BEGIN TRANSACTION;'); success: true,
settingsMigrated: 0,
errors: [],
warnings: []
};
try { try {
for (const account of accounts) { const dexieSettings = await this.getDexieSettings();
await this.db.run(`
INSERT INTO accounts (did, public_key_hex, created_at, updated_at) for (const setting of dexieSettings) {
VALUES (?, ?, ?, ?) try {
`, [ await this.migrateSetting(setting);
account.did, result.settingsMigrated++;
account.publicKeyHex, } catch (error) {
account.createdAt, result.errors.push(`Failed to migrate setting ${setting.id}: ${error}`);
account.updatedAt result.success = false;
]); }
} }
await this.db.exec('COMMIT;');
} catch (error) { } catch (error) {
await this.db.exec('ROLLBACK;'); result.errors.push(`Settings migration failed: ${error}`);
throw error; result.success = false;
} }
return result;
} }
private async verifyMigration(backup: MigrationBackup): Promise<void> { async migrateContacts(): Promise<MigrationResult> {
// Verify account count // Contact migration is handled through the contact import interface
const result = await this.db.exec('SELECT COUNT(*) as count FROM accounts'); // This provides better user control and validation
const accountCount = result[0].values[0][0]; const result: MigrationResult = {
success: true,
if (accountCount !== backup.accounts.length) { contactsMigrated: 0,
throw new StorageError( errors: [],
'Account count mismatch', warnings: []
StorageErrorCodes.VERIFICATION_FAILED };
);
try {
const dexieContacts = await this.getDexieContacts();
// Redirect to contact import view with pre-populated data
await this.redirectToContactImport(dexieContacts);
result.contactsMigrated = dexieContacts.length;
} catch (error) {
result.errors.push(`Contact migration failed: ${error}`);
result.success = false;
} }
// Verify data integrity return result;
await this.verifyDataIntegrity(backup);
} }
} }
``` ```
### 3. Rollback Strategy ### 3. Verification
```typescript ```typescript
// src/services/storage/migration/RollbackService.ts class MigrationVerification {
export class RollbackService { async verifyMigration(dexieData: MigrationData): Promise<boolean> {
async rollback(backup: MigrationBackup): Promise<void> { // Verify account count
try { const accountResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM accounts');
// 1. Stop all database operations const accountCount = accountResult[0].values[0][0];
await this.stopDatabaseOperations(); if (accountCount !== dexieData.accounts.length) {
return false;
// 2. Restore from backup }
await this.restoreFromBackup(backup);
// Verify settings count
// 3. Verify restoration const settingsResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM settings');
await this.verifyRestoration(backup); const settingsCount = settingsResult[0].values[0][0];
if (settingsCount !== dexieData.settings.length) {
// 4. Clean up absurd-sql return false;
await this.cleanupAbsurdSql(); }
} catch (error) {
throw new StorageError( // Verify data integrity
'Rollback failed', for (const account of dexieData.accounts) {
StorageErrorCodes.ROLLBACK_FAILED, const result = await this.sqliteDB.exec(
error 'SELECT * FROM accounts WHERE did = ?',
[account.did]
); );
const migratedAccount = result[0]?.values[0];
if (!migratedAccount ||
migratedAccount[1] !== account.publicKeyHex) {
return false;
}
} }
}
private async restoreFromBackup(backup: MigrationBackup): Promise<void> { return true;
const dexieDB = new Dexie('TimeSafariDB');
// Restore accounts
await dexieDB.accounts.bulkPut(backup.accounts);
// Restore settings
await dexieDB.settings.bulkPut(backup.settings);
// Restore contacts
await dexieDB.contacts.bulkPut(backup.contacts);
} }
} }
``` ```
## Migration UI ## Using the Migration Interface
```vue ### Accessing Migration Tools
<!-- src/components/MigrationProgress.vue -->
<template>
<div class="migration-progress">
<h2>Database Migration</h2>
<div class="progress-container">
<div class="progress-bar" :style="{ width: `${progress}%` }" />
<div class="progress-text">{{ progress }}%</div>
</div>
<div class="status-message">{{ statusMessage }}</div>
<div v-if="error" class="error-message">
{{ error }}
<button @click="retryMigration">Retry</button>
</div>
</div>
</template>
<script setup lang="ts">
import { ref, onMounted } from 'vue';
import { MigrationService } from '@/services/storage/migration/MigrationService';
const progress = ref(0);
const statusMessage = ref('Preparing migration...');
const error = ref<string | null>(null);
const migrationService = MigrationService.getInstance();
async function startMigration() {
try {
// 1. Preparation
statusMessage.value = 'Creating backup...';
await migrationService.prepare();
progress.value = 20;
// 2. Data migration
statusMessage.value = 'Migrating data...';
await migrationService.migrate();
progress.value = 80;
// 3. Verification
statusMessage.value = 'Verifying migration...';
await migrationService.verify();
progress.value = 100;
statusMessage.value = 'Migration completed successfully!';
} catch (err) {
error.value = err instanceof Error ? err.message : 'Migration failed';
statusMessage.value = 'Migration failed';
}
}
async function retryMigration() { 1. Navigate to the **Account** page in the TimeSafari app
error.value = null; 2. Scroll down to find the **Database Migration** link
progress.value = 0; 3. Click the link to open the migration interface
await startMigration();
}
onMounted(() => { ### Migration Steps
startMigration();
});
</script>
<style scoped> 1. **Compare Databases**
.migration-progress { - Click "Compare Databases" to see differences
padding: 2rem; - Review the comparison results
max-width: 600px; - Identify data that needs migration
margin: 0 auto;
}
.progress-container { 2. **Migrate Settings**
position: relative; - Click "Migrate Settings" to transfer user settings
height: 20px; - Verify settings are correctly transferred
background: #eee; - Check application functionality
border-radius: 10px;
overflow: hidden;
margin: 1rem 0;
}
.progress-bar { 3. **Migrate Contacts**
position: absolute; - Click "Migrate Contacts" to open contact import
height: 100%; - Review and confirm contact data
background: #4CAF50; - Complete the import process
transition: width 0.3s ease;
}
.progress-text { 4. **Verify Migration**
position: absolute; - Run comparison again to verify completion
width: 100%; - Test application functionality
text-align: center; - Export backup data if needed
line-height: 20px;
color: #000;
}
.status-message { ## Error Handling
text-align: center;
margin: 1rem 0;
}
.error-message { ### Common Issues
color: #f44336;
text-align: center;
margin: 1rem 0;
}
button { 1. **Dexie Database Not Enabled**
margin-top: 1rem; - **Error**: "Dexie database is not enabled"
padding: 0.5rem 1rem; - **Solution**: Set `USE_DEXIE_DB = true` in `constants/app.ts` temporarily
background: #2196F3;
color: white;
border: none;
border-radius: 4px;
cursor: pointer;
}
button:hover { 2. **Database Connection Issues**
background: #1976D2; - **Error**: "Failed to retrieve data"
} - **Solution**: Check database initialization and permissions
</style>
```
## Testing Strategy 3. **Migration Failures**
- **Error**: "Migration failed: [specific error]"
- **Solution**: Review error details and check data integrity
1. **Unit Tests** ### Error Recovery
```typescript
// src/services/storage/migration/__tests__/MigrationService.spec.ts
describe('MigrationService', () => {
it('should initialize absurd-sql correctly', async () => {
const service = MigrationService.getInstance();
await service.initializeAbsurdSql();
expect(service.isInitialized()).toBe(true);
expect(service.getDatabase()).toBeDefined();
});
it('should create valid backup', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
expect(backup).toBeDefined();
expect(backup.accounts).toBeInstanceOf(Array);
expect(backup.settings).toBeInstanceOf(Array);
expect(backup.contacts).toBeInstanceOf(Array);
});
it('should migrate data correctly', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
await service.migrate(backup);
// Verify migration
const accounts = await service.getMigratedAccounts();
expect(accounts).toHaveLength(backup.accounts.length);
});
it('should handle rollback correctly', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
// Simulate failed migration
await service.migrate(backup);
await service.simulateFailure();
// Perform rollback
await service.rollback(backup);
// Verify rollback
const accounts = await service.getOriginalAccounts();
expect(accounts).toHaveLength(backup.accounts.length);
});
});
```
2. **Integration Tests** 1. **Review** error messages carefully
```typescript 2. **Check** browser console for additional details
// src/services/storage/migration/__tests__/integration/Migration.spec.ts 3. **Verify** database connectivity and permissions
describe('Migration Integration', () => { 4. **Retry** the operation if appropriate
it('should handle concurrent access during migration', async () => { 5. **Export** comparison data for manual review if needed
const service = MigrationService.getInstance();
// Start migration
const migrationPromise = service.migrate();
// Simulate concurrent access
const accessPromises = Array(5).fill(null).map(() =>
service.getAccount('did:test:123')
);
// Wait for all operations
const [migrationResult, ...accessResults] = await Promise.allSettled([
migrationPromise,
...accessPromises
]);
// Verify results
expect(migrationResult.status).toBe('fulfilled');
expect(accessResults.some(r => r.status === 'rejected')).toBe(true);
});
it('should maintain data integrity during platform transition', async () => {
const service = MigrationService.getInstance();
// Simulate platform change
await service.simulatePlatformChange();
// Verify data
const accounts = await service.getAllAccounts();
const settings = await service.getAllSettings();
const contacts = await service.getAllContacts();
expect(accounts).toBeDefined();
expect(settings).toBeDefined();
expect(contacts).toBeDefined();
});
});
```
## Success Criteria ## Best Practices
1. **Data Integrity** ### Before Migration
- [ ] All accounts migrated successfully
- [ ] All settings preserved
- [ ] All contacts transferred
- [ ] No data corruption
2. **Performance** 1. **Backup** your data if possible
- [ ] Migration completes within acceptable time 2. **Test** the migration on a small dataset first
- [ ] No significant performance degradation 3. **Verify** that both databases are accessible
- [ ] Efficient storage usage 4. **Review** the comparison results before migrating
- [ ] Smooth user experience
3. **Security** ### During Migration
- [ ] Encrypted data remains secure
- [ ] Access controls maintained
- [ ] No sensitive data exposure
- [ ] Secure backup process
4. **User Experience** 1. **Don't** interrupt the migration process
- [ ] Clear migration progress 2. **Monitor** the progress and error messages
- [ ] Informative error messages 3. **Note** any warnings or skipped records
- [ ] Automatic recovery from failures 4. **Export** comparison data for reference
- [ ] No data loss
### After Migration
## Rollback Plan
1. **Verify** that data was migrated correctly
1. **Automatic Rollback** 2. **Test** the application functionality
- Triggered by migration failure 3. **Disable** Dexie database (`USE_DEXIE_DB = false`)
- Restores from verified backup 4. **Clean up** any temporary files or exports
- Maintains data consistency
- Logs rollback reason ## Performance Considerations
2. **Manual Rollback** ### 1. Migration Performance
- Available through settings - Use transactions for bulk data transfer
- Requires user confirmation - Implement progress indicators
- Preserves backup data - Process data in background when possible
- Provides rollback status
### 2. Application Performance
3. **Emergency Recovery** - Optimize SQLite queries
- Manual backup restoration - Maintain proper database indexes
- Database repair tools - Use efficient memory management
- Data recovery procedures
- Support contact information ## Security Considerations
## Post-Migration ### 1. Data Protection
- Maintain encryption standards across migration
1. **Verification** - Preserve user privacy during migration
- Data integrity checks - Log all migration operations
- Performance monitoring
- Error rate tracking ### 2. Error Handling
- User feedback collection - Handle migration failures gracefully
- Provide clear user messaging
2. **Cleanup** - Maintain rollback capabilities
- Remove old database
- Clear migration artifacts ## Testing Strategy
- Update application state
- Archive backup data ### 1. Migration Testing
```typescript
3. **Monitoring** describe('Database Migration', () => {
- Track migration success rate it('should migrate data without loss', async () => {
- Monitor performance metrics // 1. Enable Dexie
- Collect error reports // 2. Create test data
- Gather user feedback // 3. Run migration
// 4. Verify data integrity
## Support // 5. Disable Dexie
});
For assistance with migration: });
1. Check the troubleshooting guide ```
2. Review error logs
3. Contact support team ### 2. Application Testing
4. Submit issue report ```typescript
describe('Feature with Database', () => {
## Timeline it('should work with SQLite only', async () => {
// Test with USE_DEXIE_DB = false
1. **Preparation Phase** (1 week) // Verify all operations use PlatformService
- Backup system implementation });
- Migration service development });
- Testing framework setup ```
2. **Testing Phase** (2 weeks) ## Conclusion
- Unit testing
- Integration testing The migration from Dexie to absurd-sql provides:
- Performance testing - **Better Performance**: Improved query performance and storage efficiency
- Security testing - **Cross-Platform Consistency**: Unified database interface across platforms
- **Enhanced Security**: Better encryption and access controls
3. **Deployment Phase** (1 week) - **Future-Proof Architecture**: Modern SQLite-based storage system
- Staged rollout
- Monitoring The migration fence ensures a controlled and safe transition while maintaining data integrity and application stability.
- Support preparation
- Documentation updates
4. **Post-Deployment** (2 weeks)
- Monitoring
- Bug fixes
- Performance optimization
- User feedback collection
Loading…
Cancel
Save