Browse Source

docs: Update migration documentation with fence definition and security checklist - Add comprehensive migration fence definition with clear boundaries - Update migration guide to reflect current Phase 2 status - Create security audit checklist for migration process - Update README with migration status and architecture details - Document migration fence enforcement and guidelines - Add security considerations and compliance requirements

migrate-dexie-to-sqlite
Matthew Raymer 6 days ago
parent
commit
d1f61e3530
  1. 47
      README.md
  2. 272
      doc/migration-fence-definition.md
  3. 355
      doc/migration-security-checklist.md
  4. 681
      doc/migration-to-wa-sqlite.md

47
README.md

@ -3,6 +3,32 @@
[Time Safari](https://timesafari.org/) allows people to ease into collaboration: start with expressions of gratitude [Time Safari](https://timesafari.org/) allows people to ease into collaboration: start with expressions of gratitude
and expand to crowd-fund with time & money, then record and see the impact of contributions. and expand to crowd-fund with time & money, then record and see the impact of contributions.
## Database Migration Status
**Current Status**: The application is undergoing a migration from Dexie (IndexedDB) to SQLite using absurd-sql. This migration is in **Phase 2** with a well-defined migration fence in place.
### Migration Progress
- ✅ **SQLite Database Service**: Fully implemented with absurd-sql
- ✅ **Platform Service Layer**: Unified database interface across platforms
- ✅ **Settings Migration**: Core user settings transferred
- ✅ **Account Migration**: Identity and key management
- 🔄 **Contact Migration**: User contact data (via import interface)
- 📋 **Code Cleanup**: Remove unused Dexie imports
### Migration Fence
The migration is controlled by a **migration fence** that separates legacy Dexie code from the new SQLite implementation. See [Migration Fence Definition](doc/migration-fence-definition.md) for complete details.
**Key Points**:
- Legacy Dexie database is disabled by default (`USE_DEXIE_DB = false`)
- All database operations go through `PlatformService`
- Migration tools provide controlled access to both databases
- Clear separation between legacy and new code
### Migration Documentation
- [Migration Guide](doc/migration-to-wa-sqlite.md) - Complete migration process
- [Migration Fence Definition](doc/migration-fence-definition.md) - Fence boundaries and rules
- [Database Migration Guide](doc/database-migration-guide.md) - User-facing migration tools
## Roadmap ## Roadmap
See [project.task.yaml](project.task.yaml) for current priorities. See [project.task.yaml](project.task.yaml) for current priorities.
@ -21,16 +47,10 @@ npm run dev
See [BUILDING.md](BUILDING.md) for more details. See [BUILDING.md](BUILDING.md) for more details.
## Tests ## Tests
See [TESTING.md](test-playwright/TESTING.md) for detailed test instructions. See [TESTING.md](test-playwright/TESTING.md) for detailed test instructions.
## Icons ## Icons
Application icons are in the `assets` directory, processed by the `capacitor-assets` command. Application icons are in the `assets` directory, processed by the `capacitor-assets` command.
@ -66,6 +86,21 @@ Key principles:
- Common interfaces are shared through `common.ts` - Common interfaces are shared through `common.ts`
- Type definitions are generated from Zod schemas where possible - Type definitions are generated from Zod schemas where possible
### Database Architecture
The application uses a platform-agnostic database layer:
* `src/services/PlatformService.ts` - Database interface definition
* `src/services/PlatformServiceFactory.ts` - Platform-specific service factory
* `src/services/AbsurdSqlDatabaseService.ts` - SQLite implementation
* `src/db/` - Legacy Dexie database (migration in progress)
**Development Guidelines**:
- Always use `PlatformService` for database operations
- Never import Dexie directly in application code
- Test with `USE_DEXIE_DB = false` for new features
- Use migration tools for data transfer between systems
### Kudos ### Kudos
Gifts make the world go 'round! Gifts make the world go 'round!

272
doc/migration-fence-definition.md

@ -0,0 +1,272 @@
# Migration Fence Definition: Dexie to SQLite
## Overview
This document defines the **migration fence** - the boundary between the legacy Dexie (IndexedDB) storage system and the new SQLite-based storage system in TimeSafari. The fence ensures controlled migration while maintaining data integrity and application stability.
## Current Migration Status
### ✅ Completed Components
- **SQLite Database Service**: Fully implemented with absurd-sql
- **Platform Service Layer**: Unified database interface across platforms
- **Migration Tools**: Data comparison and transfer utilities
- **Schema Migration**: Complete table structure migration
- **Data Export/Import**: Backup and restore functionality
### 🔄 Active Migration Components
- **Settings Migration**: Core user settings transferred
- **Account Migration**: Identity and key management
- **Contact Migration**: User contact data (via import interface)
### ❌ Legacy Components (Fence Boundary)
- **Dexie Database**: Legacy IndexedDB storage (disabled by default)
- **Dexie-Specific Code**: Direct database access patterns
- **Legacy Migration Paths**: Old data transfer methods
## Migration Fence Definition
### 1. Configuration Boundary
```typescript
// src/constants/app.ts
export const USE_DEXIE_DB = false; // FENCE: Controls legacy database access
```
**Fence Rule**: When `USE_DEXIE_DB = false`:
- All new data operations use SQLite
- Legacy Dexie database is not initialized
- Migration tools are the only path to legacy data
**Fence Rule**: When `USE_DEXIE_DB = true`:
- Legacy database is available for migration
- Dual-write operations may be enabled
- Migration tools can access both databases
### 2. Service Layer Boundary
```typescript
// src/services/PlatformServiceFactory.ts
export class PlatformServiceFactory {
public static getInstance(): PlatformService {
// FENCE: All database operations go through platform service
// No direct Dexie access outside migration tools
}
}
```
**Fence Rule**: All database operations must use:
- `PlatformService.dbQuery()` for read operations
- `PlatformService.dbExec()` for write operations
- No direct `db.` or `accountsDBPromise` access in application code
### 3. Data Access Patterns
#### ✅ Allowed (Inside Fence)
```typescript
// Use platform service for all database operations
const platformService = PlatformServiceFactory.getInstance();
const contacts = await platformService.dbQuery(
"SELECT * FROM contacts WHERE did = ?",
[accountDid]
);
```
#### ❌ Forbidden (Outside Fence)
```typescript
// Direct Dexie access (legacy pattern)
const contacts = await db.contacts.where('did').equals(accountDid).toArray();
// Direct database reference
const result = await accountsDBPromise;
```
### 4. Migration Tool Boundary
```typescript
// src/services/indexedDBMigrationService.ts
// FENCE: Only migration tools can access both databases
export async function compareDatabases(): Promise<DataComparison> {
// This is the ONLY place where both databases are accessed
}
```
**Fence Rule**: Migration tools are the exclusive interface between:
- Legacy Dexie database
- New SQLite database
- Data comparison and transfer operations
## Migration Fence Guidelines
### 1. Code Development Rules
#### New Feature Development
- **Always** use `PlatformService` for database operations
- **Never** import or reference Dexie directly
- **Always** test with `USE_DEXIE_DB = false`
#### Legacy Code Maintenance
- **Only** modify Dexie code for migration purposes
- **Always** add migration tests for schema changes
- **Never** add new Dexie-specific features
### 2. Data Integrity Rules
#### Migration Safety
- **Always** create backups before migration
- **Always** verify data integrity after migration
- **Never** delete legacy data until verified
#### Rollback Strategy
- **Always** maintain ability to rollback to Dexie
- **Always** preserve migration logs
- **Never** assume migration is irreversible
### 3. Testing Requirements
#### Migration Testing
```typescript
// Required test pattern for migration
describe('Database Migration', () => {
it('should migrate data without loss', async () => {
// 1. Enable Dexie
// 2. Create test data
// 3. Run migration
// 4. Verify data integrity
// 5. Disable Dexie
});
});
```
#### Application Testing
```typescript
// Required test pattern for application features
describe('Feature with Database', () => {
it('should work with SQLite only', async () => {
// Test with USE_DEXIE_DB = false
// Verify all operations use PlatformService
});
});
```
## Migration Fence Enforcement
### 1. Static Analysis
#### ESLint Rules
```json
{
"rules": {
"no-restricted-imports": [
"error",
{
"patterns": [
{
"group": ["../db/index"],
"message": "Use PlatformService instead of direct Dexie access"
}
]
}
]
}
}
```
#### TypeScript Rules
```json
{
"compilerOptions": {
"strict": true,
"noImplicitAny": true
}
}
```
### 2. Runtime Checks
#### Development Mode Validation
```typescript
// Development-only fence validation
if (import.meta.env.DEV && USE_DEXIE_DB) {
console.warn('⚠️ Dexie is enabled - migration mode active');
}
```
#### Production Safety
```typescript
// Production fence enforcement
if (import.meta.env.PROD && USE_DEXIE_DB) {
throw new Error('Dexie cannot be enabled in production');
}
```
## Migration Fence Timeline
### Phase 1: Fence Establishment ✅
- [x] Define migration fence boundaries
- [x] Implement PlatformService layer
- [x] Create migration tools
- [x] Set `USE_DEXIE_DB = false` by default
### Phase 2: Data Migration 🔄
- [x] Migrate core settings
- [x] Migrate account data
- [ ] Complete contact migration
- [ ] Verify all data integrity
### Phase 3: Code Cleanup 📋
- [ ] Remove unused Dexie imports
- [ ] Clean up legacy database code
- [ ] Update all documentation
- [ ] Remove migration tools
### Phase 4: Fence Removal 🎯
- [ ] Remove `USE_DEXIE_DB` constant
- [ ] Remove Dexie dependencies
- [ ] Remove migration service
- [ ] Finalize SQLite-only architecture
## Security Considerations
### 1. Data Protection
- **Encryption**: Maintain encryption standards across migration
- **Access Control**: Preserve user privacy during migration
- **Audit Trail**: Log all migration operations
### 2. Error Handling
- **Graceful Degradation**: Handle migration failures gracefully
- **User Communication**: Clear messaging about migration status
- **Recovery Options**: Provide rollback mechanisms
## Performance Considerations
### 1. Migration Performance
- **Batch Operations**: Use transactions for bulk data transfer
- **Progress Indicators**: Show migration progress to users
- **Background Processing**: Non-blocking migration operations
### 2. Application Performance
- **Query Optimization**: Optimize SQLite queries for performance
- **Indexing Strategy**: Maintain proper database indexes
- **Memory Management**: Efficient memory usage during migration
## Documentation Requirements
### 1. Code Documentation
- **Migration Fence Comments**: Document fence boundaries in code
- **API Documentation**: Update all database API documentation
- **Migration Guides**: Comprehensive migration documentation
### 2. User Documentation
- **Migration Instructions**: Clear user migration steps
- **Troubleshooting**: Common migration issues and solutions
- **Rollback Instructions**: How to revert if needed
## Conclusion
The migration fence provides a controlled boundary between legacy and new database systems, ensuring:
- **Data Integrity**: No data loss during migration
- **Application Stability**: Consistent behavior across platforms
- **Development Clarity**: Clear guidelines for code development
- **Migration Safety**: Controlled and reversible migration process
This fence will remain in place until all data is successfully migrated and verified, at which point the legacy system can be safely removed.

355
doc/migration-security-checklist.md

@ -0,0 +1,355 @@
# Database Migration Security Audit Checklist
## Overview
This document provides a comprehensive security audit checklist for the Dexie to SQLite migration in TimeSafari. The checklist ensures that data protection, privacy, and security are maintained throughout the migration process.
## Pre-Migration Security Assessment
### 1. Data Classification and Sensitivity
- [ ] **Data Inventory**
- [ ] Identify all sensitive data types (DIDs, private keys, personal information)
- [ ] Document data retention requirements
- [ ] Map data relationships and dependencies
- [ ] Assess data sensitivity levels (public, internal, confidential, restricted)
- [ ] **Encryption Assessment**
- [ ] Verify current encryption methods for sensitive data
- [ ] Document encryption keys and their management
- [ ] Assess encryption strength and compliance
- [ ] Plan encryption migration strategy
### 2. Access Control Review
- [ ] **User Access Rights**
- [ ] Audit current user permissions and roles
- [ ] Document access control mechanisms
- [ ] Verify principle of least privilege
- [ ] Plan access control migration
- [ ] **System Access**
- [ ] Review database access patterns
- [ ] Document authentication mechanisms
- [ ] Assess session management
- [ ] Plan authentication migration
### 3. Compliance Requirements
- [ ] **Regulatory Compliance**
- [ ] Identify applicable regulations (GDPR, CCPA, etc.)
- [ ] Document data processing requirements
- [ ] Assess privacy impact
- [ ] Plan compliance verification
- [ ] **Industry Standards**
- [ ] Review security standards compliance
- [ ] Document security controls
- [ ] Assess audit requirements
- [ ] Plan standards compliance
## Migration Security Controls
### 1. Data Protection During Migration
- [ ] **Encryption in Transit**
- [ ] Verify all data transfers are encrypted
- [ ] Use secure communication protocols (TLS 1.3+)
- [ ] Implement secure API endpoints
- [ ] Monitor encryption status
- [ ] **Encryption at Rest**
- [ ] Maintain encryption for stored data
- [ ] Verify encryption key management
- [ ] Test encryption/decryption processes
- [ ] Document encryption procedures
### 2. Access Control During Migration
- [ ] **Authentication**
- [ ] Maintain user authentication during migration
- [ ] Verify session management
- [ ] Implement secure token handling
- [ ] Monitor authentication events
- [ ] **Authorization**
- [ ] Preserve user permissions during migration
- [ ] Verify role-based access control
- [ ] Implement audit logging
- [ ] Monitor access patterns
### 3. Data Integrity
- [ ] **Data Validation**
- [ ] Implement input validation for all data
- [ ] Verify data format consistency
- [ ] Test data transformation processes
- [ ] Document validation rules
- [ ] **Data Verification**
- [ ] Implement checksums for data integrity
- [ ] Verify data completeness after migration
- [ ] Test data consistency checks
- [ ] Document verification procedures
## Migration Process Security
### 1. Backup Security
- [ ] **Backup Creation**
- [ ] Create encrypted backups before migration
- [ ] Verify backup integrity
- [ ] Store backups securely
- [ ] Test backup restoration
- [ ] **Backup Access**
- [ ] Limit backup access to authorized personnel
- [ ] Implement backup access logging
- [ ] Verify backup encryption
- [ ] Document backup procedures
### 2. Migration Tool Security
- [ ] **Tool Authentication**
- [ ] Implement secure authentication for migration tools
- [ ] Verify tool access controls
- [ ] Monitor tool usage
- [ ] Document tool security
- [ ] **Tool Validation**
- [ ] Verify migration tool integrity
- [ ] Test tool security features
- [ ] Validate tool outputs
- [ ] Document tool validation
### 3. Error Handling
- [ ] **Error Security**
- [ ] Implement secure error handling
- [ ] Avoid information disclosure in errors
- [ ] Log security-relevant errors
- [ ] Document error procedures
- [ ] **Recovery Security**
- [ ] Implement secure recovery procedures
- [ ] Verify recovery data protection
- [ ] Test recovery processes
- [ ] Document recovery security
## Post-Migration Security
### 1. Data Verification
- [ ] **Data Completeness**
- [ ] Verify all data was migrated successfully
- [ ] Check for data corruption
- [ ] Validate data relationships
- [ ] Document verification results
- [ ] **Data Accuracy**
- [ ] Verify data accuracy after migration
- [ ] Test data consistency
- [ ] Validate data integrity
- [ ] Document accuracy checks
### 2. Access Control Verification
- [ ] **User Access**
- [ ] Verify user access rights after migration
- [ ] Test authentication mechanisms
- [ ] Validate authorization rules
- [ ] Document access verification
- [ ] **System Access**
- [ ] Verify system access controls
- [ ] Test API security
- [ ] Validate session management
- [ ] Document system security
### 3. Security Testing
- [ ] **Penetration Testing**
- [ ] Conduct security penetration testing
- [ ] Test for common vulnerabilities
- [ ] Verify security controls
- [ ] Document test results
- [ ] **Vulnerability Assessment**
- [ ] Scan for security vulnerabilities
- [ ] Assess security posture
- [ ] Identify security gaps
- [ ] Document assessment results
## Monitoring and Logging
### 1. Security Monitoring
- [ ] **Access Monitoring**
- [ ] Monitor database access patterns
- [ ] Track user authentication events
- [ ] Monitor system access
- [ ] Document monitoring procedures
- [ ] **Data Monitoring**
- [ ] Monitor data access patterns
- [ ] Track data modification events
- [ ] Monitor data integrity
- [ ] Document data monitoring
### 2. Security Logging
- [ ] **Audit Logging**
- [ ] Implement comprehensive audit logging
- [ ] Log all security-relevant events
- [ ] Secure log storage and access
- [ ] Document logging procedures
- [ ] **Log Analysis**
- [ ] Implement log analysis tools
- [ ] Monitor for security incidents
- [ ] Analyze security trends
- [ ] Document analysis procedures
## Incident Response
### 1. Security Incident Planning
- [ ] **Incident Response Plan**
- [ ] Develop security incident response plan
- [ ] Define incident response procedures
- [ ] Train incident response team
- [ ] Document response procedures
- [ ] **Incident Detection**
- [ ] Implement incident detection mechanisms
- [ ] Monitor for security incidents
- [ ] Establish incident reporting procedures
- [ ] Document detection procedures
### 2. Recovery Procedures
- [ ] **Data Recovery**
- [ ] Develop data recovery procedures
- [ ] Test recovery processes
- [ ] Verify recovery data integrity
- [ ] Document recovery procedures
- [ ] **System Recovery**
- [ ] Develop system recovery procedures
- [ ] Test system recovery
- [ ] Verify system security after recovery
- [ ] Document recovery procedures
## Compliance Verification
### 1. Regulatory Compliance
- [ ] **Privacy Compliance**
- [ ] Verify GDPR compliance
- [ ] Check CCPA compliance
- [ ] Assess other privacy regulations
- [ ] Document compliance status
- [ ] **Security Compliance**
- [ ] Verify security standard compliance
- [ ] Check industry requirements
- [ ] Assess security certifications
- [ ] Document compliance status
### 2. Audit Requirements
- [ ] **Audit Trail**
- [ ] Maintain comprehensive audit trail
- [ ] Verify audit log integrity
- [ ] Test audit log accessibility
- [ ] Document audit procedures
- [ ] **Audit Reporting**
- [ ] Generate audit reports
- [ ] Verify report accuracy
- [ ] Distribute reports securely
- [ ] Document reporting procedures
## Documentation and Training
### 1. Security Documentation
- [ ] **Security Procedures**
- [ ] Document security procedures
- [ ] Update security policies
- [ ] Create security guidelines
- [ ] Maintain documentation
- [ ] **Security Training**
- [ ] Develop security training materials
- [ ] Train staff on security procedures
- [ ] Verify training effectiveness
- [ ] Document training procedures
### 2. Ongoing Security
- [ ] **Security Maintenance**
- [ ] Establish security maintenance procedures
- [ ] Schedule security updates
- [ ] Monitor security trends
- [ ] Document maintenance procedures
- [ ] **Security Review**
- [ ] Conduct regular security reviews
- [ ] Update security controls
- [ ] Assess security effectiveness
- [ ] Document review procedures
## Risk Assessment
### 1. Risk Identification
- [ ] **Security Risks**
- [ ] Identify potential security risks
- [ ] Assess risk likelihood and impact
- [ ] Prioritize security risks
- [ ] Document risk assessment
- [ ] **Mitigation Strategies**
- [ ] Develop risk mitigation strategies
- [ ] Implement risk controls
- [ ] Monitor risk status
- [ ] Document mitigation procedures
### 2. Risk Monitoring
- [ ] **Risk Tracking**
- [ ] Track identified risks
- [ ] Monitor risk status
- [ ] Update risk assessments
- [ ] Document risk tracking
- [ ] **Risk Reporting**
- [ ] Generate risk reports
- [ ] Distribute risk information
- [ ] Update risk documentation
- [ ] Document reporting procedures
## Conclusion
This security audit checklist ensures that the database migration maintains the highest standards of data protection, privacy, and security. Regular review and updates of this checklist are essential to maintain security throughout the migration process and beyond.
### Security Checklist Summary
- [ ] **Pre-Migration Assessment**: Complete
- [ ] **Migration Controls**: Complete
- [ ] **Process Security**: Complete
- [ ] **Post-Migration Verification**: Complete
- [ ] **Monitoring and Logging**: Complete
- [ ] **Incident Response**: Complete
- [ ] **Compliance Verification**: Complete
- [ ] **Documentation and Training**: Complete
- [ ] **Risk Assessment**: Complete
**Overall Security Status**: [ ] Secure [ ] Needs Attention [ ] Critical Issues
**Next Review Date**: _______________
**Reviewed By**: _______________
**Approved By**: _______________

681
doc/migration-to-wa-sqlite.md

@ -4,6 +4,8 @@
This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users. This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.
**Current Status**: The migration is in **Phase 2** with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress.
## Migration Goals ## Migration Goals
1. **Data Integrity** 1. **Data Integrity**
@ -26,6 +28,17 @@ This document outlines the migration process from Dexie.js to absurd-sql for the
- Minimal downtime - Minimal downtime
- Automatic migration where possible - Automatic migration where possible
## Migration Fence
The migration is controlled by a **migration fence** that separates legacy Dexie code from the new SQLite implementation. See [Migration Fence Definition](./migration-fence-definition.md) for complete details.
### Key Fence Components
1. **Configuration Control**: `USE_DEXIE_DB = false` (default)
2. **Service Layer**: All database operations go through `PlatformService`
3. **Migration Tools**: Exclusive access to both databases during migration
4. **Code Boundaries**: Clear separation between legacy and new code
## Prerequisites ## Prerequisites
1. **Backup Requirements** 1. **Backup Requirements**
@ -62,6 +75,26 @@ This document outlines the migration process from Dexie.js to absurd-sql for the
- Android: Android 5+ with SQLite support - Android: Android 5+ with SQLite support
- Electron: Latest version with SQLite support - Electron: Latest version with SQLite support
## Current Migration Status
### ✅ Completed
- **SQLite Database Service**: Fully implemented with absurd-sql
- **Platform Service Layer**: Unified database interface
- **Migration Tools**: Data comparison and transfer utilities
- **Settings Migration**: Core user settings transferred
- **Account Migration**: Identity and key management
- **Schema Migration**: Complete table structure migration
### 🔄 In Progress
- **Contact Migration**: User contact data (via import interface)
- **Data Verification**: Comprehensive integrity checks
- **Performance Optimization**: Query optimization and indexing
### 📋 Planned
- **Code Cleanup**: Remove unused Dexie imports
- **Documentation Updates**: Complete migration guides
- **Testing**: Comprehensive migration testing
## Migration Process ## Migration Process
### 1. Preparation ### 1. Preparation
@ -72,70 +105,7 @@ import initSqlJs from '@jlongster/sql.js';
import { SQLiteFS } from 'absurd-sql'; import { SQLiteFS } from 'absurd-sql';
import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend'; import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend';
export class MigrationService { class MigrationService {
private static instance: MigrationService;
private backup: MigrationBackup | null = null;
private sql: any = null;
private db: any = null;
async prepare(): Promise<void> {
try {
// 1. Check prerequisites
await this.checkPrerequisites();
// 2. Create backup
this.backup = await this.createBackup();
// 3. Verify backup integrity
await this.verifyBackup();
// 4. Initialize absurd-sql
await this.initializeAbsurdSql();
} catch (error) {
throw new StorageError(
'Migration preparation failed',
StorageErrorCodes.MIGRATION_FAILED,
error
);
}
}
private async initializeAbsurdSql(): Promise<void> {
// Initialize SQL.js
this.sql = await initSqlJs({
locateFile: (file: string) => {
return new URL(`/node_modules/@jlongster/sql.js/dist/${file}`, import.meta.url).href;
}
});
// Setup SQLiteFS with IndexedDB backend
const sqlFS = new SQLiteFS(this.sql.FS, new IndexedDBBackend());
this.sql.register_for_idb(sqlFS);
// Create and mount filesystem
this.sql.FS.mkdir('/sql');
this.sql.FS.mount(sqlFS, {}, '/sql');
// Open database
const path = '/sql/db.sqlite';
if (typeof SharedArrayBuffer === 'undefined') {
let stream = this.sql.FS.open(path, 'a+');
await stream.node.contents.readIfFallback();
this.sql.FS.close(stream);
}
this.db = new this.sql.Database(path, { filename: true });
if (!this.db) {
throw new StorageError(
'Database initialization failed',
StorageErrorCodes.INITIALIZATION_FAILED
);
}
// Configure database
await this.db.exec(`PRAGMA journal_mode=MEMORY;`);
}
private async checkPrerequisites(): Promise<void> { private async checkPrerequisites(): Promise<void> {
// Check IndexedDB availability // Check IndexedDB availability
if (!window.indexedDB) { if (!window.indexedDB) {
@ -186,428 +156,259 @@ export class MigrationService {
```typescript ```typescript
// src/services/storage/migration/DataMigration.ts // src/services/storage/migration/DataMigration.ts
export class DataMigration { class DataMigration {
async migrate(backup: MigrationBackup): Promise<void> { async migrateAccounts(): Promise<MigrationResult> {
try { const result: MigrationResult = {
// 1. Create new database schema success: true,
await this.createSchema(); accountsMigrated: 0,
errors: [],
warnings: []
};
// 2. Migrate accounts try {
await this.migrateAccounts(backup.accounts); const dexieAccounts = await this.getDexieAccounts();
for (const account of dexieAccounts) {
try {
await this.migrateAccount(account);
result.accountsMigrated++;
} catch (error) {
result.errors.push(`Failed to migrate account ${account.did}: ${error}`);
result.success = false;
}
}
} catch (error) {
result.errors.push(`Account migration failed: ${error}`);
result.success = false;
}
// 3. Migrate settings return result;
await this.migrateSettings(backup.settings); }
// 4. Migrate contacts async migrateSettings(): Promise<MigrationResult> {
await this.migrateContacts(backup.contacts); const result: MigrationResult = {
success: true,
settingsMigrated: 0,
errors: [],
warnings: []
};
// 5. Verify migration try {
await this.verifyMigration(backup); const dexieSettings = await this.getDexieSettings();
for (const setting of dexieSettings) {
try {
await this.migrateSetting(setting);
result.settingsMigrated++;
} catch (error) {
result.errors.push(`Failed to migrate setting ${setting.id}: ${error}`);
result.success = false;
}
}
} catch (error) { } catch (error) {
// 6. Handle failure result.errors.push(`Settings migration failed: ${error}`);
await this.handleMigrationFailure(error, backup); result.success = false;
} }
return result;
} }
private async migrateAccounts(accounts: Account[]): Promise<void> { async migrateContacts(): Promise<MigrationResult> {
// Use transaction for atomicity // Contact migration is handled through the contact import interface
await this.db.exec('BEGIN TRANSACTION;'); // This provides better user control and validation
const result: MigrationResult = {
success: true,
contactsMigrated: 0,
errors: [],
warnings: []
};
try { try {
for (const account of accounts) { const dexieContacts = await this.getDexieContacts();
await this.db.run(`
INSERT INTO accounts (did, public_key_hex, created_at, updated_at) // Redirect to contact import view with pre-populated data
VALUES (?, ?, ?, ?) await this.redirectToContactImport(dexieContacts);
`, [
account.did, result.contactsMigrated = dexieContacts.length;
account.publicKeyHex,
account.createdAt,
account.updatedAt
]);
}
await this.db.exec('COMMIT;');
} catch (error) { } catch (error) {
await this.db.exec('ROLLBACK;'); result.errors.push(`Contact migration failed: ${error}`);
throw error; result.success = false;
} }
return result;
} }
}
```
private async verifyMigration(backup: MigrationBackup): Promise<void> { ### 3. Verification
```typescript
class MigrationVerification {
async verifyMigration(dexieData: MigrationData): Promise<boolean> {
// Verify account count // Verify account count
const result = await this.db.exec('SELECT COUNT(*) as count FROM accounts'); const accountResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM accounts');
const accountCount = result[0].values[0][0]; const accountCount = accountResult[0].values[0][0];
if (accountCount !== dexieData.accounts.length) {
return false;
}
if (accountCount !== backup.accounts.length) { // Verify settings count
throw new StorageError( const settingsResult = await this.sqliteDB.exec('SELECT COUNT(*) as count FROM settings');
'Account count mismatch', const settingsCount = settingsResult[0].values[0][0];
StorageErrorCodes.VERIFICATION_FAILED if (settingsCount !== dexieData.settings.length) {
); return false;
} }
// Verify data integrity // Verify data integrity
await this.verifyDataIntegrity(backup); for (const account of dexieData.accounts) {
const result = await this.sqliteDB.exec(
'SELECT * FROM accounts WHERE did = ?',
[account.did]
);
const migratedAccount = result[0]?.values[0];
if (!migratedAccount ||
migratedAccount[1] !== account.publicKeyHex) {
return false;
}
}
return true;
} }
} }
``` ```
### 3. Rollback Strategy ## Using the Migration Interface
```typescript ### Accessing Migration Tools
// src/services/storage/migration/RollbackService.ts
export class RollbackService {
async rollback(backup: MigrationBackup): Promise<void> {
try {
// 1. Stop all database operations
await this.stopDatabaseOperations();
// 2. Restore from backup 1. Navigate to the **Account** page in the TimeSafari app
await this.restoreFromBackup(backup); 2. Scroll down to find the **Database Migration** link
3. Click the link to open the migration interface
// 3. Verify restoration ### Migration Steps
await this.verifyRestoration(backup);
// 4. Clean up absurd-sql 1. **Compare Databases**
await this.cleanupAbsurdSql(); - Click "Compare Databases" to see differences
} catch (error) { - Review the comparison results
throw new StorageError( - Identify data that needs migration
'Rollback failed',
StorageErrorCodes.ROLLBACK_FAILED,
error
);
}
}
private async restoreFromBackup(backup: MigrationBackup): Promise<void> { 2. **Migrate Settings**
const dexieDB = new Dexie('TimeSafariDB'); - Click "Migrate Settings" to transfer user settings
- Verify settings are correctly transferred
- Check application functionality
// Restore accounts 3. **Migrate Contacts**
await dexieDB.accounts.bulkPut(backup.accounts); - Click "Migrate Contacts" to open contact import
- Review and confirm contact data
- Complete the import process
// Restore settings 4. **Verify Migration**
await dexieDB.settings.bulkPut(backup.settings); - Run comparison again to verify completion
- Test application functionality
- Export backup data if needed
// Restore contacts ## Error Handling
await dexieDB.contacts.bulkPut(backup.contacts);
}
}
```
## Migration UI ### Common Issues
```vue
<!-- src/components/MigrationProgress.vue -->
<template>
<div class="migration-progress">
<h2>Database Migration</h2>
<div class="progress-container">
<div class="progress-bar" :style="{ width: `${progress}%` }" />
<div class="progress-text">{{ progress }}%</div>
</div>
<div class="status-message">{{ statusMessage }}</div>
<div v-if="error" class="error-message">
{{ error }}
<button @click="retryMigration">Retry</button>
</div>
</div>
</template>
<script setup lang="ts">
import { ref, onMounted } from 'vue';
import { MigrationService } from '@/services/storage/migration/MigrationService';
const progress = ref(0);
const statusMessage = ref('Preparing migration...');
const error = ref<string | null>(null);
const migrationService = MigrationService.getInstance();
async function startMigration() {
try {
// 1. Preparation
statusMessage.value = 'Creating backup...';
await migrationService.prepare();
progress.value = 20;
// 2. Data migration
statusMessage.value = 'Migrating data...';
await migrationService.migrate();
progress.value = 80;
// 3. Verification
statusMessage.value = 'Verifying migration...';
await migrationService.verify();
progress.value = 100;
statusMessage.value = 'Migration completed successfully!';
} catch (err) {
error.value = err instanceof Error ? err.message : 'Migration failed';
statusMessage.value = 'Migration failed';
}
}
async function retryMigration() { 1. **Dexie Database Not Enabled**
error.value = null; - **Error**: "Dexie database is not enabled"
progress.value = 0; - **Solution**: Set `USE_DEXIE_DB = true` in `constants/app.ts` temporarily
await startMigration();
}
onMounted(() => { 2. **Database Connection Issues**
startMigration(); - **Error**: "Failed to retrieve data"
}); - **Solution**: Check database initialization and permissions
</script>
<style scoped> 3. **Migration Failures**
.migration-progress { - **Error**: "Migration failed: [specific error]"
padding: 2rem; - **Solution**: Review error details and check data integrity
max-width: 600px;
margin: 0 auto;
}
.progress-container { ### Error Recovery
position: relative;
height: 20px;
background: #eee;
border-radius: 10px;
overflow: hidden;
margin: 1rem 0;
}
.progress-bar { 1. **Review** error messages carefully
position: absolute; 2. **Check** browser console for additional details
height: 100%; 3. **Verify** database connectivity and permissions
background: #4CAF50; 4. **Retry** the operation if appropriate
transition: width 0.3s ease; 5. **Export** comparison data for manual review if needed
}
.progress-text { ## Best Practices
position: absolute;
width: 100%;
text-align: center;
line-height: 20px;
color: #000;
}
.status-message { ### Before Migration
text-align: center;
margin: 1rem 0;
}
.error-message { 1. **Backup** your data if possible
color: #f44336; 2. **Test** the migration on a small dataset first
text-align: center; 3. **Verify** that both databases are accessible
margin: 1rem 0; 4. **Review** the comparison results before migrating
}
button { ### During Migration
margin-top: 1rem;
padding: 0.5rem 1rem;
background: #2196F3;
color: white;
border: none;
border-radius: 4px;
cursor: pointer;
}
button:hover { 1. **Don't** interrupt the migration process
background: #1976D2; 2. **Monitor** the progress and error messages
} 3. **Note** any warnings or skipped records
</style> 4. **Export** comparison data for reference
```
## Testing Strategy ### After Migration
1. **Unit Tests** 1. **Verify** that data was migrated correctly
```typescript 2. **Test** the application functionality
// src/services/storage/migration/__tests__/MigrationService.spec.ts 3. **Disable** Dexie database (`USE_DEXIE_DB = false`)
describe('MigrationService', () => { 4. **Clean up** any temporary files or exports
it('should initialize absurd-sql correctly', async () => {
const service = MigrationService.getInstance();
await service.initializeAbsurdSql();
expect(service.isInitialized()).toBe(true);
expect(service.getDatabase()).toBeDefined();
});
it('should create valid backup', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
expect(backup).toBeDefined();
expect(backup.accounts).toBeInstanceOf(Array);
expect(backup.settings).toBeInstanceOf(Array);
expect(backup.contacts).toBeInstanceOf(Array);
});
it('should migrate data correctly', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
await service.migrate(backup);
// Verify migration
const accounts = await service.getMigratedAccounts();
expect(accounts).toHaveLength(backup.accounts.length);
});
it('should handle rollback correctly', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
// Simulate failed migration
await service.migrate(backup);
await service.simulateFailure();
// Perform rollback
await service.rollback(backup);
// Verify rollback
const accounts = await service.getOriginalAccounts();
expect(accounts).toHaveLength(backup.accounts.length);
});
});
```
2. **Integration Tests** ## Performance Considerations
```typescript
// src/services/storage/migration/__tests__/integration/Migration.spec.ts
describe('Migration Integration', () => {
it('should handle concurrent access during migration', async () => {
const service = MigrationService.getInstance();
// Start migration
const migrationPromise = service.migrate();
// Simulate concurrent access
const accessPromises = Array(5).fill(null).map(() =>
service.getAccount('did:test:123')
);
// Wait for all operations
const [migrationResult, ...accessResults] = await Promise.allSettled([
migrationPromise,
...accessPromises
]);
// Verify results
expect(migrationResult.status).toBe('fulfilled');
expect(accessResults.some(r => r.status === 'rejected')).toBe(true);
});
it('should maintain data integrity during platform transition', async () => {
const service = MigrationService.getInstance();
// Simulate platform change
await service.simulatePlatformChange();
// Verify data
const accounts = await service.getAllAccounts();
const settings = await service.getAllSettings();
const contacts = await service.getAllContacts();
expect(accounts).toBeDefined();
expect(settings).toBeDefined();
expect(contacts).toBeDefined();
});
});
```
## Success Criteria ### 1. Migration Performance
- Use transactions for bulk data transfer
- Implement progress indicators
- Process data in background when possible
1. **Data Integrity** ### 2. Application Performance
- [ ] All accounts migrated successfully - Optimize SQLite queries
- [ ] All settings preserved - Maintain proper database indexes
- [ ] All contacts transferred - Use efficient memory management
- [ ] No data corruption
2. **Performance** ## Security Considerations
- [ ] Migration completes within acceptable time
- [ ] No significant performance degradation
- [ ] Efficient storage usage
- [ ] Smooth user experience
3. **Security** ### 1. Data Protection
- [ ] Encrypted data remains secure - Maintain encryption standards across migration
- [ ] Access controls maintained - Preserve user privacy during migration
- [ ] No sensitive data exposure - Log all migration operations
- [ ] Secure backup process
4. **User Experience** ### 2. Error Handling
- [ ] Clear migration progress - Handle migration failures gracefully
- [ ] Informative error messages - Provide clear user messaging
- [ ] Automatic recovery from failures - Maintain rollback capabilities
- [ ] No data loss
## Testing Strategy
## Rollback Plan
### 1. Migration Testing
1. **Automatic Rollback** ```typescript
- Triggered by migration failure describe('Database Migration', () => {
- Restores from verified backup it('should migrate data without loss', async () => {
- Maintains data consistency // 1. Enable Dexie
- Logs rollback reason // 2. Create test data
// 3. Run migration
2. **Manual Rollback** // 4. Verify data integrity
- Available through settings // 5. Disable Dexie
- Requires user confirmation });
- Preserves backup data });
- Provides rollback status ```
3. **Emergency Recovery** ### 2. Application Testing
- Manual backup restoration ```typescript
- Database repair tools describe('Feature with Database', () => {
- Data recovery procedures it('should work with SQLite only', async () => {
- Support contact information // Test with USE_DEXIE_DB = false
// Verify all operations use PlatformService
## Post-Migration });
});
1. **Verification** ```
- Data integrity checks
- Performance monitoring ## Conclusion
- Error rate tracking
- User feedback collection The migration from Dexie to absurd-sql provides:
- **Better Performance**: Improved query performance and storage efficiency
2. **Cleanup** - **Cross-Platform Consistency**: Unified database interface across platforms
- Remove old database - **Enhanced Security**: Better encryption and access controls
- Clear migration artifacts - **Future-Proof Architecture**: Modern SQLite-based storage system
- Update application state
- Archive backup data The migration fence ensures a controlled and safe transition while maintaining data integrity and application stability.
3. **Monitoring**
- Track migration success rate
- Monitor performance metrics
- Collect error reports
- Gather user feedback
## Support
For assistance with migration:
1. Check the troubleshooting guide
2. Review error logs
3. Contact support team
4. Submit issue report
## Timeline
1. **Preparation Phase** (1 week)
- Backup system implementation
- Migration service development
- Testing framework setup
2. **Testing Phase** (2 weeks)
- Unit testing
- Integration testing
- Performance testing
- Security testing
3. **Deployment Phase** (1 week)
- Staged rollout
- Monitoring
- Support preparation
- Documentation updates
4. **Post-Deployment** (2 weeks)
- Monitoring
- Bug fixes
- Performance optimization
- User feedback collection
Loading…
Cancel
Save