Browse Source

Merge branch 'migrate-dexie-to-sqlite'

master
Trent Larson 6 days ago
parent
commit
a997d4cb92
  1. 12
      BUILDING.md
  2. 5
      CHANGELOG.md
  3. 47
      README.md
  4. 295
      doc/database-migration-guide.md
  5. 41
      doc/dexie-to-sqlite-mapping.md
  6. 272
      doc/migration-fence-definition.md
  7. 355
      doc/migration-security-checklist.md
  8. 749
      doc/migration-to-wa-sqlite.md
  9. 585
      package-lock.json
  10. 2
      package.json
  11. 75
      src/assets/icons.json
  12. 90
      src/components/IconRenderer.vue
  13. 17
      src/db-sql/migration.ts
  14. 56
      src/db/databaseUtil.ts
  15. 2
      src/db/tables/README.md
  16. 4
      src/db/tables/contacts.ts
  17. 5
      src/db/tables/settings.ts
  18. 36
      src/libs/util.ts
  19. 5
      src/router/index.ts
  20. 1373
      src/services/indexedDBMigrationService.ts
  21. 156
      src/services/migrationService.ts
  22. 2
      src/views/ContactImportView.vue
  23. 1488
      src/views/DatabaseMigration.vue
  24. 4
      src/views/HomeView.vue
  25. 44
      src/views/ImportAccountView.vue
  26. 12
      src/views/StartView.vue
  27. 9
      src/views/TestView.vue

12
BUILDING.md

@ -67,9 +67,9 @@ Install dependencies:
* Run a build to make sure package-lock version is updated, linting works, etc: `npm install && npm run build` * Run a build to make sure package-lock version is updated, linting works, etc: `npm install && npm run build`
* Put the commit hash in the changelog (which will help you remember to bump the version later). * Put the commit hash in the changelog (which will help you remember to bump the version in the step later).
* Tag with the new version, [online](https://gitea.anomalistdesign.com/trent_larson/crowd-funder-for-time-pwa/releases) or `git tag 0.3.55 && git push origin 0.3.55`. * Tag with the new version, [online](https://gitea.anomalistdesign.com/trent_larson/crowd-funder-for-time-pwa/releases) or `git tag 1.0.0 && git push origin 1.0.0`.
* For test, build the app (because test server is not yet set up to build): * For test, build the app (because test server is not yet set up to build):
@ -93,13 +93,13 @@ TIME_SAFARI_APP_TITLE="TimeSafari_Test" VITE_APP_SERVER=https://test.timesafari.
* `pkgx +npm sh` * `pkgx +npm sh`
* `cd crowd-funder-for-time-pwa && git checkout master && git pull && git checkout 0.3.55 && npm install && npm run build && cd -` * `cd crowd-funder-for-time-pwa && git checkout master && git pull && git checkout 1.0.0 && npm install && npm run build:web && cd -`
(The plain `npm run build` uses the .env.production file.) (The plain `npm run build:web` uses the .env.production file.)
* Back up the time-safari/dist folder & deploy: `mv time-safari/dist time-safari-dist-prev.0 && mv crowd-funder-for-time-pwa/dist time-safari/` * Back up the time-safari/dist folder & deploy: `mv time-safari/dist time-safari-dist-prev-1 && mv crowd-funder-for-time-pwa/dist time-safari/`
* Record the new hash in the changelog. Edit package.json to increment version & add "-beta", `npm install`, and commit. Also record what version is on production. * Record the new hash in the changelog. Edit package.json to increment version & add "-beta", `npm install`, commit, and push. Also record what version is on production.
## Docker Deployment ## Docker Deployment

5
CHANGELOG.md

@ -14,6 +14,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Feed visuals now have arrow imagery from giver to receiver - Feed visuals now have arrow imagery from giver to receiver
## [1.0.0] - 2025.06.20 - 5aa693de6337e5dbb278bfddc6bd39094bc14f73
### Added
- Web-oriented migration from IndexedDB to SQLite
## [0.4.7] ## [0.4.7]
### Fixed ### Fixed
- Cameras everywhere - Cameras everywhere

47
README.md

@ -3,6 +3,32 @@
[Time Safari](https://timesafari.org/) allows people to ease into collaboration: start with expressions of gratitude [Time Safari](https://timesafari.org/) allows people to ease into collaboration: start with expressions of gratitude
and expand to crowd-fund with time & money, then record and see the impact of contributions. and expand to crowd-fund with time & money, then record and see the impact of contributions.
## Database Migration Status
**Current Status**: The application is undergoing a migration from Dexie (IndexedDB) to SQLite using absurd-sql. This migration is in **Phase 2** with a well-defined migration fence in place.
### Migration Progress
- ✅ **SQLite Database Service**: Fully implemented with absurd-sql
- ✅ **Platform Service Layer**: Unified database interface across platforms
- ✅ **Settings Migration**: Core user settings transferred
- ✅ **Account Migration**: Identity and key management
- 🔄 **Contact Migration**: User contact data (via import interface)
- 📋 **Code Cleanup**: Remove unused Dexie imports
### Migration Fence
The migration is controlled by a **migration fence** that separates legacy Dexie code from the new SQLite implementation. See [Migration Fence Definition](doc/migration-fence-definition.md) for complete details.
**Key Points**:
- Legacy Dexie database is disabled by default (`USE_DEXIE_DB = false`)
- All database operations go through `PlatformService`
- Migration tools provide controlled access to both databases
- Clear separation between legacy and new code
### Migration Documentation
- [Migration Guide](doc/migration-to-wa-sqlite.md) - Complete migration process
- [Migration Fence Definition](doc/migration-fence-definition.md) - Fence boundaries and rules
- [Database Migration Guide](doc/database-migration-guide.md) - User-facing migration tools
## Roadmap ## Roadmap
See [project.task.yaml](project.task.yaml) for current priorities. See [project.task.yaml](project.task.yaml) for current priorities.
@ -21,16 +47,10 @@ npm run dev
See [BUILDING.md](BUILDING.md) for more details. See [BUILDING.md](BUILDING.md) for more details.
## Tests ## Tests
See [TESTING.md](test-playwright/TESTING.md) for detailed test instructions. See [TESTING.md](test-playwright/TESTING.md) for detailed test instructions.
## Icons ## Icons
Application icons are in the `assets` directory, processed by the `capacitor-assets` command. Application icons are in the `assets` directory, processed by the `capacitor-assets` command.
@ -66,6 +86,21 @@ Key principles:
- Common interfaces are shared through `common.ts` - Common interfaces are shared through `common.ts`
- Type definitions are generated from Zod schemas where possible - Type definitions are generated from Zod schemas where possible
### Database Architecture
The application uses a platform-agnostic database layer:
* `src/services/PlatformService.ts` - Database interface definition
* `src/services/PlatformServiceFactory.ts` - Platform-specific service factory
* `src/services/AbsurdSqlDatabaseService.ts` - SQLite implementation
* `src/db/` - Legacy Dexie database (migration in progress)
**Development Guidelines**:
- Always use `PlatformService` for database operations
- Never import Dexie directly in application code
- Test with `USE_DEXIE_DB = false` for new features
- Use migration tools for data transfer between systems
### Kudos ### Kudos
Gifts make the world go 'round! Gifts make the world go 'round!

295
doc/database-migration-guide.md

@ -0,0 +1,295 @@
# Database Migration Guide
## Overview
The Database Migration feature allows you to compare and migrate data between Dexie (IndexedDB) and SQLite databases in the TimeSafari application. This is particularly useful during the transition from the old Dexie-based storage system to the new SQLite-based system.
## Features
### 1. Database Comparison
- Compare data between Dexie and SQLite databases
- View detailed differences in contacts and settings
- Identify added, modified, and missing records
- Export comparison results for analysis
### 2. Data Migration
- Migrate contacts from Dexie to SQLite
- Migrate settings from Dexie to SQLite
- Option to overwrite existing records or skip them
- Comprehensive error handling and reporting
### 3. User Interface
- Modern, responsive UI built with Tailwind CSS
- Real-time loading states and progress indicators
- Clear success and error messaging
- Export functionality for comparison data
## Prerequisites
### Enable Dexie Database
Before using the migration features, you must enable the Dexie database by setting:
```typescript
// In constants/app.ts
export const USE_DEXIE_DB = true;
```
**Note**: This should only be enabled temporarily during migration. Remember to set it back to `false` after migration is complete.
## Accessing the Migration Interface
1. Navigate to the **Account** page in the TimeSafari app
2. Scroll down to find the **Database Migration** link
3. Click the link to open the migration interface
## Using the Migration Interface
### Step 1: Compare Databases
1. Click the **"Compare Databases"** button
2. The system will retrieve data from both Dexie and SQLite databases
3. Review the comparison results showing:
- Summary counts for each database
- Detailed differences (added, modified, missing records)
- Specific records that need attention
### Step 2: Review Differences
The comparison results are displayed in several sections:
#### Summary Cards
- **Dexie Contacts**: Number of contacts in Dexie database
- **SQLite Contacts**: Number of contacts in SQLite database
- **Dexie Settings**: Number of settings in Dexie database
- **SQLite Settings**: Number of settings in SQLite database
#### Contact Differences
- **Added**: Contacts in Dexie but not in SQLite
- **Modified**: Contacts that differ between databases
- **Missing**: Contacts in SQLite but not in Dexie
#### Settings Differences
- **Added**: Settings in Dexie but not in SQLite
- **Modified**: Settings that differ between databases
- **Missing**: Settings in SQLite but not in Dexie
### Step 3: Configure Migration Options
Before migrating data, configure the migration options:
- **Overwrite existing records**: When enabled, existing records in SQLite will be updated with data from Dexie. When disabled, existing records will be skipped.
### Step 4: Migrate Data
#### Migrate Contacts
1. Click the **"Migrate Contacts"** button
2. The system will transfer contacts from Dexie to SQLite
3. Review the migration results showing:
- Number of contacts successfully migrated
- Any warnings or errors encountered
#### Migrate Settings
1. Click the **"Migrate Settings"** button
2. The system will transfer settings from Dexie to SQLite
3. Review the migration results showing:
- Number of settings successfully migrated
- Any warnings or errors encountered
### Step 5: Export Comparison (Optional)
1. Click the **"Export Comparison"** button
2. A JSON file will be downloaded containing the complete comparison data
3. This file can be used for analysis or backup purposes
## Migration Process Details
### Contact Migration
The contact migration process:
1. **Retrieves** all contacts from Dexie database
2. **Checks** for existing contacts in SQLite by DID
3. **Inserts** new contacts or **updates** existing ones (if overwrite is enabled)
4. **Handles** complex fields like `contactMethods` (JSON arrays)
5. **Reports** success/failure for each contact
### Settings Migration
The settings migration process:
1. **Retrieves** all settings from Dexie database
2. **Focuses** on key user-facing settings:
- `firstName`
- `isRegistered`
- `profileImageUrl`
- `showShortcutBvc`
- `searchBoxes`
3. **Preserves** other settings in SQLite
4. **Reports** success/failure for each setting
## Error Handling
### Common Issues
#### Dexie Database Not Enabled
**Error**: "Dexie database is not enabled"
**Solution**: Set `USE_DEXIE_DB = true` in `constants/app.ts`
#### Database Connection Issues
**Error**: "Failed to retrieve Dexie contacts"
**Solution**: Check that the Dexie database is properly initialized and accessible
#### SQLite Query Errors
**Error**: "Failed to retrieve SQLite contacts"
**Solution**: Verify that the SQLite database is properly set up and the platform service is working
#### Migration Failures
**Error**: "Migration failed: [specific error]"
**Solution**: Review the error details and check data integrity in both databases
### Error Recovery
1. **Review** the error messages carefully
2. **Check** the browser console for additional details
3. **Verify** database connectivity and permissions
4. **Retry** the operation if appropriate
5. **Export** comparison data for manual review if needed
## Best Practices
### Before Migration
1. **Backup** your data if possible
2. **Test** the migration on a small dataset first
3. **Verify** that both databases are accessible
4. **Review** the comparison results before migrating
### During Migration
1. **Don't** interrupt the migration process
2. **Monitor** the progress and error messages
3. **Note** any warnings or skipped records
4. **Export** comparison data for reference
### After Migration
1. **Verify** that data was migrated correctly
2. **Test** the application functionality
3. **Disable** Dexie database (`USE_DEXIE_DB = false`)
4. **Clean up** any temporary files or exports
## Technical Details
### Database Schema
The migration handles the following data structures:
#### Contacts Table
```typescript
interface Contact {
did: string; // Decentralized Identifier
name: string; // Contact name
contactMethods: ContactMethod[]; // Array of contact methods
nextPubKeyHashB64: string; // Next public key hash
notes: string; // Contact notes
profileImageUrl: string; // Profile image URL
publicKeyBase64: string; // Public key in base64
seesMe: boolean; // Visibility flag
registered: boolean; // Registration status
}
```
#### Settings Table
```typescript
interface Settings {
id: number; // Settings ID
accountDid: string; // Account DID
activeDid: string; // Active DID
firstName: string; // User's first name
isRegistered: boolean; // Registration status
profileImageUrl: string; // Profile image URL
showShortcutBvc: boolean; // UI preference
searchBoxes: any[]; // Search configuration
// ... other fields
}
```
### Migration Logic
The migration service uses sophisticated comparison logic:
1. **Primary Key Matching**: Uses DID for contacts, ID for settings
2. **Deep Comparison**: Compares all fields including complex objects
3. **JSON Handling**: Properly handles JSON fields like `contactMethods` and `searchBoxes`
4. **Conflict Resolution**: Provides options for handling existing records
### Performance Considerations
- **Batch Processing**: Processes records one by one for reliability
- **Error Isolation**: Individual record failures don't stop the entire migration
- **Memory Management**: Handles large datasets efficiently
- **Progress Reporting**: Provides real-time feedback during migration
## Troubleshooting
### Migration Stuck
If the migration appears to be stuck:
1. **Check** the browser console for errors
2. **Refresh** the page and try again
3. **Verify** database connectivity
4. **Check** for large datasets that might take time
### Incomplete Migration
If migration doesn't complete:
1. **Review** error messages
2. **Check** data integrity in both databases
3. **Export** comparison data for manual review
4. **Consider** migrating in smaller batches
### Data Inconsistencies
If you notice data inconsistencies:
1. **Export** comparison data
2. **Review** the differences carefully
3. **Manually** verify critical records
4. **Consider** selective migration of specific records
## Support
For issues with the Database Migration feature:
1. **Check** this documentation first
2. **Review** the browser console for error details
3. **Export** comparison data for analysis
4. **Contact** the development team with specific error details
## Security Considerations
- **Data Privacy**: Migration data is processed locally and not sent to external servers
- **Access Control**: Only users with access to the account can perform migration
- **Data Integrity**: Migration preserves data integrity and handles conflicts gracefully
- **Audit Trail**: Export functionality provides an audit trail of migration operations
---
**Note**: This migration tool is designed for the transition period between database systems. Once migration is complete and verified, the Dexie database should be disabled to avoid confusion and potential data conflicts.

41
doc/dexie-to-sqlite-mapping.md

@ -3,6 +3,7 @@
## Schema Mapping ## Schema Mapping
### Current Dexie Schema ### Current Dexie Schema
```typescript ```typescript
// Current Dexie schema // Current Dexie schema
const db = new Dexie('TimeSafariDB'); const db = new Dexie('TimeSafariDB');
@ -15,6 +16,7 @@ db.version(1).stores({
``` ```
### New SQLite Schema ### New SQLite Schema
```sql ```sql
-- New SQLite schema -- New SQLite schema
CREATE TABLE accounts ( CREATE TABLE accounts (
@ -50,6 +52,7 @@ CREATE INDEX idx_settings_updated_at ON settings(updated_at);
### 1. Account Operations ### 1. Account Operations
#### Get Account by DID #### Get Account by DID
```typescript ```typescript
// Dexie // Dexie
const account = await db.accounts.get(did); const account = await db.accounts.get(did);
@ -62,6 +65,7 @@ const account = result[0]?.values[0];
``` ```
#### Get All Accounts #### Get All Accounts
```typescript ```typescript
// Dexie // Dexie
const accounts = await db.accounts.toArray(); const accounts = await db.accounts.toArray();
@ -74,6 +78,7 @@ const accounts = result[0]?.values || [];
``` ```
#### Add Account #### Add Account
```typescript ```typescript
// Dexie // Dexie
await db.accounts.add({ await db.accounts.add({
@ -91,6 +96,7 @@ await db.run(`
``` ```
#### Update Account #### Update Account
```typescript ```typescript
// Dexie // Dexie
await db.accounts.update(did, { await db.accounts.update(did, {
@ -100,7 +106,7 @@ await db.accounts.update(did, {
// absurd-sql // absurd-sql
await db.run(` await db.run(`
UPDATE accounts UPDATE accounts
SET public_key_hex = ?, updated_at = ? SET public_key_hex = ?, updated_at = ?
WHERE did = ? WHERE did = ?
`, [publicKeyHex, Date.now(), did]); `, [publicKeyHex, Date.now(), did]);
@ -109,6 +115,7 @@ await db.run(`
### 2. Settings Operations ### 2. Settings Operations
#### Get Setting #### Get Setting
```typescript ```typescript
// Dexie // Dexie
const setting = await db.settings.get(key); const setting = await db.settings.get(key);
@ -121,6 +128,7 @@ const setting = result[0]?.values[0];
``` ```
#### Set Setting #### Set Setting
```typescript ```typescript
// Dexie // Dexie
await db.settings.put({ await db.settings.put({
@ -142,6 +150,7 @@ await db.run(`
### 3. Contact Operations ### 3. Contact Operations
#### Get Contacts by Account #### Get Contacts by Account
```typescript ```typescript
// Dexie // Dexie
const contacts = await db.contacts const contacts = await db.contacts
@ -151,7 +160,7 @@ const contacts = await db.contacts
// absurd-sql // absurd-sql
const result = await db.exec(` const result = await db.exec(`
SELECT * FROM contacts SELECT * FROM contacts
WHERE did = ? WHERE did = ?
ORDER BY created_at DESC ORDER BY created_at DESC
`, [accountDid]); `, [accountDid]);
@ -159,6 +168,7 @@ const contacts = result[0]?.values || [];
``` ```
#### Add Contact #### Add Contact
```typescript ```typescript
// Dexie // Dexie
await db.contacts.add({ await db.contacts.add({
@ -179,6 +189,7 @@ await db.run(`
## Transaction Mapping ## Transaction Mapping
### Batch Operations ### Batch Operations
```typescript ```typescript
// Dexie // Dexie
await db.transaction('rw', [db.accounts, db.contacts], async () => { await db.transaction('rw', [db.accounts, db.contacts], async () => {
@ -210,10 +221,11 @@ try {
## Migration Helper Functions ## Migration Helper Functions
### 1. Data Export (Dexie to JSON) ### 1. Data Export (Dexie to JSON)
```typescript ```typescript
async function exportDexieData(): Promise<MigrationData> { async function exportDexieData(): Promise<MigrationData> {
const db = new Dexie('TimeSafariDB'); const db = new Dexie('TimeSafariDB');
return { return {
accounts: await db.accounts.toArray(), accounts: await db.accounts.toArray(),
settings: await db.settings.toArray(), settings: await db.settings.toArray(),
@ -228,6 +240,7 @@ async function exportDexieData(): Promise<MigrationData> {
``` ```
### 2. Data Import (JSON to absurd-sql) ### 2. Data Import (JSON to absurd-sql)
```typescript ```typescript
async function importToAbsurdSql(data: MigrationData): Promise<void> { async function importToAbsurdSql(data: MigrationData): Promise<void> {
await db.exec('BEGIN TRANSACTION;'); await db.exec('BEGIN TRANSACTION;');
@ -239,7 +252,7 @@ async function importToAbsurdSql(data: MigrationData): Promise<void> {
VALUES (?, ?, ?, ?) VALUES (?, ?, ?, ?)
`, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]); `, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]);
} }
// Import settings // Import settings
for (const setting of data.settings) { for (const setting of data.settings) {
await db.run(` await db.run(`
@ -247,7 +260,7 @@ async function importToAbsurdSql(data: MigrationData): Promise<void> {
VALUES (?, ?, ?) VALUES (?, ?, ?)
`, [setting.key, setting.value, setting.updatedAt]); `, [setting.key, setting.value, setting.updatedAt]);
} }
// Import contacts // Import contacts
for (const contact of data.contacts) { for (const contact of data.contacts) {
await db.run(` await db.run(`
@ -264,6 +277,7 @@ async function importToAbsurdSql(data: MigrationData): Promise<void> {
``` ```
### 3. Verification ### 3. Verification
```typescript ```typescript
async function verifyMigration(dexieData: MigrationData): Promise<boolean> { async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
// Verify account count // Verify account count
@ -272,21 +286,21 @@ async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
if (accountCount !== dexieData.accounts.length) { if (accountCount !== dexieData.accounts.length) {
return false; return false;
} }
// Verify settings count // Verify settings count
const settingsResult = await db.exec('SELECT COUNT(*) as count FROM settings'); const settingsResult = await db.exec('SELECT COUNT(*) as count FROM settings');
const settingsCount = settingsResult[0].values[0][0]; const settingsCount = settingsResult[0].values[0][0];
if (settingsCount !== dexieData.settings.length) { if (settingsCount !== dexieData.settings.length) {
return false; return false;
} }
// Verify contacts count // Verify contacts count
const contactsResult = await db.exec('SELECT COUNT(*) as count FROM contacts'); const contactsResult = await db.exec('SELECT COUNT(*) as count FROM contacts');
const contactsCount = contactsResult[0].values[0][0]; const contactsCount = contactsResult[0].values[0][0];
if (contactsCount !== dexieData.contacts.length) { if (contactsCount !== dexieData.contacts.length) {
return false; return false;
} }
// Verify data integrity // Verify data integrity
for (const account of dexieData.accounts) { for (const account of dexieData.accounts) {
const result = await db.exec( const result = await db.exec(
@ -294,12 +308,12 @@ async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
[account.did] [account.did]
); );
const migratedAccount = result[0]?.values[0]; const migratedAccount = result[0]?.values[0];
if (!migratedAccount || if (!migratedAccount ||
migratedAccount[1] !== account.publicKeyHex) { // public_key_hex is second column migratedAccount[1] !== account.publicKeyHex) { // public_key_hex is second column
return false; return false;
} }
} }
return true; return true;
} }
``` ```
@ -307,18 +321,21 @@ async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
## Performance Considerations ## Performance Considerations
### 1. Indexing ### 1. Indexing
- Dexie automatically creates indexes based on the schema - Dexie automatically creates indexes based on the schema
- absurd-sql requires explicit index creation - absurd-sql requires explicit index creation
- Added indexes for frequently queried fields - Added indexes for frequently queried fields
- Use `PRAGMA journal_mode=MEMORY;` for better performance - Use `PRAGMA journal_mode=MEMORY;` for better performance
### 2. Batch Operations ### 2. Batch Operations
- Dexie has built-in bulk operations - Dexie has built-in bulk operations
- absurd-sql uses transactions for batch operations - absurd-sql uses transactions for batch operations
- Consider chunking large datasets - Consider chunking large datasets
- Use prepared statements for repeated queries - Use prepared statements for repeated queries
### 3. Query Optimization ### 3. Query Optimization
- Dexie uses IndexedDB's native indexing - Dexie uses IndexedDB's native indexing
- absurd-sql requires explicit query optimization - absurd-sql requires explicit query optimization
- Use prepared statements for repeated queries - Use prepared statements for repeated queries
@ -327,6 +344,7 @@ async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
## Error Handling ## Error Handling
### 1. Common Errors ### 1. Common Errors
```typescript ```typescript
// Dexie errors // Dexie errors
try { try {
@ -351,6 +369,7 @@ try {
``` ```
### 2. Transaction Recovery ### 2. Transaction Recovery
```typescript ```typescript
// Dexie transaction // Dexie transaction
try { try {
@ -396,4 +415,4 @@ try {
- Remove Dexie database - Remove Dexie database
- Clear IndexedDB storage - Clear IndexedDB storage
- Update application code - Update application code
- Remove old dependencies - Remove old dependencies

272
doc/migration-fence-definition.md

@ -0,0 +1,272 @@
# Migration Fence Definition: Dexie to SQLite
## Overview
This document defines the **migration fence** - the boundary between the legacy Dexie (IndexedDB) storage system and the new SQLite-based storage system in TimeSafari. The fence ensures controlled migration while maintaining data integrity and application stability.
## Current Migration Status
### ✅ Completed Components
- **SQLite Database Service**: Fully implemented with absurd-sql
- **Platform Service Layer**: Unified database interface across platforms
- **Migration Tools**: Data comparison and transfer utilities
- **Schema Migration**: Complete table structure migration
- **Data Export/Import**: Backup and restore functionality
### 🔄 Active Migration Components
- **Settings Migration**: Core user settings transferred
- **Account Migration**: Identity and key management
- **Contact Migration**: User contact data (via import interface)
### ❌ Legacy Components (Fence Boundary)
- **Dexie Database**: Legacy IndexedDB storage (disabled by default)
- **Dexie-Specific Code**: Direct database access patterns
- **Legacy Migration Paths**: Old data transfer methods
## Migration Fence Definition
### 1. Configuration Boundary
```typescript
// src/constants/app.ts
export const USE_DEXIE_DB = false; // FENCE: Controls legacy database access
```
**Fence Rule**: When `USE_DEXIE_DB = false`:
- All new data operations use SQLite
- Legacy Dexie database is not initialized
- Migration tools are the only path to legacy data
**Fence Rule**: When `USE_DEXIE_DB = true`:
- Legacy database is available for migration
- Dual-write operations may be enabled
- Migration tools can access both databases
### 2. Service Layer Boundary
```typescript
// src/services/PlatformServiceFactory.ts
export class PlatformServiceFactory {
public static getInstance(): PlatformService {
// FENCE: All database operations go through platform service
// No direct Dexie access outside migration tools
}
}
```
**Fence Rule**: All database operations must use:
- `PlatformService.dbQuery()` for read operations
- `PlatformService.dbExec()` for write operations
- No direct `db.` or `accountsDBPromise` access in application code
### 3. Data Access Patterns
#### ✅ Allowed (Inside Fence)
```typescript
// Use platform service for all database operations
const platformService = PlatformServiceFactory.getInstance();
const contacts = await platformService.dbQuery(
"SELECT * FROM contacts WHERE did = ?",
[accountDid]
);
```
#### ❌ Forbidden (Outside Fence)
```typescript
// Direct Dexie access (legacy pattern)
const contacts = await db.contacts.where('did').equals(accountDid).toArray();
// Direct database reference
const result = await accountsDBPromise;
```
### 4. Migration Tool Boundary
```typescript
// src/services/indexedDBMigrationService.ts
// FENCE: Only migration tools can access both databases
export async function compareDatabases(): Promise<DataComparison> {
// This is the ONLY place where both databases are accessed
}
```
**Fence Rule**: Migration tools are the exclusive interface between:
- Legacy Dexie database
- New SQLite database
- Data comparison and transfer operations
## Migration Fence Guidelines
### 1. Code Development Rules
#### New Feature Development
- **Always** use `PlatformService` for database operations
- **Never** import or reference Dexie directly
- **Always** test with `USE_DEXIE_DB = false`
#### Legacy Code Maintenance
- **Only** modify Dexie code for migration purposes
- **Always** add migration tests for schema changes
- **Never** add new Dexie-specific features
### 2. Data Integrity Rules
#### Migration Safety
- **Always** create backups before migration
- **Always** verify data integrity after migration
- **Never** delete legacy data until verified
#### Rollback Strategy
- **Always** maintain ability to rollback to Dexie
- **Always** preserve migration logs
- **Never** assume migration is irreversible
### 3. Testing Requirements
#### Migration Testing
```typescript
// Required test pattern for migration
describe('Database Migration', () => {
it('should migrate data without loss', async () => {
// 1. Enable Dexie
// 2. Create test data
// 3. Run migration
// 4. Verify data integrity
// 5. Disable Dexie
});
});
```
#### Application Testing
```typescript
// Required test pattern for application features
describe('Feature with Database', () => {
it('should work with SQLite only', async () => {
// Test with USE_DEXIE_DB = false
// Verify all operations use PlatformService
});
});
```
## Migration Fence Enforcement
### 1. Static Analysis
#### ESLint Rules
```json
{
"rules": {
"no-restricted-imports": [
"error",
{
"patterns": [
{
"group": ["../db/index"],
"message": "Use PlatformService instead of direct Dexie access"
}
]
}
]
}
}
```
#### TypeScript Rules
```json
{
"compilerOptions": {
"strict": true,
"noImplicitAny": true
}
}
```
### 2. Runtime Checks
#### Development Mode Validation
```typescript
// Development-only fence validation
if (import.meta.env.DEV && USE_DEXIE_DB) {
console.warn('⚠️ Dexie is enabled - migration mode active');
}
```
#### Production Safety
```typescript
// Production fence enforcement
if (import.meta.env.PROD && USE_DEXIE_DB) {
throw new Error('Dexie cannot be enabled in production');
}
```
## Migration Fence Timeline
### Phase 1: Fence Establishment ✅
- [x] Define migration fence boundaries
- [x] Implement PlatformService layer
- [x] Create migration tools
- [x] Set `USE_DEXIE_DB = false` by default
### Phase 2: Data Migration 🔄
- [x] Migrate core settings
- [x] Migrate account data
- [ ] Complete contact migration
- [ ] Verify all data integrity
### Phase 3: Code Cleanup 📋
- [ ] Remove unused Dexie imports
- [ ] Clean up legacy database code
- [ ] Update all documentation
- [ ] Remove migration tools
### Phase 4: Fence Removal 🎯
- [ ] Remove `USE_DEXIE_DB` constant
- [ ] Remove Dexie dependencies
- [ ] Remove migration service
- [ ] Finalize SQLite-only architecture
## Security Considerations
### 1. Data Protection
- **Encryption**: Maintain encryption standards across migration
- **Access Control**: Preserve user privacy during migration
- **Audit Trail**: Log all migration operations
### 2. Error Handling
- **Graceful Degradation**: Handle migration failures gracefully
- **User Communication**: Clear messaging about migration status
- **Recovery Options**: Provide rollback mechanisms
## Performance Considerations
### 1. Migration Performance
- **Batch Operations**: Use transactions for bulk data transfer
- **Progress Indicators**: Show migration progress to users
- **Background Processing**: Non-blocking migration operations
### 2. Application Performance
- **Query Optimization**: Optimize SQLite queries for performance
- **Indexing Strategy**: Maintain proper database indexes
- **Memory Management**: Efficient memory usage during migration
## Documentation Requirements
### 1. Code Documentation
- **Migration Fence Comments**: Document fence boundaries in code
- **API Documentation**: Update all database API documentation
- **Migration Guides**: Comprehensive migration documentation
### 2. User Documentation
- **Migration Instructions**: Clear user migration steps
- **Troubleshooting**: Common migration issues and solutions
- **Rollback Instructions**: How to revert if needed
## Conclusion
The migration fence provides a controlled boundary between legacy and new database systems, ensuring:
- **Data Integrity**: No data loss during migration
- **Application Stability**: Consistent behavior across platforms
- **Development Clarity**: Clear guidelines for code development
- **Migration Safety**: Controlled and reversible migration process
This fence will remain in place until all data is successfully migrated and verified, at which point the legacy system can be safely removed.

355
doc/migration-security-checklist.md

@ -0,0 +1,355 @@
# Database Migration Security Audit Checklist
## Overview
This document provides a comprehensive security audit checklist for the Dexie to SQLite migration in TimeSafari. The checklist ensures that data protection, privacy, and security are maintained throughout the migration process.
## Pre-Migration Security Assessment
### 1. Data Classification and Sensitivity
- [ ] **Data Inventory**
- [ ] Identify all sensitive data types (DIDs, private keys, personal information)
- [ ] Document data retention requirements
- [ ] Map data relationships and dependencies
- [ ] Assess data sensitivity levels (public, internal, confidential, restricted)
- [ ] **Encryption Assessment**
- [ ] Verify current encryption methods for sensitive data
- [ ] Document encryption keys and their management
- [ ] Assess encryption strength and compliance
- [ ] Plan encryption migration strategy
### 2. Access Control Review
- [ ] **User Access Rights**
- [ ] Audit current user permissions and roles
- [ ] Document access control mechanisms
- [ ] Verify principle of least privilege
- [ ] Plan access control migration
- [ ] **System Access**
- [ ] Review database access patterns
- [ ] Document authentication mechanisms
- [ ] Assess session management
- [ ] Plan authentication migration
### 3. Compliance Requirements
- [ ] **Regulatory Compliance**
- [ ] Identify applicable regulations (GDPR, CCPA, etc.)
- [ ] Document data processing requirements
- [ ] Assess privacy impact
- [ ] Plan compliance verification
- [ ] **Industry Standards**
- [ ] Review security standards compliance
- [ ] Document security controls
- [ ] Assess audit requirements
- [ ] Plan standards compliance
## Migration Security Controls
### 1. Data Protection During Migration
- [ ] **Encryption in Transit**
- [ ] Verify all data transfers are encrypted
- [ ] Use secure communication protocols (TLS 1.3+)
- [ ] Implement secure API endpoints
- [ ] Monitor encryption status
- [ ] **Encryption at Rest**
- [ ] Maintain encryption for stored data
- [ ] Verify encryption key management
- [ ] Test encryption/decryption processes
- [ ] Document encryption procedures
### 2. Access Control During Migration
- [ ] **Authentication**
- [ ] Maintain user authentication during migration
- [ ] Verify session management
- [ ] Implement secure token handling
- [ ] Monitor authentication events
- [ ] **Authorization**
- [ ] Preserve user permissions during migration
- [ ] Verify role-based access control
- [ ] Implement audit logging
- [ ] Monitor access patterns
### 3. Data Integrity
- [ ] **Data Validation**
- [ ] Implement input validation for all data
- [ ] Verify data format consistency
- [ ] Test data transformation processes
- [ ] Document validation rules
- [ ] **Data Verification**
- [ ] Implement checksums for data integrity
- [ ] Verify data completeness after migration
- [ ] Test data consistency checks
- [ ] Document verification procedures
## Migration Process Security
### 1. Backup Security
- [ ] **Backup Creation**
- [ ] Create encrypted backups before migration
- [ ] Verify backup integrity
- [ ] Store backups securely
- [ ] Test backup restoration
- [ ] **Backup Access**
- [ ] Limit backup access to authorized personnel
- [ ] Implement backup access logging
- [ ] Verify backup encryption
- [ ] Document backup procedures
### 2. Migration Tool Security
- [ ] **Tool Authentication**
- [ ] Implement secure authentication for migration tools
- [ ] Verify tool access controls
- [ ] Monitor tool usage
- [ ] Document tool security
- [ ] **Tool Validation**
- [ ] Verify migration tool integrity
- [ ] Test tool security features
- [ ] Validate tool outputs
- [ ] Document tool validation
### 3. Error Handling
- [ ] **Error Security**
- [ ] Implement secure error handling
- [ ] Avoid information disclosure in errors
- [ ] Log security-relevant errors
- [ ] Document error procedures
- [ ] **Recovery Security**
- [ ] Implement secure recovery procedures
- [ ] Verify recovery data protection
- [ ] Test recovery processes
- [ ] Document recovery security
## Post-Migration Security
### 1. Data Verification
- [ ] **Data Completeness**
- [ ] Verify all data was migrated successfully
- [ ] Check for data corruption
- [ ] Validate data relationships
- [ ] Document verification results
- [ ] **Data Accuracy**
- [ ] Verify data accuracy after migration
- [ ] Test data consistency
- [ ] Validate data integrity
- [ ] Document accuracy checks
### 2. Access Control Verification
- [ ] **User Access**
- [ ] Verify user access rights after migration
- [ ] Test authentication mechanisms
- [ ] Validate authorization rules
- [ ] Document access verification
- [ ] **System Access**
- [ ] Verify system access controls
- [ ] Test API security
- [ ] Validate session management
- [ ] Document system security
### 3. Security Testing
- [ ] **Penetration Testing**
- [ ] Conduct security penetration testing
- [ ] Test for common vulnerabilities
- [ ] Verify security controls
- [ ] Document test results
- [ ] **Vulnerability Assessment**
- [ ] Scan for security vulnerabilities
- [ ] Assess security posture
- [ ] Identify security gaps
- [ ] Document assessment results
## Monitoring and Logging
### 1. Security Monitoring
- [ ] **Access Monitoring**
- [ ] Monitor database access patterns
- [ ] Track user authentication events
- [ ] Monitor system access
- [ ] Document monitoring procedures
- [ ] **Data Monitoring**
- [ ] Monitor data access patterns
- [ ] Track data modification events
- [ ] Monitor data integrity
- [ ] Document data monitoring
### 2. Security Logging
- [ ] **Audit Logging**
- [ ] Implement comprehensive audit logging
- [ ] Log all security-relevant events
- [ ] Secure log storage and access
- [ ] Document logging procedures
- [ ] **Log Analysis**
- [ ] Implement log analysis tools
- [ ] Monitor for security incidents
- [ ] Analyze security trends
- [ ] Document analysis procedures
## Incident Response
### 1. Security Incident Planning
- [ ] **Incident Response Plan**
- [ ] Develop security incident response plan
- [ ] Define incident response procedures
- [ ] Train incident response team
- [ ] Document response procedures
- [ ] **Incident Detection**
- [ ] Implement incident detection mechanisms
- [ ] Monitor for security incidents
- [ ] Establish incident reporting procedures
- [ ] Document detection procedures
### 2. Recovery Procedures
- [ ] **Data Recovery**
- [ ] Develop data recovery procedures
- [ ] Test recovery processes
- [ ] Verify recovery data integrity
- [ ] Document recovery procedures
- [ ] **System Recovery**
- [ ] Develop system recovery procedures
- [ ] Test system recovery
- [ ] Verify system security after recovery
- [ ] Document recovery procedures
## Compliance Verification
### 1. Regulatory Compliance
- [ ] **Privacy Compliance**
- [ ] Verify GDPR compliance
- [ ] Check CCPA compliance
- [ ] Assess other privacy regulations
- [ ] Document compliance status
- [ ] **Security Compliance**
- [ ] Verify security standard compliance
- [ ] Check industry requirements
- [ ] Assess security certifications
- [ ] Document compliance status
### 2. Audit Requirements
- [ ] **Audit Trail**
- [ ] Maintain comprehensive audit trail
- [ ] Verify audit log integrity
- [ ] Test audit log accessibility
- [ ] Document audit procedures
- [ ] **Audit Reporting**
- [ ] Generate audit reports
- [ ] Verify report accuracy
- [ ] Distribute reports securely
- [ ] Document reporting procedures
## Documentation and Training
### 1. Security Documentation
- [ ] **Security Procedures**
- [ ] Document security procedures
- [ ] Update security policies
- [ ] Create security guidelines
- [ ] Maintain documentation
- [ ] **Security Training**
- [ ] Develop security training materials
- [ ] Train staff on security procedures
- [ ] Verify training effectiveness
- [ ] Document training procedures
### 2. Ongoing Security
- [ ] **Security Maintenance**
- [ ] Establish security maintenance procedures
- [ ] Schedule security updates
- [ ] Monitor security trends
- [ ] Document maintenance procedures
- [ ] **Security Review**
- [ ] Conduct regular security reviews
- [ ] Update security controls
- [ ] Assess security effectiveness
- [ ] Document review procedures
## Risk Assessment
### 1. Risk Identification
- [ ] **Security Risks**
- [ ] Identify potential security risks
- [ ] Assess risk likelihood and impact
- [ ] Prioritize security risks
- [ ] Document risk assessment
- [ ] **Mitigation Strategies**
- [ ] Develop risk mitigation strategies
- [ ] Implement risk controls
- [ ] Monitor risk status
- [ ] Document mitigation procedures
### 2. Risk Monitoring
- [ ] **Risk Tracking**
- [ ] Track identified risks
- [ ] Monitor risk status
- [ ] Update risk assessments
- [ ] Document risk tracking
- [ ] **Risk Reporting**
- [ ] Generate risk reports
- [ ] Distribute risk information
- [ ] Update risk documentation
- [ ] Document reporting procedures
## Conclusion
This security audit checklist ensures that the database migration maintains the highest standards of data protection, privacy, and security. Regular review and updates of this checklist are essential to maintain security throughout the migration process and beyond.
### Security Checklist Summary
- [ ] **Pre-Migration Assessment**: Complete
- [ ] **Migration Controls**: Complete
- [ ] **Process Security**: Complete
- [ ] **Post-Migration Verification**: Complete
- [ ] **Monitoring and Logging**: Complete
- [ ] **Incident Response**: Complete
- [ ] **Compliance Verification**: Complete
- [ ] **Documentation and Training**: Complete
- [ ] **Risk Assessment**: Complete
**Overall Security Status**: [ ] Secure [ ] Needs Attention [ ] Critical Issues
**Next Review Date**: _______________
**Reviewed By**: _______________
**Approved By**: _______________

749
doc/migration-to-wa-sqlite.md

@ -4,610 +4,223 @@
This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users. This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.
**Current Status**: The migration is in **Phase 2** with a well-defined migration fence in place. Core settings and account data have been migrated, with contact migration in progress. **ActiveDid migration has been implemented** to ensure user identity continuity.
## Migration Goals ## Migration Goals
1. **Data Integrity** 1. **Data Integrity**
- Preserve all existing data - Preserve all existing data
- Maintain data relationships - Maintain data relationships
- Ensure data consistency - Ensure data consistency
- **Preserve user's active identity**
2. **Performance** 2. **Performance**
- Improve query performance - Improve query performance
- Reduce storage overhead - Reduce storage overhead
- Optimize for platform-specific features - Optimize for platform-specific capabilities
3. **Security**
- Maintain or improve encryption
- Preserve access controls
- Enhance data protection
4. **User Experience**
- Zero data loss
- Minimal downtime
- Automatic migration where possible
## Prerequisites
1. **Backup Requirements**
```typescript
interface MigrationBackup {
timestamp: number;
accounts: Account[];
settings: Setting[];
contacts: Contact[];
metadata: {
version: string;
platform: string;
dexieVersion: string;
};
}
```
2. **Dependencies**
```json
{
"@jlongster/sql.js": "^1.8.0",
"absurd-sql": "^1.8.0"
}
```
3. **Storage Requirements**
- Sufficient IndexedDB quota
- Available disk space for SQLite
- Backup storage space
4. **Platform Support**
- Web: Modern browser with IndexedDB support
- iOS: iOS 13+ with SQLite support
- Android: Android 5+ with SQLite support
- Electron: Latest version with SQLite support
## Migration Process 3. **User Experience**
- Seamless transition with no data loss
- Maintain user's active identity and preferences
- Preserve application state
## Migration Architecture
### Migration Fence
The migration fence is defined by the `USE_DEXIE_DB` constant in `src/constants/app.ts`:
- `USE_DEXIE_DB = false` (default): Uses SQLite database
- `USE_DEXIE_DB = true`: Uses Dexie database (for migration purposes)
### Migration Order
The migration follows a specific order to maintain data integrity:
1. **Accounts** (foundational - contains DIDs)
2. **Settings** (references accountDid, activeDid)
3. **ActiveDid** (depends on accounts and settings) ⭐ **NEW**
4. **Contacts** (independent, but migrated after accounts for consistency)
## ActiveDid Migration ⭐ **NEW FEATURE**
### 1. Preparation ### Problem Solved
Previously, the `activeDid` setting was not migrated from Dexie to SQLite, causing users to lose their active identity after migration.
### Solution Implemented
The migration now includes a dedicated step for migrating the `activeDid`:
1. **Detection**: Identifies the `activeDid` from Dexie master settings
2. **Validation**: Verifies the `activeDid` exists in SQLite accounts
3. **Migration**: Updates SQLite master settings with the `activeDid`
4. **Error Handling**: Graceful handling of missing accounts
### Implementation Details
#### New Function: `migrateActiveDid()`
```typescript ```typescript
// src/services/storage/migration/MigrationService.ts export async function migrateActiveDid(): Promise<MigrationResult> {
import initSqlJs from '@jlongster/sql.js'; // 1. Get Dexie settings to find the activeDid
import { SQLiteFS } from 'absurd-sql'; const dexieSettings = await getDexieSettings();
import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend'; const masterSettings = dexieSettings.find(setting => !setting.accountDid);
export class MigrationService { // 2. Verify the activeDid exists in SQLite accounts
private static instance: MigrationService; const accountExists = await platformService.dbQuery(
private backup: MigrationBackup | null = null; "SELECT did FROM accounts WHERE did = ?",
private sql: any = null; [dexieActiveDid],
private db: any = null; );
async prepare(): Promise<void> { // 3. Update SQLite master settings
try { await updateDefaultSettings({ activeDid: dexieActiveDid });
// 1. Check prerequisites
await this.checkPrerequisites();
// 2. Create backup
this.backup = await this.createBackup();
// 3. Verify backup integrity
await this.verifyBackup();
// 4. Initialize absurd-sql
await this.initializeAbsurdSql();
} catch (error) {
throw new StorageError(
'Migration preparation failed',
StorageErrorCodes.MIGRATION_FAILED,
error
);
}
}
private async initializeAbsurdSql(): Promise<void> {
// Initialize SQL.js
this.sql = await initSqlJs({
locateFile: (file: string) => {
return new URL(`/node_modules/@jlongster/sql.js/dist/${file}`, import.meta.url).href;
}
});
// Setup SQLiteFS with IndexedDB backend
const sqlFS = new SQLiteFS(this.sql.FS, new IndexedDBBackend());
this.sql.register_for_idb(sqlFS);
// Create and mount filesystem
this.sql.FS.mkdir('/sql');
this.sql.FS.mount(sqlFS, {}, '/sql');
// Open database
const path = '/sql/db.sqlite';
if (typeof SharedArrayBuffer === 'undefined') {
let stream = this.sql.FS.open(path, 'a+');
await stream.node.contents.readIfFallback();
this.sql.FS.close(stream);
}
this.db = new this.sql.Database(path, { filename: true });
if (!this.db) {
throw new StorageError(
'Database initialization failed',
StorageErrorCodes.INITIALIZATION_FAILED
);
}
// Configure database
await this.db.exec(`PRAGMA journal_mode=MEMORY;`);
}
private async checkPrerequisites(): Promise<void> {
// Check IndexedDB availability
if (!window.indexedDB) {
throw new StorageError(
'IndexedDB not available',
StorageErrorCodes.INITIALIZATION_FAILED
);
}
// Check storage quota
const quota = await navigator.storage.estimate();
if (quota.quota && quota.usage && quota.usage > quota.quota * 0.9) {
throw new StorageError(
'Insufficient storage space',
StorageErrorCodes.STORAGE_FULL
);
}
// Check platform support
const capabilities = await PlatformDetection.getCapabilities();
if (!capabilities.hasFileSystem) {
throw new StorageError(
'Platform does not support required features',
StorageErrorCodes.INITIALIZATION_FAILED
);
}
}
private async createBackup(): Promise<MigrationBackup> {
const dexieDB = new Dexie('TimeSafariDB');
return {
timestamp: Date.now(),
accounts: await dexieDB.accounts.toArray(),
settings: await dexieDB.settings.toArray(),
contacts: await dexieDB.contacts.toArray(),
metadata: {
version: '1.0.0',
platform: await PlatformDetection.getPlatform(),
dexieVersion: Dexie.version
}
};
}
} }
``` ```
### 2. Data Migration #### Enhanced `migrateSettings()` Function
The settings migration now includes activeDid handling:
- Extracts `activeDid` from Dexie master settings
- Validates account existence in SQLite
- Updates SQLite master settings with the `activeDid`
#### Updated `migrateAll()` Function
The complete migration now includes a dedicated step for activeDid:
```typescript ```typescript
// src/services/storage/migration/DataMigration.ts // Step 3: Migrate ActiveDid (depends on accounts and settings)
export class DataMigration { logger.info("[MigrationService] Step 3: Migrating activeDid...");
async migrate(backup: MigrationBackup): Promise<void> { const activeDidResult = await migrateActiveDid();
try {
// 1. Create new database schema
await this.createSchema();
// 2. Migrate accounts
await this.migrateAccounts(backup.accounts);
// 3. Migrate settings
await this.migrateSettings(backup.settings);
// 4. Migrate contacts
await this.migrateContacts(backup.contacts);
// 5. Verify migration
await this.verifyMigration(backup);
} catch (error) {
// 6. Handle failure
await this.handleMigrationFailure(error, backup);
}
}
private async migrateAccounts(accounts: Account[]): Promise<void> {
// Use transaction for atomicity
await this.db.exec('BEGIN TRANSACTION;');
try {
for (const account of accounts) {
await this.db.run(`
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
VALUES (?, ?, ?, ?)
`, [
account.did,
account.publicKeyHex,
account.createdAt,
account.updatedAt
]);
}
await this.db.exec('COMMIT;');
} catch (error) {
await this.db.exec('ROLLBACK;');
throw error;
}
}
private async verifyMigration(backup: MigrationBackup): Promise<void> {
// Verify account count
const result = await this.db.exec('SELECT COUNT(*) as count FROM accounts');
const accountCount = result[0].values[0][0];
if (accountCount !== backup.accounts.length) {
throw new StorageError(
'Account count mismatch',
StorageErrorCodes.VERIFICATION_FAILED
);
}
// Verify data integrity
await this.verifyDataIntegrity(backup);
}
}
``` ```
### 3. Rollback Strategy ### Benefits
- ✅ **User Identity Preservation**: Users maintain their active identity
- ✅ **Seamless Experience**: No need to manually select identity after migration
- ✅ **Data Consistency**: Ensures all identity-related settings are preserved
- ✅ **Error Resilience**: Graceful handling of edge cases
## Migration Process
### Phase 1: Preparation ✅
- [x] Enable Dexie database access
- [x] Implement data comparison tools
- [x] Create migration service structure
### Phase 2: Core Migration ✅
- [x] Account migration with `importFromMnemonic`
- [x] Settings migration (excluding activeDid)
- [x] **ActiveDid migration** ⭐ **COMPLETED**
- [x] Contact migration framework
### Phase 3: Validation and Cleanup 🔄
- [ ] Comprehensive data validation
- [ ] Performance testing
- [ ] User acceptance testing
- [ ] Dexie removal
## Usage
### Manual Migration
```typescript ```typescript
// src/services/storage/migration/RollbackService.ts import { migrateAll, migrateActiveDid } from '../services/indexedDBMigrationService';
export class RollbackService {
async rollback(backup: MigrationBackup): Promise<void> { // Complete migration
try { const result = await migrateAll();
// 1. Stop all database operations
await this.stopDatabaseOperations(); // Or migrate just the activeDid
const activeDidResult = await migrateActiveDid();
// 2. Restore from backup
await this.restoreFromBackup(backup);
// 3. Verify restoration
await this.verifyRestoration(backup);
// 4. Clean up absurd-sql
await this.cleanupAbsurdSql();
} catch (error) {
throw new StorageError(
'Rollback failed',
StorageErrorCodes.ROLLBACK_FAILED,
error
);
}
}
private async restoreFromBackup(backup: MigrationBackup): Promise<void> {
const dexieDB = new Dexie('TimeSafariDB');
// Restore accounts
await dexieDB.accounts.bulkPut(backup.accounts);
// Restore settings
await dexieDB.settings.bulkPut(backup.settings);
// Restore contacts
await dexieDB.contacts.bulkPut(backup.contacts);
}
}
``` ```
## Migration UI ### Migration Verification
```typescript
```vue import { compareDatabases } from '../services/indexedDBMigrationService';
<!-- src/components/MigrationProgress.vue -->
<template>
<div class="migration-progress">
<h2>Database Migration</h2>
<div class="progress-container">
<div class="progress-bar" :style="{ width: `${progress}%` }" />
<div class="progress-text">{{ progress }}%</div>
</div>
<div class="status-message">{{ statusMessage }}</div>
<div v-if="error" class="error-message">
{{ error }}
<button @click="retryMigration">Retry</button>
</div>
</div>
</template>
<script setup lang="ts">
import { ref, onMounted } from 'vue';
import { MigrationService } from '@/services/storage/migration/MigrationService';
const progress = ref(0);
const statusMessage = ref('Preparing migration...');
const error = ref<string | null>(null);
const migrationService = MigrationService.getInstance();
async function startMigration() {
try {
// 1. Preparation
statusMessage.value = 'Creating backup...';
await migrationService.prepare();
progress.value = 20;
// 2. Data migration
statusMessage.value = 'Migrating data...';
await migrationService.migrate();
progress.value = 80;
// 3. Verification
statusMessage.value = 'Verifying migration...';
await migrationService.verify();
progress.value = 100;
statusMessage.value = 'Migration completed successfully!';
} catch (err) {
error.value = err instanceof Error ? err.message : 'Migration failed';
statusMessage.value = 'Migration failed';
}
}
async function retryMigration() { const comparison = await compareDatabases();
error.value = null; console.log('Migration differences:', comparison.differences);
progress.value = 0; ```
await startMigration();
}
onMounted(() => { ## Error Handling
startMigration();
});
</script>
<style scoped> ### ActiveDid Migration Errors
.migration-progress { - **Missing Account**: If the `activeDid` from Dexie doesn't exist in SQLite accounts
padding: 2rem; - **Database Errors**: Connection or query failures
max-width: 600px; - **Settings Update Failures**: Issues updating SQLite master settings
margin: 0 auto;
}
.progress-container { ### Recovery Strategies
position: relative; 1. **Automatic Recovery**: Migration continues even if activeDid migration fails
height: 20px; 2. **Manual Recovery**: Users can manually select their identity after migration
background: #eee; 3. **Fallback**: System creates new identity if none exists
border-radius: 10px;
overflow: hidden;
margin: 1rem 0;
}
.progress-bar { ## Security Considerations
position: absolute;
height: 100%;
background: #4CAF50;
transition: width 0.3s ease;
}
.progress-text { ### Data Protection
position: absolute; - All sensitive data (mnemonics, private keys) are encrypted
width: 100%; - Migration preserves encryption standards
text-align: center; - No plaintext data exposure during migration
line-height: 20px;
color: #000;
}
.status-message { ### Identity Verification
text-align: center; - ActiveDid migration validates account existence
margin: 1rem 0; - Prevents setting non-existent identities as active
} - Maintains cryptographic integrity
.error-message { ## Testing
color: #f44336;
text-align: center;
margin: 1rem 0;
}
button { ### Migration Testing
margin-top: 1rem; ```bash
padding: 0.5rem 1rem; # Enable Dexie for testing
background: #2196F3; # Set USE_DEXIE_DB = true in constants/app.ts
color: white;
border: none;
border-radius: 4px;
cursor: pointer;
}
button:hover { # Run migration
background: #1976D2; npm run migrate
}
</style> # Verify results
npm run test:migration
```
### ActiveDid Testing
```typescript
// Test activeDid migration specifically
const result = await migrateActiveDid();
expect(result.success).toBe(true);
expect(result.warnings).toContain('Successfully migrated activeDid');
``` ```
## Testing Strategy ## Troubleshooting
1. **Unit Tests**
```typescript
// src/services/storage/migration/__tests__/MigrationService.spec.ts
describe('MigrationService', () => {
it('should initialize absurd-sql correctly', async () => {
const service = MigrationService.getInstance();
await service.initializeAbsurdSql();
expect(service.isInitialized()).toBe(true);
expect(service.getDatabase()).toBeDefined();
});
it('should create valid backup', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
expect(backup).toBeDefined();
expect(backup.accounts).toBeInstanceOf(Array);
expect(backup.settings).toBeInstanceOf(Array);
expect(backup.contacts).toBeInstanceOf(Array);
});
it('should migrate data correctly', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
await service.migrate(backup);
// Verify migration
const accounts = await service.getMigratedAccounts();
expect(accounts).toHaveLength(backup.accounts.length);
});
it('should handle rollback correctly', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
// Simulate failed migration
await service.migrate(backup);
await service.simulateFailure();
// Perform rollback
await service.rollback(backup);
// Verify rollback
const accounts = await service.getOriginalAccounts();
expect(accounts).toHaveLength(backup.accounts.length);
});
});
```
2. **Integration Tests**
```typescript
// src/services/storage/migration/__tests__/integration/Migration.spec.ts
describe('Migration Integration', () => {
it('should handle concurrent access during migration', async () => {
const service = MigrationService.getInstance();
// Start migration
const migrationPromise = service.migrate();
// Simulate concurrent access
const accessPromises = Array(5).fill(null).map(() =>
service.getAccount('did:test:123')
);
// Wait for all operations
const [migrationResult, ...accessResults] = await Promise.allSettled([
migrationPromise,
...accessPromises
]);
// Verify results
expect(migrationResult.status).toBe('fulfilled');
expect(accessResults.some(r => r.status === 'rejected')).toBe(true);
});
it('should maintain data integrity during platform transition', async () => {
const service = MigrationService.getInstance();
// Simulate platform change
await service.simulatePlatformChange();
// Verify data
const accounts = await service.getAllAccounts();
const settings = await service.getAllSettings();
const contacts = await service.getAllContacts();
expect(accounts).toBeDefined();
expect(settings).toBeDefined();
expect(contacts).toBeDefined();
});
});
```
## Success Criteria
1. **Data Integrity** ### Common Issues
- [ ] All accounts migrated successfully
- [ ] All settings preserved
- [ ] All contacts transferred
- [ ] No data corruption
2. **Performance** 1. **ActiveDid Not Found**
- [ ] Migration completes within acceptable time - Ensure accounts were migrated before activeDid migration
- [ ] No significant performance degradation - Check that the Dexie activeDid exists in SQLite accounts
- [ ] Efficient storage usage
- [ ] Smooth user experience 2. **Migration Failures**
- Verify Dexie database is accessible
3. **Security** - Check SQLite database permissions
- [ ] Encrypted data remains secure - Review migration logs for specific errors
- [ ] Access controls maintained
- [ ] No sensitive data exposure 3. **Data Inconsistencies**
- [ ] Secure backup process - Use `compareDatabases()` to identify differences
- Re-run migration if necessary
4. **User Experience** - Check for duplicate or conflicting records
- [ ] Clear migration progress
- [ ] Informative error messages ### Debugging
- [ ] Automatic recovery from failures ```typescript
- [ ] No data loss // Enable detailed logging
logger.setLevel('debug');
## Rollback Plan
// Check migration status
1. **Automatic Rollback** const comparison = await compareDatabases();
- Triggered by migration failure console.log('Settings differences:', comparison.differences.settings);
- Restores from verified backup ```
- Maintains data consistency
- Logs rollback reason ## Future Enhancements
2. **Manual Rollback** ### Planned Improvements
- Available through settings 1. **Batch Processing**: Optimize for large datasets
- Requires user confirmation 2. **Incremental Migration**: Support partial migrations
- Preserves backup data 3. **Rollback Capability**: Ability to revert migration
- Provides rollback status 4. **Progress Tracking**: Real-time migration progress
3. **Emergency Recovery** ### Performance Optimizations
- Manual backup restoration 1. **Parallel Processing**: Migrate independent data concurrently
- Database repair tools 2. **Memory Management**: Optimize for large datasets
- Data recovery procedures 3. **Transaction Batching**: Reduce database round trips
- Support contact information
## Conclusion
## Post-Migration
The Dexie to SQLite migration provides a robust, secure, and user-friendly transition path. The addition of activeDid migration ensures that users maintain their identity continuity throughout the migration process, significantly improving the user experience.
1. **Verification**
- Data integrity checks The migration fence architecture allows for controlled, reversible migration while maintaining application stability and data integrity.
- Performance monitoring
- Error rate tracking
- User feedback collection
2. **Cleanup**
- Remove old database
- Clear migration artifacts
- Update application state
- Archive backup data
3. **Monitoring**
- Track migration success rate
- Monitor performance metrics
- Collect error reports
- Gather user feedback
## Support
For assistance with migration:
1. Check the troubleshooting guide
2. Review error logs
3. Contact support team
4. Submit issue report
## Timeline
1. **Preparation Phase** (1 week)
- Backup system implementation
- Migration service development
- Testing framework setup
2. **Testing Phase** (2 weeks)
- Unit testing
- Integration testing
- Performance testing
- Security testing
3. **Deployment Phase** (1 week)
- Staged rollout
- Monitoring
- Support preparation
- Documentation updates
4. **Post-Deployment** (2 weeks)
- Monitoring
- Bug fixes
- Performance optimization
- User feedback collection

585
package-lock.json

File diff suppressed because it is too large

2
package.json

@ -1,6 +1,6 @@
{ {
"name": "timesafari", "name": "timesafari",
"version": "0.5.8", "version": "1.0.1-beta",
"description": "Time Safari Application", "description": "Time Safari Application",
"author": { "author": {
"name": "Time Safari Team" "name": "Time Safari Team"

75
src/assets/icons.json

@ -0,0 +1,75 @@
{
"warning": {
"fillRule": "evenodd",
"d": "M8.257 3.099c.765-1.36 2.722-1.36 3.486 0l5.58 9.92c.75 1.334-.213 2.98-1.742 2.98H4.42c-1.53 0-2.493-1.646-1.743-2.98l5.58-9.92zM11 13a1 1 0 11-2 0 1 1 0 012 0zm-1-8a1 1 0 00-1 1v3a1 1 0 002 0V6a1 1 0 00-1-1z",
"clipRule": "evenodd"
},
"spinner": {
"d": "M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z"
},
"chart": {
"strokeLinecap": "round",
"strokeLinejoin": "round",
"strokeWidth": "2",
"d": "M9 19v-6a2 2 0 00-2-2H5a2 2 0 00-2 2v6a2 2 0 002 2h2a2 2 0 002-2zm0 0V9a2 2 0 012-2h2a2 2 0 012 2v10m-6 0a2 2 0 002 2h2a2 2 0 002-2m0 0V5a2 2 0 012-2h2a2 2 0 012 2v14a2 2 0 01-2 2h-2a2 2 0 01-2-2z"
},
"plus": {
"strokeLinecap": "round",
"strokeLinejoin": "round",
"strokeWidth": "2",
"d": "M12 4v16m8-8H4"
},
"settings": {
"strokeLinecap": "round",
"strokeLinejoin": "round",
"strokeWidth": "2",
"d": "M10.325 4.317c.426-1.756 2.924-1.756 3.35 0a1.724 1.724 0 002.573 1.066c1.543-.94 3.31.826 2.37 2.37a1.724 1.724 0 001.065 2.572c1.756.426 1.756 2.924 0 3.35a1.724 1.724 0 00-1.066 2.573c.94 1.543-.826 3.31-2.37 2.37a1.724 1.724 0 00-2.572 1.065c-.426 1.756-2.924 1.756-3.35 0a1.724 1.724 0 00-2.573-1.066c-1.543.94-3.31-.826-2.37-2.37a1.724 1.724 0 00-1.065-2.572c-1.756-.426-1.756-2.924 0-3.35a1.724 1.724 0 001.066-2.573c-.94-1.543.826-3.31 2.37-2.37.996.608 2.296.07 2.572-1.065z"
},
"settingsDot": {
"strokeLinecap": "round",
"strokeLinejoin": "round",
"strokeWidth": "2",
"d": "M15 12a3 3 0 11-6 0 3 3 0 016 0z"
},
"lock": {
"strokeLinecap": "round",
"strokeLinejoin": "round",
"strokeWidth": "2",
"d": "M12 15v2m-6 4h12a2 2 0 002-2v-6a2 2 0 00-2-2H6a2 2 0 00-2 2v6a2 2 0 002 2zm10-10V7a4 4 0 00-8 0v4h8z"
},
"download": {
"strokeLinecap": "round",
"strokeLinejoin": "round",
"strokeWidth": "2",
"d": "M12 10v6m0 0l-3-3m3 3l3-3m2 8H7a2 2 0 01-2-2V5a2 2 0 012-2h5.586a1 1 0 01.707.293l5.414 5.414a1 1 0 01.293.707V19a2 2 0 01-2 2z"
},
"check": {
"strokeLinecap": "round",
"strokeLinejoin": "round",
"strokeWidth": "2",
"d": "M9 12l2 2 4-4m6 2a9 9 0 11-18 0 9 9 0 0118 0z"
},
"edit": {
"strokeLinecap": "round",
"strokeLinejoin": "round",
"strokeWidth": "2",
"d": "M11 5H6a2 2 0 00-2 2v11a2 2 0 002 2h11a2 2 0 002-2v-5m-1.414-9.414a2 2 0 112.828 2.828L11.828 15H9v-2.828l8.586-8.586z"
},
"trash": {
"strokeLinecap": "round",
"strokeLinejoin": "round",
"strokeWidth": "2",
"d": "M19 7l-.867 12.142A2 2 0 0116.138 21H7.862a2 2 0 01-1.995-1.858L5 7m5 4v6m4-6v6m1-10V4a1 1 0 00-1-1h-4a1 1 0 00-1 1v3M4 7h16"
},
"plusCircle": {
"strokeLinecap": "round",
"strokeLinejoin": "round",
"strokeWidth": "2",
"d": "M12 6v6m0 0v6m0-6h6m-6 0H6"
},
"info": {
"fillRule": "evenodd",
"d": "M18 10a8 8 0 11-16 0 8 8 0 0116 0zm-7-4a1 1 0 11-2 0 1 1 0 012 0zM9 9a1 1 0 000 2v3a1 1 0 001 1h1a1 1 0 100-2v-3a1 1 0 00-1-1H9z",
"clipRule": "evenodd"
}
}

90
src/components/IconRenderer.vue

@ -0,0 +1,90 @@
<template>
<svg
v-if="iconData"
:class="svgClass"
:fill="fill"
:stroke="stroke"
:viewBox="viewBox"
xmlns="http://www.w3.org/2000/svg"
>
<path v-for="(path, index) in iconData.paths" :key="index" v-bind="path" />
</svg>
</template>
<script lang="ts">
import { Component, Prop, Vue } from "vue-facing-decorator";
import icons from "../assets/icons.json";
import { logger } from "../utils/logger";
/**
* Icon path interface
*/
interface IconPath {
d: string;
fillRule?: string;
clipRule?: string;
strokeLinecap?: string;
strokeLinejoin?: string;
strokeWidth?: string | number;
fill?: string;
stroke?: string;
}
/**
* Icon data interface
*/
interface IconData {
paths: IconPath[];
}
/**
* Icons JSON structure
*/
interface IconsJson {
[key: string]: IconPath | IconData;
}
/**
* Icon Renderer Component
*
* This component loads SVG icon definitions from a JSON file and renders them
* as SVG elements. It provides a clean way to use icons without cluttering
* templates with long SVG path definitions.
*
* @author Matthew Raymer
* @version 1.0.0
* @since 2024
*/
@Component({
name: "IconRenderer",
})
export default class IconRenderer extends Vue {
@Prop({ required: true }) readonly iconName!: string;
@Prop({ default: "h-5 w-5" }) readonly svgClass!: string;
@Prop({ default: "none" }) readonly fill!: string;
@Prop({ default: "currentColor" }) readonly stroke!: string;
@Prop({ default: "0 0 24 24" }) readonly viewBox!: string;
/**
* Get the icon data for the specified icon name
*
* @returns {IconData | null} The icon data object or null if not found
*/
get iconData(): IconData | null {
const icon = (icons as IconsJson)[this.iconName];
if (!icon) {
logger.warn(`Icon "${this.iconName}" not found in icons.json`);
return null;
}
// Convert single path to array format for consistency
if ("d" in icon) {
return {
paths: [icon as IconPath],
};
}
return icon as IconData;
}
}
</script>

17
src/db-sql/migration.ts

@ -1,4 +1,7 @@
import migrationService from "../services/migrationService"; import {
registerMigration,
runMigrations as runMigrationsService,
} from "../services/migrationService";
import { DEFAULT_ENDORSER_API_SERVER } from "@/constants/app"; import { DEFAULT_ENDORSER_API_SERVER } from "@/constants/app";
import { arrayBufferToBase64 } from "@/libs/crypto"; import { arrayBufferToBase64 } from "@/libs/crypto";
@ -123,16 +126,12 @@ const MIGRATIONS = [
* @param extractMigrationNames - A function that extracts the names (string array) from "select name from migrations" * @param extractMigrationNames - A function that extracts the names (string array) from "select name from migrations"
*/ */
export async function runMigrations<T>( export async function runMigrations<T>(
sqlExec: (sql: string) => Promise<unknown>, sqlExec: (sql: string, params?: unknown[]) => Promise<unknown>,
sqlQuery: (sql: string) => Promise<T>, sqlQuery: (sql: string, params?: unknown[]) => Promise<T>,
extractMigrationNames: (result: T) => Set<string>, extractMigrationNames: (result: T) => Set<string>,
): Promise<void> { ): Promise<void> {
for (const migration of MIGRATIONS) { for (const migration of MIGRATIONS) {
migrationService.registerMigration(migration); registerMigration(migration);
} }
await migrationService.runMigrations( await runMigrationsService(sqlExec, sqlQuery, extractMigrationNames);
sqlExec,
sqlQuery,
extractMigrationNames,
);
} }

56
src/db/databaseUtil.ts

@ -227,10 +227,28 @@ export async function logConsoleAndDb(
} }
/** /**
* Generates an SQL INSERT statement and parameters from a model object. * Generates SQL INSERT statement and parameters from a model object
* @param model The model object containing fields to update *
* @param tableName The name of the table to update * This helper function creates a parameterized SQL INSERT statement
* @returns Object containing the SQL statement and parameters array * from a JavaScript object. It filters out undefined values and
* creates the appropriate SQL syntax with placeholders.
*
* The function is used internally by the migration functions to
* safely insert data into the SQLite database.
*
* @function generateInsertStatement
* @param {Record<string, unknown>} model - The model object containing fields to insert
* @param {string} tableName - The name of the table to insert into
* @returns {Object} Object containing the SQL statement and parameters array
* @returns {string} returns.sql - The SQL INSERT statement
* @returns {unknown[]} returns.params - Array of parameter values
* @example
* ```typescript
* const contact = { did: 'did:example:123', name: 'John Doe' };
* const { sql, params } = generateInsertStatement(contact, 'contacts');
* // sql: "INSERT INTO contacts (did, name) VALUES (?, ?)"
* // params: ['did:example:123', 'John Doe']
* ```
*/ */
export function generateInsertStatement( export function generateInsertStatement(
model: Record<string, unknown>, model: Record<string, unknown>,
@ -248,12 +266,30 @@ export function generateInsertStatement(
} }
/** /**
* Generates an SQL UPDATE statement and parameters from a model object. * Generates SQL UPDATE statement and parameters from a model object
* @param model The model object containing fields to update *
* @param tableName The name of the table to update * This helper function creates a parameterized SQL UPDATE statement
* @param whereClause The WHERE clause for the update (e.g. "id = ?") * from a JavaScript object. It filters out undefined values and
* @param whereParams Parameters for the WHERE clause * creates the appropriate SQL syntax with placeholders.
* @returns Object containing the SQL statement and parameters array *
* The function is used internally by the migration functions to
* safely update data in the SQLite database.
*
* @function generateUpdateStatement
* @param {Record<string, unknown>} model - The model object containing fields to update
* @param {string} tableName - The name of the table to update
* @param {string} whereClause - The WHERE clause for the update (e.g. "id = ?")
* @param {unknown[]} [whereParams=[]] - Parameters for the WHERE clause
* @returns {Object} Object containing the SQL statement and parameters array
* @returns {string} returns.sql - The SQL UPDATE statement
* @returns {unknown[]} returns.params - Array of parameter values
* @example
* ```typescript
* const contact = { name: 'Jane Doe' };
* const { sql, params } = generateUpdateStatement(contact, 'contacts', 'did = ?', ['did:example:123']);
* // sql: "UPDATE contacts SET name = ? WHERE did = ?"
* // params: ['Jane Doe', 'did:example:123']
* ```
*/ */
export function generateUpdateStatement( export function generateUpdateStatement(
model: Record<string, unknown>, model: Record<string, unknown>,

2
src/db/tables/README.md

@ -1 +1 @@
Check the contact & settings export to see whether you want your new table to be included in it. # Check the contact & settings export to see whether you want your new table to be included in it

4
src/db/tables/contacts.ts

@ -19,6 +19,10 @@ export interface Contact {
registered?: boolean; // cached value of the server setting registered?: boolean; // cached value of the server setting
} }
export type ContactWithJsonStrings = Contact & {
contactMethods?: string;
};
export const ContactSchema = { export const ContactSchema = {
contacts: "&did, name", // no need to key by other things contacts: "&did, name", // no need to key by other things
}; };

5
src/db/tables/settings.ts

@ -64,6 +64,11 @@ export type Settings = {
webPushServer?: string; // Web Push server URL webPushServer?: string; // Web Push server URL
}; };
// type of settings where the searchBoxes are JSON strings instead of objects
export type SettingsWithJsonStrings = Settings & {
searchBoxes: string;
};
export function checkIsAnyFeedFilterOn(settings: Settings): boolean { export function checkIsAnyFeedFilterOn(settings: Settings): boolean {
return !!(settings?.filterFeedByNearby || settings?.filterFeedByVisible); return !!(settings?.filterFeedByNearby || settings?.filterFeedByVisible);
} }

36
src/libs/util.ts

@ -45,6 +45,7 @@ import { PlatformServiceFactory } from "@/services/PlatformServiceFactory";
import { sha256 } from "ethereum-cryptography/sha256"; import { sha256 } from "ethereum-cryptography/sha256";
import { IIdentifier } from "@veramo/core"; import { IIdentifier } from "@veramo/core";
import { insertDidSpecificSettings, parseJsonField } from "../db/databaseUtil"; import { insertDidSpecificSettings, parseJsonField } from "../db/databaseUtil";
import { DEFAULT_ROOT_DERIVATION_PATH } from "./crypto";
export interface GiverReceiverInputInfo { export interface GiverReceiverInputInfo {
did?: string; did?: string;
@ -998,3 +999,38 @@ export const contactsToExportJson = (contacts: Contact[]): DatabaseExport => {
}, },
}; };
}; };
/**
* Imports an account from a mnemonic phrase
* @param mnemonic - The seed phrase to import from
* @param derivationPath - The derivation path to use (defaults to DEFAULT_ROOT_DERIVATION_PATH)
* @param shouldErase - Whether to erase existing accounts before importing
* @returns Promise that resolves when import is complete
* @throws Error if mnemonic is invalid or import fails
*/
export async function importFromMnemonic(
mnemonic: string,
derivationPath: string = DEFAULT_ROOT_DERIVATION_PATH,
shouldErase: boolean = false,
): Promise<void> {
const mne: string = mnemonic.trim().toLowerCase();
// Derive address and keys from mnemonic
const [address, privateHex, publicHex] = deriveAddress(mne, derivationPath);
// Create new identifier
const newId = newIdentifier(address, publicHex, privateHex, derivationPath);
// Handle erasures
if (shouldErase) {
const platformService = PlatformServiceFactory.getInstance();
await platformService.dbExec("DELETE FROM accounts");
if (USE_DEXIE_DB) {
const accountsDB = await accountsDBPromise;
await accountsDB.accounts.clear();
}
}
// Save the new identity
await saveNewIdentity(newId, mne, derivationPath);
}

5
src/router/index.ts

@ -148,6 +148,11 @@ const routes: Array<RouteRecordRaw> = [
name: "logs", name: "logs",
component: () => import("../views/LogView.vue"), component: () => import("../views/LogView.vue"),
}, },
{
path: "/database-migration",
name: "database-migration",
component: () => import("../views/DatabaseMigration.vue"),
},
{ {
path: "/new-activity", path: "/new-activity",
name: "new-activity", name: "new-activity",

1373
src/services/indexedDBMigrationService.ts

File diff suppressed because it is too large

156
src/services/migrationService.ts

@ -1,60 +1,150 @@
/**
* Manage database migrations as people upgrade their app over time
*/
import { logger } from "../utils/logger";
/**
* Migration interface for database schema migrations
*/
interface Migration { interface Migration {
name: string; name: string;
sql: string; sql: string;
} }
export class MigrationService { /**
private static instance: MigrationService; * Migration registry to store and manage database migrations
*/
class MigrationRegistry {
private migrations: Migration[] = []; private migrations: Migration[] = [];
private constructor() {} /**
* Register a migration with the registry
static getInstance(): MigrationService { *
if (!MigrationService.instance) { * @param migration - The migration to register
MigrationService.instance = new MigrationService(); */
} registerMigration(migration: Migration): void {
return MigrationService.instance; this.migrations.push(migration);
logger.info(`[MigrationService] Registered migration: ${migration.name}`);
} }
registerMigration(migration: Migration) { /**
this.migrations.push(migration); * Get all registered migrations
*
* @returns Array of registered migrations
*/
getMigrations(): Migration[] {
return this.migrations;
} }
/** /**
* @param sqlExec - A function that executes a SQL statement and returns some update result * Clear all registered migrations
* @param sqlQuery - A function that executes a SQL query and returns the result in some format
* @param extractMigrationNames - A function that extracts the names (string array) from a "select name from migrations" query
*/ */
async runMigrations<T>( clearMigrations(): void {
// note that this does not take parameters because the Capacitor SQLite 'execute' is different this.migrations = [];
sqlExec: (sql: string) => Promise<unknown>, logger.info("[MigrationService] Cleared all registered migrations");
sqlQuery: (sql: string) => Promise<T>, }
extractMigrationNames: (result: T) => Set<string>, }
): Promise<void> {
// Create a singleton instance of the migration registry
const migrationRegistry = new MigrationRegistry();
/**
* Register a migration with the migration service
*
* This function is used by the migration system to register database
* schema migrations that need to be applied to the database.
*
* @param migration - The migration to register
*/
export function registerMigration(migration: Migration): void {
migrationRegistry.registerMigration(migration);
}
/**
* Run all registered migrations against the database
*
* This function executes all registered migrations in order, checking
* which ones have already been applied to avoid duplicate execution.
* It creates a migrations table if it doesn't exist to track applied
* migrations.
*
* @param sqlExec - Function to execute SQL statements
* @param sqlQuery - Function to query SQL data
* @param extractMigrationNames - Function to extract migration names from query results
* @returns Promise that resolves when all migrations are complete
*/
export async function runMigrations<T>(
sqlExec: (sql: string, params?: unknown[]) => Promise<unknown>,
sqlQuery: (sql: string, params?: unknown[]) => Promise<T>,
extractMigrationNames: (result: T) => Set<string>,
): Promise<void> {
try {
// Create migrations table if it doesn't exist // Create migrations table if it doesn't exist
await sqlExec(` await sqlExec(`
CREATE TABLE IF NOT EXISTS migrations ( CREATE TABLE IF NOT EXISTS migrations (
id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE, applied_at TEXT DEFAULT CURRENT_TIMESTAMP
executed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
); );
`); `);
// Get list of executed migrations // Get list of already applied migrations
const result1: T = await sqlQuery("SELECT name FROM migrations;"); const appliedMigrationsResult = await sqlQuery(
const executedMigrations = extractMigrationNames(result1); "SELECT name FROM migrations",
);
const appliedMigrations = extractMigrationNames(appliedMigrationsResult);
logger.info(
`[MigrationService] Found ${appliedMigrations.size} applied migrations`,
);
// Get all registered migrations
const migrations = migrationRegistry.getMigrations();
if (migrations.length === 0) {
logger.warn("[MigrationService] No migrations registered");
return;
}
logger.info(
`[MigrationService] Running ${migrations.length} registered migrations`,
);
// Run each migration that hasn't been applied yet
for (const migration of migrations) {
if (appliedMigrations.has(migration.name)) {
logger.info(
`[MigrationService] Skipping already applied migration: ${migration.name}`,
);
continue;
}
// Run pending migrations in order logger.info(`[MigrationService] Applying migration: ${migration.name}`);
for (const migration of this.migrations) {
if (!executedMigrations.has(migration.name)) { try {
// Execute the migration SQL
await sqlExec(migration.sql); await sqlExec(migration.sql);
await sqlExec( // Record that the migration was applied
`INSERT INTO migrations (name) VALUES ('${migration.name}')`, await sqlExec("INSERT INTO migrations (name) VALUES (?)", [
migration.name,
]);
logger.info(
`[MigrationService] Successfully applied migration: ${migration.name}`,
); );
} catch (error) {
logger.error(
`[MigrationService] Failed to apply migration ${migration.name}:`,
error,
);
throw new Error(`Migration ${migration.name} failed: ${error}`);
} }
} }
logger.info("[MigrationService] All migrations completed successfully");
} catch (error) {
logger.error("[MigrationService] Migration process failed:", error);
throw error;
} }
} }
export default MigrationService.getInstance();

2
src/views/ContactImportView.vue

@ -566,7 +566,7 @@ export default class ContactImportView extends Vue {
this.checkingImports = true; this.checkingImports = true;
try { try {
const jwt: string = getContactJwtFromJwtUrl(jwtInput); const jwt: string = getContactJwtFromJwtUrl(jwtInput) || "";
const payload = decodeEndorserJwt(jwt).payload; const payload = decodeEndorserJwt(jwt).payload;
if (Array.isArray(payload.contacts)) { if (Array.isArray(payload.contacts)) {

1488
src/views/DatabaseMigration.vue

File diff suppressed because it is too large

4
src/views/HomeView.vue

@ -106,12 +106,12 @@ Raymer * @version 1.0.0 */
</button> </button>
</div> </div>
<UserNameDialog ref="userNameDialog" /> <UserNameDialog ref="userNameDialog" />
<div v-if="PASSKEYS_ENABLED" class="flex justify-end w-full"> <div class="flex justify-end w-full">
<router-link <router-link
:to="{ name: 'start' }" :to="{ name: 'start' }"
class="block text-right text-md font-bold bg-gradient-to-b from-blue-400 to-blue-700 shadow-[inset_0_-1px_0_0_rgba(0,0,0,0.5)] text-white mt-2 px-2 py-3 rounded-md" class="block text-right text-md font-bold bg-gradient-to-b from-blue-400 to-blue-700 shadow-[inset_0_-1px_0_0_rgba(0,0,0,0.5)] text-white mt-2 px-2 py-3 rounded-md"
> >
See all your options first See advanced options
</router-link> </router-link>
</div> </div>
</div> </div>

44
src/views/ImportAccountView.vue

@ -51,7 +51,7 @@
<div v-if="numAccounts == 1" class="mt-4"> <div v-if="numAccounts == 1" class="mt-4">
<input v-model="shouldErase" type="checkbox" class="mr-2" /> <input v-model="shouldErase" type="checkbox" class="mr-2" />
<label>Erase the previous identifier.</label> <label>Erase previous identifiers.</label>
</div> </div>
<div v-if="isNotProdServer()" class="mt-4 text-blue-500"> <div v-if="isNotProdServer()" class="mt-4 text-blue-500">
@ -88,17 +88,9 @@ import { Router } from "vue-router";
import { AppString, NotificationIface, USE_DEXIE_DB } from "../constants/app"; import { AppString, NotificationIface, USE_DEXIE_DB } from "../constants/app";
import * as databaseUtil from "../db/databaseUtil"; import * as databaseUtil from "../db/databaseUtil";
import { import { retrieveSettingsForActiveAccount } from "../db/index";
accountsDBPromise, import { DEFAULT_ROOT_DERIVATION_PATH } from "../libs/crypto";
retrieveSettingsForActiveAccount, import { retrieveAccountCount, importFromMnemonic } from "../libs/util";
} from "../db/index";
import {
DEFAULT_ROOT_DERIVATION_PATH,
deriveAddress,
newIdentifier,
} from "../libs/crypto";
import { retrieveAccountCount, saveNewIdentity } from "../libs/util";
import { PlatformServiceFactory } from "@/services/PlatformServiceFactory";
import { logger } from "../utils/logger"; import { logger } from "../utils/logger";
@Component({ @Component({
@ -115,12 +107,9 @@ export default class ImportAccountView extends Vue {
$router!: Router; $router!: Router;
apiServer = ""; apiServer = "";
address = "";
derivationPath = DEFAULT_ROOT_DERIVATION_PATH; derivationPath = DEFAULT_ROOT_DERIVATION_PATH;
mnemonic = ""; mnemonic = "";
numAccounts = 0; numAccounts = 0;
privateHex = "";
publicHex = "";
showAdvanced = false; showAdvanced = false;
shouldErase = false; shouldErase = false;
@ -143,33 +132,16 @@ export default class ImportAccountView extends Vue {
} }
public async fromMnemonic() { public async fromMnemonic() {
const mne: string = this.mnemonic.trim().toLowerCase();
try { try {
[this.address, this.privateHex, this.publicHex] = deriveAddress( await importFromMnemonic(
mne, this.mnemonic,
this.derivationPath, this.derivationPath,
this.shouldErase,
); );
const newId = newIdentifier(
this.address,
this.publicHex,
this.privateHex,
this.derivationPath,
);
const accountsDB = await accountsDBPromise;
if (this.shouldErase) {
const platformService = PlatformServiceFactory.getInstance();
await platformService.dbExec("DELETE FROM accounts");
if (USE_DEXIE_DB) {
await accountsDB.accounts.clear();
}
}
await saveNewIdentity(newId, mne, this.derivationPath);
this.$router.push({ name: "account" }); this.$router.push({ name: "account" });
// eslint-disable-next-line @typescript-eslint/no-explicit-any // eslint-disable-next-line @typescript-eslint/no-explicit-any
} catch (err: any) { } catch (err: any) {
logger.error("Error saving mnemonic & updating settings:", err); logger.error("Error importing from mnemonic:", err);
if (err == "Error: invalid mnemonic") { if (err == "Error: invalid mnemonic") {
this.$notify( this.$notify(
{ {

12
src/views/StartView.vue

@ -82,6 +82,18 @@
Derive new address from existing seed Derive new address from existing seed
</a> </a>
</div> </div>
<!-- Database Migration Section -->
<div class="mt-8 pt-6 border-t border-gray-200">
<div class="flex justify-center">
<router-link
:to="{ name: 'database-migration' }"
class="block w-fit text-center text-md bg-gradient-to-b from-slate-400 to-slate-700 shadow-[inset_0_-1px_0_0_rgba(0,0,0,0.5)] text-white px-4 py-2 rounded-md"
>
Migrate My Old Data
</router-link>
</div>
</div>
</div> </div>
</div> </div>
</section> </section>

9
src/views/TestView.vue

@ -182,6 +182,15 @@
> >
Accounts Accounts
</button> </button>
<button
class="text-sm text-blue-600 hover:text-blue-800 underline"
@click="
sqlQuery = 'SELECT * FROM contacts;';
executeSql();
"
>
Contacts
</button>
<button <button
class="text-sm text-blue-600 hover:text-blue-800 underline" class="text-sm text-blue-600 hover:text-blue-800 underline"
@click=" @click="

Loading…
Cancel
Save