trent_larson
/
crowd-funder-for-time-pwa
4
1
Fork 2
Code Pull Requests 12 Releases 5 Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1917 Commits
72 Branches
23 Tags
449 MiB
 
 
 
 
 
 
Tree: 4162208b7f
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '4162208b7f'
${ noResults }
crowd-funder-for-time-pwa/doc/database-migration-guide.md

9.1 KiB
Raw Blame History

Database Migration Guide

Overview

The Database Migration feature allows you to compare and migrate data between Dexie (IndexedDB) and SQLite databases in the TimeSafari application. This is particularly useful during the transition from the old Dexie-based storage system to the new SQLite-based system.

Features

1. Database Comparison

  • Compare data between Dexie and SQLite databases
  • View detailed differences in contacts and settings
  • Identify added, modified, and missing records
  • Export comparison results for analysis

2. Data Migration

  • Migrate contacts from Dexie to SQLite
  • Migrate settings from Dexie to SQLite
  • Option to overwrite existing records or skip them
  • Comprehensive error handling and reporting

3. User Interface

  • Modern, responsive UI built with Tailwind CSS
  • Real-time loading states and progress indicators
  • Clear success and error messaging
  • Export functionality for comparison data

Prerequisites

Enable Dexie Database

Before using the migration features, you must enable the Dexie database by setting:

// In constants/app.ts
export const USE_DEXIE_DB = true;

Note: This should only be enabled temporarily during migration. Remember to set it back to false after migration is complete.

Accessing the Migration Interface

  1. Navigate to the Account page in the TimeSafari app
  2. Scroll down to find the Database Migration link
  3. Click the link to open the migration interface

Using the Migration Interface

Step 1: Compare Databases

  1. Click the "Compare Databases" button
  2. The system will retrieve data from both Dexie and SQLite databases
  3. Review the comparison results showing:
    • Summary counts for each database
    • Detailed differences (added, modified, missing records)
    • Specific records that need attention

Step 2: Review Differences

The comparison results are displayed in several sections:

Summary Cards

  • Dexie Contacts: Number of contacts in Dexie database
  • SQLite Contacts: Number of contacts in SQLite database
  • Dexie Settings: Number of settings in Dexie database
  • SQLite Settings: Number of settings in SQLite database

Contact Differences

  • Added: Contacts in Dexie but not in SQLite
  • Modified: Contacts that differ between databases
  • Missing: Contacts in SQLite but not in Dexie

Settings Differences

  • Added: Settings in Dexie but not in SQLite
  • Modified: Settings that differ between databases
  • Missing: Settings in SQLite but not in Dexie

Step 3: Configure Migration Options

Before migrating data, configure the migration options:

  • Overwrite existing records: When enabled, existing records in SQLite will be updated with data from Dexie. When disabled, existing records will be skipped.

Step 4: Migrate Data

Migrate Contacts

  1. Click the "Migrate Contacts" button
  2. The system will transfer contacts from Dexie to SQLite
  3. Review the migration results showing:
    • Number of contacts successfully migrated
    • Any warnings or errors encountered

Migrate Settings

  1. Click the "Migrate Settings" button
  2. The system will transfer settings from Dexie to SQLite
  3. Review the migration results showing:
    • Number of settings successfully migrated
    • Any warnings or errors encountered

Step 5: Export Comparison (Optional)

  1. Click the "Export Comparison" button
  2. A JSON file will be downloaded containing the complete comparison data
  3. This file can be used for analysis or backup purposes

Migration Process Details

Contact Migration

The contact migration process:

  1. Retrieves all contacts from Dexie database
  2. Checks for existing contacts in SQLite by DID
  3. Inserts new contacts or updates existing ones (if overwrite is enabled)
  4. Handles complex fields like contactMethods (JSON arrays)
  5. Reports success/failure for each contact

Settings Migration

The settings migration process:

  1. Retrieves all settings from Dexie database
  2. Focuses on key user-facing settings:
    • firstName
    • isRegistered
    • profileImageUrl
    • showShortcutBvc
    • searchBoxes
  3. Preserves other settings in SQLite
  4. Reports success/failure for each setting

Error Handling

Common Issues

Dexie Database Not Enabled

Error: "Dexie database is not enabled" Solution: Set USE_DEXIE_DB = true in constants/app.ts

Database Connection Issues

Error: "Failed to retrieve Dexie contacts" Solution: Check that the Dexie database is properly initialized and accessible

SQLite Query Errors

Error: "Failed to retrieve SQLite contacts" Solution: Verify that the SQLite database is properly set up and the platform service is working

Migration Failures

Error: "Migration failed: [specific error]" Solution: Review the error details and check data integrity in both databases

Error Recovery

  1. Review the error messages carefully
  2. Check the browser console for additional details
  3. Verify database connectivity and permissions
  4. Retry the operation if appropriate
  5. Export comparison data for manual review if needed

Best Practices

Before Migration

  1. Backup your data if possible
  2. Test the migration on a small dataset first
  3. Verify that both databases are accessible
  4. Review the comparison results before migrating

During Migration

  1. Don't interrupt the migration process
  2. Monitor the progress and error messages
  3. Note any warnings or skipped records
  4. Export comparison data for reference

After Migration

  1. Verify that data was migrated correctly
  2. Test the application functionality
  3. Disable Dexie database (USE_DEXIE_DB = false)
  4. Clean up any temporary files or exports

Technical Details

Database Schema

The migration handles the following data structures:

Contacts Table

interface Contact {
  did: string;                    // Decentralized Identifier
  name: string;                   // Contact name
  contactMethods: ContactMethod[]; // Array of contact methods
  nextPubKeyHashB64: string;      // Next public key hash
  notes: string;                  // Contact notes
  profileImageUrl: string;        // Profile image URL
  publicKeyBase64: string;        // Public key in base64
  seesMe: boolean;                // Visibility flag
  registered: boolean;            // Registration status
}

Settings Table

interface Settings {
  id: number;                     // Settings ID
  accountDid: string;             // Account DID
  activeDid: string;              // Active DID
  firstName: string;              // User's first name
  isRegistered: boolean;          // Registration status
  profileImageUrl: string;        // Profile image URL
  showShortcutBvc: boolean;       // UI preference
  searchBoxes: any[];             // Search configuration
  // ... other fields
}

Migration Logic

The migration service uses sophisticated comparison logic:

  1. Primary Key Matching: Uses DID for contacts, ID for settings
  2. Deep Comparison: Compares all fields including complex objects
  3. JSON Handling: Properly handles JSON fields like contactMethods and searchBoxes
  4. Conflict Resolution: Provides options for handling existing records

Performance Considerations

  • Batch Processing: Processes records one by one for reliability
  • Error Isolation: Individual record failures don't stop the entire migration
  • Memory Management: Handles large datasets efficiently
  • Progress Reporting: Provides real-time feedback during migration

Troubleshooting

Migration Stuck

If the migration appears to be stuck:

  1. Check the browser console for errors
  2. Refresh the page and try again
  3. Verify database connectivity
  4. Check for large datasets that might take time

Incomplete Migration

If migration doesn't complete:

  1. Review error messages
  2. Check data integrity in both databases
  3. Export comparison data for manual review
  4. Consider migrating in smaller batches

Data Inconsistencies

If you notice data inconsistencies:

  1. Export comparison data
  2. Review the differences carefully
  3. Manually verify critical records
  4. Consider selective migration of specific records

Support

For issues with the Database Migration feature:

  1. Check this documentation first
  2. Review the browser console for error details
  3. Export comparison data for analysis
  4. Contact the development team with specific error details

Security Considerations

  • Data Privacy: Migration data is processed locally and not sent to external servers
  • Access Control: Only users with access to the account can perform migration
  • Data Integrity: Migration preserves data integrity and handles conflicts gracefully
  • Audit Trail: Export functionality provides an audit trail of migration operations

Note: This migration tool is designed for the transition period between database systems. Once migration is complete and verified, the Dexie database should be disabled to avoid confusion and potential data conflicts.