Browse Source

feat: Add comprehensive database migration service for Dexie to SQLite

- Add migrationService.ts with functions to compare and transfer data between Dexie and SQLite
- Implement data comparison with detailed difference analysis (added/modified/missing)
- Add contact migration with overwrite options and error handling
- Add settings migration focusing on key user fields (firstName, isRegistered, profileImageUrl, showShortcutBvc, searchBoxes)
- Include YAML export functionality for data inspection
- Add comprehensive JSDoc documentation with examples and usage instructions
- Support both INSERT and UPDATE operations with parameterized SQL generation
- Include detailed logging and error reporting for migration operations

This service enables safe migration of user data from the legacy Dexie (IndexedDB)
database to the new SQLite implementation, with full comparison capabilities
and rollback safety through detailed reporting.
migrate-dexie-to-sqlite
Matthew Raymer 1 week ago
parent
commit
25c1d6ef4e
  1. 41
      doc/dexie-to-sqlite-mapping.md
  2. 2
      src/db/tables/README.md
  3. 1019
      src/services/migrationService.ts

41
doc/dexie-to-sqlite-mapping.md

@ -3,6 +3,7 @@
## Schema Mapping
### Current Dexie Schema
```typescript
// Current Dexie schema
const db = new Dexie('TimeSafariDB');
@ -15,6 +16,7 @@ db.version(1).stores({
```
### New SQLite Schema
```sql
-- New SQLite schema
CREATE TABLE accounts (
@ -50,6 +52,7 @@ CREATE INDEX idx_settings_updated_at ON settings(updated_at);
### 1. Account Operations
#### Get Account by DID
```typescript
// Dexie
const account = await db.accounts.get(did);
@ -62,6 +65,7 @@ const account = result[0]?.values[0];
```
#### Get All Accounts
```typescript
// Dexie
const accounts = await db.accounts.toArray();
@ -74,6 +78,7 @@ const accounts = result[0]?.values || [];
```
#### Add Account
```typescript
// Dexie
await db.accounts.add({
@ -91,6 +96,7 @@ await db.run(`
```
#### Update Account
```typescript
// Dexie
await db.accounts.update(did, {
@ -100,7 +106,7 @@ await db.accounts.update(did, {
// absurd-sql
await db.run(`
UPDATE accounts
UPDATE accounts
SET public_key_hex = ?, updated_at = ?
WHERE did = ?
`, [publicKeyHex, Date.now(), did]);
@ -109,6 +115,7 @@ await db.run(`
### 2. Settings Operations
#### Get Setting
```typescript
// Dexie
const setting = await db.settings.get(key);
@ -121,6 +128,7 @@ const setting = result[0]?.values[0];
```
#### Set Setting
```typescript
// Dexie
await db.settings.put({
@ -142,6 +150,7 @@ await db.run(`
### 3. Contact Operations
#### Get Contacts by Account
```typescript
// Dexie
const contacts = await db.contacts
@ -151,7 +160,7 @@ const contacts = await db.contacts
// absurd-sql
const result = await db.exec(`
SELECT * FROM contacts
SELECT * FROM contacts
WHERE did = ?
ORDER BY created_at DESC
`, [accountDid]);
@ -159,6 +168,7 @@ const contacts = result[0]?.values || [];
```
#### Add Contact
```typescript
// Dexie
await db.contacts.add({
@ -179,6 +189,7 @@ await db.run(`
## Transaction Mapping
### Batch Operations
```typescript
// Dexie
await db.transaction('rw', [db.accounts, db.contacts], async () => {
@ -210,10 +221,11 @@ try {
## Migration Helper Functions
### 1. Data Export (Dexie to JSON)
```typescript
async function exportDexieData(): Promise<MigrationData> {
const db = new Dexie('TimeSafariDB');
return {
accounts: await db.accounts.toArray(),
settings: await db.settings.toArray(),
@ -228,6 +240,7 @@ async function exportDexieData(): Promise<MigrationData> {
```
### 2. Data Import (JSON to absurd-sql)
```typescript
async function importToAbsurdSql(data: MigrationData): Promise<void> {
await db.exec('BEGIN TRANSACTION;');
@ -239,7 +252,7 @@ async function importToAbsurdSql(data: MigrationData): Promise<void> {
VALUES (?, ?, ?, ?)
`, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]);
}
// Import settings
for (const setting of data.settings) {
await db.run(`
@ -247,7 +260,7 @@ async function importToAbsurdSql(data: MigrationData): Promise<void> {
VALUES (?, ?, ?)
`, [setting.key, setting.value, setting.updatedAt]);
}
// Import contacts
for (const contact of data.contacts) {
await db.run(`
@ -264,6 +277,7 @@ async function importToAbsurdSql(data: MigrationData): Promise<void> {
```
### 3. Verification
```typescript
async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
// Verify account count
@ -272,21 +286,21 @@ async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
if (accountCount !== dexieData.accounts.length) {
return false;
}
// Verify settings count
const settingsResult = await db.exec('SELECT COUNT(*) as count FROM settings');
const settingsCount = settingsResult[0].values[0][0];
if (settingsCount !== dexieData.settings.length) {
return false;
}
// Verify contacts count
const contactsResult = await db.exec('SELECT COUNT(*) as count FROM contacts');
const contactsCount = contactsResult[0].values[0][0];
if (contactsCount !== dexieData.contacts.length) {
return false;
}
// Verify data integrity
for (const account of dexieData.accounts) {
const result = await db.exec(
@ -294,12 +308,12 @@ async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
[account.did]
);
const migratedAccount = result[0]?.values[0];
if (!migratedAccount ||
if (!migratedAccount ||
migratedAccount[1] !== account.publicKeyHex) { // public_key_hex is second column
return false;
}
}
return true;
}
```
@ -307,18 +321,21 @@ async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
## Performance Considerations
### 1. Indexing
- Dexie automatically creates indexes based on the schema
- absurd-sql requires explicit index creation
- Added indexes for frequently queried fields
- Use `PRAGMA journal_mode=MEMORY;` for better performance
### 2. Batch Operations
- Dexie has built-in bulk operations
- absurd-sql uses transactions for batch operations
- Consider chunking large datasets
- Use prepared statements for repeated queries
### 3. Query Optimization
- Dexie uses IndexedDB's native indexing
- absurd-sql requires explicit query optimization
- Use prepared statements for repeated queries
@ -327,6 +344,7 @@ async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
## Error Handling
### 1. Common Errors
```typescript
// Dexie errors
try {
@ -351,6 +369,7 @@ try {
```
### 2. Transaction Recovery
```typescript
// Dexie transaction
try {
@ -396,4 +415,4 @@ try {
- Remove Dexie database
- Clear IndexedDB storage
- Update application code
- Remove old dependencies
- Remove old dependencies

2
src/db/tables/README.md

@ -1 +1 @@
Check the contact & settings export to see whether you want your new table to be included in it.
# Check the contact & settings export to see whether you want your new table to be included in it

1019
src/services/migrationService.ts

File diff suppressed because it is too large
Loading…
Cancel
Save