Browse Source

docs: add comprehensive migration guide for Dexie to wa-sqlite

- Add detailed migration process documentation including preparation, data migration, and rollback strategies\n- Include TypeScript implementation examples for MigrationService, DataMigration, and RollbackService\n- Add Vue component for migration progress tracking with error handling\n- Document testing strategy with unit and integration test examples\n- Define clear success criteria and timeline for migration\n- Include platform-specific considerations and prerequisites\n- Add post-migration verification and monitoring guidelines
new-storage
Matt Raymer 1 week ago
parent
commit
28e848e386
  1. 267
      .cursor/rules/wa-sqlite.mdc
  2. 554
      docs/migration-to-wa-sqlite.md

267
.cursor/rules/wa-sqlite.mdc

@ -0,0 +1,267 @@
---
description:
globs:
alwaysApply: true
---
# wa-sqlite Usage Guide
## Table of Contents
- [1. Overview](#1-overview)
- [2. Installation](#2-installation)
- [3. Basic Setup](#3-basic-setup)
- [3.1 Import and Initialize](#31-import-and-initialize)
- [3.2 Basic Database Operations](#32-basic-database-operations)
- [4. Virtual File Systems (VFS)](#4-virtual-file-systems-vfs)
- [4.1 Available VFS Options](#41-available-vfs-options)
- [4.2 Using a VFS](#42-using-a-vfs)
- [5. Best Practices](#5-best-practices)
- [5.1 Error Handling](#51-error-handling)
- [5.2 Transaction Management](#52-transaction-management)
- [5.3 Prepared Statements](#53-prepared-statements)
- [6. Performance Considerations](#6-performance-considerations)
- [7. Common Issues and Solutions](#7-common-issues-and-solutions)
- [8. TypeScript Support](#8-typescript-support)
## 1. Overview
wa-sqlite is a WebAssembly build of SQLite that enables SQLite database operations in web browsers and JavaScript environments. It provides both synchronous and asynchronous builds, with support for custom virtual file systems (VFS) for persistent storage.
## 2. Installation
```bash
npm install wa-sqlite
# or
yarn add wa-sqlite
```
## 3. Basic Setup
### 3.1 Import and Initialize
```javascript
// Choose one of these imports based on your needs:
// - wa-sqlite.mjs: Synchronous build
// - wa-sqlite-async.mjs: Asynchronous build (required for async VFS)
// - wa-sqlite-jspi.mjs: JSPI-based async build (experimental, Chromium only)
import SQLiteESMFactory from 'wa-sqlite/dist/wa-sqlite.mjs';
import * as SQLite from 'wa-sqlite';
async function initDatabase() {
// Initialize SQLite module
const module = await SQLiteESMFactory();
const sqlite3 = SQLite.Factory(module);
// Open database (returns a Promise)
const db = await sqlite3.open_v2('myDatabase');
return { sqlite3, db };
}
```
### 3.2 Basic Database Operations
```javascript
async function basicOperations() {
const { sqlite3, db } = await initDatabase();
try {
// Create a table
await sqlite3.exec(db, `
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
email TEXT UNIQUE
)
`);
// Insert data
await sqlite3.exec(db, `
INSERT INTO users (name, email)
VALUES ('John Doe', 'john@example.com')
`);
// Query data
const results = [];
await sqlite3.exec(db, 'SELECT * FROM users', (row, columns) => {
results.push({ row, columns });
});
return results;
} finally {
// Always close the database when done
await sqlite3.close(db);
}
}
```
## 4. Virtual File Systems (VFS)
### 4.1 Available VFS Options
wa-sqlite provides several VFS implementations for persistent storage:
1. **IDBBatchAtomicVFS** (Recommended for general use)
- Uses IndexedDB with batch atomic writes
- Works in all contexts (Window, Worker, Service Worker)
- Supports WAL mode
- Best performance with `PRAGMA synchronous=normal`
2. **IDBMirrorVFS**
- Keeps files in memory, persists to IndexedDB
- Works in all contexts
- Good for smaller databases
3. **OPFS-based VFS** (Origin Private File System)
- Various implementations available:
- AccessHandlePoolVFS
- OPFSAdaptiveVFS
- OPFSCoopSyncVFS
- OPFSPermutedVFS
- Better performance but limited to Worker contexts
### 4.2 Using a VFS
```javascript
import { IDBBatchAtomicVFS } from 'wa-sqlite/src/examples/IDBBatchAtomicVFS.js';
import SQLiteESMFactory from 'wa-sqlite/dist/wa-sqlite-async.mjs';
import * as SQLite from 'wa-sqlite';
async function initDatabaseWithVFS() {
const module = await SQLiteESMFactory();
const sqlite3 = SQLite.Factory(module);
// Register VFS
const vfs = await IDBBatchAtomicVFS.create('myApp', module);
sqlite3.vfs_register(vfs, true);
// Open database with VFS
const db = await sqlite3.open_v2('myDatabase');
// Configure for better performance
await sqlite3.exec(db, 'PRAGMA synchronous = normal');
await sqlite3.exec(db, 'PRAGMA journal_mode = WAL');
return { sqlite3, db };
}
```
## 5. Best Practices
### 5.1 Error Handling
```javascript
async function safeDatabaseOperation() {
const { sqlite3, db } = await initDatabase();
try {
await sqlite3.exec(db, 'SELECT * FROM non_existent_table');
} catch (error) {
if (error.code === SQLite.SQLITE_ERROR) {
console.error('SQL error:', error.message);
} else {
console.error('Database error:', error);
}
} finally {
await sqlite3.close(db);
}
}
```
### 5.2 Transaction Management
```javascript
async function transactionExample() {
const { sqlite3, db } = await initDatabase();
try {
await sqlite3.exec(db, 'BEGIN TRANSACTION');
// Perform multiple operations
await sqlite3.exec(db, 'INSERT INTO users (name) VALUES (?)', ['Alice']);
await sqlite3.exec(db, 'INSERT INTO users (name) VALUES (?)', ['Bob']);
await sqlite3.exec(db, 'COMMIT');
} catch (error) {
await sqlite3.exec(db, 'ROLLBACK');
throw error;
} finally {
await sqlite3.close(db);
}
}
```
### 5.3 Prepared Statements
```javascript
async function preparedStatementExample() {
const { sqlite3, db } = await initDatabase();
try {
// Prepare statement
const stmt = await sqlite3.prepare(db, 'SELECT * FROM users WHERE id = ?');
// Execute with different parameters
await sqlite3.bind(stmt, 1, 1);
while (await sqlite3.step(stmt) === SQLite.SQLITE_ROW) {
const row = sqlite3.row(stmt);
console.log(row);
}
// Reset and reuse
await sqlite3.reset(stmt);
await sqlite3.bind(stmt, 1, 2);
// ... execute again
await sqlite3.finalize(stmt);
} finally {
await sqlite3.close(db);
}
}
```
## 6. Performance Considerations
1. **VFS Selection**
- Use IDBBatchAtomicVFS for general-purpose applications
- Consider OPFS-based VFS for better performance in Worker contexts
- Use MemoryVFS for temporary databases
2. **Configuration**
- Set appropriate page size (default is usually fine)
- Use WAL mode for better concurrency
- Consider `PRAGMA synchronous=normal` for better performance
- Adjust cache size based on your needs
3. **Concurrency**
- Use transactions for multiple operations
- Be aware of VFS-specific concurrency limitations
- Consider using Web Workers for heavy database operations
## 7. Common Issues and Solutions
1. **Database Locking**
- Use appropriate transaction isolation levels
- Implement retry logic for busy errors
- Consider using WAL mode
2. **Storage Limitations**
- Be aware of browser storage quotas
- Implement cleanup strategies
- Monitor database size
3. **Cross-Context Access**
- Use appropriate VFS for your context
- Consider message passing for cross-context communication
- Be aware of storage access limitations
## 8. TypeScript Support
wa-sqlite includes TypeScript definitions. The main types are:
```typescript
type SQLiteCompatibleType = number | string | Uint8Array | Array<number> | bigint | null;
interface SQLiteAPI {
open_v2(filename: string, flags?: number, zVfs?: string): Promise<number>;
exec(db: number, sql: string, callback?: (row: any[], columns: string[]) => void): Promise<number>;
close(db: number): Promise<number>;
// ... other methods
}
```
## Additional Resources
- [Official GitHub Repository](https://github.com/rhashimoto/wa-sqlite)
- [Online Demo](https://rhashimoto.github.io/wa-sqlite/demo/)
- [API Reference](https://rhashimoto.github.io/wa-sqlite/docs/)
- [FAQ](https://github.com/rhashimoto/wa-sqlite/issues?q=is%3Aissue+label%3Afaq+)
- [Discussion Forums](https://github.com/rhashimoto/wa-sqlite/discussions)

554
docs/migration-to-wa-sqlite.md

@ -0,0 +1,554 @@
# Migration Guide: Dexie to wa-sqlite
## Overview
This document outlines the migration process from Dexie.js to wa-sqlite for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.
## Migration Goals
1. **Data Integrity**
- Preserve all existing data
- Maintain data relationships
- Ensure data consistency
2. **Performance**
- Improve query performance
- Reduce storage overhead
- Optimize for platform-specific features
3. **Security**
- Maintain or improve encryption
- Preserve access controls
- Enhance data protection
4. **User Experience**
- Zero data loss
- Minimal downtime
- Automatic migration where possible
## Prerequisites
1. **Backup Requirements**
```typescript
interface MigrationBackup {
timestamp: number;
accounts: Account[];
settings: Setting[];
contacts: Contact[];
metadata: {
version: string;
platform: string;
dexieVersion: string;
};
}
```
2. **Storage Requirements**
- Sufficient IndexedDB quota
- Available disk space for SQLite
- Backup storage space
3. **Platform Support**
- Web: Modern browser with IndexedDB support
- iOS: iOS 13+ with SQLite support
- Android: Android 5+ with SQLite support
- Electron: Latest version with SQLite support
## Migration Process
### 1. Preparation
```typescript
// src/services/storage/migration/MigrationService.ts
export class MigrationService {
private static instance: MigrationService;
private backup: MigrationBackup | null = null;
async prepare(): Promise<void> {
try {
// 1. Check prerequisites
await this.checkPrerequisites();
// 2. Create backup
this.backup = await this.createBackup();
// 3. Verify backup integrity
await this.verifyBackup();
// 4. Initialize wa-sqlite
await this.initializeWaSqlite();
} catch (error) {
throw new StorageError(
'Migration preparation failed',
StorageErrorCodes.MIGRATION_FAILED,
error
);
}
}
private async checkPrerequisites(): Promise<void> {
// Check IndexedDB availability
if (!window.indexedDB) {
throw new StorageError(
'IndexedDB not available',
StorageErrorCodes.INITIALIZATION_FAILED
);
}
// Check storage quota
const quota = await navigator.storage.estimate();
if (quota.quota && quota.usage && quota.usage > quota.quota * 0.9) {
throw new StorageError(
'Insufficient storage space',
StorageErrorCodes.STORAGE_FULL
);
}
// Check platform support
const capabilities = await PlatformDetection.getCapabilities();
if (!capabilities.hasFileSystem) {
throw new StorageError(
'Platform does not support required features',
StorageErrorCodes.INITIALIZATION_FAILED
);
}
}
private async createBackup(): Promise<MigrationBackup> {
const dexieDB = new Dexie('TimeSafariDB');
return {
timestamp: Date.now(),
accounts: await dexieDB.accounts.toArray(),
settings: await dexieDB.settings.toArray(),
contacts: await dexieDB.contacts.toArray(),
metadata: {
version: '1.0.0',
platform: await PlatformDetection.getPlatform(),
dexieVersion: Dexie.version
}
};
}
}
```
### 2. Data Migration
```typescript
// src/services/storage/migration/DataMigration.ts
export class DataMigration {
async migrate(backup: MigrationBackup): Promise<void> {
try {
// 1. Create new database schema
await this.createSchema();
// 2. Migrate accounts
await this.migrateAccounts(backup.accounts);
// 3. Migrate settings
await this.migrateSettings(backup.settings);
// 4. Migrate contacts
await this.migrateContacts(backup.contacts);
// 5. Verify migration
await this.verifyMigration(backup);
} catch (error) {
// 6. Handle failure
await this.handleMigrationFailure(error, backup);
}
}
private async migrateAccounts(accounts: Account[]): Promise<void> {
const db = await this.getWaSqliteConnection();
// Use transaction for atomicity
await db.transaction(async (tx) => {
for (const account of accounts) {
await tx.execute(`
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
VALUES (?, ?, ?, ?)
`, [
account.did,
account.publicKeyHex,
account.createdAt,
account.updatedAt
]);
}
});
}
private async verifyMigration(backup: MigrationBackup): Promise<void> {
const db = await this.getWaSqliteConnection();
// Verify account count
const accountCount = await db.selectValue(
'SELECT COUNT(*) FROM accounts'
);
if (accountCount !== backup.accounts.length) {
throw new StorageError(
'Account count mismatch',
StorageErrorCodes.VERIFICATION_FAILED
);
}
// Verify data integrity
await this.verifyDataIntegrity(backup);
}
}
```
### 3. Rollback Strategy
```typescript
// src/services/storage/migration/RollbackService.ts
export class RollbackService {
async rollback(backup: MigrationBackup): Promise<void> {
try {
// 1. Stop all database operations
await this.stopDatabaseOperations();
// 2. Restore from backup
await this.restoreFromBackup(backup);
// 3. Verify restoration
await this.verifyRestoration(backup);
// 4. Clean up wa-sqlite
await this.cleanupWaSqlite();
} catch (error) {
throw new StorageError(
'Rollback failed',
StorageErrorCodes.ROLLBACK_FAILED,
error
);
}
}
private async restoreFromBackup(backup: MigrationBackup): Promise<void> {
const dexieDB = new Dexie('TimeSafariDB');
// Restore accounts
await dexieDB.accounts.bulkPut(backup.accounts);
// Restore settings
await dexieDB.settings.bulkPut(backup.settings);
// Restore contacts
await dexieDB.contacts.bulkPut(backup.contacts);
}
}
```
## Migration UI
```vue
<!-- src/components/MigrationProgress.vue -->
<template>
<div class="migration-progress">
<h2>Database Migration</h2>
<div class="progress-container">
<div class="progress-bar" :style="{ width: `${progress}%` }" />
<div class="progress-text">{{ progress }}%</div>
</div>
<div class="status-message">{{ statusMessage }}</div>
<div v-if="error" class="error-message">
{{ error }}
<button @click="retryMigration">Retry</button>
</div>
</div>
</template>
<script setup lang="ts">
import { ref, onMounted } from 'vue';
import { MigrationService } from '@/services/storage/migration/MigrationService';
const progress = ref(0);
const statusMessage = ref('Preparing migration...');
const error = ref<string | null>(null);
const migrationService = MigrationService.getInstance();
async function startMigration() {
try {
// 1. Preparation
statusMessage.value = 'Creating backup...';
await migrationService.prepare();
progress.value = 20;
// 2. Data migration
statusMessage.value = 'Migrating data...';
await migrationService.migrate();
progress.value = 80;
// 3. Verification
statusMessage.value = 'Verifying migration...';
await migrationService.verify();
progress.value = 100;
statusMessage.value = 'Migration completed successfully!';
} catch (err) {
error.value = err instanceof Error ? err.message : 'Migration failed';
statusMessage.value = 'Migration failed';
}
}
async function retryMigration() {
error.value = null;
progress.value = 0;
await startMigration();
}
onMounted(() => {
startMigration();
});
</script>
<style scoped>
.migration-progress {
padding: 2rem;
max-width: 600px;
margin: 0 auto;
}
.progress-container {
position: relative;
height: 20px;
background: #eee;
border-radius: 10px;
overflow: hidden;
margin: 1rem 0;
}
.progress-bar {
position: absolute;
height: 100%;
background: #4CAF50;
transition: width 0.3s ease;
}
.progress-text {
position: absolute;
width: 100%;
text-align: center;
line-height: 20px;
color: #000;
}
.status-message {
text-align: center;
margin: 1rem 0;
}
.error-message {
color: #f44336;
text-align: center;
margin: 1rem 0;
}
button {
margin-top: 1rem;
padding: 0.5rem 1rem;
background: #2196F3;
color: white;
border: none;
border-radius: 4px;
cursor: pointer;
}
button:hover {
background: #1976D2;
}
</style>
```
## Testing Strategy
1. **Unit Tests**
```typescript
// src/services/storage/migration/__tests__/MigrationService.spec.ts
describe('MigrationService', () => {
it('should create valid backup', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
expect(backup).toBeDefined();
expect(backup.accounts).toBeInstanceOf(Array);
expect(backup.settings).toBeInstanceOf(Array);
expect(backup.contacts).toBeInstanceOf(Array);
});
it('should migrate data correctly', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
await service.migrate(backup);
// Verify migration
const accounts = await service.getMigratedAccounts();
expect(accounts).toHaveLength(backup.accounts.length);
});
it('should handle rollback correctly', async () => {
const service = MigrationService.getInstance();
const backup = await service.createBackup();
// Simulate failed migration
await service.migrate(backup);
await service.simulateFailure();
// Perform rollback
await service.rollback(backup);
// Verify rollback
const accounts = await service.getOriginalAccounts();
expect(accounts).toHaveLength(backup.accounts.length);
});
});
```
2. **Integration Tests**
```typescript
// src/services/storage/migration/__tests__/integration/Migration.spec.ts
describe('Migration Integration', () => {
it('should handle concurrent access during migration', async () => {
const service = MigrationService.getInstance();
// Start migration
const migrationPromise = service.migrate();
// Simulate concurrent access
const accessPromises = Array(5).fill(null).map(() =>
service.getAccount('did:test:123')
);
// Wait for all operations
const [migrationResult, ...accessResults] = await Promise.allSettled([
migrationPromise,
...accessPromises
]);
// Verify results
expect(migrationResult.status).toBe('fulfilled');
expect(accessResults.some(r => r.status === 'rejected')).toBe(true);
});
it('should maintain data integrity during platform transition', async () => {
const service = MigrationService.getInstance();
// Simulate platform change
await service.simulatePlatformChange();
// Verify data
const accounts = await service.getAllAccounts();
const settings = await service.getAllSettings();
const contacts = await service.getAllContacts();
expect(accounts).toBeDefined();
expect(settings).toBeDefined();
expect(contacts).toBeDefined();
});
});
```
## Success Criteria
1. **Data Integrity**
- [ ] All accounts migrated successfully
- [ ] All settings preserved
- [ ] All contacts transferred
- [ ] No data corruption
2. **Performance**
- [ ] Migration completes within acceptable time
- [ ] No significant performance degradation
- [ ] Efficient storage usage
- [ ] Smooth user experience
3. **Security**
- [ ] Encrypted data remains secure
- [ ] Access controls maintained
- [ ] No sensitive data exposure
- [ ] Secure backup process
4. **User Experience**
- [ ] Clear migration progress
- [ ] Informative error messages
- [ ] Automatic recovery from failures
- [ ] No data loss
## Rollback Plan
1. **Automatic Rollback**
- Triggered by migration failure
- Restores from verified backup
- Maintains data consistency
- Logs rollback reason
2. **Manual Rollback**
- Available through settings
- Requires user confirmation
- Preserves backup data
- Provides rollback status
3. **Emergency Recovery**
- Manual backup restoration
- Database repair tools
- Data recovery procedures
- Support contact information
## Post-Migration
1. **Verification**
- Data integrity checks
- Performance monitoring
- Error rate tracking
- User feedback collection
2. **Cleanup**
- Remove old database
- Clear migration artifacts
- Update application state
- Archive backup data
3. **Monitoring**
- Track migration success rate
- Monitor performance metrics
- Collect error reports
- Gather user feedback
## Support
For assistance with migration:
1. Check the troubleshooting guide
2. Review error logs
3. Contact support team
4. Submit issue report
## Timeline
1. **Preparation Phase** (1 week)
- Backup system implementation
- Migration service development
- Testing framework setup
2. **Testing Phase** (2 weeks)
- Unit testing
- Integration testing
- Performance testing
- Security testing
3. **Deployment Phase** (1 week)
- Staged rollout
- Monitoring
- Support preparation
- Documentation updates
4. **Post-Deployment** (2 weeks)
- Monitoring
- Bug fixes
- Performance optimization
- User feedback collection
Loading…
Cancel
Save