feat(db): Implement SQLite database layer with migration support

Add SQLite database implementation with comprehensive features:

- Core database functionality:
  - Connection management and pooling
  - Schema creation and validation
  - Transaction support with rollback
  - Backup and restore capabilities
  - Health checks and integrity verification

- Data migration:
  - Migration utilities from Dexie to SQLite
  - Data transformation and validation
  - Migration verification and rollback
  - Backup before migration

- CRUD operations for all entities:
  - Accounts, contacts, and contact methods
  - Settings and secrets
  - Logging and audit trails

- Type safety and error handling:
  - Full TypeScript type definitions
  - Runtime data validation
  - Comprehensive error handling
  - Transaction safety

Note: Requires @wa-sqlite/sql.js package to be installed
This commit is contained in:
Matt Raymer
2025-05-25 04:52:16 -04:00
parent 861ed9dd2d
commit a903a6d664
8 changed files with 2160 additions and 244 deletions

View File

@@ -0,0 +1,389 @@
# Dexie to SQLite Mapping Guide
## Schema Mapping
### Current Dexie Schema
```typescript
// Current Dexie schema
const db = new Dexie('TimeSafariDB');
db.version(1).stores({
accounts: 'did, publicKeyHex, createdAt, updatedAt',
settings: 'key, value, updatedAt',
contacts: 'id, did, name, createdAt, updatedAt'
});
```
### New SQLite Schema
```sql
-- New SQLite schema
CREATE TABLE accounts (
did TEXT PRIMARY KEY,
public_key_hex TEXT NOT NULL,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
);
CREATE TABLE settings (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
updated_at INTEGER NOT NULL
);
CREATE TABLE contacts (
id TEXT PRIMARY KEY,
did TEXT NOT NULL,
name TEXT,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL,
FOREIGN KEY (did) REFERENCES accounts(did)
);
-- Indexes for performance
CREATE INDEX idx_accounts_created_at ON accounts(created_at);
CREATE INDEX idx_contacts_did ON contacts(did);
CREATE INDEX idx_settings_updated_at ON settings(updated_at);
```
## Query Mapping
### 1. Account Operations
#### Get Account by DID
```typescript
// Dexie
const account = await db.accounts.get(did);
// SQLite
const account = await db.selectOne(`
SELECT * FROM accounts WHERE did = ?
`, [did]);
```
#### Get All Accounts
```typescript
// Dexie
const accounts = await db.accounts.toArray();
// SQLite
const accounts = await db.selectAll(`
SELECT * FROM accounts ORDER BY created_at DESC
`);
```
#### Add Account
```typescript
// Dexie
await db.accounts.add({
did,
publicKeyHex,
createdAt: Date.now(),
updatedAt: Date.now()
});
// SQLite
await db.execute(`
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
VALUES (?, ?, ?, ?)
`, [did, publicKeyHex, Date.now(), Date.now()]);
```
#### Update Account
```typescript
// Dexie
await db.accounts.update(did, {
publicKeyHex,
updatedAt: Date.now()
});
// SQLite
await db.execute(`
UPDATE accounts
SET public_key_hex = ?, updated_at = ?
WHERE did = ?
`, [publicKeyHex, Date.now(), did]);
```
### 2. Settings Operations
#### Get Setting
```typescript
// Dexie
const setting = await db.settings.get(key);
// SQLite
const setting = await db.selectOne(`
SELECT * FROM settings WHERE key = ?
`, [key]);
```
#### Set Setting
```typescript
// Dexie
await db.settings.put({
key,
value,
updatedAt: Date.now()
});
// SQLite
await db.execute(`
INSERT INTO settings (key, value, updated_at)
VALUES (?, ?, ?)
ON CONFLICT(key) DO UPDATE SET
value = excluded.value,
updated_at = excluded.updated_at
`, [key, value, Date.now()]);
```
### 3. Contact Operations
#### Get Contacts by Account
```typescript
// Dexie
const contacts = await db.contacts
.where('did')
.equals(accountDid)
.toArray();
// SQLite
const contacts = await db.selectAll(`
SELECT * FROM contacts
WHERE did = ?
ORDER BY created_at DESC
`, [accountDid]);
```
#### Add Contact
```typescript
// Dexie
await db.contacts.add({
id: generateId(),
did: accountDid,
name,
createdAt: Date.now(),
updatedAt: Date.now()
});
// SQLite
await db.execute(`
INSERT INTO contacts (id, did, name, created_at, updated_at)
VALUES (?, ?, ?, ?, ?)
`, [generateId(), accountDid, name, Date.now(), Date.now()]);
```
## Transaction Mapping
### Batch Operations
```typescript
// Dexie
await db.transaction('rw', [db.accounts, db.contacts], async () => {
await db.accounts.add(account);
await db.contacts.bulkAdd(contacts);
});
// SQLite
await db.transaction(async (tx) => {
await tx.execute(`
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
VALUES (?, ?, ?, ?)
`, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]);
for (const contact of contacts) {
await tx.execute(`
INSERT INTO contacts (id, did, name, created_at, updated_at)
VALUES (?, ?, ?, ?, ?)
`, [contact.id, contact.did, contact.name, contact.createdAt, contact.updatedAt]);
}
});
```
## Migration Helper Functions
### 1. Data Export (Dexie to JSON)
```typescript
async function exportDexieData(): Promise<MigrationData> {
const db = new Dexie('TimeSafariDB');
return {
accounts: await db.accounts.toArray(),
settings: await db.settings.toArray(),
contacts: await db.contacts.toArray(),
metadata: {
version: '1.0.0',
timestamp: Date.now(),
dexieVersion: Dexie.version
}
};
}
```
### 2. Data Import (JSON to SQLite)
```typescript
async function importToSQLite(data: MigrationData): Promise<void> {
const db = await getSQLiteConnection();
await db.transaction(async (tx) => {
// Import accounts
for (const account of data.accounts) {
await tx.execute(`
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
VALUES (?, ?, ?, ?)
`, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]);
}
// Import settings
for (const setting of data.settings) {
await tx.execute(`
INSERT INTO settings (key, value, updated_at)
VALUES (?, ?, ?)
`, [setting.key, setting.value, setting.updatedAt]);
}
// Import contacts
for (const contact of data.contacts) {
await tx.execute(`
INSERT INTO contacts (id, did, name, created_at, updated_at)
VALUES (?, ?, ?, ?, ?)
`, [contact.id, contact.did, contact.name, contact.createdAt, contact.updatedAt]);
}
});
}
```
### 3. Verification
```typescript
async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
const db = await getSQLiteConnection();
// Verify account count
const accountCount = await db.selectValue(
'SELECT COUNT(*) FROM accounts'
);
if (accountCount !== dexieData.accounts.length) {
return false;
}
// Verify settings count
const settingsCount = await db.selectValue(
'SELECT COUNT(*) FROM settings'
);
if (settingsCount !== dexieData.settings.length) {
return false;
}
// Verify contacts count
const contactsCount = await db.selectValue(
'SELECT COUNT(*) FROM contacts'
);
if (contactsCount !== dexieData.contacts.length) {
return false;
}
// Verify data integrity
for (const account of dexieData.accounts) {
const migratedAccount = await db.selectOne(
'SELECT * FROM accounts WHERE did = ?',
[account.did]
);
if (!migratedAccount ||
migratedAccount.public_key_hex !== account.publicKeyHex) {
return false;
}
}
return true;
}
```
## Performance Considerations
### 1. Indexing
- Dexie automatically creates indexes based on the schema
- SQLite requires explicit index creation
- Added indexes for frequently queried fields
### 2. Batch Operations
- Dexie has built-in bulk operations
- SQLite uses transactions for batch operations
- Consider chunking large datasets
### 3. Query Optimization
- Dexie uses IndexedDB's native indexing
- SQLite requires explicit query optimization
- Use prepared statements for repeated queries
## Error Handling
### 1. Common Errors
```typescript
// Dexie errors
try {
await db.accounts.add(account);
} catch (error) {
if (error instanceof Dexie.ConstraintError) {
// Handle duplicate key
}
}
// SQLite errors
try {
await db.execute(`
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
VALUES (?, ?, ?, ?)
`, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]);
} catch (error) {
if (error.code === 'SQLITE_CONSTRAINT') {
// Handle duplicate key
}
}
```
### 2. Transaction Recovery
```typescript
// Dexie transaction
try {
await db.transaction('rw', db.accounts, async () => {
// Operations
});
} catch (error) {
// Dexie automatically rolls back
}
// SQLite transaction
const db = await getSQLiteConnection();
try {
await db.transaction(async (tx) => {
// Operations
});
} catch (error) {
// SQLite automatically rolls back
await db.execute('ROLLBACK');
}
```
## Migration Strategy
1. **Preparation**
- Export all Dexie data
- Verify data integrity
- Create SQLite schema
- Setup indexes
2. **Migration**
- Import data in transactions
- Verify each batch
- Handle errors gracefully
- Maintain backup
3. **Verification**
- Compare record counts
- Verify data integrity
- Test common queries
- Validate relationships
4. **Cleanup**
- Remove Dexie database
- Clear IndexedDB storage
- Update application code
- Remove old dependencies

View File

@@ -0,0 +1,306 @@
# Storage Implementation Checklist
## Core Services
### 1. Storage Service Layer
- [ ] Create base `StorageService` interface
- [ ] Define common methods for all platforms
- [ ] Add platform-specific method signatures
- [ ] Include error handling types
- [ ] Add migration support methods
- [ ] Implement platform-specific services
- [ ] `WebSQLiteService` (wa-sqlite)
- [ ] Database initialization
- [ ] VFS setup
- [ ] Connection management
- [ ] Query builder
- [ ] `NativeSQLiteService` (iOS/Android)
- [ ] SQLCipher integration
- [ ] Native bridge setup
- [ ] File system access
- [ ] `ElectronSQLiteService`
- [ ] Node SQLite integration
- [ ] IPC communication
- [ ] File system access
### 2. Migration Services
- [ ] Implement `MigrationService`
- [ ] Backup creation
- [ ] Data verification
- [ ] Rollback procedures
- [ ] Progress tracking
- [ ] Create `MigrationUI` components
- [ ] Progress indicators
- [ ] Error handling
- [ ] User notifications
- [ ] Manual triggers
### 3. Security Layer
- [ ] Implement `EncryptionService`
- [ ] Key management
- [ ] Encryption/decryption
- [ ] Secure storage
- [ ] Add `BiometricService`
- [ ] Platform detection
- [ ] Authentication flow
- [ ] Fallback mechanisms
## Platform-Specific Implementation
### Web Platform
- [ ] Setup wa-sqlite
- [ ] Install dependencies
```json
{
"@wa-sqlite/sql.js": "^0.8.12",
"@wa-sqlite/sql.js-httpvfs": "^0.8.12"
}
```
- [ ] Configure VFS
- [ ] Setup worker threads
- [ ] Implement connection pooling
- [ ] Update build configuration
- [ ] Modify `vite.config.ts`
- [ ] Add worker configuration
- [ ] Update chunk splitting
- [ ] Configure asset handling
- [ ] Implement IndexedDB fallback
- [ ] Create fallback service
- [ ] Add data synchronization
- [ ] Handle quota exceeded
### iOS Platform
- [ ] Setup SQLCipher
- [ ] Install pod dependencies
- [ ] Configure encryption
- [ ] Setup keychain access
- [ ] Implement secure storage
- [ ] Update Capacitor config
- [ ] Modify `capacitor.config.ts`
- [ ] Add iOS permissions
- [ ] Configure backup
- [ ] Setup app groups
### Android Platform
- [ ] Setup SQLCipher
- [ ] Add Gradle dependencies
- [ ] Configure encryption
- [ ] Setup keystore
- [ ] Implement secure storage
- [ ] Update Capacitor config
- [ ] Modify `capacitor.config.ts`
- [ ] Add Android permissions
- [ ] Configure backup
- [ ] Setup file provider
### Electron Platform
- [ ] Setup Node SQLite
- [ ] Install dependencies
- [ ] Configure IPC
- [ ] Setup file system access
- [ ] Implement secure storage
- [ ] Update Electron config
- [ ] Modify `electron.config.ts`
- [ ] Add security policies
- [ ] Configure file access
- [ ] Setup auto-updates
## Data Models and Types
### 1. Database Schema
- [ ] Define tables
```sql
-- Accounts table
CREATE TABLE accounts (
did TEXT PRIMARY KEY,
public_key_hex TEXT NOT NULL,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
);
-- Settings table
CREATE TABLE settings (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
updated_at INTEGER NOT NULL
);
-- Contacts table
CREATE TABLE contacts (
id TEXT PRIMARY KEY,
did TEXT NOT NULL,
name TEXT,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL,
FOREIGN KEY (did) REFERENCES accounts(did)
);
```
- [ ] Create indexes
- [ ] Define constraints
- [ ] Add triggers
- [ ] Setup migrations
### 2. Type Definitions
- [ ] Create interfaces
```typescript
interface Account {
did: string;
publicKeyHex: string;
createdAt: number;
updatedAt: number;
}
interface Setting {
key: string;
value: string;
updatedAt: number;
}
interface Contact {
id: string;
did: string;
name?: string;
createdAt: number;
updatedAt: number;
}
```
- [ ] Add validation
- [ ] Create DTOs
- [ ] Define enums
- [ ] Add type guards
## UI Components
### 1. Migration UI
- [ ] Create components
- [ ] `MigrationProgress.vue`
- [ ] `MigrationError.vue`
- [ ] `MigrationSettings.vue`
- [ ] `MigrationStatus.vue`
### 2. Settings UI
- [ ] Update components
- [ ] Add storage settings
- [ ] Add migration controls
- [ ] Add backup options
- [ ] Add security settings
### 3. Error Handling UI
- [ ] Create components
- [ ] `StorageError.vue`
- [ ] `QuotaExceeded.vue`
- [ ] `MigrationFailed.vue`
- [ ] `RecoveryOptions.vue`
## Testing
### 1. Unit Tests
- [ ] Test services
- [ ] Storage service tests
- [ ] Migration service tests
- [ ] Security service tests
- [ ] Platform detection tests
### 2. Integration Tests
- [ ] Test migrations
- [ ] Web platform tests
- [ ] iOS platform tests
- [ ] Android platform tests
- [ ] Electron platform tests
### 3. E2E Tests
- [ ] Test workflows
- [ ] Account management
- [ ] Settings management
- [ ] Contact management
- [ ] Migration process
## Documentation
### 1. Technical Documentation
- [ ] Update architecture docs
- [ ] Add API documentation
- [ ] Create migration guides
- [ ] Document security measures
### 2. User Documentation
- [ ] Update user guides
- [ ] Add troubleshooting guides
- [ ] Create FAQ
- [ ] Document new features
## Deployment
### 1. Build Process
- [ ] Update build scripts
- [ ] Add platform-specific builds
- [ ] Configure CI/CD
- [ ] Setup automated testing
### 2. Release Process
- [ ] Create release checklist
- [ ] Add version management
- [ ] Setup rollback procedures
- [ ] Configure monitoring
## Monitoring and Analytics
### 1. Error Tracking
- [ ] Setup error logging
- [ ] Add performance monitoring
- [ ] Configure alerts
- [ ] Create dashboards
### 2. Usage Analytics
- [ ] Add storage metrics
- [ ] Track migration success
- [ ] Monitor performance
- [ ] Collect user feedback
## Security Audit
### 1. Code Review
- [ ] Review encryption
- [ ] Check access controls
- [ ] Verify data handling
- [ ] Audit dependencies
### 2. Penetration Testing
- [ ] Test data access
- [ ] Verify encryption
- [ ] Check authentication
- [ ] Review permissions
## Success Criteria
### 1. Performance
- [ ] Query response time < 100ms
- [ ] Migration time < 5s per 1000 records
- [ ] Storage overhead < 10%
- [ ] Memory usage < 50MB
### 2. Reliability
- [ ] 99.9% uptime
- [ ] Zero data loss
- [ ] Automatic recovery
- [ ] Backup verification
### 3. Security
- [ ] AES-256 encryption
- [ ] Secure key storage
- [ ] Access control
- [ ] Audit logging
### 4. User Experience
- [ ] Smooth migration
- [ ] Clear error messages
- [ ] Progress indicators
- [ ] Recovery options