diff --git a/.cursor/rules/absurd-sql.mdc b/.cursor/rules/absurd-sql.mdc new file mode 100644 index 00000000..56729c2a --- /dev/null +++ b/.cursor/rules/absurd-sql.mdc @@ -0,0 +1,153 @@ +--- +description: +globs: +alwaysApply: true +--- +# Absurd SQL - Cursor Development Guide + +## Project Overview +Absurd SQL is a backend implementation for sql.js that enables persistent SQLite databases in the browser by using IndexedDB as a block storage system. This guide provides rules and best practices for developing with this project in Cursor. + +## Project Structure +``` +absurd-sql/ +├── src/ # Source code +├── dist/ # Built files +├── package.json # Dependencies and scripts +├── rollup.config.js # Build configuration +└── jest.config.js # Test configuration +``` + +## Development Rules + +### 1. Worker Thread Requirements +- All SQL operations MUST be performed in a worker thread +- Main thread should only handle worker initialization and communication +- Never block the main thread with database operations + +### 2. Code Organization +- Keep worker code in separate files (e.g., `*.worker.js`) +- Use ES modules for imports/exports +- Follow the project's existing module structure + +### 3. Required Headers +When developing locally or deploying, ensure these headers are set: +``` +Cross-Origin-Opener-Policy: same-origin +Cross-Origin-Embedder-Policy: require-corp +``` + +### 4. Browser Compatibility +- Primary target: Modern browsers with SharedArrayBuffer support +- Fallback mode: Safari (with limitations) +- Always test in both modes + +### 5. Database Configuration +Recommended database settings: +```sql +PRAGMA journal_mode=MEMORY; +PRAGMA page_size=8192; -- Optional, but recommended +``` + +### 6. Development Workflow +1. Install dependencies: + ```bash + yarn add @jlongster/sql.js absurd-sql + ``` + +2. Development commands: + - `yarn build` - Build the project + - `yarn jest` - Run tests + - `yarn serve` - Start development server + +### 7. Testing Guidelines +- Write tests for both SharedArrayBuffer and fallback modes +- Use Jest for testing +- Include performance benchmarks for critical operations + +### 8. Performance Considerations +- Use bulk operations when possible +- Monitor read/write performance +- Consider using transactions for multiple operations +- Avoid unnecessary database connections + +### 9. Error Handling +- Implement proper error handling for: + - Worker initialization failures + - Database connection issues + - Concurrent access conflicts (in fallback mode) + - Storage quota exceeded scenarios + +### 10. Security Best Practices +- Never expose database operations directly to the client +- Validate all SQL queries +- Implement proper access controls +- Handle sensitive data appropriately + +### 11. Code Style +- Follow ESLint configuration +- Use async/await for asynchronous operations +- Document complex database operations +- Include comments for non-obvious optimizations + +### 12. Debugging +- Use `jest-debug` for debugging tests +- Monitor IndexedDB usage in browser dev tools +- Check worker communication in console +- Use performance monitoring tools + +## Common Patterns + +### Worker Initialization +```javascript +// Main thread +import { initBackend } from 'absurd-sql/dist/indexeddb-main-thread'; + +function init() { + let worker = new Worker(new URL('./index.worker.js', import.meta.url)); + initBackend(worker); +} +``` + +### Database Setup +```javascript +// Worker thread +import initSqlJs from '@jlongster/sql.js'; +import { SQLiteFS } from 'absurd-sql'; +import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend'; + +async function setupDatabase() { + let SQL = await initSqlJs({ locateFile: file => file }); + let sqlFS = new SQLiteFS(SQL.FS, new IndexedDBBackend()); + SQL.register_for_idb(sqlFS); + + SQL.FS.mkdir('/sql'); + SQL.FS.mount(sqlFS, {}, '/sql'); + + return new SQL.Database('/sql/db.sqlite', { filename: true }); +} +``` + +## Troubleshooting + +### Common Issues +1. SharedArrayBuffer not available + - Check COOP/COEP headers + - Verify browser support + - Test fallback mode + +2. Worker initialization failures + - Check file paths + - Verify module imports + - Check browser console for errors + +3. Performance issues + - Monitor IndexedDB usage + - Check for unnecessary operations + - Verify transaction usage + +## Resources +- [Project Demo](https://priceless-keller-d097e5.netlify.app/) +- [Example Project](https://github.com/jlongster/absurd-example-project) +- [Blog Post](https://jlongster.com/future-sql-web) +- [SQL.js Documentation](https://github.com/sql-js/sql.js/) \ No newline at end of file diff --git a/docs/dexie-to-sqlite-mapping.md b/doc/dexie-to-sqlite-mapping.md similarity index 76% rename from docs/dexie-to-sqlite-mapping.md rename to doc/dexie-to-sqlite-mapping.md index 8ffddc90..893b4670 100644 --- a/docs/dexie-to-sqlite-mapping.md +++ b/doc/dexie-to-sqlite-mapping.md @@ -1,4 +1,4 @@ -# Dexie to SQLite Mapping Guide +# Dexie to absurd-sql Mapping Guide ## Schema Mapping @@ -54,10 +54,11 @@ CREATE INDEX idx_settings_updated_at ON settings(updated_at); // Dexie const account = await db.accounts.get(did); -// SQLite -const account = await db.selectOne(` +// absurd-sql +const result = await db.exec(` SELECT * FROM accounts WHERE did = ? `, [did]); +const account = result[0]?.values[0]; ``` #### Get All Accounts @@ -65,10 +66,11 @@ const account = await db.selectOne(` // Dexie const accounts = await db.accounts.toArray(); -// SQLite -const accounts = await db.selectAll(` +// absurd-sql +const result = await db.exec(` SELECT * FROM accounts ORDER BY created_at DESC `); +const accounts = result[0]?.values || []; ``` #### Add Account @@ -81,8 +83,8 @@ await db.accounts.add({ updatedAt: Date.now() }); -// SQLite -await db.execute(` +// absurd-sql +await db.run(` INSERT INTO accounts (did, public_key_hex, created_at, updated_at) VALUES (?, ?, ?, ?) `, [did, publicKeyHex, Date.now(), Date.now()]); @@ -96,8 +98,8 @@ await db.accounts.update(did, { updatedAt: Date.now() }); -// SQLite -await db.execute(` +// absurd-sql +await db.run(` UPDATE accounts SET public_key_hex = ?, updated_at = ? WHERE did = ? @@ -111,10 +113,11 @@ await db.execute(` // Dexie const setting = await db.settings.get(key); -// SQLite -const setting = await db.selectOne(` +// absurd-sql +const result = await db.exec(` SELECT * FROM settings WHERE key = ? `, [key]); +const setting = result[0]?.values[0]; ``` #### Set Setting @@ -126,8 +129,8 @@ await db.settings.put({ updatedAt: Date.now() }); -// SQLite -await db.execute(` +// absurd-sql +await db.run(` INSERT INTO settings (key, value, updated_at) VALUES (?, ?, ?) ON CONFLICT(key) DO UPDATE SET @@ -146,12 +149,13 @@ const contacts = await db.contacts .equals(accountDid) .toArray(); -// SQLite -const contacts = await db.selectAll(` +// absurd-sql +const result = await db.exec(` SELECT * FROM contacts WHERE did = ? ORDER BY created_at DESC `, [accountDid]); +const contacts = result[0]?.values || []; ``` #### Add Contact @@ -165,8 +169,8 @@ await db.contacts.add({ updatedAt: Date.now() }); -// SQLite -await db.execute(` +// absurd-sql +await db.run(` INSERT INTO contacts (id, did, name, created_at, updated_at) VALUES (?, ?, ?, ?, ?) `, [generateId(), accountDid, name, Date.now(), Date.now()]); @@ -182,20 +186,25 @@ await db.transaction('rw', [db.accounts, db.contacts], async () => { await db.contacts.bulkAdd(contacts); }); -// SQLite -await db.transaction(async (tx) => { - await tx.execute(` +// absurd-sql +await db.exec('BEGIN TRANSACTION;'); +try { + await db.run(` INSERT INTO accounts (did, public_key_hex, created_at, updated_at) VALUES (?, ?, ?, ?) `, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]); for (const contact of contacts) { - await tx.execute(` + await db.run(` INSERT INTO contacts (id, did, name, created_at, updated_at) VALUES (?, ?, ?, ?, ?) `, [contact.id, contact.did, contact.name, contact.createdAt, contact.updatedAt]); } -}); + await db.exec('COMMIT;'); +} catch (error) { + await db.exec('ROLLBACK;'); + throw error; +} ``` ## Migration Helper Functions @@ -218,15 +227,14 @@ async function exportDexieData(): Promise { } ``` -### 2. Data Import (JSON to SQLite) +### 2. Data Import (JSON to absurd-sql) ```typescript -async function importToSQLite(data: MigrationData): Promise { - const db = await getSQLiteConnection(); - - await db.transaction(async (tx) => { +async function importToAbsurdSql(data: MigrationData): Promise { + await db.exec('BEGIN TRANSACTION;'); + try { // Import accounts for (const account of data.accounts) { - await tx.execute(` + await db.run(` INSERT INTO accounts (did, public_key_hex, created_at, updated_at) VALUES (?, ?, ?, ?) `, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]); @@ -234,7 +242,7 @@ async function importToSQLite(data: MigrationData): Promise { // Import settings for (const setting of data.settings) { - await tx.execute(` + await db.run(` INSERT INTO settings (key, value, updated_at) VALUES (?, ?, ?) `, [setting.key, setting.value, setting.updatedAt]); @@ -242,52 +250,52 @@ async function importToSQLite(data: MigrationData): Promise { // Import contacts for (const contact of data.contacts) { - await tx.execute(` + await db.run(` INSERT INTO contacts (id, did, name, created_at, updated_at) VALUES (?, ?, ?, ?, ?) `, [contact.id, contact.did, contact.name, contact.createdAt, contact.updatedAt]); } - }); + await db.exec('COMMIT;'); + } catch (error) { + await db.exec('ROLLBACK;'); + throw error; + } } ``` ### 3. Verification ```typescript async function verifyMigration(dexieData: MigrationData): Promise { - const db = await getSQLiteConnection(); - // Verify account count - const accountCount = await db.selectValue( - 'SELECT COUNT(*) FROM accounts' - ); + const accountResult = await db.exec('SELECT COUNT(*) as count FROM accounts'); + const accountCount = accountResult[0].values[0][0]; if (accountCount !== dexieData.accounts.length) { return false; } // Verify settings count - const settingsCount = await db.selectValue( - 'SELECT COUNT(*) FROM settings' - ); + const settingsResult = await db.exec('SELECT COUNT(*) as count FROM settings'); + const settingsCount = settingsResult[0].values[0][0]; if (settingsCount !== dexieData.settings.length) { return false; } // Verify contacts count - const contactsCount = await db.selectValue( - 'SELECT COUNT(*) FROM contacts' - ); + const contactsResult = await db.exec('SELECT COUNT(*) as count FROM contacts'); + const contactsCount = contactsResult[0].values[0][0]; if (contactsCount !== dexieData.contacts.length) { return false; } // Verify data integrity for (const account of dexieData.accounts) { - const migratedAccount = await db.selectOne( + const result = await db.exec( 'SELECT * FROM accounts WHERE did = ?', [account.did] ); + const migratedAccount = result[0]?.values[0]; if (!migratedAccount || - migratedAccount.public_key_hex !== account.publicKeyHex) { + migratedAccount[1] !== account.publicKeyHex) { // public_key_hex is second column return false; } } @@ -300,18 +308,21 @@ async function verifyMigration(dexieData: MigrationData): Promise { ### 1. Indexing - Dexie automatically creates indexes based on the schema -- SQLite requires explicit index creation +- absurd-sql requires explicit index creation - Added indexes for frequently queried fields +- Use `PRAGMA journal_mode=MEMORY;` for better performance ### 2. Batch Operations - Dexie has built-in bulk operations -- SQLite uses transactions for batch operations +- absurd-sql uses transactions for batch operations - Consider chunking large datasets +- Use prepared statements for repeated queries ### 3. Query Optimization - Dexie uses IndexedDB's native indexing -- SQLite requires explicit query optimization +- absurd-sql requires explicit query optimization - Use prepared statements for repeated queries +- Consider using `PRAGMA synchronous=NORMAL;` for better performance ## Error Handling @@ -326,14 +337,14 @@ try { } } -// SQLite errors +// absurd-sql errors try { - await db.execute(` + await db.run(` INSERT INTO accounts (did, public_key_hex, created_at, updated_at) VALUES (?, ?, ?, ?) `, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]); } catch (error) { - if (error.code === 'SQLITE_CONSTRAINT') { + if (error.message.includes('UNIQUE constraint failed')) { // Handle duplicate key } } @@ -350,15 +361,14 @@ try { // Dexie automatically rolls back } -// SQLite transaction -const db = await getSQLiteConnection(); +// absurd-sql transaction try { - await db.transaction(async (tx) => { - // Operations - }); + await db.exec('BEGIN TRANSACTION;'); + // Operations + await db.exec('COMMIT;'); } catch (error) { - // SQLite automatically rolls back - await db.execute('ROLLBACK'); + await db.exec('ROLLBACK;'); + throw error; } ``` diff --git a/docs/migration-to-wa-sqlite.md b/doc/migration-to-wa-sqlite.md similarity index 83% rename from docs/migration-to-wa-sqlite.md rename to doc/migration-to-wa-sqlite.md index b837c00e..627c112d 100644 --- a/docs/migration-to-wa-sqlite.md +++ b/doc/migration-to-wa-sqlite.md @@ -1,8 +1,8 @@ -# Migration Guide: Dexie to wa-sqlite +# Migration Guide: Dexie to absurd-sql ## Overview -This document outlines the migration process from Dexie.js to wa-sqlite for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users. +This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users. ## Migration Goals @@ -43,12 +43,20 @@ This document outlines the migration process from Dexie.js to wa-sqlite for the } ``` -2. **Storage Requirements** +2. **Dependencies** + ```json + { + "@jlongster/sql.js": "^1.8.0", + "absurd-sql": "^1.8.0" + } + ``` + +3. **Storage Requirements** - Sufficient IndexedDB quota - Available disk space for SQLite - Backup storage space -3. **Platform Support** +4. **Platform Support** - Web: Modern browser with IndexedDB support - iOS: iOS 13+ with SQLite support - Android: Android 5+ with SQLite support @@ -60,9 +68,15 @@ This document outlines the migration process from Dexie.js to wa-sqlite for the ```typescript // src/services/storage/migration/MigrationService.ts +import initSqlJs from '@jlongster/sql.js'; +import { SQLiteFS } from 'absurd-sql'; +import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend'; + export class MigrationService { private static instance: MigrationService; private backup: MigrationBackup | null = null; + private sql: any = null; + private db: any = null; async prepare(): Promise { try { @@ -75,8 +89,8 @@ export class MigrationService { // 3. Verify backup integrity await this.verifyBackup(); - // 4. Initialize wa-sqlite - await this.initializeWaSqlite(); + // 4. Initialize absurd-sql + await this.initializeAbsurdSql(); } catch (error) { throw new StorageError( 'Migration preparation failed', @@ -86,6 +100,42 @@ export class MigrationService { } } + private async initializeAbsurdSql(): Promise { + // Initialize SQL.js + this.sql = await initSqlJs({ + locateFile: (file: string) => { + return new URL(`/node_modules/@jlongster/sql.js/dist/${file}`, import.meta.url).href; + } + }); + + // Setup SQLiteFS with IndexedDB backend + const sqlFS = new SQLiteFS(this.sql.FS, new IndexedDBBackend()); + this.sql.register_for_idb(sqlFS); + + // Create and mount filesystem + this.sql.FS.mkdir('/sql'); + this.sql.FS.mount(sqlFS, {}, '/sql'); + + // Open database + const path = '/sql/db.sqlite'; + if (typeof SharedArrayBuffer === 'undefined') { + let stream = this.sql.FS.open(path, 'a+'); + await stream.node.contents.readIfFallback(); + this.sql.FS.close(stream); + } + + this.db = new this.sql.Database(path, { filename: true }); + if (!this.db) { + throw new StorageError( + 'Database initialization failed', + StorageErrorCodes.INITIALIZATION_FAILED + ); + } + + // Configure database + await this.db.exec(`PRAGMA journal_mode=MEMORY;`); + } + private async checkPrerequisites(): Promise { // Check IndexedDB availability if (!window.indexedDB) { @@ -160,12 +210,11 @@ export class DataMigration { } private async migrateAccounts(accounts: Account[]): Promise { - const db = await this.getWaSqliteConnection(); - // Use transaction for atomicity - await db.transaction(async (tx) => { + await this.db.exec('BEGIN TRANSACTION;'); + try { for (const account of accounts) { - await tx.execute(` + await this.db.run(` INSERT INTO accounts (did, public_key_hex, created_at, updated_at) VALUES (?, ?, ?, ?) `, [ @@ -175,16 +224,18 @@ export class DataMigration { account.updatedAt ]); } - }); + await this.db.exec('COMMIT;'); + } catch (error) { + await this.db.exec('ROLLBACK;'); + throw error; + } } private async verifyMigration(backup: MigrationBackup): Promise { - const db = await this.getWaSqliteConnection(); - // Verify account count - const accountCount = await db.selectValue( - 'SELECT COUNT(*) FROM accounts' - ); + const result = await this.db.exec('SELECT COUNT(*) as count FROM accounts'); + const accountCount = result[0].values[0][0]; + if (accountCount !== backup.accounts.length) { throw new StorageError( 'Account count mismatch', @@ -214,8 +265,8 @@ export class RollbackService { // 3. Verify restoration await this.verifyRestoration(backup); - // 4. Clean up wa-sqlite - await this.cleanupWaSqlite(); + // 4. Clean up absurd-sql + await this.cleanupAbsurdSql(); } catch (error) { throw new StorageError( 'Rollback failed', @@ -371,6 +422,14 @@ button:hover { ```typescript // src/services/storage/migration/__tests__/MigrationService.spec.ts describe('MigrationService', () => { + it('should initialize absurd-sql correctly', async () => { + const service = MigrationService.getInstance(); + await service.initializeAbsurdSql(); + + expect(service.isInitialized()).toBe(true); + expect(service.getDatabase()).toBeDefined(); + }); + it('should create valid backup', async () => { const service = MigrationService.getInstance(); const backup = await service.createBackup(); diff --git a/docs/secure-storage-implementation.md b/doc/secure-storage-implementation.md similarity index 100% rename from docs/secure-storage-implementation.md rename to doc/secure-storage-implementation.md diff --git a/docs/storage-implementation-checklist.md b/doc/storage-implementation-checklist.md similarity index 88% rename from docs/storage-implementation-checklist.md rename to doc/storage-implementation-checklist.md index b608dfcf..cfcda815 100644 --- a/docs/storage-implementation-checklist.md +++ b/doc/storage-implementation-checklist.md @@ -10,9 +10,9 @@ - [ ] Add migration support methods - [ ] Implement platform-specific services - - [ ] `WebSQLiteService` (wa-sqlite) + - [ ] `WebSQLiteService` (absurd-sql) - [ ] Database initialization - - [ ] VFS setup + - [ ] VFS setup with IndexedDB backend - [ ] Connection management - [ ] Query builder - [ ] `NativeSQLiteService` (iOS/Android) @@ -49,17 +49,24 @@ ## Platform-Specific Implementation ### Web Platform -- [ ] Setup wa-sqlite +- [ ] Setup absurd-sql - [ ] Install dependencies ```json { - "@wa-sqlite/sql.js": "^0.8.12", - "@wa-sqlite/sql.js-httpvfs": "^0.8.12" + "@jlongster/sql.js": "^1.8.0", + "absurd-sql": "^1.8.0" } ``` - - [ ] Configure VFS + - [ ] Configure VFS with IndexedDB backend - [ ] Setup worker threads - [ ] Implement connection pooling + - [ ] Configure database pragmas + ```sql + PRAGMA journal_mode=MEMORY; + PRAGMA synchronous=NORMAL; + PRAGMA foreign_keys=ON; + PRAGMA busy_timeout=5000; + ``` - [ ] Update build configuration - [ ] Modify `vite.config.ts` @@ -71,6 +78,7 @@ - [ ] Create fallback service - [ ] Add data synchronization - [ ] Handle quota exceeded + - [ ] Implement atomic operations ### iOS Platform - [ ] Setup SQLCipher @@ -140,6 +148,11 @@ updated_at INTEGER NOT NULL, FOREIGN KEY (did) REFERENCES accounts(did) ); + + -- Indexes for performance + CREATE INDEX idx_accounts_created_at ON accounts(created_at); + CREATE INDEX idx_contacts_did ON contacts(did); + CREATE INDEX idx_settings_updated_at ON settings(updated_at); ``` - [ ] Create indexes @@ -286,12 +299,16 @@ - [ ] Migration time < 5s per 1000 records - [ ] Storage overhead < 10% - [ ] Memory usage < 50MB +- [ ] Atomic operations complete successfully +- [ ] Transaction performance meets requirements ### 2. Reliability - [ ] 99.9% uptime - [ ] Zero data loss - [ ] Automatic recovery - [ ] Backup verification +- [ ] Transaction atomicity +- [ ] Data consistency ### 3. Security - [ ] AES-256 encryption