chore: update migration documents and move to new home
This commit is contained in:
153
.cursor/rules/absurd-sql.mdc
Normal file
153
.cursor/rules/absurd-sql.mdc
Normal file
@@ -0,0 +1,153 @@
|
|||||||
|
---
|
||||||
|
description:
|
||||||
|
globs:
|
||||||
|
alwaysApply: true
|
||||||
|
---
|
||||||
|
# Absurd SQL - Cursor Development Guide
|
||||||
|
|
||||||
|
## Project Overview
|
||||||
|
Absurd SQL is a backend implementation for sql.js that enables persistent SQLite databases in the browser by using IndexedDB as a block storage system. This guide provides rules and best practices for developing with this project in Cursor.
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
```
|
||||||
|
absurd-sql/
|
||||||
|
├── src/ # Source code
|
||||||
|
├── dist/ # Built files
|
||||||
|
├── package.json # Dependencies and scripts
|
||||||
|
├── rollup.config.js # Build configuration
|
||||||
|
└── jest.config.js # Test configuration
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development Rules
|
||||||
|
|
||||||
|
### 1. Worker Thread Requirements
|
||||||
|
- All SQL operations MUST be performed in a worker thread
|
||||||
|
- Main thread should only handle worker initialization and communication
|
||||||
|
- Never block the main thread with database operations
|
||||||
|
|
||||||
|
### 2. Code Organization
|
||||||
|
- Keep worker code in separate files (e.g., `*.worker.js`)
|
||||||
|
- Use ES modules for imports/exports
|
||||||
|
- Follow the project's existing module structure
|
||||||
|
|
||||||
|
### 3. Required Headers
|
||||||
|
When developing locally or deploying, ensure these headers are set:
|
||||||
|
```
|
||||||
|
Cross-Origin-Opener-Policy: same-origin
|
||||||
|
Cross-Origin-Embedder-Policy: require-corp
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Browser Compatibility
|
||||||
|
- Primary target: Modern browsers with SharedArrayBuffer support
|
||||||
|
- Fallback mode: Safari (with limitations)
|
||||||
|
- Always test in both modes
|
||||||
|
|
||||||
|
### 5. Database Configuration
|
||||||
|
Recommended database settings:
|
||||||
|
```sql
|
||||||
|
PRAGMA journal_mode=MEMORY;
|
||||||
|
PRAGMA page_size=8192; -- Optional, but recommended
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Development Workflow
|
||||||
|
1. Install dependencies:
|
||||||
|
```bash
|
||||||
|
yarn add @jlongster/sql.js absurd-sql
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Development commands:
|
||||||
|
- `yarn build` - Build the project
|
||||||
|
- `yarn jest` - Run tests
|
||||||
|
- `yarn serve` - Start development server
|
||||||
|
|
||||||
|
### 7. Testing Guidelines
|
||||||
|
- Write tests for both SharedArrayBuffer and fallback modes
|
||||||
|
- Use Jest for testing
|
||||||
|
- Include performance benchmarks for critical operations
|
||||||
|
|
||||||
|
### 8. Performance Considerations
|
||||||
|
- Use bulk operations when possible
|
||||||
|
- Monitor read/write performance
|
||||||
|
- Consider using transactions for multiple operations
|
||||||
|
- Avoid unnecessary database connections
|
||||||
|
|
||||||
|
### 9. Error Handling
|
||||||
|
- Implement proper error handling for:
|
||||||
|
- Worker initialization failures
|
||||||
|
- Database connection issues
|
||||||
|
- Concurrent access conflicts (in fallback mode)
|
||||||
|
- Storage quota exceeded scenarios
|
||||||
|
|
||||||
|
### 10. Security Best Practices
|
||||||
|
- Never expose database operations directly to the client
|
||||||
|
- Validate all SQL queries
|
||||||
|
- Implement proper access controls
|
||||||
|
- Handle sensitive data appropriately
|
||||||
|
|
||||||
|
### 11. Code Style
|
||||||
|
- Follow ESLint configuration
|
||||||
|
- Use async/await for asynchronous operations
|
||||||
|
- Document complex database operations
|
||||||
|
- Include comments for non-obvious optimizations
|
||||||
|
|
||||||
|
### 12. Debugging
|
||||||
|
- Use `jest-debug` for debugging tests
|
||||||
|
- Monitor IndexedDB usage in browser dev tools
|
||||||
|
- Check worker communication in console
|
||||||
|
- Use performance monitoring tools
|
||||||
|
|
||||||
|
## Common Patterns
|
||||||
|
|
||||||
|
### Worker Initialization
|
||||||
|
```javascript
|
||||||
|
// Main thread
|
||||||
|
import { initBackend } from 'absurd-sql/dist/indexeddb-main-thread';
|
||||||
|
|
||||||
|
function init() {
|
||||||
|
let worker = new Worker(new URL('./index.worker.js', import.meta.url));
|
||||||
|
initBackend(worker);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Database Setup
|
||||||
|
```javascript
|
||||||
|
// Worker thread
|
||||||
|
import initSqlJs from '@jlongster/sql.js';
|
||||||
|
import { SQLiteFS } from 'absurd-sql';
|
||||||
|
import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend';
|
||||||
|
|
||||||
|
async function setupDatabase() {
|
||||||
|
let SQL = await initSqlJs({ locateFile: file => file });
|
||||||
|
let sqlFS = new SQLiteFS(SQL.FS, new IndexedDBBackend());
|
||||||
|
SQL.register_for_idb(sqlFS);
|
||||||
|
|
||||||
|
SQL.FS.mkdir('/sql');
|
||||||
|
SQL.FS.mount(sqlFS, {}, '/sql');
|
||||||
|
|
||||||
|
return new SQL.Database('/sql/db.sqlite', { filename: true });
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
1. SharedArrayBuffer not available
|
||||||
|
- Check COOP/COEP headers
|
||||||
|
- Verify browser support
|
||||||
|
- Test fallback mode
|
||||||
|
|
||||||
|
2. Worker initialization failures
|
||||||
|
- Check file paths
|
||||||
|
- Verify module imports
|
||||||
|
- Check browser console for errors
|
||||||
|
|
||||||
|
3. Performance issues
|
||||||
|
- Monitor IndexedDB usage
|
||||||
|
- Check for unnecessary operations
|
||||||
|
- Verify transaction usage
|
||||||
|
|
||||||
|
## Resources
|
||||||
|
- [Project Demo](https://priceless-keller-d097e5.netlify.app/)
|
||||||
|
- [Example Project](https://github.com/jlongster/absurd-example-project)
|
||||||
|
- [Blog Post](https://jlongster.com/future-sql-web)
|
||||||
|
- [SQL.js Documentation](https://github.com/sql-js/sql.js/)
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
# Dexie to SQLite Mapping Guide
|
# Dexie to absurd-sql Mapping Guide
|
||||||
|
|
||||||
## Schema Mapping
|
## Schema Mapping
|
||||||
|
|
||||||
@@ -54,10 +54,11 @@ CREATE INDEX idx_settings_updated_at ON settings(updated_at);
|
|||||||
// Dexie
|
// Dexie
|
||||||
const account = await db.accounts.get(did);
|
const account = await db.accounts.get(did);
|
||||||
|
|
||||||
// SQLite
|
// absurd-sql
|
||||||
const account = await db.selectOne(`
|
const result = await db.exec(`
|
||||||
SELECT * FROM accounts WHERE did = ?
|
SELECT * FROM accounts WHERE did = ?
|
||||||
`, [did]);
|
`, [did]);
|
||||||
|
const account = result[0]?.values[0];
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Get All Accounts
|
#### Get All Accounts
|
||||||
@@ -65,10 +66,11 @@ const account = await db.selectOne(`
|
|||||||
// Dexie
|
// Dexie
|
||||||
const accounts = await db.accounts.toArray();
|
const accounts = await db.accounts.toArray();
|
||||||
|
|
||||||
// SQLite
|
// absurd-sql
|
||||||
const accounts = await db.selectAll(`
|
const result = await db.exec(`
|
||||||
SELECT * FROM accounts ORDER BY created_at DESC
|
SELECT * FROM accounts ORDER BY created_at DESC
|
||||||
`);
|
`);
|
||||||
|
const accounts = result[0]?.values || [];
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Add Account
|
#### Add Account
|
||||||
@@ -81,8 +83,8 @@ await db.accounts.add({
|
|||||||
updatedAt: Date.now()
|
updatedAt: Date.now()
|
||||||
});
|
});
|
||||||
|
|
||||||
// SQLite
|
// absurd-sql
|
||||||
await db.execute(`
|
await db.run(`
|
||||||
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
|
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
|
||||||
VALUES (?, ?, ?, ?)
|
VALUES (?, ?, ?, ?)
|
||||||
`, [did, publicKeyHex, Date.now(), Date.now()]);
|
`, [did, publicKeyHex, Date.now(), Date.now()]);
|
||||||
@@ -96,8 +98,8 @@ await db.accounts.update(did, {
|
|||||||
updatedAt: Date.now()
|
updatedAt: Date.now()
|
||||||
});
|
});
|
||||||
|
|
||||||
// SQLite
|
// absurd-sql
|
||||||
await db.execute(`
|
await db.run(`
|
||||||
UPDATE accounts
|
UPDATE accounts
|
||||||
SET public_key_hex = ?, updated_at = ?
|
SET public_key_hex = ?, updated_at = ?
|
||||||
WHERE did = ?
|
WHERE did = ?
|
||||||
@@ -111,10 +113,11 @@ await db.execute(`
|
|||||||
// Dexie
|
// Dexie
|
||||||
const setting = await db.settings.get(key);
|
const setting = await db.settings.get(key);
|
||||||
|
|
||||||
// SQLite
|
// absurd-sql
|
||||||
const setting = await db.selectOne(`
|
const result = await db.exec(`
|
||||||
SELECT * FROM settings WHERE key = ?
|
SELECT * FROM settings WHERE key = ?
|
||||||
`, [key]);
|
`, [key]);
|
||||||
|
const setting = result[0]?.values[0];
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Set Setting
|
#### Set Setting
|
||||||
@@ -126,8 +129,8 @@ await db.settings.put({
|
|||||||
updatedAt: Date.now()
|
updatedAt: Date.now()
|
||||||
});
|
});
|
||||||
|
|
||||||
// SQLite
|
// absurd-sql
|
||||||
await db.execute(`
|
await db.run(`
|
||||||
INSERT INTO settings (key, value, updated_at)
|
INSERT INTO settings (key, value, updated_at)
|
||||||
VALUES (?, ?, ?)
|
VALUES (?, ?, ?)
|
||||||
ON CONFLICT(key) DO UPDATE SET
|
ON CONFLICT(key) DO UPDATE SET
|
||||||
@@ -146,12 +149,13 @@ const contacts = await db.contacts
|
|||||||
.equals(accountDid)
|
.equals(accountDid)
|
||||||
.toArray();
|
.toArray();
|
||||||
|
|
||||||
// SQLite
|
// absurd-sql
|
||||||
const contacts = await db.selectAll(`
|
const result = await db.exec(`
|
||||||
SELECT * FROM contacts
|
SELECT * FROM contacts
|
||||||
WHERE did = ?
|
WHERE did = ?
|
||||||
ORDER BY created_at DESC
|
ORDER BY created_at DESC
|
||||||
`, [accountDid]);
|
`, [accountDid]);
|
||||||
|
const contacts = result[0]?.values || [];
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Add Contact
|
#### Add Contact
|
||||||
@@ -165,8 +169,8 @@ await db.contacts.add({
|
|||||||
updatedAt: Date.now()
|
updatedAt: Date.now()
|
||||||
});
|
});
|
||||||
|
|
||||||
// SQLite
|
// absurd-sql
|
||||||
await db.execute(`
|
await db.run(`
|
||||||
INSERT INTO contacts (id, did, name, created_at, updated_at)
|
INSERT INTO contacts (id, did, name, created_at, updated_at)
|
||||||
VALUES (?, ?, ?, ?, ?)
|
VALUES (?, ?, ?, ?, ?)
|
||||||
`, [generateId(), accountDid, name, Date.now(), Date.now()]);
|
`, [generateId(), accountDid, name, Date.now(), Date.now()]);
|
||||||
@@ -182,20 +186,25 @@ await db.transaction('rw', [db.accounts, db.contacts], async () => {
|
|||||||
await db.contacts.bulkAdd(contacts);
|
await db.contacts.bulkAdd(contacts);
|
||||||
});
|
});
|
||||||
|
|
||||||
// SQLite
|
// absurd-sql
|
||||||
await db.transaction(async (tx) => {
|
await db.exec('BEGIN TRANSACTION;');
|
||||||
await tx.execute(`
|
try {
|
||||||
|
await db.run(`
|
||||||
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
|
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
|
||||||
VALUES (?, ?, ?, ?)
|
VALUES (?, ?, ?, ?)
|
||||||
`, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]);
|
`, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]);
|
||||||
|
|
||||||
for (const contact of contacts) {
|
for (const contact of contacts) {
|
||||||
await tx.execute(`
|
await db.run(`
|
||||||
INSERT INTO contacts (id, did, name, created_at, updated_at)
|
INSERT INTO contacts (id, did, name, created_at, updated_at)
|
||||||
VALUES (?, ?, ?, ?, ?)
|
VALUES (?, ?, ?, ?, ?)
|
||||||
`, [contact.id, contact.did, contact.name, contact.createdAt, contact.updatedAt]);
|
`, [contact.id, contact.did, contact.name, contact.createdAt, contact.updatedAt]);
|
||||||
}
|
}
|
||||||
});
|
await db.exec('COMMIT;');
|
||||||
|
} catch (error) {
|
||||||
|
await db.exec('ROLLBACK;');
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
## Migration Helper Functions
|
## Migration Helper Functions
|
||||||
@@ -218,15 +227,14 @@ async function exportDexieData(): Promise<MigrationData> {
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. Data Import (JSON to SQLite)
|
### 2. Data Import (JSON to absurd-sql)
|
||||||
```typescript
|
```typescript
|
||||||
async function importToSQLite(data: MigrationData): Promise<void> {
|
async function importToAbsurdSql(data: MigrationData): Promise<void> {
|
||||||
const db = await getSQLiteConnection();
|
await db.exec('BEGIN TRANSACTION;');
|
||||||
|
try {
|
||||||
await db.transaction(async (tx) => {
|
|
||||||
// Import accounts
|
// Import accounts
|
||||||
for (const account of data.accounts) {
|
for (const account of data.accounts) {
|
||||||
await tx.execute(`
|
await db.run(`
|
||||||
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
|
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
|
||||||
VALUES (?, ?, ?, ?)
|
VALUES (?, ?, ?, ?)
|
||||||
`, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]);
|
`, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]);
|
||||||
@@ -234,7 +242,7 @@ async function importToSQLite(data: MigrationData): Promise<void> {
|
|||||||
|
|
||||||
// Import settings
|
// Import settings
|
||||||
for (const setting of data.settings) {
|
for (const setting of data.settings) {
|
||||||
await tx.execute(`
|
await db.run(`
|
||||||
INSERT INTO settings (key, value, updated_at)
|
INSERT INTO settings (key, value, updated_at)
|
||||||
VALUES (?, ?, ?)
|
VALUES (?, ?, ?)
|
||||||
`, [setting.key, setting.value, setting.updatedAt]);
|
`, [setting.key, setting.value, setting.updatedAt]);
|
||||||
@@ -242,52 +250,52 @@ async function importToSQLite(data: MigrationData): Promise<void> {
|
|||||||
|
|
||||||
// Import contacts
|
// Import contacts
|
||||||
for (const contact of data.contacts) {
|
for (const contact of data.contacts) {
|
||||||
await tx.execute(`
|
await db.run(`
|
||||||
INSERT INTO contacts (id, did, name, created_at, updated_at)
|
INSERT INTO contacts (id, did, name, created_at, updated_at)
|
||||||
VALUES (?, ?, ?, ?, ?)
|
VALUES (?, ?, ?, ?, ?)
|
||||||
`, [contact.id, contact.did, contact.name, contact.createdAt, contact.updatedAt]);
|
`, [contact.id, contact.did, contact.name, contact.createdAt, contact.updatedAt]);
|
||||||
}
|
}
|
||||||
});
|
await db.exec('COMMIT;');
|
||||||
|
} catch (error) {
|
||||||
|
await db.exec('ROLLBACK;');
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Verification
|
### 3. Verification
|
||||||
```typescript
|
```typescript
|
||||||
async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
|
async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
|
||||||
const db = await getSQLiteConnection();
|
|
||||||
|
|
||||||
// Verify account count
|
// Verify account count
|
||||||
const accountCount = await db.selectValue(
|
const accountResult = await db.exec('SELECT COUNT(*) as count FROM accounts');
|
||||||
'SELECT COUNT(*) FROM accounts'
|
const accountCount = accountResult[0].values[0][0];
|
||||||
);
|
|
||||||
if (accountCount !== dexieData.accounts.length) {
|
if (accountCount !== dexieData.accounts.length) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Verify settings count
|
// Verify settings count
|
||||||
const settingsCount = await db.selectValue(
|
const settingsResult = await db.exec('SELECT COUNT(*) as count FROM settings');
|
||||||
'SELECT COUNT(*) FROM settings'
|
const settingsCount = settingsResult[0].values[0][0];
|
||||||
);
|
|
||||||
if (settingsCount !== dexieData.settings.length) {
|
if (settingsCount !== dexieData.settings.length) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Verify contacts count
|
// Verify contacts count
|
||||||
const contactsCount = await db.selectValue(
|
const contactsResult = await db.exec('SELECT COUNT(*) as count FROM contacts');
|
||||||
'SELECT COUNT(*) FROM contacts'
|
const contactsCount = contactsResult[0].values[0][0];
|
||||||
);
|
|
||||||
if (contactsCount !== dexieData.contacts.length) {
|
if (contactsCount !== dexieData.contacts.length) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Verify data integrity
|
// Verify data integrity
|
||||||
for (const account of dexieData.accounts) {
|
for (const account of dexieData.accounts) {
|
||||||
const migratedAccount = await db.selectOne(
|
const result = await db.exec(
|
||||||
'SELECT * FROM accounts WHERE did = ?',
|
'SELECT * FROM accounts WHERE did = ?',
|
||||||
[account.did]
|
[account.did]
|
||||||
);
|
);
|
||||||
|
const migratedAccount = result[0]?.values[0];
|
||||||
if (!migratedAccount ||
|
if (!migratedAccount ||
|
||||||
migratedAccount.public_key_hex !== account.publicKeyHex) {
|
migratedAccount[1] !== account.publicKeyHex) { // public_key_hex is second column
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -300,18 +308,21 @@ async function verifyMigration(dexieData: MigrationData): Promise<boolean> {
|
|||||||
|
|
||||||
### 1. Indexing
|
### 1. Indexing
|
||||||
- Dexie automatically creates indexes based on the schema
|
- Dexie automatically creates indexes based on the schema
|
||||||
- SQLite requires explicit index creation
|
- absurd-sql requires explicit index creation
|
||||||
- Added indexes for frequently queried fields
|
- Added indexes for frequently queried fields
|
||||||
|
- Use `PRAGMA journal_mode=MEMORY;` for better performance
|
||||||
|
|
||||||
### 2. Batch Operations
|
### 2. Batch Operations
|
||||||
- Dexie has built-in bulk operations
|
- Dexie has built-in bulk operations
|
||||||
- SQLite uses transactions for batch operations
|
- absurd-sql uses transactions for batch operations
|
||||||
- Consider chunking large datasets
|
- Consider chunking large datasets
|
||||||
|
- Use prepared statements for repeated queries
|
||||||
|
|
||||||
### 3. Query Optimization
|
### 3. Query Optimization
|
||||||
- Dexie uses IndexedDB's native indexing
|
- Dexie uses IndexedDB's native indexing
|
||||||
- SQLite requires explicit query optimization
|
- absurd-sql requires explicit query optimization
|
||||||
- Use prepared statements for repeated queries
|
- Use prepared statements for repeated queries
|
||||||
|
- Consider using `PRAGMA synchronous=NORMAL;` for better performance
|
||||||
|
|
||||||
## Error Handling
|
## Error Handling
|
||||||
|
|
||||||
@@ -326,14 +337,14 @@ try {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// SQLite errors
|
// absurd-sql errors
|
||||||
try {
|
try {
|
||||||
await db.execute(`
|
await db.run(`
|
||||||
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
|
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
|
||||||
VALUES (?, ?, ?, ?)
|
VALUES (?, ?, ?, ?)
|
||||||
`, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]);
|
`, [account.did, account.publicKeyHex, account.createdAt, account.updatedAt]);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
if (error.code === 'SQLITE_CONSTRAINT') {
|
if (error.message.includes('UNIQUE constraint failed')) {
|
||||||
// Handle duplicate key
|
// Handle duplicate key
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -350,15 +361,14 @@ try {
|
|||||||
// Dexie automatically rolls back
|
// Dexie automatically rolls back
|
||||||
}
|
}
|
||||||
|
|
||||||
// SQLite transaction
|
// absurd-sql transaction
|
||||||
const db = await getSQLiteConnection();
|
|
||||||
try {
|
try {
|
||||||
await db.transaction(async (tx) => {
|
await db.exec('BEGIN TRANSACTION;');
|
||||||
// Operations
|
// Operations
|
||||||
});
|
await db.exec('COMMIT;');
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
// SQLite automatically rolls back
|
await db.exec('ROLLBACK;');
|
||||||
await db.execute('ROLLBACK');
|
throw error;
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -1,8 +1,8 @@
|
|||||||
# Migration Guide: Dexie to wa-sqlite
|
# Migration Guide: Dexie to absurd-sql
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
This document outlines the migration process from Dexie.js to wa-sqlite for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.
|
This document outlines the migration process from Dexie.js to absurd-sql for the TimeSafari app's storage implementation. The migration aims to provide a consistent SQLite-based storage solution across all platforms while maintaining data integrity and ensuring a smooth transition for users.
|
||||||
|
|
||||||
## Migration Goals
|
## Migration Goals
|
||||||
|
|
||||||
@@ -43,12 +43,20 @@ This document outlines the migration process from Dexie.js to wa-sqlite for the
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
2. **Storage Requirements**
|
2. **Dependencies**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"@jlongster/sql.js": "^1.8.0",
|
||||||
|
"absurd-sql": "^1.8.0"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Storage Requirements**
|
||||||
- Sufficient IndexedDB quota
|
- Sufficient IndexedDB quota
|
||||||
- Available disk space for SQLite
|
- Available disk space for SQLite
|
||||||
- Backup storage space
|
- Backup storage space
|
||||||
|
|
||||||
3. **Platform Support**
|
4. **Platform Support**
|
||||||
- Web: Modern browser with IndexedDB support
|
- Web: Modern browser with IndexedDB support
|
||||||
- iOS: iOS 13+ with SQLite support
|
- iOS: iOS 13+ with SQLite support
|
||||||
- Android: Android 5+ with SQLite support
|
- Android: Android 5+ with SQLite support
|
||||||
@@ -60,9 +68,15 @@ This document outlines the migration process from Dexie.js to wa-sqlite for the
|
|||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// src/services/storage/migration/MigrationService.ts
|
// src/services/storage/migration/MigrationService.ts
|
||||||
|
import initSqlJs from '@jlongster/sql.js';
|
||||||
|
import { SQLiteFS } from 'absurd-sql';
|
||||||
|
import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend';
|
||||||
|
|
||||||
export class MigrationService {
|
export class MigrationService {
|
||||||
private static instance: MigrationService;
|
private static instance: MigrationService;
|
||||||
private backup: MigrationBackup | null = null;
|
private backup: MigrationBackup | null = null;
|
||||||
|
private sql: any = null;
|
||||||
|
private db: any = null;
|
||||||
|
|
||||||
async prepare(): Promise<void> {
|
async prepare(): Promise<void> {
|
||||||
try {
|
try {
|
||||||
@@ -75,8 +89,8 @@ export class MigrationService {
|
|||||||
// 3. Verify backup integrity
|
// 3. Verify backup integrity
|
||||||
await this.verifyBackup();
|
await this.verifyBackup();
|
||||||
|
|
||||||
// 4. Initialize wa-sqlite
|
// 4. Initialize absurd-sql
|
||||||
await this.initializeWaSqlite();
|
await this.initializeAbsurdSql();
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
throw new StorageError(
|
throw new StorageError(
|
||||||
'Migration preparation failed',
|
'Migration preparation failed',
|
||||||
@@ -86,6 +100,42 @@ export class MigrationService {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private async initializeAbsurdSql(): Promise<void> {
|
||||||
|
// Initialize SQL.js
|
||||||
|
this.sql = await initSqlJs({
|
||||||
|
locateFile: (file: string) => {
|
||||||
|
return new URL(`/node_modules/@jlongster/sql.js/dist/${file}`, import.meta.url).href;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Setup SQLiteFS with IndexedDB backend
|
||||||
|
const sqlFS = new SQLiteFS(this.sql.FS, new IndexedDBBackend());
|
||||||
|
this.sql.register_for_idb(sqlFS);
|
||||||
|
|
||||||
|
// Create and mount filesystem
|
||||||
|
this.sql.FS.mkdir('/sql');
|
||||||
|
this.sql.FS.mount(sqlFS, {}, '/sql');
|
||||||
|
|
||||||
|
// Open database
|
||||||
|
const path = '/sql/db.sqlite';
|
||||||
|
if (typeof SharedArrayBuffer === 'undefined') {
|
||||||
|
let stream = this.sql.FS.open(path, 'a+');
|
||||||
|
await stream.node.contents.readIfFallback();
|
||||||
|
this.sql.FS.close(stream);
|
||||||
|
}
|
||||||
|
|
||||||
|
this.db = new this.sql.Database(path, { filename: true });
|
||||||
|
if (!this.db) {
|
||||||
|
throw new StorageError(
|
||||||
|
'Database initialization failed',
|
||||||
|
StorageErrorCodes.INITIALIZATION_FAILED
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Configure database
|
||||||
|
await this.db.exec(`PRAGMA journal_mode=MEMORY;`);
|
||||||
|
}
|
||||||
|
|
||||||
private async checkPrerequisites(): Promise<void> {
|
private async checkPrerequisites(): Promise<void> {
|
||||||
// Check IndexedDB availability
|
// Check IndexedDB availability
|
||||||
if (!window.indexedDB) {
|
if (!window.indexedDB) {
|
||||||
@@ -160,12 +210,11 @@ export class DataMigration {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private async migrateAccounts(accounts: Account[]): Promise<void> {
|
private async migrateAccounts(accounts: Account[]): Promise<void> {
|
||||||
const db = await this.getWaSqliteConnection();
|
|
||||||
|
|
||||||
// Use transaction for atomicity
|
// Use transaction for atomicity
|
||||||
await db.transaction(async (tx) => {
|
await this.db.exec('BEGIN TRANSACTION;');
|
||||||
|
try {
|
||||||
for (const account of accounts) {
|
for (const account of accounts) {
|
||||||
await tx.execute(`
|
await this.db.run(`
|
||||||
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
|
INSERT INTO accounts (did, public_key_hex, created_at, updated_at)
|
||||||
VALUES (?, ?, ?, ?)
|
VALUES (?, ?, ?, ?)
|
||||||
`, [
|
`, [
|
||||||
@@ -175,16 +224,18 @@ export class DataMigration {
|
|||||||
account.updatedAt
|
account.updatedAt
|
||||||
]);
|
]);
|
||||||
}
|
}
|
||||||
});
|
await this.db.exec('COMMIT;');
|
||||||
|
} catch (error) {
|
||||||
|
await this.db.exec('ROLLBACK;');
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private async verifyMigration(backup: MigrationBackup): Promise<void> {
|
private async verifyMigration(backup: MigrationBackup): Promise<void> {
|
||||||
const db = await this.getWaSqliteConnection();
|
|
||||||
|
|
||||||
// Verify account count
|
// Verify account count
|
||||||
const accountCount = await db.selectValue(
|
const result = await this.db.exec('SELECT COUNT(*) as count FROM accounts');
|
||||||
'SELECT COUNT(*) FROM accounts'
|
const accountCount = result[0].values[0][0];
|
||||||
);
|
|
||||||
if (accountCount !== backup.accounts.length) {
|
if (accountCount !== backup.accounts.length) {
|
||||||
throw new StorageError(
|
throw new StorageError(
|
||||||
'Account count mismatch',
|
'Account count mismatch',
|
||||||
@@ -214,8 +265,8 @@ export class RollbackService {
|
|||||||
// 3. Verify restoration
|
// 3. Verify restoration
|
||||||
await this.verifyRestoration(backup);
|
await this.verifyRestoration(backup);
|
||||||
|
|
||||||
// 4. Clean up wa-sqlite
|
// 4. Clean up absurd-sql
|
||||||
await this.cleanupWaSqlite();
|
await this.cleanupAbsurdSql();
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
throw new StorageError(
|
throw new StorageError(
|
||||||
'Rollback failed',
|
'Rollback failed',
|
||||||
@@ -371,6 +422,14 @@ button:hover {
|
|||||||
```typescript
|
```typescript
|
||||||
// src/services/storage/migration/__tests__/MigrationService.spec.ts
|
// src/services/storage/migration/__tests__/MigrationService.spec.ts
|
||||||
describe('MigrationService', () => {
|
describe('MigrationService', () => {
|
||||||
|
it('should initialize absurd-sql correctly', async () => {
|
||||||
|
const service = MigrationService.getInstance();
|
||||||
|
await service.initializeAbsurdSql();
|
||||||
|
|
||||||
|
expect(service.isInitialized()).toBe(true);
|
||||||
|
expect(service.getDatabase()).toBeDefined();
|
||||||
|
});
|
||||||
|
|
||||||
it('should create valid backup', async () => {
|
it('should create valid backup', async () => {
|
||||||
const service = MigrationService.getInstance();
|
const service = MigrationService.getInstance();
|
||||||
const backup = await service.createBackup();
|
const backup = await service.createBackup();
|
||||||
@@ -10,9 +10,9 @@
|
|||||||
- [ ] Add migration support methods
|
- [ ] Add migration support methods
|
||||||
|
|
||||||
- [ ] Implement platform-specific services
|
- [ ] Implement platform-specific services
|
||||||
- [ ] `WebSQLiteService` (wa-sqlite)
|
- [ ] `WebSQLiteService` (absurd-sql)
|
||||||
- [ ] Database initialization
|
- [ ] Database initialization
|
||||||
- [ ] VFS setup
|
- [ ] VFS setup with IndexedDB backend
|
||||||
- [ ] Connection management
|
- [ ] Connection management
|
||||||
- [ ] Query builder
|
- [ ] Query builder
|
||||||
- [ ] `NativeSQLiteService` (iOS/Android)
|
- [ ] `NativeSQLiteService` (iOS/Android)
|
||||||
@@ -49,17 +49,24 @@
|
|||||||
## Platform-Specific Implementation
|
## Platform-Specific Implementation
|
||||||
|
|
||||||
### Web Platform
|
### Web Platform
|
||||||
- [ ] Setup wa-sqlite
|
- [ ] Setup absurd-sql
|
||||||
- [ ] Install dependencies
|
- [ ] Install dependencies
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"@wa-sqlite/sql.js": "^0.8.12",
|
"@jlongster/sql.js": "^1.8.0",
|
||||||
"@wa-sqlite/sql.js-httpvfs": "^0.8.12"
|
"absurd-sql": "^1.8.0"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
- [ ] Configure VFS
|
- [ ] Configure VFS with IndexedDB backend
|
||||||
- [ ] Setup worker threads
|
- [ ] Setup worker threads
|
||||||
- [ ] Implement connection pooling
|
- [ ] Implement connection pooling
|
||||||
|
- [ ] Configure database pragmas
|
||||||
|
```sql
|
||||||
|
PRAGMA journal_mode=MEMORY;
|
||||||
|
PRAGMA synchronous=NORMAL;
|
||||||
|
PRAGMA foreign_keys=ON;
|
||||||
|
PRAGMA busy_timeout=5000;
|
||||||
|
```
|
||||||
|
|
||||||
- [ ] Update build configuration
|
- [ ] Update build configuration
|
||||||
- [ ] Modify `vite.config.ts`
|
- [ ] Modify `vite.config.ts`
|
||||||
@@ -71,6 +78,7 @@
|
|||||||
- [ ] Create fallback service
|
- [ ] Create fallback service
|
||||||
- [ ] Add data synchronization
|
- [ ] Add data synchronization
|
||||||
- [ ] Handle quota exceeded
|
- [ ] Handle quota exceeded
|
||||||
|
- [ ] Implement atomic operations
|
||||||
|
|
||||||
### iOS Platform
|
### iOS Platform
|
||||||
- [ ] Setup SQLCipher
|
- [ ] Setup SQLCipher
|
||||||
@@ -140,6 +148,11 @@
|
|||||||
updated_at INTEGER NOT NULL,
|
updated_at INTEGER NOT NULL,
|
||||||
FOREIGN KEY (did) REFERENCES accounts(did)
|
FOREIGN KEY (did) REFERENCES accounts(did)
|
||||||
);
|
);
|
||||||
|
|
||||||
|
-- Indexes for performance
|
||||||
|
CREATE INDEX idx_accounts_created_at ON accounts(created_at);
|
||||||
|
CREATE INDEX idx_contacts_did ON contacts(did);
|
||||||
|
CREATE INDEX idx_settings_updated_at ON settings(updated_at);
|
||||||
```
|
```
|
||||||
|
|
||||||
- [ ] Create indexes
|
- [ ] Create indexes
|
||||||
@@ -286,12 +299,16 @@
|
|||||||
- [ ] Migration time < 5s per 1000 records
|
- [ ] Migration time < 5s per 1000 records
|
||||||
- [ ] Storage overhead < 10%
|
- [ ] Storage overhead < 10%
|
||||||
- [ ] Memory usage < 50MB
|
- [ ] Memory usage < 50MB
|
||||||
|
- [ ] Atomic operations complete successfully
|
||||||
|
- [ ] Transaction performance meets requirements
|
||||||
|
|
||||||
### 2. Reliability
|
### 2. Reliability
|
||||||
- [ ] 99.9% uptime
|
- [ ] 99.9% uptime
|
||||||
- [ ] Zero data loss
|
- [ ] Zero data loss
|
||||||
- [ ] Automatic recovery
|
- [ ] Automatic recovery
|
||||||
- [ ] Backup verification
|
- [ ] Backup verification
|
||||||
|
- [ ] Transaction atomicity
|
||||||
|
- [ ] Data consistency
|
||||||
|
|
||||||
### 3. Security
|
### 3. Security
|
||||||
- [ ] AES-256 encryption
|
- [ ] AES-256 encryption
|
||||||
Reference in New Issue
Block a user