performance-optimizations-testing #171

Open
anomalist wants to merge 27 commits from performance-optimizations-testing into master
4 changed files with 882 additions and 351 deletions
Showing only changes of commit 294034d9b4 - Show all commits

View File

@@ -3,50 +3,50 @@ description:
globs:
alwaysApply: true
---
# Absurd SQL - Cursor Development Guide
# Absurd SQL - Cursor Development Guide (Directive Style)
## Project Overview
Absurd SQL is a backend implementation for sql.js that enables persistent SQLite databases in the browser by using IndexedDB as a block storage system. This guide provides rules and best practices for developing with this project in Cursor.
Implement persistent SQLite databases in the browser using **Absurd SQL** with IndexedDB as block storage. Execute all SQL operations according to the following directives.
## Project Structure
```
absurd-sql/
├── src/ # Source code
├── dist/ # Built files
├── package.json # Dependencies and scripts
├── rollup.config.js # Build configuration
└── jest.config.js # Test configuration
├── src/ # Place source code here
├── dist/ # Place built files here
├── package.json # Maintain dependencies and scripts here
├── rollup.config.js # Maintain build configuration here
└── jest.config.js # Maintain test configuration here
```
## Development Rules
## Directives
### 1. Worker Thread Requirements
- All SQL operations MUST be performed in a worker thread
- Main thread should only handle worker initialization and communication
- Never block the main thread with database operations
### 1. Worker Thread Execution
- Execute **all SQL operations** inside worker threads.
- Restrict the main thread to **initialization** and **communication only**.
- Block **no operations** on the main thread.
### 2. Code Organization
- Keep worker code in separate files (e.g., `*.worker.js`)
- Use ES modules for imports/exports
- Follow the project's existing module structure
- Store worker logic in dedicated files: `*.worker.js`.
- Use **ES modules** exclusively.
- Conform to the existing **module structure**.
### 3. Required Headers
When developing locally or deploying, ensure these headers are set:
### 3. Headers Enforcement
Always set the following headers:
```
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
```
### 4. Browser Compatibility
- Primary target: Modern browsers with SharedArrayBuffer support
- Fallback mode: Safari (with limitations)
- Always test in both modes
- Target **modern browsers with SharedArrayBuffer support**.
- Activate fallback mode for Safari when required.
- Test in **both primary and fallback modes** without exception.
### 5. Database Configuration
Recommended database settings:
Apply the following PRAGMA settings immediately:
```sql
PRAGMA journal_mode=MEMORY;
PRAGMA page_size=8192; -- Optional, but recommended
PRAGMA page_size=8192;
```
### 6. Development Workflow
@@ -54,100 +54,96 @@ PRAGMA page_size=8192; -- Optional, but recommended
```bash
yarn add @jlongster/sql.js absurd-sql
```
2. Execute commands as follows:
- `yarn build` → build the project
- `yarn jest` → run all tests
- `yarn serve` → launch development server
2. Development commands:
- `yarn build` - Build the project
- `yarn jest` - Run tests
- `yarn serve` - Start development server
### 7. Testing
- Write tests for both **SharedArrayBuffer** and **fallback modes**.
- Use **Jest** exclusively.
- Include **performance benchmarks** for critical paths.
### 7. Testing Guidelines
- Write tests for both SharedArrayBuffer and fallback modes
- Use Jest for testing
- Include performance benchmarks for critical operations
### 8. Performance Considerations
- Use bulk operations when possible
- Monitor read/write performance
- Consider using transactions for multiple operations
- Avoid unnecessary database connections
### 8. Performance Optimization
- Execute bulk operations when available.
- Enforce **transactions** for multi-step operations.
- Monitor read/write throughput continuously.
- Reuse database connections. Do **not** open unnecessary ones.
### 9. Error Handling
- Implement proper error handling for:
- Worker initialization failures
- Database connection issues
- Concurrent access conflicts (in fallback mode)
- Storage quota exceeded scenarios
Implement error handling for:
- Worker initialization failures
- Database connection issues
- Concurrent access conflicts (fallback mode)
- Storage quota exceeded scenarios
### 10. Security Best Practices
- Never expose database operations directly to the client
- Validate all SQL queries
- Implement proper access controls
- Handle sensitive data appropriately
### 10. Security
- Forbid direct client access to database operations.
- Validate every SQL query.
- Enforce access control measures.
- Handle sensitive data with strict isolation.
### 11. Code Style
- Follow ESLint configuration
- Use async/await for asynchronous operations
- Document complex database operations
- Include comments for non-obvious optimizations
- Follow ESLint configuration.
- Use `async/await` for asynchronous operations.
- Document complex operations thoroughly.
- Comment all optimizations that are not obvious.
### 12. Debugging
- Use `jest-debug` for debugging tests
- Monitor IndexedDB usage in browser dev tools
- Check worker communication in console
- Use performance monitoring tools
- Use `jest-debug` for test debugging.
- Inspect IndexedDB in browser developer tools.
- Trace worker communication in console logs.
- Apply browser performance monitoring tools.
## Common Patterns
## Required Patterns
### Worker Initialization
```javascript
// Main thread
import { initBackend } from 'absurd-sql/dist/indexeddb-main-thread';
function init() {
let worker = new Worker(new URL('./index.worker.js', import.meta.url));
const worker = new Worker(new URL('./index.worker.js', import.meta.url));
initBackend(worker);
}
```
### Database Setup
```javascript
// Worker thread
import initSqlJs from '@jlongster/sql.js';
import { SQLiteFS } from 'absurd-sql';
import IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend';
async function setupDatabase() {
let SQL = await initSqlJs({ locateFile: file => file });
let sqlFS = new SQLiteFS(SQL.FS, new IndexedDBBackend());
const SQL = await initSqlJs({ locateFile: f => f });
const sqlFS = new SQLiteFS(SQL.FS, new IndexedDBBackend());
SQL.register_for_idb(sqlFS);
SQL.FS.mkdir('/sql');
SQL.FS.mount(sqlFS, {}, '/sql');
return new SQL.Database('/sql/db.sqlite', { filename: true });
}
```
## Troubleshooting
## Troubleshooting Directives
### Common Issues
1. SharedArrayBuffer not available
- Check COOP/COEP headers
- Verify browser support
- Test fallback mode
### If SharedArrayBuffer is unavailable:
- Verify COOP/COEP headers.
- Check browser support.
- Activate fallback mode.
2. Worker initialization failures
- Check file paths
- Verify module imports
- Check browser console for errors
### If worker initialization fails:
- Verify file paths.
- Confirm module imports.
- Inspect browser console for errors.
3. Performance issues
- Monitor IndexedDB usage
- Check for unnecessary operations
- Verify transaction usage
### If performance degrades:
- Inspect IndexedDB usage.
- Eliminate redundant operations.
- Confirm transaction enforcement.
## Resources
- [Project Demo](https://priceless-keller-d097e5.netlify.app/)
- [Example Project](https://github.com/jlongster/absurd-example-project)
- [Blog Post](https://jlongster.com/future-sql-web)
- [SQL.js Documentation](https://github.com/sql-js/sql.js/)
## Reference Materials
- [Project Demo](https://priceless-keller-d097e5.netlify.app/)
- [Example Project](https://github.com/jlongster/absurd-example-project)
- [Blog Post](https://jlongster.com/future-sql-web)
- [SQL.js Documentation](https://github.com/sql-js/sql.js/)

View File

@@ -123,74 +123,222 @@
<script lang="ts">
/**
* @file Contact Import View Component
* @author Matthew Raymer
*
* This component handles the import of contacts into the TimeSafari app.
* It supports multiple import methods and handles duplicate detection,
* contact validation, and visibility settings.
*
* Import Methods:
* 1. Direct URL Query Parameters:
* Example: /contact-import?contacts=[{"did":"did:example:123","name":"Alice"}]
*
* 2. JWT in URL Path:
* Example: /contact-import/eyJhbGciOiJFUzI1NksifQ...
* - Supports both single and bulk imports
* - JWT payload can be either:
* a) Array format: { contacts: [{did: "...", name: "..."}, ...] }
* b) Single contact: { own: true, did: "...", name: "..." }
*
* 3. Manual JWT Input:
* - Accepts pasted JWT strings
* - Validates format and content before processing
*
* URL Examples:
* ```
* # Bulk import via query params
* /contact-import?contacts=[
* {"did":"did:example:123","name":"Alice"},
* {"did":"did:example:456","name":"Bob"}
* ]
*
* # Single contact via JWT
* /contact-import/eyJhbGciOiJFUzI1NksifQ.eyJvd24iOnRydWUsImRpZCI6ImRpZDpleGFtcGxlOjEyMyJ9...
*
* # Bulk import via JWT
* /contact-import/eyJhbGciOiJFUzI1NksifQ.eyJjb250YWN0cyI6W3siZGlkIjoiZGlkOmV4YW1wbGU6MTIzIn1dfQ...
*
* # Redirect to contacts page (single contact)
* /contacts?contactJwt=eyJhbGciOiJFUzI1NksifQ...
* ```
*
* Features:
* - Automatic duplicate detection
* - Field-by-field comparison for existing contacts
* - Batch visibility settings
* - Auto-import for single new contacts
* - Error handling and validation
*
* State Management:
* - Tracks existing contacts
* - Maintains selection state for bulk imports
* - Records differences for duplicate contacts
* - Manages visibility settings
*
* Security Considerations:
* - JWT validation for imported contacts
* - Visibility control per contact
* - Error handling for malformed data
*
* @example
* // Component usage in router
* {
* path: "/contact-import/:jwt?",
* name: "contact-import",
* component: ContactImportView
* ContactImportView - Contact Import and Batch Processing Page
*
* This component handles the batch import of contacts with comprehensive
* validation, duplicate detection, and field comparison capabilities.
* It provides users with detailed information about each contact before
* importing, allowing them to make informed decisions about their contact list.
*
* ## How the Contact Import Page Works
*
* ### Page Entry and Data Processing
*
* **Entry Points**:
* - **URL Parameters**: Direct navigation with contact data in URL
* - **Contact Input Form**: Redirected from ContactsView with parsed data
* - **Manual Entry**: Users can input contact data directly
*
* **Data Processing Pipeline**:
* 1. **Input Validation**: Parse and validate contact data format
* 2. **Contact Analysis**: Check each contact against existing database
* 3. **Duplicate Detection**: Identify existing contacts and compare fields
* 4. **UI Preparation**: Prepare contact list with status indicators
*
* ### Contact Analysis and Display
*
* **Contact Status Classification**:
* - **New Contacts** (Green): Contacts not in database
* - **Existing Contacts** (Orange): Contacts already in database
* - **Identical Contacts**: Existing contacts with no field differences
*
* **Field Comparison System**:
* - **Automatic Detection**: Compare all contact fields
* - **Difference Display**: Show old vs new values in table format
* - **User Decision**: Allow users to see what will be updated
*
* **Contact List Structure**:
* ```typescript
* interface ContactImportItem {
* did: string; // Decentralized identifier
* name?: string; // Display name
* publicKey?: string; // Public key
* publicKeyBase64?: string; // Base64 encoded key
* status: 'new' | 'existing'; // Import status
* differences?: FieldDifferences; // Field comparison results
* }
*
* @see {@link Contact} for contact data structure
* @see {@link setVisibilityUtil} for visibility management
* ```
*
* ### User Interface Components
*
* **Header Section**:
* - **Back Navigation**: Return to previous page
* - **Page Title**: "Contact Import" heading
* - **Loading State**: Spinner during data processing
*
* **Visibility Settings**:
* - **Activity Visibility Checkbox**: Control activity sharing with imported contacts
* - **Global Setting**: Applies to all contacts being imported
*
* **Contact List Display**:
* - **Contact Cards**: Individual contact information with:
* - Selection checkbox for import control
* - Contact name and DID display
* - Status indicator (New/Existing)
* - Field comparison table for existing contacts
*
* **Field Comparison Table**:
* - **Three-Column Layout**: Field name, old value, new value
* - **Difference Highlighting**: Clear visual indication of changes
* - **Comprehensive Coverage**: All contact fields are compared
*
* **Import Controls**:
* - **Select All/None**: Bulk selection controls
* - **Individual Selection**: Per-contact import control
* - **Import Button**: Execute the selected imports
*
* ### Data Processing Logic
*
* **Contact Validation**:
* ```typescript
* // Validate DID format
* const isValidDid = (did: string): boolean => {
* return did.startsWith('did:') && did.length > 10;
* };
*
* // Check for existing contact
* const existingContact = await $getContact(did);
* const isExisting = existingContact !== null;
* ```
*
* **Field Comparison Algorithm**:
* ```typescript
* // Compare contact fields
* const compareFields = (existing: Contact, importing: Contact) => {
* const differences: FieldDifferences = {};
*
* for (const field of ['name', 'publicKey', 'publicKeyBase64']) {
* if (existing[field] !== importing[field]) {
* differences[field] = {
* old: existing[field] || '',
* new: importing[field] || ''
* };
* }
* }
*
* return differences;
* };
* ```
*
* **Import Decision Logic**:
* - **New Contact**: Add to database with all provided fields
* - **Existing Contact with Differences**: Update with new field values
* - **Existing Contact without Differences**: Skip import (already identical)
* - **Invalid Contact**: Skip import and show error
*
* ### Batch Import Process
*
* **Pre-Import Validation**:
* - Verify all selected contacts are valid
* - Check database constraints
* - Validate visibility settings
*
* **Database Transaction**:
* ```typescript
* // Execute batch import
* const importContacts = async () => {
* const selectedContacts = contactsImporting.filter((_, index) =>
* contactsSelected[index]
* );
*
* await $beginTransaction();
*
* try {
* for (const contact of selectedContacts) {
* if (contactsExisting[contact.did]) {
* await $updateContact(contact.did, contact);
* } else {
* await $addContact(contact);
* }
* }
*
* await $commitTransaction();
* notify.success('Contacts imported successfully');
* } catch (error) {
* await $rollbackTransaction();
* notify.error('Import failed: ' + error.message);
* }
* };
* ```
*
* **Post-Import Actions**:
* - Update contact list in parent component
* - Apply visibility settings if enabled
* - Navigate back to contacts list
* - Display success/error notifications
*
* ### Error Handling and Edge Cases
*
* **Input Validation Errors**:
* - Malformed JSON data
* - Invalid DID format
* - Missing required fields
* - Empty contact arrays
*
* **Database Errors**:
* - Constraint violations
* - Storage quota exceeded
* - Concurrent access conflicts
* - Transaction failures
*
* **UI Error Recovery**:
* - Graceful handling of network failures
* - Retry mechanisms for failed operations
* - Clear error messages for users
* - Fallback options for unsupported features
*
* ### Performance Optimizations
*
* **Efficient Processing**:
* - Batch database operations
* - Optimized field comparison algorithms
* - Lazy loading of contact details
* - Debounced UI updates
*
* **Memory Management**:
* - Cleanup of temporary data structures
* - Proper disposal of event listeners
* - Efficient state management
* - Garbage collection optimization
*
* **UI Responsiveness**:
* - Asynchronous data processing
* - Progressive loading of contact data
* - Non-blocking UI updates
* - Optimized rendering for large lists
*
* ### Integration Points
*
* **Database Integration**:
* - PlatformServiceMixin for database operations
* - Transaction-based data integrity
* - Optimized queries for contact retrieval
* - Proper error handling and rollback
*
* **Navigation Integration**:
* - Route-based data passing
* - Deep linking support
* - Back navigation handling
* - Modal dialog management
*
* **Notification System**:
* - Success/error message display
* - Progress indication during import
* - User feedback for all operations
* - Accessibility-compliant notifications
*
* @author Matthew Raymer
* @date 2025-08-04
*/
import * as R from "ramda";

View File

@@ -123,6 +123,144 @@
</template>
<script lang="ts">
/**
* ContactsView - Main Contacts Management Page
*
* This component serves as the central hub for contact management in Time Safari.
* It provides a comprehensive interface for viewing, adding, importing, and managing
* contacts with various input methods and bulk operations.
*
* ## How the Contacts Page Works
*
* ### Contact Input and Import Workflow
*
* **ContactInputForm Component**:
* - **Input Field**: Accepts contact data in multiple formats:
* - Individual contact: `"did:ethr:0x..., Alice, publicKey"`
* - JSON array: `"Paste this: [{"did":"did:ethr:0x...","name":"Alice"}]"`
* - URL with contact data: `"https://example.com/contact-data"`
* - **Add Button**: Triggers contact processing and validation
* - **QR Scanner**: Alternative input method for mobile devices
* - **Real-time Validation**: Checks DID format and required fields
*
* **Contact Processing Logic**:
* 1. **Input Parsing**: The system parses the input to determine format
* 2. **Data Validation**: Validates DID format and required fields
* 3. **Duplicate Detection**: Checks if contact already exists
* 4. **Import Decision**:
* - Single contact: Direct addition to database
* - Multiple contacts: Redirect to ContactImportView for batch processing
* - Invalid data: Display error message
*
* **Import Workflow**:
* - **Single Contact**: Added directly with success notification
* - **Multiple Contacts**: Redirected to ContactImportView for:
* - Contact comparison and selection
* - Field difference display
* - Batch import execution
* - Visibility settings configuration
*
* ### Contact List Management
*
* **ContactListItem Components**:
* - **Contact Display**: Name, DID, and identicon
* - **Selection Checkboxes**: For bulk operations
* - **Action Buttons**: Gift, offer, and contact management
* - **Status Indicators**: Online/offline status, activity visibility
*
* **Bulk Operations**:
* - **Select All**: Toggle selection of all contacts
* - **Copy Selected**: Export selected contacts as JSON/CSV
* - **Bulk Actions**: Gift amounts, visibility settings
*
* **Contact Actions**:
* - **Gift Dialog**: Record gifts given to/received from contact
* - **Offer Dialog**: Create and manage offers
* - **Contact Edit**: Modify contact information
* - **Large Identicon**: View full-size contact identicon
*
* ### Data Flow and State Management
*
* **Contact Data Structure**:
* ```typescript
* interface Contact {
* did: string; // Decentralized identifier
* name?: string; // Display name (optional)
* publicKey?: string; // Public key for verification
* publicKeyBase64?: string; // Base64 encoded public key
* visibility?: boolean; // Activity visibility setting
* }
* ```
*
* **State Management**:
* - **Contact List**: Reactive list of all user contacts
* - **Selection State**: Track selected contacts for bulk operations
* - **UI State**: Toggle visibility of give totals, actions, etc.
* - **Modal State**: Manage dialog visibility and data
*
* **Database Operations**:
* - **Contact Addition**: Add new contacts with validation
* - **Contact Updates**: Modify existing contact information
* - **Contact Deletion**: Remove contacts (with confirmation)
* - **Bulk Operations**: Process multiple contacts efficiently
*
* ### Error Handling and User Feedback
*
* **Input Validation Errors**:
* - Invalid DID format
* - Missing required fields
* - Malformed JSON data
* - Network errors for URL-based imports
*
* **User Notifications**:
* - Success messages for successful operations
* - Error messages with specific details
* - Warning messages for potential issues
* - Confirmation dialogs for destructive actions
*
* **Error Recovery**:
* - Graceful handling of network failures
* - Retry mechanisms for failed operations
* - Fallback options for unsupported features
*
* ### Performance Optimizations
*
* **Contact List Rendering**:
* - Virtual scrolling for large contact lists
* - Efficient filtering and sorting
* - Lazy loading of contact details
*
* **Database Operations**:
* - Batch processing for multiple contacts
* - Transaction-based updates for data integrity
* - Optimized queries for contact retrieval
*
* **UI Responsiveness**:
* - Debounced input validation
* - Asynchronous contact processing
* - Progressive loading of contact data
*
* ### Integration Points
*
* **Platform Services**:
* - Database operations via PlatformServiceMixin
* - QR code scanning via platform-specific implementations
* - File system access for contact export
*
* **External Services**:
* - Endorser.ch for contact verification
* - JWT token processing for secure imports
* - URL-based contact data retrieval
*
* **Navigation Integration**:
* - Deep linking to contact import
* - Route-based contact filtering
* - Modal dialog management
*
* @author Matthew Raymer
* @date 2025-08-04
*/
import { AxiosError } from "axios";
import { Buffer } from "buffer/";
import { IndexableType } from "dexie";

File diff suppressed because it is too large Load Diff