WIP: Debugging sqlite-run/dbExec hanging issue
- Add renderer-to-main process log forwarding
- Implement operation tracking with unique IDs
- Add detailed timing and retry logging
- Enhance error capture and formatting
- Move logs to app user data directory
- Add exponential backoff for retries
This commit adds comprehensive logging to help diagnose
why dbExec operations are hanging when sent through the
sqlite-run channel. Changes include:
- Forward all renderer process logs to main process
- Track SQLite operations with unique IDs
- Log operation timing and retry attempts
- Capture detailed error information
- Implement exponential backoff for retries
- Centralize logs in app user data directory
Security:
- Logs are stored in app user data directory
- Sensitive data is sanitized in logs
- Error stacks are properly captured
Testing:
- Manual testing of logging system
- Verify log capture in both processes
- Check log rotation and file sizes
TODO:
- Monitor logs to identify root cause
- Add specific logging for settings table
- Consider connection pooling if needed
- Refactor dbExec in ElectronPlatformService to use proper connection management
- Add comprehensive logging throughout API server update flow
- Fix connection handling in database operations
- Add user feedback notifications for success/failure
- Add verification step after settings update
The main issue was that dbExec wasn't using the same robust connection
management as dbQuery, leading to "SQLite not initialized" errors. Now both
methods use the same connection lifecycle management through enqueueOperation.
Also added detailed logging at each step to help diagnose any future issues:
- AccountViewView component logging
- Database utility operations logging
- Connection state tracking
- Update verification
This should make the API server update more reliable and easier to debug.
- Replace generic execute method with specific IPC handlers
- Fix database operations by using proper IPC methods (createConnection, query, run, execute)
- Update type definitions to match actual IPC bridge interface
- Fix "Must provide statements" error by using correct method signatures
This change ensures proper communication between renderer and main processes
for SQLite operations, resolving database initialization and query issues.
BREAKING CHANGE: Removes database encryption in favor of simpler implementation
- Remove encryption from SQLite initialization and connection options
- Change journal mode from WAL to MEMORY to match sacred-sql
- Simplify PRAGMA settings and remove WAL-specific configurations
- Remove secret table and encryption-related migrations
- Update database schema to use non-encrypted storage
- Clean up database initialization process
This change aligns the TimeSafari Electron SQLite implementation with
sacred-sql, improving compatibility and simplifying the database layer.
Existing databases will need to be cleared and recreated due to the
removal of encryption support.
Migration:
1. Delete existing database at ~/Databases/TimeSafari/timesafariSQLite.db
2. Restart application to create fresh database with new schema
The app is failing to initialize encryption because:
- Database is created with 'no-encryption' mode
- Capacitor SQLite plugin's encryption methods are available but unused
- Secret table exists but encryption isn't properly initialized
This commit will:
- Enable encryption in database connection options
- Initialize encryption secret before database open
- Use Capacitor SQLite plugin's encryption methods
- Ensure secret table is properly initialized
This fixes the "No initial encryption supported" error that occurs
when trying to save new identities or access encrypted data.
Technical details:
- Changes connection options to use 'secret' encryption mode
- Adds setEncryptionSecret call before database open
- Maintains existing secret table structure
- Uses Capacitor SQLite plugin's native encryption support
- Add sqlite-status to valid IPC channels
- Fix SQLite ready signal handling
- Improve database status tracking
- Add proper error handling for status updates
- Keep database connection open during initialization
Technical Details:
- Added sqlite-status to VALID_CHANNELS.invoke list
- Implemented sqlite-status handler with proper verification
- Added database open state verification
- Improved error handling and logging
- Fixed premature database closing
Testing Notes:
- Verify SQLite ready signal is received correctly
- Confirm database stays open after initialization
- Check status updates are processed properly
- Verify error handling for invalid states
Security:
- Validates all IPC channels
- Verifies database state before operations
- Maintains proper connection lifecycle
- Implements proper error boundaries
Author: Matthew Raymer
This commit significantly improves SQLite database management and IPC communication
in the TimeSafari Electron app. Key changes include:
- Add new IPC handlers for database lifecycle management:
- sqlite-open: Open database connections
- sqlite-close: Close database connections
- sqlite-is-db-open: Check database connection status
- get-path: Retrieve database path
- get-base-path: Get base directory path
- Enhance SQLite initialization with:
- Improved error handling and recovery mechanisms
- Detailed logging for all database operations
- State verification and tracking
- Proper cleanup of IPC handlers
- Transaction state management
- Security improvements:
- Validate all IPC channels
- Implement proper file permissions (0o755)
- Add connection state verification
- Secure error handling and logging
- Performance optimizations:
- Implement WAL journal mode
- Configure optimal PRAGMA settings
- Add connection pooling support
- Implement retry logic with exponential backoff
Technical details:
- Add SQLiteError class for detailed error tracking
- Implement handler registration tracking
- Add comprehensive logging with operation tagging
- Update preload script with new valid channels
- Add type definitions for all SQLite operations
Testing:
- All handlers include proper error handling
- State verification before operations
- Recovery mechanisms for failed operations
- Logging for debugging and monitoring
Author: Matthew Raymer
- Comment out all info and debug logs in sqlite-migrations.ts
- Maintain logging structure for future debugging
- Reduce console output during normal operation
- Keep error handling and logging infrastructure intact
This change reduces noise in the console while maintaining the ability
to re-enable detailed logging for debugging purposes.
- Add structured SQLite configuration in main process with separate settings for
initialization and operations
- Implement proper retry logic with configurable attempts and delays
- Add connection pool management to prevent resource exhaustion
- Reduce initialization timeout from 30s to 10s for faster feedback
- Add proper cleanup of timeouts and resources
- Maintain consistent retry behavior in preload script
This change addresses the cascade of SQLite timeout errors seen in the logs
by implementing proper timing controls and resource management. The main
process now handles initialization more robustly with configurable retries,
while the preload script maintains its existing retry behavior for
compatibility.
Security Impact:
- No security implications
- Improves application stability
- Prevents resource exhaustion
Testing:
- Verify SQLite initialization completes within new timeout
- Confirm retry behavior works as expected
- Check that connection pool limits are respected
- Ensure proper cleanup of resources
Current Status:
- SQLite plugin successfully initializes in main process
- Preload script and context bridge working correctly
- IPC handlers for SQLite operations not registered
- Type definitions out of sync with implementation
Changes Made:
- Added comprehensive debug logging in preload script
- Implemented retry logic for SQLite operations (3 attempts, 1s delay)
- Added proper type definitions for SQLite connection options
- Defined strict channel validation for IPC communication
- Enhanced error handling and logging throughout
Type Definitions Updates:
- Aligned ElectronAPI interface with actual implementation
- Added proper typing for SQLite operations
- Structured IPC renderer interface with correct method signatures
Next Steps:
- Register missing IPC handlers in main process
- Update type definitions to match implementation
- Add proper error recovery for SQLite operations
- Address Content Security Policy warnings
Affected Files:
- electron/src/preload.ts
- src/types/electron.d.ts
- src/utils/debug-electron.ts
- src/services/platforms/ElectronPlatformService.ts
Note: This is a transitional commit. While the structure is improved,
database operations are not yet functional due to missing IPC handlers.
This commit updates the Electron preload script and type definitions to improve
SQLite integration and IPC communication. The changes include:
- Enhanced preload script (electron/src/preload.ts):
* Added detailed logging for SQLite operations and IPC communication
* Implemented retry logic for SQLite operations (3 attempts, 1s delay)
* Added proper type definitions for SQLite connection options
* Defined strict channel validation for IPC communication
* Improved error handling and logging throughout
- Updated type definitions (src/types/electron.d.ts):
* Aligned ElectronAPI interface with actual implementation
* Added proper typing for all SQLite operations
* Added environment variables (platform, isDev)
* Structured IPC renderer interface with proper method signatures
Current Status:
- Preload script initializes successfully
- SQLite availability check works (returns true)
- SQLite ready signal is properly received
- Database operations are failing with two types of errors:
1. "CapacitorSQLite not available" during initialization
2. "Cannot read properties of undefined" for SQLite methods
Next Steps:
- Verify context bridge exposure in renderer process
- Check main process SQLite handlers
- Debug database initialization
- Address Content Security Policy warning
Affected Files:
- Modified: electron/src/preload.ts
- Modified: src/types/electron.d.ts
Note: This is a transitional commit. While the preload script and type
definitions are now properly structured, database operations are not yet
functional. Further debugging and fixes are required to resolve the
SQLite integration issues.
- Consolidate preload script with secure IPC bridge and channel validation
- Ensure single initialization path in main process
- Add robust error handling and user feedback
- Fix race conditions in window creation and SQLite ready signal
Current state:
- SQLite initializes successfully in main process
- IPC bridge is established and events are transmitted
- Window creation and loading sequence is correct
- Renderer receives ready signal and mounts app
- Database operations still fail in renderer due to connection issues
Known issues:
- SQLite proxy not properly initialized in renderer
- Database connection not established in renderer
- Error logging attempts to use database before ready
- Connection state management needs improvement
This commit represents a stable point where IPC communication
is working but database operations need to be fixed.
- Remove connection object and connection pool logic
- Call all database methods directly on window.CapacitorSQLite with db name
- Refactor migrations, queries, and exec to match Capacitor SQLite Electron API
- Ensure preload script exposes both window.electron and window.CapacitorSQLite
- Fixes runtime errors related to missing query/run methods on connection
- Improves security and cross-platform compatibility
Co-authored-by: Matthew Raymer
- Add robust logging and error tracking to migration system
- Implement idempotent migrations with transaction safety
- Add detailed progress tracking and state verification
- Improve error handling with recoverable/non-recoverable states
- Add migration version validation and sequential checks
- Implement proper rollback handling with error recording
- Add table state verification and debugging
- Fix migration SQL parsing and parameter handling
- Add connection pool management and retry logic
- Add proper transaction isolation and state tracking
The migration system now provides:
- Atomic transactions per migration
- Automatic rollback on failure
- Detailed error logging and context
- State verification before/after operations
- Proper connection management
- Idempotent operations for safety
This commit improves database reliability and makes debugging
easier while maintaining proper process isolation. The changes
are focused on the migration system and do not require
restructuring the existing ElectronPlatformService architecture.
Technical details:
- Added MigrationError interface for better error tracking
- Added logMigrationProgress helper for consistent logging
- Added debugTableState for verification
- Added executeWithRetry for connection resilience
- Added validateMigrationVersions for safety
- Enhanced SQL parsing with better error handling
- Added proper transaction state management
- Added connection pool with retry logic
- Added detailed logging throughout migration process
Note: This commit addresses the database initialization issues
while maintaining the current architecture. Further improvements
to the ElectronPlatformService initialization will be handled
in a separate commit to maintain clear separation of concerns.
Add proper secret generation using Node's crypto module and initial data setup
for the electron environment. This commit:
- Implements secure random secret generation using crypto.randomBytes()
- Adds initial data migrations (002) with:
- Secret table with cryptographically secure random key
- Settings table with default API server
- Contacts, logs, and temp tables
- Improves SQL parameter handling for migrations
- Adds proper transaction safety and rollback support
- Includes comprehensive logging and error handling
Security:
- Uses Node's crypto module for secure random generation
- Implements proper base64 encoding for secrets
- Maintains transaction safety for all operations
Testing:
- Verified database structure via sqlite3 CLI
- Confirmed successful migration execution
- Validated initial data insertion
- Checked index creation and constraints
Note: This is a temporary solution for secret storage until a more
secure storage mechanism is implemented.
Split initial schema into two sequential migrations to prevent duplicate
table creation and improve migration clarity.
Changes:
- Separate initial schema into two distinct migrations:
* 001_initial_accounts (v1): Create accounts table & index
* 002_secret_and_settings (v2): Create remaining tables (secret, settings, contacts, logs, temp)
- Add version conflict detection to prevent duplicate migration versions
- Ensure migrations are sequential (no gaps)
- Update rollback scripts to only drop relevant tables
Technical Details:
- Add validateMigrationVersions() to check for:
* Duplicate version numbers
* Sequential version ordering
* Gaps in version numbers
- Validate migrations both at definition time and runtime
- Update schema_version tracking to reflect new versioning
Testing:
- Verified no duplicate table creation
- Confirmed migrations run in correct order
- Validated rollback procedures
- Checked version conflict detection
Add robust SQLite migration system with initial schema for TimeSafari desktop app.
Includes comprehensive error handling, transaction safety, and detailed logging.
Key Changes:
- Add migration system with version tracking and rollback support
- Implement initial schema with accounts, secret, settings, contacts tables
- Configure SQLite PRAGMAs for optimal performance and reliability
- Add detailed logging and state verification
- Set up WAL journal mode and connection pooling
Technical Details:
- Use @capacitor-community/sqlite for native SQLite integration
- Implement atomic transactions per migration
- Add SQL validation and parsing utilities
- Configure PRAGMAs:
* foreign_keys = ON
* journal_mode = WAL
* synchronous = NORMAL
* temp_store = MEMORY
* page_size = 4096
* cache_size = 2000
* busy_timeout = 15000
* wal_autocheckpoint = 1000
Note: Current version has duplicate migration v1 entries that need to be
addressed in a follow-up commit to ensure proper versioning.
Testing:
- Verified migrations run successfully
- Confirmed table creation and index setup
- Validated transaction safety and rollback
- Checked logging and error handling
- Add comprehensive logging for database operations
- Implement proper database path handling and permissions
- Set up WAL journal mode and PRAGMA configurations
- Create initial database schema with tables and triggers
- Add retry logic for database operations
- Implement proper error handling and state management
Current state:
- Database initialization works in main process
- Connection creation succeeds with proper permissions
- Schema creation and table setup complete
- Logging system fully implemented
- Known issue: Property name mismatch between main process and renderer
causing read-only mode conflicts (to be fixed in next commit)
Technical details:
- Uses WAL journal mode for better concurrency
- Implements proper file permissions checking
- Sets up foreign key constraints
- Creates tables: users, time_entries, time_goals, time_goal_entries
- Adds automatic timestamp triggers
- Implements proper connection lifecycle management
Security:
- Proper file permissions (755 for directory)
- No hardcoded credentials
- Proper error handling and logging
- Safe file path handling
Author: Matthew Raymer
Changes to sqlite-init.ts:
- Standardized connection options format between main and renderer processes
- Added explicit mode: 'rwc' to force read-write-create mode
- Added connection registration verification
- Added detailed logging of actual file paths
- Added journal mode verification to detect read-only state
- Removed redundant PRAGMA settings (now handled in main process)
- Added more detailed error reporting for connection failures
Security:
- Ensures consistent read-write permissions across processes
- Verifies database is not opened in read-only mode
- Maintains proper file permissions (644) and directory permissions (755)
Testing:
- Verified connection creation in both main and renderer processes
- Confirmed journal mode is set correctly
- Validated connection registration
- Tested error handling for invalid states
Author: Matthew Raymer
- Implement XDG Base Directory Specification for data storage
- Use $XDG_DATA_HOME (defaults to ~/.local/share) for data files
- Add proper directory permissions (700) for security
- Fallback to ~/.timesafari if XDG paths fail
- Add graceful degradation for SQLite failures
- Allow app to boot even if SQLite initialization fails
- Track and expose initialization errors via IPC
- Add availability checks to all SQLite operations
- Improve error reporting and logging
- Security improvements
- Set secure permissions (700) on data directories
- Verify directory permissions on existing paths
- Add proper error handling for permission issues
TODO:
- Fix database creation
- Add retry logic for initialization
- Add reinitialization capability
- Add more detailed error reporting
- Consider fallback storage options
Matthew Raymer
Matthew Raymer