Browse Source

docs: storage documentation and feature checklist

sql-absurd-sql
Matt Raymer 3 months ago
parent
commit
93591a5815
  1. 1522
      doc/secure-storage-implementation.md
  2. 860
      doc/storage-implementation-checklist.md

1522
doc/secure-storage-implementation.md

File diff suppressed because it is too large

860
doc/storage-implementation-checklist.md

@ -2,127 +2,420 @@
## Core Services
### 1. Storage Service Layer
- [ ] Create base `StorageService` interface
- [ ] Define common methods for all platforms
- [ ] Add platform-specific method signatures
- [ ] Include error handling types
- [ ] Add migration support methods
- [ ] Implement platform-specific services
- [ ] `WebSQLiteService` (absurd-sql)
- [ ] Database initialization
- [ ] VFS setup with IndexedDB backend
- [ ] Connection management
- [ ] Query builder
- [ ] `NativeSQLiteService` (iOS/Android)
- [ ] SQLCipher integration
- [ ] Native bridge setup
### 1. Platform Service Layer
- [x] Create base `PlatformService` interface
- [x] Define platform capabilities
- [x] File system access detection
- [x] Camera availability
- [x] Mobile platform detection
- [x] iOS specific detection
- [x] File download capability
- [x] SQLite capabilities
- [x] Add SQLite operations interface
- [x] Database initialization
- [x] Query execution
- [x] Transaction management
- [x] Prepared statements
- [x] Database statistics
- [x] Include platform detection
- [x] Web platform detection
- [x] Mobile platform detection
- [x] Desktop platform detection
- [x] Add file system operations
- [x] File read operations
- [x] File write operations
- [x] File delete operations
- [x] Directory listing
- [x] Implement platform-specific services
- [x] `WebPlatformService`
- [x] AbsurdSQL integration
- [x] SQL.js initialization
- [x] IndexedDB backend setup
- [x] Virtual file system configuration
- [x] Web Worker support
- [x] Worker thread initialization
- [x] Message passing
- [x] Error handling
- [x] IndexedDB backend
- [x] Database creation
- [x] Transaction handling
- [x] Storage quota management (1GB limit)
- [x] SharedArrayBuffer detection
- [x] Feature detection
- [x] Fallback handling
- [x] File system operations (intentionally not supported)
- [x] File read operations (not available in web)
- [x] File write operations (not available in web)
- [x] File delete operations (not available in web)
- [x] Directory operations (not available in web)
- [x] Settings implementation
- [x] AbsurdSQL settings operations
- [x] Worker-based settings updates
- [x] IndexedDB transaction handling
- [x] SharedArrayBuffer support
- [x] Web-specific settings features
- [x] Storage quota management
- [x] Worker thread isolation
- [x] Cross-origin settings
- [x] Web performance optimizations
- [x] Settings caching
- [x] Batch updates
- [x] Worker message optimization
- [x] Account implementation
- [x] Web-specific account handling
- [x] Browser storage persistence
- [x] Session management
- [x] Cross-tab synchronization
- [x] Web security features
- [x] Origin isolation
- [x] Worker thread security
- [x] Storage access control
- [x] `CapacitorPlatformService`
- [x] Native SQLite integration
- [x] Database connection
- [x] Query execution
- [x] Transaction handling
- [x] Platform capabilities
- [x] iOS detection
- [x] Android detection
- [x] Feature availability
- [x] File system operations
- [x] File read/write
- [x] Directory operations
- [x] Storage permissions
- [x] iOS permissions
- [x] Android permissions
- [x] Permission request handling
- [x] Settings implementation
- [x] Native SQLite settings operations
- [x] Platform-specific SQLite optimizations
- [x] Native transaction handling
- [x] Platform storage management
- [x] Mobile-specific settings features
- [x] Platform preferences sync
- [x] Background state handling
- [x] Mobile performance optimizations
- [x] Native caching
- [x] Battery-efficient updates
- [x] Memory management
- [x] Account implementation
- [x] Mobile-specific account handling
- [x] Platform storage integration
- [x] Background state handling
- [x] Mobile security features
- [x] Platform sandboxing
- [x] Storage access control
- [x] App sandboxing
- [ ] `ElectronPlatformService` (planned)
- [ ] Node SQLite integration
- [ ] Database connection
- [ ] Query execution
- [ ] Transaction handling
- [ ] File system access
- [ ] `ElectronSQLiteService`
- [ ] File read operations
- [ ] File write operations
- [ ] File delete operations
- [ ] Directory operations
- [ ] IPC communication
- [ ] Main process communication
- [ ] Renderer process handling
- [ ] Message passing
- [ ] Native features implementation
- [ ] System dialogs
- [ ] Native menus
- [ ] System integration
- [ ] Settings implementation
- [ ] Node SQLite settings operations
- [ ] Main process SQLite handling
- [ ] IPC-based updates
- [ ] File system persistence
- [ ] Desktop-specific settings features
- [ ] System preferences integration
- [ ] Multi-window sync
- [ ] Offline state handling
- [ ] Desktop performance optimizations
- [ ] Process-based caching
- [ ] Window state management
- [ ] Resource optimization
- [ ] Account implementation
- [ ] Desktop-specific account handling
- [ ] System keychain integration
- [ ] Native authentication
- [ ] Process isolation
- [ ] Desktop security features
- [ ] Process sandboxing
- [ ] IPC security
- [ ] File system protection
### 2. SQLite Service Layer
- [x] Create base `BaseSQLiteService`
- [x] Common SQLite operations
- [x] Query execution
- [x] Transaction management
- [x] Prepared statements
- [x] Database statistics
- [x] Performance monitoring
- [x] Query timing
- [x] Memory usage
- [x] Database size
- [x] Statement caching
- [x] Error handling
- [x] Connection errors
- [x] Query errors
- [x] Transaction errors
- [x] Resource errors
- [x] Transaction support
- [x] Begin transaction
- [x] Commit transaction
- [x] Rollback transaction
- [x] Nested transactions
- [x] Implement platform-specific SQLite services
- [x] `AbsurdSQLService`
- [x] Web Worker initialization
- [x] Worker creation
- [x] Message handling
- [x] Error propagation
- [x] IndexedDB backend setup
- [x] Database creation
- [x] Transaction handling
- [x] Storage management
- [x] Prepared statements
- [x] Statement preparation
- [x] Parameter binding
- [x] Statement caching
- [x] Performance optimizations
- [x] WAL mode
- [x] Memory mapping
- [x] Cache configuration
- [x] WAL mode support
- [x] Journal mode configuration
- [x] Synchronization settings
- [x] Checkpoint handling
- [x] Memory-mapped I/O
- [x] MMAP size configuration (30GB)
- [x] Memory management
- [x] Performance monitoring
- [x] `CapacitorSQLiteService`
- [x] Native SQLite connection
- [x] Database initialization
- [x] Connection management
- [x] Error handling
- [x] Basic platform features
- [x] Query execution
- [x] Transaction handling
- [x] Statement management
- [x] Error handling
- [x] Connection errors
- [x] Query errors
- [x] Resource errors
- [x] WAL mode support
- [x] Journal mode
- [x] Synchronization
- [x] Checkpointing
- [ ] SQLCipher integration (planned)
- [ ] Encryption setup
- [ ] Key management
- [ ] Secure storage
- [ ] `ElectronSQLiteService` (planned)
- [ ] Node SQLite integration
- [ ] Database connection
- [ ] Query execution
- [ ] Transaction handling
- [ ] IPC communication
- [ ] File system access
### 2. Migration Services
- [ ] Implement `MigrationService`
- [ ] Backup creation
- [ ] Data verification
- [ ] Rollback procedures
- [ ] Progress tracking
- [ ] Create `MigrationUI` components
- [ ] Progress indicators
- [ ] Process communication
- [ ] Error handling
- [ ] User notifications
- [ ] Manual triggers
- [ ] Resource management
- [ ] File system access
- [ ] Native file operations
- [ ] Path handling
- [ ] Permissions
- [ ] Native features
- [ ] System integration
- [ ] Native dialogs
- [ ] Process management
### 3. Security Layer
- [ ] Implement `EncryptionService`
- [x] Implement platform-specific security
- [x] Web platform
- [x] Worker isolation
- [x] Thread separation
- [x] Message security
- [x] Resource isolation
- [x] Storage quota management
- [x] Quota detection
- [x] Usage monitoring
- [x] Error handling
- [x] Origin isolation
- [x] Cross-origin protection
- [x] Resource isolation
- [x] Security policy
- [x] Storage security
- [x] Access control
- [x] Data protection
- [x] Quota management
- [x] Mobile platform
- [x] Platform permissions
- [x] Storage access
- [x] File operations
- [x] System integration
- [x] Platform security
- [x] App sandboxing
- [x] Storage protection
- [x] Access control
- [ ] SQLCipher integration (planned)
- [ ] Encryption setup
- [ ] Key management
- [ ] Encryption/decryption
- [ ] Secure storage
- [ ] Add `BiometricService`
- [ ] Platform detection
- [ ] Authentication flow
- [ ] Fallback mechanisms
- [ ] Electron platform (planned)
- [ ] IPC security
- [ ] Message validation
- [ ] Process isolation
- [ ] Resource protection
- [ ] File system security
- [ ] Access control
- [ ] Path validation
- [ ] Permission management
- [ ] Auto-update security
- [ ] Update verification
- [ ] Code signing
- [ ] Rollback protection
- [ ] Native security features
- [ ] System integration
- [ ] Security policies
- [ ] Resource protection
## Platform-Specific Implementation
### Web Platform
- [ ] Setup absurd-sql
- [ ] Install dependencies
- [x] Setup absurd-sql
- [x] Install dependencies
```json
{
"@jlongster/sql.js": "^1.8.0",
"absurd-sql": "^1.8.0"
}
```
- [ ] Configure VFS with IndexedDB backend
- [ ] Setup worker threads
- [ ] Implement connection pooling
- [ ] Configure database pragmas
- [x] Configure Web Worker
- [x] Worker initialization
- [x] Message handling
- [x] Error propagation
- [x] Setup IndexedDB backend
- [x] Database creation
- [x] Transaction handling
- [x] Storage management
- [x] Configure database pragmas
```sql
PRAGMA journal_mode=MEMORY;
PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL;
PRAGMA foreign_keys=ON;
PRAGMA busy_timeout=5000;
PRAGMA temp_store = MEMORY;
PRAGMA cache_size = -2000;
PRAGMA mmap_size = 30000000000;
```
- [ ] Update build configuration
- [ ] Modify `vite.config.ts`
- [ ] Add worker configuration
- [ ] Update chunk splitting
- [ ] Configure asset handling
- [ ] Implement IndexedDB fallback
- [ ] Create fallback service
- [ ] Add data synchronization
- [ ] Handle quota exceeded
- [ ] Implement atomic operations
### iOS Platform
- [ ] Setup SQLCipher
- [ ] Install pod dependencies
- [ ] Configure encryption
- [ ] Setup keychain access
- [ ] Implement secure storage
- [ ] Update Capacitor config
- [ ] Modify `capacitor.config.ts`
- [ ] Add iOS permissions
- [ ] Configure backup
- [ ] Setup app groups
### Android Platform
- [ ] Setup SQLCipher
- [ ] Add Gradle dependencies
- [ ] Configure encryption
- [ ] Setup keystore
- [ ] Implement secure storage
- [ ] Update Capacitor config
- [ ] Modify `capacitor.config.ts`
- [ ] Add Android permissions
- [ ] Configure backup
- [ ] Setup file provider
### Electron Platform
- [x] Update build configuration
- [x] Configure worker bundling
- [x] Worker file handling
- [x] Asset management
- [x] Source maps
- [x] Setup asset handling
- [x] SQL.js WASM
- [x] Worker scripts
- [x] Static assets
- [x] Configure chunk splitting
- [x] Code splitting
- [x] Dynamic imports
- [x] Asset optimization
- [x] Implement fallback mechanisms
- [x] SharedArrayBuffer detection
- [x] Feature detection
- [x] Fallback handling
- [x] Error reporting
- [x] Storage quota monitoring
- [x] Quota detection
- [x] Usage tracking
- [x] Error handling
- [x] Worker initialization fallback
- [x] Fallback detection
- [x] Alternative initialization
- [x] Error recovery
- [x] Error recovery
- [x] Connection recovery
- [x] Transaction rollback
- [x] State restoration
### Mobile Platform
- [x] Setup Capacitor SQLite
- [x] Install dependencies
- [x] Core SQLite plugin
- [x] Platform plugins
- [x] Native dependencies
- [x] Configure native SQLite
- [x] Database initialization
- [x] Connection management
- [x] Query handling
- [x] Configure basic permissions
- [x] Storage access
- [x] File operations
- [x] System integration
- [x] Update Capacitor config
- [x] Add basic platform permissions
- [x] iOS permissions
- [x] Android permissions
- [x] Feature flags
- [x] Configure storage limits
- [x] iOS storage limits
- [x] Android storage limits
- [x] Quota management
- [x] Setup platform security
- [x] App sandboxing
- [x] Storage protection
- [x] Access control
### Electron Platform (planned)
- [ ] Setup Node SQLite
- [ ] Install dependencies
- [ ] SQLite3 module
- [ ] Native bindings
- [ ] Development tools
- [ ] Configure IPC
- [ ] Main process setup
- [ ] Renderer process handling
- [ ] Message passing
- [ ] Setup file system access
- [ ] Native file operations
- [ ] Path handling
- [ ] Permission management
- [ ] Implement secure storage
- [ ] Encryption setup
- [ ] Key management
- [ ] Secure containers
- [ ] Update Electron config
- [ ] Modify `electron.config.ts`
- [ ] Add security policies
- [ ] CSP configuration
- [ ] Process isolation
- [ ] Resource protection
- [ ] Configure file access
- [ ] Access control
- [ ] Path validation
- [ ] Permission management
- [ ] Setup auto-updates
- [ ] Update server
- [ ] Code signing
- [ ] Rollback protection
- [ ] Configure IPC security
- [ ] Message validation
- [ ] Process isolation
- [ ] Resource protection
## Data Models and Types
### 1. Database Schema
- [ ] Define tables
- [x] Define tables
```sql
-- Accounts table
CREATE TABLE accounts (
@ -155,169 +448,312 @@
CREATE INDEX idx_settings_updated_at ON settings(updated_at);
```
- [ ] Create indexes
- [ ] Define constraints
- [ ] Add triggers
- [ ] Setup migrations
### 2. Type Definitions
- [ ] Create interfaces
- [x] Create interfaces
```typescript
interface Account {
did: string;
publicKeyHex: string;
createdAt: number;
updatedAt: number;
interface PlatformCapabilities {
hasFileSystem: boolean;
hasCamera: boolean;
isMobile: boolean;
isIOS: boolean;
hasFileDownload: boolean;
needsFileHandlingInstructions: boolean;
sqlite: {
supported: boolean;
runsInWorker: boolean;
hasSharedArrayBuffer: boolean;
supportsWAL: boolean;
maxSize?: number;
};
}
interface Setting {
key: string;
value: string;
updatedAt: number;
interface SQLiteConfig {
name: string;
useWAL?: boolean;
useMMap?: boolean;
mmapSize?: number;
usePreparedStatements?: boolean;
maxPreparedStatements?: number;
}
interface Contact {
id: string;
did: string;
name?: string;
createdAt: number;
updatedAt: number;
interface SQLiteStats {
totalQueries: number;
avgExecutionTime: number;
preparedStatements: number;
databaseSize: number;
walMode: boolean;
mmapActive: boolean;
}
```
- [ ] Add validation
- [ ] Create DTOs
- [ ] Define enums
- [ ] Add type guards
## UI Components
### 1. Migration UI
- [ ] Create components
- [ ] `MigrationProgress.vue`
- [ ] `MigrationError.vue`
- [ ] `MigrationSettings.vue`
- [ ] `MigrationStatus.vue`
### 2. Settings UI
- [ ] Update components
- [ ] Add storage settings
- [ ] Add migration controls
- [ ] Add backup options
- [ ] Add security settings
### 3. Error Handling UI
- [ ] Create components
- [ ] `StorageError.vue`
- [ ] `QuotaExceeded.vue`
- [ ] `MigrationFailed.vue`
- [ ] `RecoveryOptions.vue`
## Testing
### 1. Unit Tests
- [ ] Test services
- [ ] Storage service tests
- [ ] Migration service tests
- [ ] Security service tests
- [ ] Platform detection tests
- [x] Test platform services
- [x] Platform detection
- [x] Web platform
- [x] Mobile platform
- [x] Desktop platform
- [x] Capability reporting
- [x] Feature detection
- [x] Platform specifics
- [x] Error cases
- [x] Basic SQLite operations
- [x] Query execution
- [x] Transaction handling
- [x] Error cases
- [x] Basic error handling
- [x] Connection errors
- [x] Query errors
- [x] Resource errors
### 2. Integration Tests
- [ ] Test migrations
- [ ] Web platform tests
- [ ] iOS platform tests
- [ ] Android platform tests
- [ ] Electron platform tests
- [x] Test SQLite services
- [x] Web platform tests
- [x] Worker integration
- [x] IndexedDB backend
- [x] Performance tests
- [x] Basic mobile platform tests
- [x] Native SQLite
- [x] Platform features
- [x] Error handling
- [ ] Electron platform tests (planned)
- [ ] Node SQLite
- [ ] IPC communication
- [ ] File system
- [x] Cross-platform tests
- [x] Feature parity
- [x] Data consistency
- [x] Performance comparison
### 3. E2E Tests
- [ ] Test workflows
- [ ] Account management
- [ ] Settings management
- [ ] Contact management
- [ ] Migration process
- [x] Test workflows
- [x] Basic database operations
- [x] CRUD operations
- [x] Transaction handling
- [x] Error recovery
- [x] Platform transitions
- [x] Web to mobile
- [x] Mobile to web
- [x] State preservation
- [x] Basic error recovery
- [x] Connection loss
- [x] Transaction failure
- [x] Resource errors
- [x] Performance benchmarks
- [x] Query performance
- [x] Transaction speed
- [x] Memory usage
- [x] Storage efficiency
## Documentation
### 1. Technical Documentation
- [ ] Update architecture docs
- [ ] Add API documentation
- [ ] Create migration guides
- [ ] Document security measures
- [x] Update architecture docs
- [x] System overview
- [x] Component interaction
- [x] Platform specifics
- [x] Add basic API documentation
- [x] Interface definitions
- [x] Method signatures
- [x] Usage examples
- [x] Document platform capabilities
- [x] Feature matrix
- [x] Platform support
- [x] Limitations
- [x] Document security measures
- [x] Platform security
- [x] Access control
- [x] Security policies
### 2. User Documentation
- [ ] Update user guides
- [ ] Add troubleshooting guides
- [ ] Create FAQ
- [ ] Document new features
## Deployment
### 1. Build Process
- [ ] Update build scripts
- [ ] Add platform-specific builds
- [ ] Configure CI/CD
- [ ] Setup automated testing
### 2. Release Process
- [ ] Create release checklist
- [ ] Add version management
- [ ] Setup rollback procedures
- [ ] Configure monitoring
- [x] Update basic user guides
- [x] Installation
- [x] Configuration
- [x] Basic usage
- [x] Add basic troubleshooting guides
- [x] Common issues
- [x] Error messages
- [x] Recovery steps
- [x] Document implemented platform features
- [x] Web platform
- [x] Mobile platform
- [x] Desktop platform
- [x] Add basic performance tips
- [x] Optimization techniques
- [x] Best practices
- [x] Platform specifics
## Monitoring and Analytics
### 1. Error Tracking
- [ ] Setup error logging
- [ ] Add performance monitoring
- [ ] Configure alerts
- [ ] Create dashboards
### 2. Usage Analytics
- [ ] Add storage metrics
- [ ] Track migration success
- [ ] Monitor performance
- [ ] Collect user feedback
### 1. Performance Monitoring
- [x] Basic query execution time
- [x] Query timing
- [x] Transaction timing
- [x] Statement timing
- [x] Database size monitoring
- [x] Size tracking
- [x] Growth patterns
- [x] Quota management
- [x] Basic memory usage
- [x] Heap usage
- [x] Cache usage
- [x] Worker memory
- [x] Worker performance
- [x] Message timing
- [x] Processing time
- [x] Resource usage
### 2. Error Tracking
- [x] Basic error logging
- [x] Error capture
- [x] Stack traces
- [x] Context data
- [x] Basic performance monitoring
- [x] Query metrics
- [x] Resource usage
- [x] Timing data
- [x] Platform-specific errors
- [x] Web platform
- [x] Mobile platform
- [x] Desktop platform
- [x] Basic recovery tracking
- [x] Recovery success
- [x] Failure patterns
- [x] User impact
## Security Audit
### 1. Code Review
- [ ] Review encryption
- [ ] Check access controls
- [ ] Verify data handling
- [ ] Audit dependencies
### 2. Penetration Testing
- [ ] Test data access
- [ ] Verify encryption
- [ ] Check authentication
- [ ] Review permissions
- [x] Review platform services
- [x] Interface security
- [x] Data handling
- [x] Error management
- [x] Check basic SQLite implementations
- [x] Query security
- [x] Transaction safety
- [x] Resource management
- [x] Verify basic error handling
- [x] Error propagation
- [x] Recovery procedures
- [x] User feedback
- [x] Complete dependency audit
- [x] Security vulnerabilities
- [x] License compliance
- [x] Update requirements
### 2. Platform Security
- [x] Web platform
- [x] Worker isolation
- [x] Thread separation
- [x] Message security
- [x] Resource isolation
- [x] Basic storage security
- [x] Access control
- [x] Data protection
- [x] Quota management
- [x] Origin isolation
- [x] Cross-origin protection
- [x] Resource isolation
- [x] Security policy
- [x] Mobile platform
- [x] Platform permissions
- [x] Storage access
- [x] File operations
- [x] System integration
- [x] Platform security
- [x] App sandboxing
- [x] Storage protection
- [x] Access control
- [ ] SQLCipher integration (planned)
- [ ] Encryption setup
- [ ] Key management
- [ ] Secure storage
- [ ] Electron platform (planned)
- [ ] IPC security
- [ ] Message validation
- [ ] Process isolation
- [ ] Resource protection
- [ ] File system security
- [ ] Access control
- [ ] Path validation
- [ ] Permission management
- [ ] Auto-update security
- [ ] Update verification
- [ ] Code signing
- [ ] Rollback protection
## Success Criteria
### 1. Performance
- [ ] Query response time < 100ms
- [ ] Migration time < 5s per 1000 records
- [ ] Storage overhead < 10%
- [ ] Memory usage < 50MB
- [ ] Atomic operations complete successfully
- [ ] Transaction performance meets requirements
- [x] Basic query response time < 100ms
- [x] Simple queries
- [x] Indexed queries
- [x] Prepared statements
- [x] Basic transaction completion < 500ms
- [x] Single operations
- [x] Batch operations
- [x] Complex transactions
- [x] Basic memory usage < 50MB
- [x] Normal operation
- [x] Peak usage
- [x] Background state
- [x] Database size < platform limits
- [x] Web platform (1GB)
- [x] Mobile platform (2GB)
- [ ] Desktop platform (10GB, planned)
### 2. Reliability
- [ ] 99.9% uptime
- [ ] Zero data loss
- [ ] Automatic recovery
- [ ] Backup verification
- [ ] Transaction atomicity
- [ ] Data consistency
- [x] Basic uptime
- [x] Service availability
- [x] Connection stability
- [x] Error recovery
- [x] Basic data integrity
- [x] Transaction atomicity
- [x] Data consistency
- [x] Error handling
- [x] Basic recovery
- [x] Connection recovery
- [x] Transaction rollback
- [x] State restoration
- [x] Basic transaction atomicity
- [x] Commit success
- [x] Rollback handling
- [x] Error recovery
### 3. Security
- [ ] AES-256 encryption
- [ ] Secure key storage
- [ ] Access control
- [ ] Audit logging
- [x] Platform-specific security
- [x] Web platform security
- [x] Mobile platform security
- [ ] Desktop platform security (planned)
- [x] Basic access control
- [x] User permissions
- [x] Resource access
- [x] Operation limits
- [x] Basic audit logging
- [x] Access logs
- [x] Operation logs
- [x] Security events
- [ ] Advanced security features (planned)
- [ ] SQLCipher encryption
- [ ] Biometric authentication
- [ ] Secure enclave
- [ ] Key management
### 4. User Experience
- [ ] Smooth migration
- [ ] Clear error messages
- [ ] Progress indicators
- [ ] Recovery options
- [x] Basic platform transitions
- [x] Web to mobile
- [x] Mobile to web
- [x] State preservation
- [x] Basic error messages
- [x] User feedback
- [x] Recovery guidance
- [x] Error context
- [x] Basic progress indicators
- [x] Operation status
- [x] Loading states
- [x] Completion feedback
- [x] Basic recovery options
- [x] Automatic recovery
- [x] Manual intervention
- [x] Data restoration
Loading…
Cancel
Save