Files
daily-notification-plugin/docs/android-implementation-directive-phase2.md
Matthew Raymer 35babb3126 docs(alarms): unify and enhance alarm directive documentation stack
Create unified alarm documentation system with strict role separation:
- Doc A: Platform capability reference (canonical OS facts)
- Doc B: Plugin behavior exploration (executable test harness)
- Doc C: Plugin requirements (guarantees, JS/TS contract, traceability)

Changes:
- Add canonical rule to Doc A preventing platform fact duplication
- Convert Doc B to pure executable test spec with scenario tables
- Complete Doc C with guarantees matrix, JS/TS API contract, recovery
  contract, unsupported behaviors, and traceability matrix
- Remove implementation details from unified directive
- Add compliance milestone tracking and iOS parity gates
- Add deprecation banners to legacy platform docs

All documents now enforce strict role separation with cross-references
to prevent duplication and ensure single source of truth.
2025-11-25 10:09:46 +00:00

26 KiB

Android Implementation Directive: Phase 2 - Force Stop Detection & Recovery

Author: Matthew Raymer
Date: November 2025
Status: Phase 2 - Force Stop Recovery
Version: 1.0.0
Last Synced With Plugin Version: v1.1.0

Implements: Plugin Requirements §3.1.4 - Force Stop Recovery

Purpose

Phase 2 implements force stop detection and comprehensive recovery. This handles the scenario where the user force-stops the app, causing all alarms to be cancelled by the OS.

⚠️ IMPORTANT: This phase modifies and extends the ReactivationManager introduced in Phase 1. Do not create a second copy; update the existing class.

Prerequisites: Phase 1 must be complete (cold start recovery implemented).

Scope: Force stop detection, scenario differentiation, and full alarm recovery.

Reference:


1. Acceptance Criteria

1.1 Definition of Done

Phase 2 is complete when:

  1. Force stop scenario is detected correctly

    • Detection: (DB schedules count > 0) && (AlarmManager alarms count == 0)
    • Detection runs on app launch (via ReactivationManager)
    • False positives avoided (distinguishes from first launch)
  2. All past alarms are marked as missed

    • All schedules with nextRunAt < currentTime marked as missed
    • Missed notifications created/updated in database
    • History records created for each missed alarm
  3. All future alarms are rescheduled

    • All schedules with nextRunAt >= currentTime rescheduled
    • Repeating schedules calculate next occurrence correctly
    • No duplicate alarms created
  4. Recovery handles both notify and fetch schedules

    • notify schedules rescheduled via AlarmManager
    • fetch schedules rescheduled via WorkManager
    • Both types recovered completely
  5. Recovery never crashes the app

    • All exceptions caught and logged
    • Partial recovery logged if some schedules fail
    • App continues normally even if recovery fails

1.2 Success Metrics

Metric Target Measurement
Force stop detection accuracy 100% Manual verification via logs
Past alarm recovery rate 100% All past alarms marked as missed
Future alarm recovery rate > 95% History table outcome field
Recovery execution time < 3 seconds Log timestamp difference
Crash rate 0% No exceptions propagate to app

1.3 Out of Scope (Phase 2)

  • Warm start optimization (Phase 3)
  • Boot receiver missed alarm handling (Phase 3)
  • Callback event emission (Phase 3)
  • User notification of missed alarms (Phase 3)

2. Implementation: Force Stop Detection

2.1 Update ReactivationManager

File: android/src/main/java/com/timesafari/dailynotification/ReactivationManager.kt

Location: Add scenario detection after Phase 1 implementation

2.2 Scenario Detection

⚠️ Canonical Source: This method supersedes any earlier scenario detection code shown in the full directive.

/**
 * Detect recovery scenario based on AlarmManager state vs database
 * 
 * Phase 2: Adds force stop detection
 * 
 * This is the normative implementation of scenario detection.
 * 
 * @return RecoveryScenario enum value
 */
private suspend fun detectScenario(): RecoveryScenario {
    val db = DailyNotificationDatabase.getDatabase(context)
    val dbSchedules = db.scheduleDao().getEnabled()
    
    // Check for first launch (empty DB)
    if (dbSchedules.isEmpty()) {
        Log.d(TAG, "No schedules in database - first launch (NONE)")
        return RecoveryScenario.NONE
    }
    
    // Check for boot recovery (set by BootReceiver)
    if (isBootRecovery()) {
        Log.i(TAG, "Boot recovery detected")
        return RecoveryScenario.BOOT
    }
    
    // Check for force stop: DB has schedules but no alarms exist
    if (!alarmsExist()) {
        Log.i(TAG, "Force stop detected: DB has ${dbSchedules.size} schedules, but no alarms exist")
        return RecoveryScenario.FORCE_STOP
    }
    
    // Normal cold start: DB has schedules and alarms exist
    // (Alarms may have fired or may be future alarms - need to verify/resync)
    Log.d(TAG, "Cold start: DB has ${dbSchedules.size} schedules, alarms exist")
    return RecoveryScenario.COLD_START
}

/**
 * Check if this is a boot recovery scenario
 * 
 * BootReceiver sets a flag in SharedPreferences when boot completes.
 * This allows ReactivationManager to detect boot scenario.
 * 
 * @return true if boot recovery, false otherwise
 */
private fun isBootRecovery(): Boolean {
    val prefs = context.getSharedPreferences("dailynotification_recovery", Context.MODE_PRIVATE)
    val lastBootAt = prefs.getLong("last_boot_at", 0)
    val currentTime = System.currentTimeMillis()
    
    // Boot flag is valid for 60 seconds after boot
    // This prevents false positives from stale flags
    if (lastBootAt > 0 && (currentTime - lastBootAt) < 60000) {
        // Clear the flag after reading
        prefs.edit().remove("last_boot_at").apply()
        return true
    }
    
    return false
}

/**
 * Check if alarms exist in AlarmManager
 * 
 * **Correction**: Replaces unreliable nextAlarmClock check with PendingIntent check.
 * This eliminates false positives from nextAlarmClock.
 * 
 * @return true if at least one alarm exists, false otherwise
 */
private fun alarmsExist(): Boolean {
    return try {
        // Check if any PendingIntent for our receiver exists
        // This is more reliable than nextAlarmClock
        val intent = Intent(context, DailyNotificationReceiver::class.java).apply {
            action = "com.timesafari.daily.NOTIFICATION"
        }
        val pendingIntent = PendingIntent.getBroadcast(
            context,
            0, // Use 0 to check for any alarm
            intent,
            PendingIntent.FLAG_NO_CREATE or PendingIntent.FLAG_IMMUTABLE
        )
        val exists = pendingIntent != null
        Log.d(TAG, "Alarm check: alarms exist = $exists")
        exists
    } catch (e: Exception) {
        Log.e(TAG, "Error checking if alarms exist", e)
        // On error, assume no alarms (conservative for force stop detection)
        false
    }
}

/**
 * Recovery scenario enum
 * 
 * **Corrected Model**: Only these four scenarios are supported
 */
enum class RecoveryScenario {
    COLD_START,    // Process killed, alarms may or may not exist
    FORCE_STOP,    // Alarms cleared, DB still populated
    BOOT,          // Device reboot
    NONE           // No recovery required (warm resume or first launch)
}

2.3 Update performRecovery()

/**
 * Perform recovery on app launch
 * Phase 2: Adds force stop handling
 */
fun performRecovery() {
    CoroutineScope(Dispatchers.IO).launch {
        try {
            withTimeout(TimeUnit.SECONDS.toMillis(RECOVERY_TIMEOUT_SECONDS)) {
                Log.i(TAG, "Starting app launch recovery (Phase 2)")
                
                // Step 1: Detect scenario
                val scenario = detectScenario()
                Log.i(TAG, "Detected scenario: $scenario")
                
                // Step 2: Handle based on scenario
                when (scenario) {
                    RecoveryScenario.FORCE_STOP -> {
                        // Phase 2: Force stop recovery (new in this phase)
                        val result = performForceStopRecovery()
                        Log.i(TAG, "Force stop recovery completed: $result")
                    }
                    RecoveryScenario.COLD_START -> {
                        // Phase 1: Cold start recovery (reuse existing implementation)
                        val result = performColdStartRecovery()
                        Log.i(TAG, "Cold start recovery completed: $result")
                    }
                    RecoveryScenario.BOOT -> {
                        // Phase 3: Boot recovery (handled via ReactivationManager)
                        // Boot recovery uses same logic as force stop (all alarms wiped)
                        val result = performForceStopRecovery()
                        Log.i(TAG, "Boot recovery completed: $result")
                    }
                    RecoveryScenario.NONE -> {
                        // No recovery needed (warm resume or first launch)
                        Log.d(TAG, "No recovery needed (NONE scenario)")
                    }
                }
                
                Log.i(TAG, "App launch recovery completed")
            }
        } catch (e: Exception) {
            Log.e(TAG, "Recovery failed (non-fatal): ${e.message}", e)
            try {
                recordRecoveryFailure(e)
            } catch (historyError: Exception) {
                Log.w(TAG, "Failed to record recovery failure in history", historyError)
            }
        }
    }
}

3. Implementation: Force Stop Recovery

3.1 Force Stop Recovery Method

/**
 * Perform force stop recovery
 * 
 * Force stop scenario: ALL alarms were cancelled by OS
 * Need to:
 * 1. Mark all past alarms as missed
 * 2. Reschedule all future alarms
 * 3. Handle both notify and fetch schedules
 * 
 * @return RecoveryResult with counts
 */
private suspend fun performForceStopRecovery(): RecoveryResult {
    val db = DailyNotificationDatabase.getDatabase(context)
    val currentTime = System.currentTimeMillis()
    
    Log.i(TAG, "Force stop recovery: recovering all schedules")
    
    val dbSchedules = try {
        db.scheduleDao().getEnabled()
    } catch (e: Exception) {
        Log.e(TAG, "Failed to query schedules", e)
        return RecoveryResult(0, 0, 0, 1)
    }
    
    var missedCount = 0
    var rescheduledCount = 0
    var errors = 0
    
    dbSchedules.forEach { schedule ->
        try {
            when (schedule.kind) {
                "notify" -> {
                    val result = recoverNotifySchedule(schedule, currentTime, db)
                    missedCount += result.missedCount
                    rescheduledCount += result.rescheduledCount
                    errors += result.errors
                }
                "fetch" -> {
                    val result = recoverFetchSchedule(schedule, currentTime, db)
                    rescheduledCount += result.rescheduledCount
                    errors += result.errors
                }
                else -> {
                    Log.w(TAG, "Unknown schedule kind: ${schedule.kind}")
                }
            }
        } catch (e: Exception) {
            errors++
            Log.e(TAG, "Failed to recover schedule: ${schedule.id}", e)
        }
    }
    
    val result = RecoveryResult(
        missedCount = missedCount,
        rescheduledCount = rescheduledCount,
        verifiedCount = 0, // Not applicable for force stop
        errors = errors
    )
    
    recordRecoveryHistory(db, "force_stop", result)
    
    Log.i(TAG, "Force stop recovery complete: $result")
    return result
}

/**
 * Data class for schedule recovery results
 */
private data class ScheduleRecoveryResult(
    val missedCount: Int = 0,
    val rescheduledCount: Int = 0,
    val errors: Int = 0
)

3.2 Recover Notify Schedule

Behavior: Handles kind == "notify" schedules. Reschedules via AlarmManager.

/**
 * Recover a notify schedule after force stop
 * 
 * Handles notify schedules (kind == "notify")
 * 
 * @param schedule Schedule to recover
 * @param currentTime Current time in milliseconds
 * @param db Database instance
 * @return ScheduleRecoveryResult
 */
private suspend fun recoverNotifySchedule(
    schedule: Schedule,
    currentTime: Long,
    db: DailyNotificationDatabase
): ScheduleRecoveryResult {
    
    // Data integrity check
    if (schedule.id.isBlank()) {
        Log.w(TAG, "Skipping invalid schedule: empty ID")
        return ScheduleRecoveryResult(errors = 1)
    }
    
    var missedCount = 0
    var rescheduledCount = 0
    var errors = 0
    
    // Calculate next run time
    val nextRunTime = calculateNextRunTime(schedule, currentTime)
    
    if (nextRunTime < currentTime) {
        // Past alarm - was missed during force stop
        Log.i(TAG, "Past alarm detected: ${schedule.id} scheduled for $nextRunTime")
        
        try {
            // Mark as missed
            markMissedNotification(schedule, nextRunTime, db)
            missedCount++
        } catch (e: Exception) {
            errors++
            Log.e(TAG, "Failed to mark missed notification: ${schedule.id}", e)
        }
        
        // Reschedule next occurrence if repeating
        if (isRepeating(schedule)) {
            try {
                val nextOccurrence = calculateNextOccurrence(schedule, currentTime)
                rescheduleAlarm(schedule, nextOccurrence, db)
                rescheduledCount++
                Log.i(TAG, "Rescheduled next occurrence: ${schedule.id} for $nextOccurrence")
            } catch (e: Exception) {
                errors++
                Log.e(TAG, "Failed to reschedule next occurrence: ${schedule.id}", e)
            }
        }
    } else {
        // Future alarm - reschedule immediately
        Log.i(TAG, "Future alarm detected: ${schedule.id} scheduled for $nextRunTime")
        
        try {
            rescheduleAlarm(schedule, nextRunTime, db)
            rescheduledCount++
        } catch (e: Exception) {
            errors++
            Log.e(TAG, "Failed to reschedule future alarm: ${schedule.id}", e)
        }
    }
    
    return ScheduleRecoveryResult(missedCount, rescheduledCount, errors)
}

3.3 Recover Fetch Schedule

Behavior: Handles kind == "fetch" schedules. Reschedules via WorkManager.

/**
 * Recover a fetch schedule after force stop
 * 
 * Handles fetch schedules (kind == "fetch")
 * 
 * @param schedule Schedule to recover
 * @param currentTime Current time in milliseconds
 * @param db Database instance
 * @return ScheduleRecoveryResult
 */
private suspend fun recoverFetchSchedule(
    schedule: Schedule,
    currentTime: Long,
    db: DailyNotificationDatabase
): ScheduleRecoveryResult {
    
    // Data integrity check
    if (schedule.id.isBlank()) {
        Log.w(TAG, "Skipping invalid schedule: empty ID")
        return ScheduleRecoveryResult(errors = 1)
    }
    
    var rescheduledCount = 0
    var errors = 0
    
    try {
        // Reschedule fetch work via WorkManager
        val config = ContentFetchConfig(
            enabled = schedule.enabled,
            schedule = schedule.cron ?: schedule.clockTime ?: "0 9 * * *",
            url = null, // Will use registered native fetcher
            timeout = 30000,
            retryAttempts = 3,
            retryDelay = 1000,
            callbacks = CallbackConfig()
        )
        
        FetchWorker.scheduleFetch(context, config)
        rescheduledCount++
        
        Log.i(TAG, "Rescheduled fetch: ${schedule.id}")
    } catch (e: Exception) {
        errors++
        Log.e(TAG, "Failed to reschedule fetch: ${schedule.id}", e)
    }
    
    return ScheduleRecoveryResult(rescheduledCount = rescheduledCount, errors = errors)
}

3.4 Helper Methods

/**
 * Mark a notification as missed
 * 
 * @param schedule Schedule that was missed
 * @param scheduledTime When the notification was scheduled
 * @param db Database instance
 */
private suspend fun markMissedNotification(
    schedule: Schedule,
    scheduledTime: Long,
    db: DailyNotificationDatabase
) {
    try {
        // Try to find existing NotificationContentEntity
        val notificationId = schedule.id
        val existingNotification = db.notificationContentDao().getNotificationById(notificationId)
        
        if (existingNotification != null) {
            // Update existing notification
            existingNotification.deliveryStatus = "missed"
            existingNotification.lastDeliveryAttempt = System.currentTimeMillis()
            existingNotification.deliveryAttempts = (existingNotification.deliveryAttempts ?: 0) + 1
            db.notificationContentDao().updateNotification(existingNotification)
            Log.d(TAG, "Updated existing notification as missed: $notificationId")
        } else {
            // Create missed notification entry
            // Note: This may not have full content, but marks the missed event
            Log.w(TAG, "No NotificationContentEntity found for schedule: $notificationId")
            // Could create a minimal entry here if needed
        }
    } catch (e: Exception) {
        Log.e(TAG, "Failed to mark missed notification: ${schedule.id}", e)
        throw e
    }
}

/**
 * Calculate next run time from schedule
 * Uses existing BootReceiver logic if available
 * 
 * @param schedule Schedule to calculate for
 * @param currentTime Current time in milliseconds
 * @return Next run time in milliseconds
 */
private fun calculateNextRunTime(schedule: Schedule, currentTime: Long): Long {
    // Prefer nextRunAt if set
    if (schedule.nextRunAt != null) {
        return schedule.nextRunAt!!
    }
    
    // Calculate from cron or clockTime
    // For now, simplified: use BootReceiver logic if available
    // Otherwise, default to next day at 9 AM
    return when {
        schedule.cron != null -> {
            // TODO: Parse cron and calculate next run
            // For now, return next day at 9 AM
            currentTime + (24 * 60 * 60 * 1000L)
        }
        schedule.clockTime != null -> {
            // TODO: Parse HH:mm and calculate next run
            // For now, return next day at specified time
            currentTime + (24 * 60 * 60 * 1000L)
        }
        else -> {
            // Default to next day at 9 AM
            currentTime + (24 * 60 * 60 * 1000L)
        }
    }
}

/**
 * Check if schedule is repeating
 * 
 * **Helper Consistency Note**: This helper must remain consistent with any
 * equivalent methods used in `BootReceiver` (Phase 3). If updated, update both places.
 * 
 * @param schedule Schedule to check
 * @return true if repeating, false if one-time
 */
private fun isRepeating(schedule: Schedule): Boolean {
    // Schedules with cron or clockTime are repeating
    return schedule.cron != null || schedule.clockTime != null
}

/**
 * Calculate next occurrence for repeating schedule
 * 
 * **Helper Consistency Note**: This helper must remain consistent with any
 * equivalent methods used in `BootReceiver` (Phase 3). If updated, update both places.
 * 
 * @param schedule Schedule to calculate for
 * @param fromTime Calculate next occurrence after this time
 * @return Next occurrence time in milliseconds
 */
private fun calculateNextOccurrence(schedule: Schedule, fromTime: Long): Long {
    // TODO: Implement proper calculation based on cron/clockTime
    // For now, simplified: daily schedules add 24 hours
    return fromTime + (24 * 60 * 60 * 1000L)
}

4. Data Integrity Checks

4.1 Force Stop Detection Validation

False Positive Prevention:

  • First launch: DB schedules count == 0 → Not force stop
  • Normal cold start: AlarmManager has alarms → Not force stop
  • Only detect force stop when: DB schedules > 0 && AlarmManager alarms == 0

Edge Cases:

  • All alarms already fired: Still detect as force stop if AlarmManager is empty
  • Partial alarm cancellation: Not detected as force stop (handled by cold start recovery)

4.2 Schedule Validation

Notify Schedule Validation:

  • id must not be blank
  • kind must be "notify"
  • nextRunAt or cron/clockTime must be set

Fetch Schedule Validation:

  • id must not be blank
  • kind must be "fetch"
  • cron or clockTime must be set

5. Rollback Safety

5.1 No-Crash Guarantee

All force stop recovery operations must:

  1. Catch all exceptions - Never propagate exceptions to app
  2. Continue processing - One schedule failure doesn't stop recovery
  3. Log errors - All failures logged with context
  4. Partial recovery - Some schedules can recover even if others fail

5.2 Error Handling Strategy

Error Type Handling Log Level
Schedule query failure Return empty result, log error ERROR
Invalid schedule data Skip schedule, continue WARN
Alarm reschedule failure Log error, continue to next ERROR
Fetch reschedule failure Log error, continue to next ERROR
Missed notification marking failure Log error, continue ERROR
History recording failure Log warning, don't fail WARN

6. Testing Requirements

6.1 Test 1: Force Stop Detection

Purpose: Verify force stop scenario is detected correctly.

Steps:

  1. Schedule 3 notifications (2 minutes, 5 minutes, 10 minutes in future)
  2. Verify alarms scheduled: adb shell dumpsys alarm | grep timesafari
  3. Force stop app: adb shell am force-stop com.timesafari.dailynotification
  4. Verify alarms cancelled: adb shell dumpsys alarm | grep timesafari (should be empty)
  5. Launch app: adb shell am start -n com.timesafari.dailynotification/.MainActivity
  6. Check logs: adb logcat -d | grep DNP-REACTIVATION

Expected:

  • Log shows "Force stop detected: DB has X schedules, AlarmManager has 0 alarms"
  • Log shows "Detected scenario: FORCE_STOP"
  • Log shows "Force stop recovery: recovering all schedules"

Pass Criteria: Force stop correctly detected.

6.2 Test 2: Past Alarm Recovery

Purpose: Verify past alarms are marked as missed.

Steps:

  1. Schedule notification for 2 minutes in future
  2. Force stop app
  3. Wait 5 minutes (past scheduled time)
  4. Launch app
  5. Check database: delivery_status = 'missed' for past alarm

Expected:

  • Past alarm marked as missed in database
  • History entry created
  • Log shows "Past alarm detected" and "Marked missed notification"

Pass Criteria: Past alarms correctly marked as missed.

6.3 Test 3: Future Alarm Recovery

Purpose: Verify future alarms are rescheduled.

Steps:

  1. Schedule 3 notifications (5, 10, 15 minutes in future)
  2. Force stop app
  3. Launch app immediately
  4. Verify alarms rescheduled: adb shell dumpsys alarm | grep timesafari

Expected:

  • All 3 alarms rescheduled in AlarmManager
  • Log shows "Future alarm detected" and "Rescheduled alarm"
  • No duplicate alarms created

Pass Criteria: Future alarms correctly rescheduled.

6.4 Test 4: Repeating Schedule Recovery

Purpose: Verify repeating schedules calculate next occurrence correctly.

Steps:

  1. Schedule daily notification (cron: "0 9 * * *")
  2. Force stop app
  3. Wait past scheduled time (e.g., wait until 10 AM)
  4. Launch app
  5. Verify next occurrence scheduled for tomorrow 9 AM

Expected:

  • Past occurrence marked as missed
  • Next occurrence scheduled for tomorrow
  • Log shows "Rescheduled next occurrence"

Pass Criteria: Repeating schedules correctly calculate next occurrence.

6.5 Test 5: Fetch Schedule Recovery

Purpose: Verify fetch schedules are recovered.

Steps:

  1. Schedule fetch work (cron: "0 9 * * *")
  2. Force stop app
  3. Launch app
  4. Check WorkManager: adb shell dumpsys jobscheduler | grep timesafari

Expected:

  • Fetch work rescheduled in WorkManager
  • Log shows "Rescheduled fetch"

Pass Criteria: Fetch schedules correctly recovered.


7. Implementation Checklist

  • Add detectScenario() method to ReactivationManager
  • Add alarmsExist() method (replaces getActiveAlarmCount)
  • Add isBootRecovery() method
  • Add RecoveryScenario enum
  • Update performRecovery() to handle force stop
  • Implement performForceStopRecovery()
  • Implement recoverNotifySchedule()
  • Implement recoverFetchSchedule()
  • Implement markMissedNotification()
  • Implement calculateNextRunTime() (or reuse BootReceiver logic)
  • Implement isRepeating()
  • Implement calculateNextOccurrence()
  • Add data integrity checks
  • Add error handling
  • Test force stop detection
  • Test past alarm recovery
  • Test future alarm recovery
  • Test repeating schedule recovery
  • Test fetch schedule recovery
  • Verify no duplicate alarms

8. Code References

Existing Code to Reuse:

  • NotifyReceiver.scheduleExactNotification() - Line 92
  • FetchWorker.scheduleFetch() - Line 31
  • BootReceiver.calculateNextRunTime() - Line 103 (for next run calculation)
  • ScheduleDao.getEnabled() - Line 298
  • NotificationContentDao.getNotificationById() - Line 69

New Code to Create:

  • detectScenario() - Add to ReactivationManager
  • alarmsExist() - Add to ReactivationManager (replaces getActiveAlarmCount)
  • isBootRecovery() - Add to ReactivationManager
  • performForceStopRecovery() - Add to ReactivationManager
  • recoverNotifySchedule() - Add to ReactivationManager
  • recoverFetchSchedule() - Add to ReactivationManager

9. Success Criteria Summary

Phase 2 is complete when:

  1. Force stop scenario detected correctly
  2. All past alarms marked as missed
  3. All future alarms rescheduled
  4. Both notify and fetch schedules recovered
  5. Repeating schedules calculate next occurrence correctly
  6. Recovery never crashes app
  7. All tests pass


Notes

  • Prerequisite: Phase 1 must be complete before starting Phase 2
  • Detection accuracy: Force stop detection uses best available method (nextAlarmClock)
  • Comprehensive recovery: Force stop recovery handles ALL schedules (past and future)
  • Safety first: All recovery operations are non-blocking and non-fatal