# Daily Notification Plugin — Test Operator Runbook

**Last Updated:** 2025-01-XX  
**Purpose:** Complete guide for operators running Phase 1, 2, and 3 test suites  
**Audience:** Test operators, QA engineers, developers running manual tests

---

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Quick Start](#quick-start)
3. [Phase 1: Daily Rollover & Recovery](#phase-1-daily-rollover--recovery)
4. [Phase 2: Force Stop Recovery](#phase-2-force-stop-recovery)
5. [Phase 3: Boot Recovery](#phase-3-boot-recovery)
6. [Evidence & Artifacts](#evidence--artifacts)
7. [Interpreting Verdicts](#interpreting-verdicts)
8. [Common Failures & Fixes](#common-failures--fixes)
9. [Troubleshooting](#troubleshooting)

---

## Prerequisites

### Required Tools

- **ADB (Android Debug Bridge)** — Must be in PATH
  ```bash
  which adb
  adb version
  ```
- **Android Emulator or Physical Device** — Connected and accessible
  ```bash
  adb devices
  ```
- **Bash 4.0+** — For script execution
  ```bash
  bash --version
  ```
- **Gradle** — For building test app (included via `gradlew`)
- **Java JDK 11+** — For Android builds

### Environment Setup

1. **Navigate to test directory:**
   ```bash
   cd test-apps/android-test-app
   ```

2. **Verify scripts are executable:**
   ```bash
   ls -la test-phase*.sh
   chmod +x test-phase*.sh alarm-test-lib.sh
   ```

3. **Check ADB connection:**
   ```bash
   adb devices
   # Should show your device/emulator
   ```

### Pre-Flight Checks

Before running any test phase, verify:

- [ ] ADB device is connected (`adb devices` shows device)
- [ ] Emulator/device is unlocked and responsive
- [ ] Test app can be built (`./gradlew assembleDebug` succeeds)
- [ ] Previous test runs cleaned up (optional but recommended)

---

## Quick Start

### Run All Phases (Advisory Mode)

```bash
# Phase 1: Daily rollover and recovery
./test-phase1.sh --all

# Phase 2: Force stop recovery
./test-phase2.sh

# Phase 3: Boot recovery
./test-phase3.sh
```

### Run Specific Tests

```bash
# Phase 1: Run only TEST 0 (Daily Rollover)
./test-phase1.sh 0

# Phase 2: Run TEST 1 and TEST 3
./test-phase2.sh 1 3

# Phase 3: Run only TEST 4 (Silent Boot Recovery)
./test-phase3.sh 4
```

### Run in Release-Gating Mode

```bash
# Phase 3: Fail fast on any test failure
./test-phase3.sh --gate-phase3

# Or via environment variable
RELEASE_GATE_PHASE3=1 ./test-phase3.sh
```

---

## Phase 1: Daily Rollover & Recovery

**Purpose:** Verify daily rollover behavior, force-stop recovery, schedule updates, and error handling.

**Expected Duration:** 30-45 minutes (all tests)

### Test Modes

```bash
# Setup only (pre-flight checks, app install, permissions)
./test-phase1.sh --setup

# Run all tests
./test-phase1.sh --all

# Smoke test (minimal verification)
./test-phase1.sh --smoke

# CI mode (non-interactive, fail fast)
./test-phase1.sh --ci

# Run specific test
./test-phase1.sh 0  # TEST 0: Daily Rollover
./test-phase1.sh 1  # TEST 1: Force-Stop Recovery
./test-phase1.sh 2  # TEST 2: Schedule Update
./test-phase1.sh 3  # TEST 3: Recovery Timeout
./test-phase1.sh 4  # TEST 4: Invalid Data Handling
```

### Test Descriptions

**TEST 0: Daily Rollover Verification**
- **Time:** 5-8 minutes
- **Automatable:** Partial (requires manual verification)
- **What it tests:** Daily rollover at midnight, schedule advancement
- **Key steps:**
  1. Schedule notification for future
  2. Advance system time past midnight
  3. Verify rollover occurred and next day scheduled

**TEST 1: Force-Stop Recovery - Database Restoration**
- **Time:** 8-12 minutes
- **Automatable:** Partial (requires manual force-stop)
- **What it tests:** Recovery after force-stop, database restoration
- **Key steps:**
  1. Schedule notification
  2. Force-stop app
  3. Relaunch and verify recovery

**TEST 2: Schedule Update Verification**
- **Time:** 5-8 minutes
- **Automatable:** Partial
- **What it tests:** Schedule updates, one-per-day semantics
- **Key steps:**
  1. Schedule notification
  2. Update schedule
  3. Verify only one alarm exists

**TEST 3: Recovery Timeout**
- **Time:** 3-5 minutes
- **Automatable:** Yes (code verification)
- **What it tests:** Recovery timeout handling
- **Key steps:**
  1. Verify timeout logic in code
  2. Check timeout behavior

**TEST 4: Invalid Data Handling**
- **Time:** 5-8 minutes
- **Automatable:** Partial
- **What it tests:** Graceful handling of invalid data
- **Key steps:**
  1. Inject invalid data
  2. Verify graceful recovery

### Phase 1 Evidence Location

All evidence is saved to: `runs/<RUN_ID>/`

- **Alarms:** `runs/<RUN_ID>/alarms/phase1_*.txt`
- **Logs:** `runs/<RUN_ID>/logs/phase1_*.txt`
- **Screenshots:** `runs/<RUN_ID>/screens/phase1_*.png`

---

## Phase 2: Force Stop Recovery

**Purpose:** Verify force-stop detection, alarm rescheduling, and recovery scenarios.

**Expected Duration:** 15-20 minutes (all tests)

### Test Modes

```bash
# Run all tests (default)
./test-phase2.sh

# Run specific tests
./test-phase2.sh 1      # TEST 1: Force Stop with Cleared Alarms
./test-phase2.sh 2      # TEST 2: Force Stop with Intact Alarms
./test-phase2.sh 3      # TEST 3: First Launch / No Schedules

# With strictness policy
STRICTNESS=soft ./test-phase2.sh   # Default: minor quirks = warn
STRICTNESS=hard ./test-phase2.sh    # Any issue = fail
```

### Test Descriptions

**TEST 1: Force Stop – Alarms Cleared**
- **Time:** 5-8 minutes
- **Automatable:** Partial (requires manual force-stop)
- **What it tests:** Force-stop detection when alarms are cleared
- **Key steps:**
  1. Schedule notification
  2. Force-stop app (clears alarms on many devices)
  3. Relaunch and verify FORCE_STOP scenario detected
  4. Verify alarms rescheduled

**TEST 2: Force Stop / Process Stop – Alarms Intact**
- **Time:** 4-6 minutes
- **Automatable:** Partial
- **What it tests:** Recovery when alarms remain intact
- **Key steps:**
  1. Schedule notification
  2. Soft-kill app (alarms remain)
  3. Relaunch and verify FORCE_STOP scenario did NOT run

**TEST 3: First Launch / No Schedules Safeguard**
- **Time:** 3-5 minutes
- **Automatable:** Yes
- **What it tests:** No recovery on empty database
- **Key steps:**
  1. Uninstall app
  2. Reinstall app
  3. Reboot (without scheduling)
  4. Verify no recovery logs or NONE scenario

### Strictness Policy

**`STRICTNESS=soft` (default):**
- Minor device quirks = `verdict_warn`
- Alarms still present after force-stop = warn
- FORCE_STOP scenario not detected = warn

**`STRICTNESS=hard`:**
- Any unexpected behavior = `verdict_fail`
- Alarms still present after force-stop = fail
- FORCE_STOP scenario not detected = fail
- Recovery errors = fail

### Phase 2 Evidence Location

- **Alarms:** `runs/<RUN_ID>/alarms/phase2_*.txt`
- **Logs:** `runs/<RUN_ID>/logs/phase2_*.txt`
- **Screenshots:** `runs/<RUN_ID>/screens/phase2_*.png`

---

## Phase 3: Boot Recovery

**Purpose:** Verify boot recovery, missed alarm detection, and silent recovery.

**Expected Duration:** 12-18 minutes (all tests)

### Test Modes

```bash
# Run all tests (advisory mode, default)
./test-phase3.sh

# Run specific tests
./test-phase3.sh 1      # TEST 1: Boot with Future Alarms
./test-phase3.sh 2      # TEST 2: Boot with Past Alarms
./test-phase3.sh 3      # TEST 3: Boot with No Schedules
./test-phase3.sh 4      # TEST 4: Silent Boot Recovery

# Release-gating mode (failures exit with non-zero)
./test-phase3.sh --gate-phase3
RELEASE_GATE_PHASE3=1 ./test-phase3.sh
```

### Test Descriptions

**TEST 1: Boot with Future Alarms**
- **Time:** 2-3 minutes (includes 30-60s reboot)
- **Automatable:** Partial (requires manual reboot confirmation)
- **What it tests:** Boot recovery with future alarms
- **Key steps:**
  1. Schedule notification for future
  2. Reboot emulator
  3. Verify BOOT scenario detected
  4. Verify alarms rescheduled

**TEST 2: Boot with Past Alarms**
- **Time:** 5-6 minutes (includes 3min wait + 30-60s reboot)
- **Automatable:** Partial (requires manual time advancement or wait)
- **What it tests:** Missed alarm detection and rescheduling
- **Key steps:**
  1. Schedule notification for 2 minutes future
  2. Wait 3 minutes (alarm time passes)
  3. Reboot emulator
  4. Verify missed alarms detected
  5. Verify next occurrence scheduled

**TEST 3: Boot with No Schedules**
- **Time:** 2-3 minutes (includes 30-60s reboot)
- **Automatable:** Yes
- **What it tests:** Graceful handling of empty database
- **Key steps:**
  1. Uninstall app
  2. Reinstall app
  3. Reboot (without scheduling)
  4. Verify no recovery logs or NONE scenario

**TEST 4: Silent Boot Recovery (App Never Opened)**
- **Time:** 2-3 minutes (includes 30-60s reboot)
- **Automatable:** Partial (requires manual verification app not opened)
- **What it tests:** Boot recovery without app launch
- **Key steps:**
  1. Schedule notification
  2. Reboot emulator
  3. **DO NOT open app** after reboot
  4. Verify boot recovery occurred silently
  5. Verify alarms recreated

### Release Gating

**Advisory Mode (default):**
- Failures become warnings
- Script continues to next test
- Use for development/testing

**Release-Blocking Mode (`--gate-phase3` or `RELEASE_GATE_PHASE3=1`):**
- Failures cause script to exit with non-zero
- Use for CI/CD or release validation
- First failure stops execution

### Phase 3 Evidence Location

- **Alarms:** `runs/<RUN_ID>/alarms/phase3_*.txt`
- **Logs:** `runs/<RUN_ID>/logs/phase3_*.txt`
- **Screenshots:** `runs/<RUN_ID>/screens/phase3_*.png`

---

## Evidence & Artifacts

### Run Directory Structure

Each test run creates a timestamped directory:

```
runs/
  └── 20250124_143022_abc123/
      ├── alarms/
      │   ├── phase1_test0_initial.txt
      │   ├── phase1_test0_before_schedule.txt
      │   └── ...
      ├── logs/
      │   ├── phase1_test0_initial.txt
      │   ├── phase1_test0_after_schedule.txt
      │   └── ...
      ├── screens/
      │   ├── phase1_test0_after_schedule_screenshot.png
      │   └── ...
      └── notes/
          └── (manual notes can be added here)
```

### Evidence Types

**Alarm Dumps:**
- Captured via `adb shell dumpsys alarm`
- Shows all scheduled alarms
- Used to verify plugin alarms exist

**Logcat Logs:**
- Captured via `adb logcat`
- Filtered by tag (e.g., `DNP`, `DNP-REACTIVATION`)
- Used to verify recovery scenarios and errors

**Screenshots:**
- Captured via `adb exec-out screencap`
- Shows app state at key points
- Used for visual verification

### Accessing Evidence

```bash
# Get current run directory
cd test-apps/android-test-app
ls -la runs/

# View latest run
ls -la runs/$(ls -t runs/ | head -1)

# View specific evidence
cat runs/<RUN_ID>/alarms/phase1_test0_initial.txt
cat runs/<RUN_ID>/logs/phase1_test0_after_schedule.txt
```

---

## Interpreting Verdicts

### Verdict Types

**✅ PASS (`verdict_pass`):**
- Test passed all checks
- Continue to next test
- Evidence saved for reference

**⚠️ WARN (`verdict_warn`):**
- Test had minor issues or device quirks
- May be acceptable depending on context
- Review evidence and decide if action needed
- Script continues

**❌ FAIL (`verdict_fail`):**
- Test failed critical checks
- Review evidence immediately
- In advisory mode: script continues
- In release-gating mode: script exits with non-zero

### Verdict Format

```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ VERDICT: PASS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Test ID: phase1_test0_daily_rollover
Status:  PASS
Message: Daily rollover verified successfully
Evidence: runs/20250124_143022_abc123
Next:    Continue to next test
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```

### Evidence Block Format

```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 EVIDENCE: phase1_test0_daily_rollover
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Run ID: 20250124_143022_abc123
Evidence directory: runs/20250124_143022_abc123

Artifacts:
  • Alarms:    runs/20250124_143022_abc123/alarms/
  • Logs:      runs/20250124_143022_abc123/logs/
  • Screens:   runs/20250124_143022_abc123/screens/
  • Notes:     runs/20250124_143022_abc123/notes/
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```

---

## Common Failures & Fixes

### Phase 1 Failures

**"No plugin alarms found before rollover"**
- **Cause:** Notification not scheduled correctly
- **Fix:** Verify plugin is configured, check app UI shows scheduled notification
- **Evidence:** Check `alarms/phase1_test0_before_schedule.txt`

**"Rollover did not occur"**
- **Cause:** System time not advanced correctly, or rollover logic issue
- **Fix:** Verify system time was advanced past midnight, check logs for rollover activity
- **Evidence:** Check `logs/phase1_test0_after_rollover.txt`

**"Recovery did not restore database"**
- **Cause:** Force-stop recovery not triggered, or database restoration failed
- **Fix:** Verify force-stop actually cleared alarms, check recovery logs
- **Evidence:** Check `logs/phase1_test1_after_recovery.txt`

### Phase 2 Failures

**"FORCE_STOP scenario not detected"**
- **Cause:** Device/emulator didn't clear alarms on force-stop, or scenario detection logic issue
- **Fix:** Check if device clears alarms on force-stop (device-specific), verify boot flag cleared
- **Evidence:** Check `logs/phase2_test1_after_recovery.txt`, verify `scenario=FORCE_STOP`

**"Alarms still present after force-stop"**
- **Cause:** Device/emulator doesn't clear alarms on force-stop (common on some devices)
- **Fix:** This is device-specific behavior. In `STRICTNESS=soft` mode, this is a warning. In `STRICTNESS=hard` mode, this is a failure.
- **Evidence:** Check `alarms/phase2_test1_after_force_stop.txt`

**"rescheduled>0 on first launch"**
- **Cause:** Boot recovery misfiring on empty database
- **Fix:** Check recovery logic, verify database is actually empty
- **Evidence:** Check `logs/phase2_test3_after_reboot.txt`

### Phase 3 Failures

**"Boot recovery not detected"**
- **Cause:** Boot receiver not registered, or BOOT_COMPLETED permission missing
- **Fix:** Verify `AndroidManifest.xml` has boot receiver and permission
- **Evidence:** Check `logs/phase3_test1_after_reboot.txt`, verify boot receiver logs

**"No missed alarms detected"**
- **Cause:** Alarm time didn't actually pass before reboot
- **Fix:** Verify system time was advanced, or wait longer before reboot
- **Evidence:** Check `alarms/phase3_test2_after_wait.txt`, verify alarm time is in past

**"Boot recovery ran but alarms not recreated"**
- **Cause:** Recovery succeeded but alarm scheduling failed
- **Fix:** Check alarm scheduling logic, verify permissions
- **Evidence:** Check `logs/phase3_test4_after_reboot.txt`, verify `rescheduled>0` but `after_count=0`

### General Failures

**"ADB device not found"**
- **Cause:** Device disconnected, ADB not in PATH, or device not authorized
- **Fix:** Run `adb devices`, verify device shows as "device" (not "unauthorized"), reconnect if needed

**"App build failed"**
- **Cause:** Gradle issues, missing dependencies, or Java version mismatch
- **Fix:** Run `./gradlew clean`, verify Java version, check `build.gradle` dependencies

**"Permission denied"**
- **Cause:** App doesn't have required permissions
- **Fix:** Grant permissions via app UI or `adb shell pm grant`

---

## Troubleshooting

### Script Won't Run

**"Permission denied: ./test-phase1.sh"**
```bash
chmod +x test-phase1.sh test-phase2.sh test-phase3.sh alarm-test-lib.sh
```

**"Command not found: adb"**
```bash
# Add Android SDK platform-tools to PATH
export PATH="$PATH:$ANDROID_HOME/platform-tools"
```

**"Syntax error near unexpected token"**
```bash
# Verify bash version
bash --version  # Should be 4.0+

# Check script syntax
bash -n test-phase1.sh
```

### Evidence Not Captured

**"Run directory not initialized"**
- **Cause:** `ensure_run_dir()` failed
- **Fix:** Check write permissions in test directory, verify `runs/` directory can be created

**"Screenshot capture failed"**
- **Cause:** `ENABLE_SCREENSHOTS=0` or device doesn't support screencap
- **Fix:** Set `ENABLE_SCREENSHOTS=1` (default), verify device supports `adb exec-out screencap`

**"Logcat capture empty"**
- **Cause:** Logs cleared before capture, or filter pattern doesn't match
- **Fix:** Verify log tag matches filter pattern, check logs weren't cleared

### Device/Emulator Issues

**"Emulator not responding"**
```bash
# Restart emulator
adb kill-server
adb start-server
adb devices
```

**"Device unauthorized"**
- **Cause:** USB debugging authorization not granted
- **Fix:** Check device screen for authorization prompt, click "Allow"

**"Alarms not clearing on force-stop"**
- **Cause:** Device-specific behavior (some devices don't clear alarms)
- **Fix:** This is expected on some devices. Use `STRICTNESS=soft` mode to treat as warning.

### Test-Specific Issues

**"Phase 3 reboot takes too long"**
- **Cause:** Emulator is slow or hung
- **Fix:** Wait longer (60-90 seconds), or restart emulator if completely hung

**"Time advancement not working"**
- **Cause:** System time can't be advanced (requires root or emulator)
- **Fix:** Use emulator with root, or manually advance time via emulator settings

**"Plugin not configured"**
- **Cause:** Plugin setup not completed
- **Fix:** Run `./test-phase1.sh --setup` to configure plugin, or manually configure in app UI

---

## Quick Reference

### Command Cheat Sheet

```bash
# Phase 1
./test-phase1.sh --setup          # Setup only
./test-phase1.sh --all            # All tests
./test-phase1.sh --smoke          # Smoke test
./test-phase1.sh 0                # TEST 0 only

# Phase 2
./test-phase2.sh                  # All tests (soft mode)
STRICTNESS=hard ./test-phase2.sh  # All tests (hard mode)
./test-phase2.sh 1 3              # TEST 1 and 3

# Phase 3
./test-phase3.sh                  # All tests (advisory)
./test-phase3.sh --gate-phase3    # All tests (release-blocking)
./test-phase3.sh 2                # TEST 2 only
```

### Evidence Locations

```bash
# Latest run
ls -la runs/$(ls -t runs/ | head -1)

# Specific evidence
cat runs/<RUN_ID>/alarms/phase1_test0_initial.txt
cat runs/<RUN_ID>/logs/phase1_test0_after_schedule.txt
```

### Verdict Meanings

- **✅ PASS:** Test passed, continue
- **⚠️ WARN:** Minor issue, review evidence, continue
- **❌ FAIL:** Critical failure, review evidence, may exit (if gating enabled)

---

## Support & Feedback

**Issues or Questions?**
- Check evidence files in `runs/<RUN_ID>/`
- Review test logs for detailed error messages
- Consult platform-specific documentation in `doc/platform/android/`

**Improving This Runbook:**
- Document new failure patterns as they're discovered
- Update time estimates based on actual test runs
- Add automation hints for new test scenarios

---

**Last Updated:** 2025-01-XX  
**Version:** 1.0  
**Maintainer:** Test Engineering Team

