docs(test): add Phase 2 force stop recovery testing infrastructure
Adds documentation and test harness for Phase 2 (Force Stop Detection & Recovery). Changes: - Add PHASE2-EMULATOR-TESTING.md with detailed test procedures - Add PHASE2-VERIFICATION.md with test matrix and verification template - Add test-phase2.sh automated test harness Test harness features: - 3 test cases: force stop with cleared alarms, intact alarms, empty DB - Automatic force stop simulation via adb - Log parsing for scenario detection and recovery results - UI prompts for plugin configuration and scheduling Related: - Directive: android-implementation-directive-phase2.md - Requirements: docs/alarms/03-plugin-requirements.md §3.1.4 - Testing: docs/alarms/PHASE2-EMULATOR-TESTING.md - Verification: docs/alarms/PHASE2-VERIFICATION.md
This commit is contained in:
Matthew Raymer
358
docs/alarms/PHASE2-EMULATOR-TESTING.md
Normal file
358
docs/alarms/PHASE2-EMULATOR-TESTING.md
Normal file
@@ -0,0 +1,358 @@
|
||||
# PHASE 2 – EMULATOR TESTING
|
||||
|
||||
**Force Stop Detection & Recovery**
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Phase 2 verifies that the Daily Notification Plugin correctly:
|
||||
|
||||
1. Detects **force stop** scenarios (where alarms may be cleared by the OS).
|
||||
2. **Reschedules** future notifications when alarms are missing but schedules remain in the database.
|
||||
3. **Avoids heavy recovery** when alarms are still intact.
|
||||
4. **Does not misfire** force-stop recovery on first launch / empty database.
|
||||
|
||||
This document defines the emulator test procedure for Phase 2 using the script:
|
||||
|
||||
```bash
|
||||
test-apps/android-test-app/test-phase2.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Prerequisites
|
||||
|
||||
* Android emulator or device, e.g.:
|
||||
* Pixel 8 / API 34 (recommended baseline)
|
||||
* ADB available in `PATH` (or `ADB_BIN` exported)
|
||||
* Project built with:
|
||||
* Daily Notification Plugin integrated
|
||||
* Test app at: `test-apps/android-test-app`
|
||||
* Debug APK path:
|
||||
* `app/build/outputs/apk/debug/app-debug.apk`
|
||||
* Phase 1 behavior already implemented and verified:
|
||||
* Cold start detection
|
||||
* Missed notification marking
|
||||
|
||||
> **Note:** Some OS/device combinations do not clear alarms on `am force-stop`. In such cases, TEST 1 may partially skip or only verify scenario logging.
|
||||
|
||||
---
|
||||
|
||||
## 3. How to Run
|
||||
|
||||
From the `android-test-app` directory:
|
||||
|
||||
```bash
|
||||
cd test-apps/android-test-app
|
||||
chmod +x test-phase2.sh # first time only
|
||||
./test-phase2.sh
|
||||
```
|
||||
|
||||
The script will:
|
||||
|
||||
1. Perform pre-flight checks (ADB/emulator).
|
||||
2. Build and install the debug APK.
|
||||
3. Guide you through three tests:
|
||||
* TEST 1: Force stop – alarms cleared
|
||||
* TEST 2: Force stop / process stop – alarms intact
|
||||
* TEST 3: First launch / empty DB safeguard
|
||||
4. Print parsed recovery summaries from `DNP-REACTIVATION` logs.
|
||||
|
||||
You will be prompted for **UI actions** at each step (e.g., configuring the plugin, pressing "Test Notification").
|
||||
|
||||
---
|
||||
|
||||
## 4. Test Cases
|
||||
|
||||
### 4.1 TEST 1 – Force Stop with Cleared Alarms
|
||||
|
||||
**Goal:**
|
||||
|
||||
Verify that when a force stop clears alarms, the plugin:
|
||||
|
||||
* Detects the **FORCE_STOP** scenario.
|
||||
* Reschedules future notifications.
|
||||
* Completes recovery with `errors=0`.
|
||||
|
||||
**Steps (Script Flow):**
|
||||
|
||||
1. **Launch & configure plugin**
|
||||
* Script launches the app.
|
||||
* In the app UI, confirm:
|
||||
* `⚙️ Plugin Settings: ✅ Configured`
|
||||
* `🔌 Native Fetcher: ✅ Configured`
|
||||
* If not, press **Configure Plugin** and wait until both are ✅.
|
||||
|
||||
2. **Schedule a future notification**
|
||||
* Click **Test Notification** (or equivalent) to schedule a notification a few minutes in the future.
|
||||
|
||||
3. **Verify alarms are scheduled**
|
||||
* Script runs:
|
||||
```bash
|
||||
adb shell dumpsys alarm | grep com.timesafari.dailynotification
|
||||
```
|
||||
* Confirm at least one `RTC_WAKEUP` alarm for `com.timesafari.dailynotification`.
|
||||
|
||||
4. **Force stop the app**
|
||||
* Script executes:
|
||||
```bash
|
||||
adb shell am force-stop com.timesafari.dailynotification
|
||||
```
|
||||
|
||||
5. **Confirm alarms after force stop**
|
||||
* Script re-runs `dumpsys alarm`.
|
||||
* Ideal test case: **0** alarms for `com.timesafari.dailynotification` (alarms cleared).
|
||||
|
||||
6. **Trigger recovery**
|
||||
* Script clears logcat and launches the app.
|
||||
* Wait ~5 seconds for recovery.
|
||||
|
||||
7. **Collect and inspect logs**
|
||||
* Script collects `DNP-REACTIVATION` logs and parses:
|
||||
* `scenario=<value>`
|
||||
* `missed=<n>`
|
||||
* `rescheduled=<n>`
|
||||
* `verified=<n>`
|
||||
* `errors=<n>`
|
||||
|
||||
**Expected Logs (Ideal Case):**
|
||||
|
||||
* Scenario:
|
||||
```text
|
||||
DNP-REACTIVATION: Detected scenario: FORCE_STOP
|
||||
```
|
||||
|
||||
* Alarm handling:
|
||||
```text
|
||||
DNP-REACTIVATION: Rescheduled alarm: daily_<id> for <time>
|
||||
DNP-REACTIVATION: Rescheduled missing alarm: daily_<id> at <time>
|
||||
```
|
||||
|
||||
* Summary:
|
||||
```text
|
||||
DNP-REACTIVATION: Force stop recovery completed: missed=1, rescheduled=1, verified=0, errors=0
|
||||
```
|
||||
|
||||
**Pass Criteria:**
|
||||
|
||||
* `scenario=FORCE_STOP`
|
||||
* `rescheduled > 0`
|
||||
* `errors = 0`
|
||||
* Script prints:
|
||||
> `✅ TEST 1 PASSED: Force stop detected and alarms rescheduled (scenario=FORCE_STOP, rescheduled=1).`
|
||||
|
||||
**Partial / Edge Cases:**
|
||||
|
||||
* If alarms remain after `force-stop` on this device:
|
||||
* Script warns that FORCE_STOP may not fully trigger.
|
||||
* Mark this as **environment-limited** rather than a plugin failure.
|
||||
|
||||
---
|
||||
|
||||
### 4.2 TEST 2 – Force Stop / Process Stop with Intact Alarms
|
||||
|
||||
**Goal:**
|
||||
|
||||
Ensure we **do not run heavy force-stop recovery** when alarms are still intact.
|
||||
|
||||
**Steps (Script Flow):**
|
||||
|
||||
1. **Launch & configure plugin**
|
||||
* Same as TEST 1 (ensure both plugin statuses are ✅).
|
||||
|
||||
2. **Schedule another future notification**
|
||||
* Click **Test Notification** again to create a second schedule.
|
||||
|
||||
3. **Verify alarms are scheduled**
|
||||
* Confirm multiple alarms for `com.timesafari.dailynotification` via `dumpsys alarm`.
|
||||
|
||||
4. **Simulate a "soft stop"**
|
||||
* Script runs:
|
||||
```bash
|
||||
adb shell am kill com.timesafari.dailynotification
|
||||
```
|
||||
* Intent: stop the process but **not** clear alarms (actual behavior may vary by OS).
|
||||
|
||||
5. **Check alarms after soft stop**
|
||||
* Ensure alarms are still present in `dumpsys alarm`.
|
||||
|
||||
6. **Trigger recovery**
|
||||
* Script clears logcat and relaunches the app.
|
||||
* Wait ~5 seconds.
|
||||
|
||||
7. **Collect logs**
|
||||
* Script parses `DNP-REACTIVATION` logs for:
|
||||
* `scenario`
|
||||
* `rescheduled`
|
||||
* `errors`
|
||||
|
||||
**Expected Behavior:**
|
||||
|
||||
* `scenario` should **not** be `FORCE_STOP` (e.g., `COLD_START` or another non-force-stop scenario, depending on your implementation).
|
||||
* `rescheduled = 0`.
|
||||
* `errors = 0`.
|
||||
|
||||
**Pass Criteria:**
|
||||
|
||||
* Alarms still present after soft stop.
|
||||
* Recovery summary indicates **no force-stop-level rescheduling** when alarms are intact:
|
||||
* `scenario != FORCE_STOP`
|
||||
* `rescheduled = 0`
|
||||
* Script prints:
|
||||
> `✅ TEST 2 PASSED: No heavy force-stop recovery when alarms intact (scenario=<value>, rescheduled=0).`
|
||||
|
||||
If `scenario=FORCE_STOP` and `rescheduled>0` here, treat this as a **warning** and review scenario detection logic.
|
||||
|
||||
---
|
||||
|
||||
### 4.3 TEST 3 – First Launch / Empty DB Safeguard
|
||||
|
||||
**Goal:**
|
||||
|
||||
Ensure **force-stop recovery is not mis-triggered** when the app is freshly installed with **no schedules**.
|
||||
|
||||
**Steps (Script Flow):**
|
||||
|
||||
1. **Clear state**
|
||||
* Script uninstalls the app to clear DB/state:
|
||||
```bash
|
||||
adb uninstall com.timesafari.dailynotification
|
||||
```
|
||||
|
||||
2. **Reinstall APK**
|
||||
* Script reinstalls `app-debug.apk`.
|
||||
|
||||
3. **Launch app without scheduling anything**
|
||||
* Script launches the app.
|
||||
* Do **not** schedule notifications or configure plugin beyond initial display.
|
||||
|
||||
4. **Collect logs**
|
||||
* Script grabs `DNP-REACTIVATION` logs and parses:
|
||||
* `scenario`
|
||||
* `rescheduled`
|
||||
|
||||
**Expected Behavior:**
|
||||
|
||||
* Either:
|
||||
* No `DNP-REACTIVATION` logs at all (no recovery run), **or**
|
||||
* A specific "no schedules" scenario, e.g.:
|
||||
```text
|
||||
DNP-REACTIVATION: No schedules present — skipping recovery (first launch)
|
||||
```
|
||||
or
|
||||
```text
|
||||
DNP-REACTIVATION: Detected scenario: NONE
|
||||
```
|
||||
|
||||
* In both cases:
|
||||
* `rescheduled = 0`.
|
||||
|
||||
**Pass Criteria:**
|
||||
|
||||
* If **no logs**:
|
||||
* Pass: recovery correctly doesn't run at all on empty DB.
|
||||
* If logs present:
|
||||
* `scenario=NONE` (or equivalent) **and** `rescheduled=0`.
|
||||
|
||||
Script will report success when:
|
||||
|
||||
* `scenario == NONE_SCENARIO_VALUE` and `rescheduled=0`, or
|
||||
* No recovery logs are found.
|
||||
|
||||
---
|
||||
|
||||
## 5. Latest Known Good Run (Template)
|
||||
|
||||
Fill this in after your first successful emulator run.
|
||||
|
||||
```markdown
|
||||
---
|
||||
## Latest Known Good Run (Emulator)
|
||||
|
||||
**Environment**
|
||||
|
||||
- Device: Pixel 8 API 34 (Android 14)
|
||||
- App ID: `com.timesafari.dailynotification`
|
||||
- Build: Debug APK (`app-debug.apk`) from commit `<GIT_HASH>`
|
||||
- Script: `./test-phase2.sh`
|
||||
- Date: 2025-11-XX
|
||||
|
||||
**Results**
|
||||
|
||||
- ✅ TEST 1: Force Stop – Alarms Cleared
|
||||
- `scenario=FORCE_STOP`
|
||||
- `missed=1, rescheduled=1, verified=0, errors=0`
|
||||
|
||||
- ✅ TEST 2: Force Stop / Process Stop – Alarms Intact
|
||||
- `scenario=COLD_START` (or equivalent non-force-stop scenario)
|
||||
- `rescheduled=0, errors=0`
|
||||
|
||||
- ✅ TEST 3: First Launch / No Schedules
|
||||
- `scenario=NONE` (or no logs)
|
||||
- `rescheduled=0`
|
||||
|
||||
**Conclusion:**
|
||||
Phase 2 **Force Stop Detection & Recovery** is verified on the emulator using `test-phase2.sh`. This run is the canonical reference for future regression testing.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Troubleshooting
|
||||
|
||||
### Alarms Not Cleared on Force Stop
|
||||
|
||||
**Symptom**: `am force-stop` doesn't clear alarms in AlarmManager
|
||||
|
||||
**Cause**: Some Android versions/emulators don't clear alarms on force stop
|
||||
|
||||
**Solution**:
|
||||
- This is expected behavior on some systems
|
||||
- TEST 1 will run as Phase 1 (cold start) recovery
|
||||
- For full force stop validation, test on a device/OS that clears alarms
|
||||
- Script will report this as an environment limitation, not a failure
|
||||
|
||||
### Scenario Not Detected as FORCE_STOP
|
||||
|
||||
**Symptom**: Logs show `COLD_START` even when alarms were cleared
|
||||
|
||||
**Possible Causes**:
|
||||
1. Scenario detection logic not implemented (Phase 2 not complete)
|
||||
2. Alarm count check failing (`alarmsExist()` returning true when it shouldn't)
|
||||
3. Database query timing issue
|
||||
|
||||
**Solution**:
|
||||
- Verify Phase 2 implementation is complete
|
||||
- Check `ReactivationManager.detectScenario()` implementation
|
||||
- Review logs for alarm existence checks
|
||||
- Verify `alarmsExist()` uses PendingIntent check (not `nextAlarmClock`)
|
||||
|
||||
### Recovery Doesn't Reschedule Alarms
|
||||
|
||||
**Symptom**: `rescheduled=0` even when alarms were cleared
|
||||
|
||||
**Possible Causes**:
|
||||
1. Schedules not in database
|
||||
2. Reschedule logic failing
|
||||
3. Alarm scheduling permissions missing
|
||||
|
||||
**Solution**:
|
||||
- Verify schedules exist in database
|
||||
- Check logs for reschedule errors
|
||||
- Verify exact alarm permission is granted
|
||||
- Review `performForceStopRecovery()` implementation
|
||||
|
||||
---
|
||||
|
||||
## 7. Related Documentation
|
||||
|
||||
- [Phase 2 Directive](../android-implementation-directive-phase2.md) - Implementation details
|
||||
- [Phase 2 Verification](./PHASE2-VERIFICATION.md) - Verification report
|
||||
- [Phase 1 Testing Guide](./PHASE1-EMULATOR-TESTING.md) - Prerequisite testing
|
||||
- [Activation Guide](./ACTIVATION-GUIDE.md) - How to use directives
|
||||
- [Plugin Requirements](./03-plugin-requirements.md) - Requirements Phase 2 implements
|
||||
|
||||
---
|
||||
|
||||
**Status**: Ready for testing (Phase 2 implementation pending)
|
||||
**Last Updated**: November 2025
|
||||
Reference in New Issue
Block a user