trent_larson/daily-notification-plugin
4
0
Fork 0
Code Issues Pull Requests 2 Projects Releases Wiki Activity

test(android-test-app): unify presentation framework with evidence collection

Browse Source 
Implement P0-P5 directives for operator clarity, consistent outcomes, and
easy evidence capture across all test phases.

Changes:
- alarm-test-lib.sh: Add evidence collection (capture_alarms, capture_logcat,
  capture_screenshot), verdict functions (verdict_pass/warn/fail), run directory
  management, and release gating support (RELEASE_GATE_PHASE3)

- test-phase1.sh: Refactor to unified framework with CLI modes (--setup,
  --run, --smoke, --all, --ci), micro-prompts, evidence capture, and verdict
  blocks for all 5 tests

- test-phase2.sh: Add evidence capture, verdict blocks, and STRICTNESS policy
  (soft/hard) for warn vs fail behavior

- test-phase3.sh: Add evidence capture, verdict blocks, release gating
  (--gate-phase3), and fatigue reduction (time estimates, automation hints)

- RUNBOOK-TESTING.md: New comprehensive operator guide (669 lines) covering
  prerequisites, all phases, evidence locations, verdict interpretation,
  common failures, and troubleshooting

All test scripts now use consistent UI helpers (section, substep, info, ok,
warn, error), standardized evidence collection, and clear verdict reporting.
Evidence is saved to timestamped run directories (runs/<RUN_ID>/) with alarms,
logs, and screenshots organized by test phase and scenario.

Tests pass with consistent presentation and reproducible evidence collection.
This commit is contained in:
Matthew Raymer
2025-12-24 12:01:16 +00:00
parent 973af9b688
commit ac39255672
5 changed files with 2373 additions and 870 deletions
Download Patch File Download Diff File Expand all files Collapse all files

669
test-apps/android-test-app/docs/RUNBOOK-TESTING.md Normal file
View File

@@ -0,0 +1,669 @@
# Daily Notification Plugin — Test Operator Runbook
**Last Updated:** 2025-01-XX
**Purpose:** Complete guide for operators running Phase 1, 2, and 3 test suites
**Audience:** Test operators, QA engineers, developers running manual tests
---
## Table of Contents
1. [Prerequisites](#prerequisites)
2. [Quick Start](#quick-start)
3. [Phase 1: Daily Rollover & Recovery](#phase-1-daily-rollover--recovery)
4. [Phase 2: Force Stop Recovery](#phase-2-force-stop-recovery)
5. [Phase 3: Boot Recovery](#phase-3-boot-recovery)
6. [Evidence & Artifacts](#evidence--artifacts)
7. [Interpreting Verdicts](#interpreting-verdicts)
8. [Common Failures & Fixes](#common-failures--fixes)
9. [Troubleshooting](#troubleshooting)
---
## Prerequisites
### Required Tools
- **ADB (Android Debug Bridge)** — Must be in PATH
```bash
which adb
adb version
```
- **Android Emulator or Physical Device** — Connected and accessible
```bash
adb devices
```
- **Bash 4.0+** — For script execution
```bash
bash --version
```
- **Gradle** — For building test app (included via `gradlew`)
- **Java JDK 11+** — For Android builds
### Environment Setup
1. **Navigate to test directory:**
```bash
cd test-apps/android-test-app
```
2. **Verify scripts are executable:**
```bash
ls -la test-phase*.sh
chmod +x test-phase*.sh alarm-test-lib.sh
```
3. **Check ADB connection:**
```bash
adb devices
# Should show your device/emulator
```
### Pre-Flight Checks
Before running any test phase, verify:
- [ ] ADB device is connected (`adb devices` shows device)
- [ ] Emulator/device is unlocked and responsive
- [ ] Test app can be built (`./gradlew assembleDebug` succeeds)
- [ ] Previous test runs cleaned up (optional but recommended)
---
## Quick Start
### Run All Phases (Advisory Mode)
```bash
# Phase 1: Daily rollover and recovery
./test-phase1.sh --all
# Phase 2: Force stop recovery
./test-phase2.sh
# Phase 3: Boot recovery
./test-phase3.sh
```
### Run Specific Tests
```bash
# Phase 1: Run only TEST 0 (Daily Rollover)
./test-phase1.sh 0
# Phase 2: Run TEST 1 and TEST 3
./test-phase2.sh 1 3
# Phase 3: Run only TEST 4 (Silent Boot Recovery)
./test-phase3.sh 4
```
### Run in Release-Gating Mode
```bash
# Phase 3: Fail fast on any test failure
./test-phase3.sh --gate-phase3
# Or via environment variable
RELEASE_GATE_PHASE3=1 ./test-phase3.sh
```
---
## Phase 1: Daily Rollover & Recovery
**Purpose:** Verify daily rollover behavior, force-stop recovery, schedule updates, and error handling.
**Expected Duration:** 30-45 minutes (all tests)
### Test Modes
```bash
# Setup only (pre-flight checks, app install, permissions)
./test-phase1.sh --setup
# Run all tests
./test-phase1.sh --all
# Smoke test (minimal verification)
./test-phase1.sh --smoke
# CI mode (non-interactive, fail fast)
./test-phase1.sh --ci
# Run specific test
./test-phase1.sh 0 # TEST 0: Daily Rollover
./test-phase1.sh 1 # TEST 1: Force-Stop Recovery
./test-phase1.sh 2 # TEST 2: Schedule Update
./test-phase1.sh 3 # TEST 3: Recovery Timeout
./test-phase1.sh 4 # TEST 4: Invalid Data Handling
```
### Test Descriptions
**TEST 0: Daily Rollover Verification**
- **Time:** 5-8 minutes
- **Automatable:** Partial (requires manual verification)
- **What it tests:** Daily rollover at midnight, schedule advancement
- **Key steps:**
1. Schedule notification for future
2. Advance system time past midnight
3. Verify rollover occurred and next day scheduled
**TEST 1: Force-Stop Recovery - Database Restoration**
- **Time:** 8-12 minutes
- **Automatable:** Partial (requires manual force-stop)
- **What it tests:** Recovery after force-stop, database restoration
- **Key steps:**
1. Schedule notification
2. Force-stop app
3. Relaunch and verify recovery
**TEST 2: Schedule Update Verification**
- **Time:** 5-8 minutes
- **Automatable:** Partial
- **What it tests:** Schedule updates, one-per-day semantics
- **Key steps:**
1. Schedule notification
2. Update schedule
3. Verify only one alarm exists
**TEST 3: Recovery Timeout**
- **Time:** 3-5 minutes
- **Automatable:** Yes (code verification)
- **What it tests:** Recovery timeout handling
- **Key steps:**
1. Verify timeout logic in code
2. Check timeout behavior
**TEST 4: Invalid Data Handling**
- **Time:** 5-8 minutes
- **Automatable:** Partial
- **What it tests:** Graceful handling of invalid data
- **Key steps:**
1. Inject invalid data
2. Verify graceful recovery
### Phase 1 Evidence Location
All evidence is saved to: `runs/<RUN_ID>/`
- **Alarms:** `runs/<RUN_ID>/alarms/phase1_*.txt`
- **Logs:** `runs/<RUN_ID>/logs/phase1_*.txt`
- **Screenshots:** `runs/<RUN_ID>/screens/phase1_*.png`
---
## Phase 2: Force Stop Recovery
**Purpose:** Verify force-stop detection, alarm rescheduling, and recovery scenarios.
**Expected Duration:** 15-20 minutes (all tests)
### Test Modes
```bash
# Run all tests (default)
./test-phase2.sh
# Run specific tests
./test-phase2.sh 1 # TEST 1: Force Stop with Cleared Alarms
./test-phase2.sh 2 # TEST 2: Force Stop with Intact Alarms
./test-phase2.sh 3 # TEST 3: First Launch / No Schedules
# With strictness policy
STRICTNESS=soft ./test-phase2.sh # Default: minor quirks = warn
STRICTNESS=hard ./test-phase2.sh # Any issue = fail
```
### Test Descriptions
**TEST 1: Force Stop – Alarms Cleared**
- **Time:** 5-8 minutes
- **Automatable:** Partial (requires manual force-stop)
- **What it tests:** Force-stop detection when alarms are cleared
- **Key steps:**
1. Schedule notification
2. Force-stop app (clears alarms on many devices)
3. Relaunch and verify FORCE_STOP scenario detected
4. Verify alarms rescheduled
**TEST 2: Force Stop / Process Stop – Alarms Intact**
- **Time:** 4-6 minutes
- **Automatable:** Partial
- **What it tests:** Recovery when alarms remain intact
- **Key steps:**
1. Schedule notification
2. Soft-kill app (alarms remain)
3. Relaunch and verify FORCE_STOP scenario did NOT run
**TEST 3: First Launch / No Schedules Safeguard**
- **Time:** 3-5 minutes
- **Automatable:** Yes
- **What it tests:** No recovery on empty database
- **Key steps:**
1. Uninstall app
2. Reinstall app
3. Reboot (without scheduling)
4. Verify no recovery logs or NONE scenario
### Strictness Policy
**`STRICTNESS=soft` (default):**
- Minor device quirks = `verdict_warn`
- Alarms still present after force-stop = warn
- FORCE_STOP scenario not detected = warn
**`STRICTNESS=hard`:**
- Any unexpected behavior = `verdict_fail`
- Alarms still present after force-stop = fail
- FORCE_STOP scenario not detected = fail
- Recovery errors = fail
### Phase 2 Evidence Location
- **Alarms:** `runs/<RUN_ID>/alarms/phase2_*.txt`
- **Logs:** `runs/<RUN_ID>/logs/phase2_*.txt`
- **Screenshots:** `runs/<RUN_ID>/screens/phase2_*.png`
---
## Phase 3: Boot Recovery
**Purpose:** Verify boot recovery, missed alarm detection, and silent recovery.
**Expected Duration:** 12-18 minutes (all tests)
### Test Modes
```bash
# Run all tests (advisory mode, default)
./test-phase3.sh
# Run specific tests
./test-phase3.sh 1 # TEST 1: Boot with Future Alarms
./test-phase3.sh 2 # TEST 2: Boot with Past Alarms
./test-phase3.sh 3 # TEST 3: Boot with No Schedules
./test-phase3.sh 4 # TEST 4: Silent Boot Recovery
# Release-gating mode (failures exit with non-zero)
./test-phase3.sh --gate-phase3
RELEASE_GATE_PHASE3=1 ./test-phase3.sh
```
### Test Descriptions
**TEST 1: Boot with Future Alarms**
- **Time:** 2-3 minutes (includes 30-60s reboot)
- **Automatable:** Partial (requires manual reboot confirmation)
- **What it tests:** Boot recovery with future alarms
- **Key steps:**
1. Schedule notification for future
2. Reboot emulator
3. Verify BOOT scenario detected
4. Verify alarms rescheduled
**TEST 2: Boot with Past Alarms**
- **Time:** 5-6 minutes (includes 3min wait + 30-60s reboot)
- **Automatable:** Partial (requires manual time advancement or wait)
- **What it tests:** Missed alarm detection and rescheduling
- **Key steps:**
1. Schedule notification for 2 minutes future
2. Wait 3 minutes (alarm time passes)
3. Reboot emulator
4. Verify missed alarms detected
5. Verify next occurrence scheduled
**TEST 3: Boot with No Schedules**
- **Time:** 2-3 minutes (includes 30-60s reboot)
- **Automatable:** Yes
- **What it tests:** Graceful handling of empty database
- **Key steps:**
1. Uninstall app
2. Reinstall app
3. Reboot (without scheduling)
4. Verify no recovery logs or NONE scenario
**TEST 4: Silent Boot Recovery (App Never Opened)**
- **Time:** 2-3 minutes (includes 30-60s reboot)
- **Automatable:** Partial (requires manual verification app not opened)
- **What it tests:** Boot recovery without app launch
- **Key steps:**
1. Schedule notification
2. Reboot emulator
3. **DO NOT open app** after reboot
4. Verify boot recovery occurred silently
5. Verify alarms recreated
### Release Gating
**Advisory Mode (default):**
- Failures become warnings
- Script continues to next test
- Use for development/testing
**Release-Blocking Mode (`--gate-phase3` or `RELEASE_GATE_PHASE3=1`):**
- Failures cause script to exit with non-zero
- Use for CI/CD or release validation
- First failure stops execution
### Phase 3 Evidence Location
- **Alarms:** `runs/<RUN_ID>/alarms/phase3_*.txt`
- **Logs:** `runs/<RUN_ID>/logs/phase3_*.txt`
- **Screenshots:** `runs/<RUN_ID>/screens/phase3_*.png`
---
## Evidence & Artifacts
### Run Directory Structure
Each test run creates a timestamped directory:
```
runs/
└── 20250124_143022_abc123/
├── alarms/
│ ├── phase1_test0_initial.txt
│ ├── phase1_test0_before_schedule.txt
│ └── ...
├── logs/
│ ├── phase1_test0_initial.txt
│ ├── phase1_test0_after_schedule.txt
│ └── ...
├── screens/
│ ├── phase1_test0_after_schedule_screenshot.png
│ └── ...
└── notes/
└── (manual notes can be added here)
```
### Evidence Types
**Alarm Dumps:**
- Captured via `adb shell dumpsys alarm`
- Shows all scheduled alarms
- Used to verify plugin alarms exist
**Logcat Logs:**
- Captured via `adb logcat`
- Filtered by tag (e.g., `DNP`, `DNP-REACTIVATION`)
- Used to verify recovery scenarios and errors
**Screenshots:**
- Captured via `adb exec-out screencap`
- Shows app state at key points
- Used for visual verification
### Accessing Evidence
```bash
# Get current run directory
cd test-apps/android-test-app
ls -la runs/
# View latest run
ls -la runs/$(ls -t runs/ | head -1)
# View specific evidence
cat runs/<RUN_ID>/alarms/phase1_test0_initial.txt
cat runs/<RUN_ID>/logs/phase1_test0_after_schedule.txt
```
---
## Interpreting Verdicts
### Verdict Types
**✅ PASS (`verdict_pass`):**
- Test passed all checks
- Continue to next test
- Evidence saved for reference
**⚠️ WARN (`verdict_warn`):**
- Test had minor issues or device quirks
- May be acceptable depending on context
- Review evidence and decide if action needed
- Script continues
**❌ FAIL (`verdict_fail`):**
- Test failed critical checks
- Review evidence immediately
- In advisory mode: script continues
- In release-gating mode: script exits with non-zero
### Verdict Format
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ VERDICT: PASS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Test ID: phase1_test0_daily_rollover
Status: PASS
Message: Daily rollover verified successfully
Evidence: runs/20250124_143022_abc123
Next: Continue to next test
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```
### Evidence Block Format
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 EVIDENCE: phase1_test0_daily_rollover
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Run ID: 20250124_143022_abc123
Evidence directory: runs/20250124_143022_abc123
Artifacts:
• Alarms: runs/20250124_143022_abc123/alarms/
• Logs: runs/20250124_143022_abc123/logs/
• Screens: runs/20250124_143022_abc123/screens/
• Notes: runs/20250124_143022_abc123/notes/
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```
---
## Common Failures & Fixes
### Phase 1 Failures
**"No plugin alarms found before rollover"**
- **Cause:** Notification not scheduled correctly
- **Fix:** Verify plugin is configured, check app UI shows scheduled notification
- **Evidence:** Check `alarms/phase1_test0_before_schedule.txt`
**"Rollover did not occur"**
- **Cause:** System time not advanced correctly, or rollover logic issue
- **Fix:** Verify system time was advanced past midnight, check logs for rollover activity
- **Evidence:** Check `logs/phase1_test0_after_rollover.txt`
**"Recovery did not restore database"**
- **Cause:** Force-stop recovery not triggered, or database restoration failed
- **Fix:** Verify force-stop actually cleared alarms, check recovery logs
- **Evidence:** Check `logs/phase1_test1_after_recovery.txt`
### Phase 2 Failures
**"FORCE_STOP scenario not detected"**
- **Cause:** Device/emulator didn't clear alarms on force-stop, or scenario detection logic issue
- **Fix:** Check if device clears alarms on force-stop (device-specific), verify boot flag cleared
- **Evidence:** Check `logs/phase2_test1_after_recovery.txt`, verify `scenario=FORCE_STOP`
**"Alarms still present after force-stop"**
- **Cause:** Device/emulator doesn't clear alarms on force-stop (common on some devices)
- **Fix:** This is device-specific behavior. In `STRICTNESS=soft` mode, this is a warning. In `STRICTNESS=hard` mode, this is a failure.
- **Evidence:** Check `alarms/phase2_test1_after_force_stop.txt`
**"rescheduled>0 on first launch"**
- **Cause:** Boot recovery misfiring on empty database
- **Fix:** Check recovery logic, verify database is actually empty
- **Evidence:** Check `logs/phase2_test3_after_reboot.txt`
### Phase 3 Failures
**"Boot recovery not detected"**
- **Cause:** Boot receiver not registered, or BOOT_COMPLETED permission missing
- **Fix:** Verify `AndroidManifest.xml` has boot receiver and permission
- **Evidence:** Check `logs/phase3_test1_after_reboot.txt`, verify boot receiver logs
**"No missed alarms detected"**
- **Cause:** Alarm time didn't actually pass before reboot
- **Fix:** Verify system time was advanced, or wait longer before reboot
- **Evidence:** Check `alarms/phase3_test2_after_wait.txt`, verify alarm time is in past
**"Boot recovery ran but alarms not recreated"**
- **Cause:** Recovery succeeded but alarm scheduling failed
- **Fix:** Check alarm scheduling logic, verify permissions
- **Evidence:** Check `logs/phase3_test4_after_reboot.txt`, verify `rescheduled>0` but `after_count=0`
### General Failures
**"ADB device not found"**
- **Cause:** Device disconnected, ADB not in PATH, or device not authorized
- **Fix:** Run `adb devices`, verify device shows as "device" (not "unauthorized"), reconnect if needed
**"App build failed"**
- **Cause:** Gradle issues, missing dependencies, or Java version mismatch
- **Fix:** Run `./gradlew clean`, verify Java version, check `build.gradle` dependencies
**"Permission denied"**
- **Cause:** App doesn't have required permissions
- **Fix:** Grant permissions via app UI or `adb shell pm grant`
---
## Troubleshooting
### Script Won't Run
**"Permission denied: ./test-phase1.sh"**
```bash
chmod +x test-phase1.sh test-phase2.sh test-phase3.sh alarm-test-lib.sh
```
**"Command not found: adb"**
```bash
# Add Android SDK platform-tools to PATH
export PATH="$PATH:$ANDROID_HOME/platform-tools"
```
**"Syntax error near unexpected token"**
```bash
# Verify bash version
bash --version # Should be 4.0+
# Check script syntax
bash -n test-phase1.sh
```
### Evidence Not Captured
**"Run directory not initialized"**
- **Cause:** `ensure_run_dir()` failed
- **Fix:** Check write permissions in test directory, verify `runs/` directory can be created
**"Screenshot capture failed"**
- **Cause:** `ENABLE_SCREENSHOTS=0` or device doesn't support screencap
- **Fix:** Set `ENABLE_SCREENSHOTS=1` (default), verify device supports `adb exec-out screencap`
**"Logcat capture empty"**
- **Cause:** Logs cleared before capture, or filter pattern doesn't match
- **Fix:** Verify log tag matches filter pattern, check logs weren't cleared
### Device/Emulator Issues
**"Emulator not responding"**
```bash
# Restart emulator
adb kill-server
adb start-server
adb devices
```
**"Device unauthorized"**
- **Cause:** USB debugging authorization not granted
- **Fix:** Check device screen for authorization prompt, click "Allow"
**"Alarms not clearing on force-stop"**
- **Cause:** Device-specific behavior (some devices don't clear alarms)
- **Fix:** This is expected on some devices. Use `STRICTNESS=soft` mode to treat as warning.
### Test-Specific Issues
**"Phase 3 reboot takes too long"**
- **Cause:** Emulator is slow or hung
- **Fix:** Wait longer (60-90 seconds), or restart emulator if completely hung
**"Time advancement not working"**
- **Cause:** System time can't be advanced (requires root or emulator)
- **Fix:** Use emulator with root, or manually advance time via emulator settings
**"Plugin not configured"**
- **Cause:** Plugin setup not completed
- **Fix:** Run `./test-phase1.sh --setup` to configure plugin, or manually configure in app UI
---
## Quick Reference
### Command Cheat Sheet
```bash
# Phase 1
./test-phase1.sh --setup # Setup only
./test-phase1.sh --all # All tests
./test-phase1.sh --smoke # Smoke test
./test-phase1.sh 0 # TEST 0 only
# Phase 2
./test-phase2.sh # All tests (soft mode)
STRICTNESS=hard ./test-phase2.sh # All tests (hard mode)
./test-phase2.sh 1 3 # TEST 1 and 3
# Phase 3
./test-phase3.sh # All tests (advisory)
./test-phase3.sh --gate-phase3 # All tests (release-blocking)
./test-phase3.sh 2 # TEST 2 only
```
### Evidence Locations
```bash
# Latest run
ls -la runs/$(ls -t runs/ | head -1)
# Specific evidence
cat runs/<RUN_ID>/alarms/phase1_test0_initial.txt
cat runs/<RUN_ID>/logs/phase1_test0_after_schedule.txt
```
### Verdict Meanings
- **✅ PASS:** Test passed, continue
- **⚠️ WARN:** Minor issue, review evidence, continue
- **❌ FAIL:** Critical failure, review evidence, may exit (if gating enabled)
---
## Support & Feedback
**Issues or Questions?**
- Check evidence files in `runs/<RUN_ID>/`
- Review test logs for detailed error messages
- Consult platform-specific documentation in `docs/platform/android/`
**Improving This Runbook:**
- Document new failure patterns as they're discovered
- Update time estimates based on actual test runs
- Add automation hints for new test scenarios
---
**Last Updated:** 2025-01-XX
**Version:** 1.0
**Maintainer:** Test Engineering Team