# Test Implementation Alignment with Documentation

**Last Updated:** 2025-12-29  
**Purpose:** Document how test scripts align with golden run specifications and runbook guidance

---

## Overview

The test implementation is guided by three types of documentation:

1. **Golden Run Documents** (`PHASE1_TEST0_GOLDEN.md`, `PHASE1_TEST1_GOLDEN.md`)
   - Define **what a successful test looks like**
   - Specify expected outputs, UI states, logcat patterns
   - Provide pass/fail checklists
   - **These are the test specifications**

2. **Runbook** (`RUNBOOK-TESTING.md`)
   - Provides **operator guidance**
   - Documents how to run tests, interpret results
   - Troubleshooting and common issues

3. **Console Documentation** (`CONSOLE-USAGE.md`, `CONSOLE-REMAINING-WORK.md`)
   - Documents the operator console UI
   - Event-driven test execution

---

## How Golden Runs Guide Implementation

### PHASE1_TEST0_GOLDEN.md Requirements

**Step 4 (lines 54-59) specifies prerequisites:**
```
4. Confirmed plugin status in the UI:
   - ⚙️ Plugin Settings: ✅ Configured
   - 🔌 Native Fetcher: ✅ Configured
   - 🔔 Notifications: ✅ Granted
   - ⏰ Exact Alarms: ✅ Granted
   - 📢 Channel: ✅ Enabled (High)
```

**Current Implementation:**
- ✅ Checks notification permissions (`check_permissions()`)
- ✅ Checks plugin configuration (`check_plugin_configured()`)
- ✅ Verifies exact alarms permission (via `dumpsys package`)
- ✅ Verifies channel status (via logcat)
- ✅ Comprehensive `verify_all_prerequisites()` function added
- ✅ Final UI verification prompt for all 5 items (aligned with golden run step 4)

**Alignment Status:**
✅ **FULLY ALIGNED** - Test 0 now verifies all 5 prerequisites as specified in the golden run:
1. Plugin Settings: Configured (via `check_plugin_configured()`)
2. Native Fetcher: Configured (via logs + plugin config check)
3. Notifications: Granted (via `check_permissions()`)
4. Exact Alarms: Granted (via `dumpsys package`)
5. Channel: Enabled (High) (via logcat + UI verification)

The implementation includes both programmatic checks and a final UI verification prompt that matches the golden run's step 4.

### Pass/Fail Checklist Alignment

**Golden Run Checklist (lines 172-194):**

1. **Script Output:**
   - ✅ "Found 1 notification alarm (expected: 1)" - **Implemented**
   - ✅ "Notification alarms after rollover: 1" - **Implemented**
   - ✅ "TEST 0 PASSED" verdict - **Implemented**

2. **UI State:**
   - ⚠️ "Before scheduling: Active Schedules: No" - **Not explicitly checked**
   - ⚠️ "After scheduling: Active Schedules: Yes" - **Not explicitly checked**
   - ⚠️ "After rollover: Active Schedules: Yes" - **Not explicitly checked**

3. **dumpsys alarm:**
   - ✅ Exactly one RTC_WAKEUP alarm - **Implemented**
   - ✅ origWhen timestamp 24h later - **Not explicitly verified**

4. **logcat:**
   - ⚠️ `source=TEST_NOTIFICATION` sequence - **Not explicitly checked**
   - ⚠️ `source=ROLLOVER_ON_FIRE` sequence - **Not explicitly checked**
   - ✅ No duplicate DNP-SCHEDULE entries - **Partially checked**

**Current Implementation Status:**
- Core functionality: ✅ Aligned
- Detailed verification: ⚠️ Partial alignment
- UI state checks: ❌ Not implemented
- Logcat pattern verification: ⚠️ Partial

---

## How Runbook Guides Implementation

### RUNBOOK-TESTING.md Structure

**Section 3: Phase 1: Daily Rollover & Recovery**

**Test Descriptions (lines 144-186):**
- Defines what each test should do
- Provides time estimates
- Lists key steps

**Current Implementation:**
- ✅ Test purposes match runbook descriptions
- ✅ Test steps align with runbook key steps
- ✅ Time estimates are documented

**Evidence Location (lines 187-194):**
- Specifies where evidence should be saved
- Current implementation: ✅ Aligned (`runs/<RUN_ID>/`)

---

## Alignment Recommendations

### High Priority

1. ✅ **Add Prerequisite Verification to Test 0** - **COMPLETED**
   - ✅ Added `verify_all_prerequisites()` function
   - ✅ Verifies all 5 UI status items (Plugin Settings, Native Fetcher, Notifications, Exact Alarms, Channel)
   - ✅ Includes programmatic checks and final UI verification prompt
   - ✅ Aligned with golden run step 4

2. **Add UI State Checks**
   - Verify "Active Schedules" state before/after scheduling
   - Verify "Next Notification" time updates correctly
   - Can be done via UI inspection or plugin API

3. **Add Logcat Pattern Verification**
   - Check for `source=TEST_NOTIFICATION` sequence
   - Check for `source=ROLLOVER_ON_FIRE` sequence
   - Verify timing relationships match golden run

### Medium Priority

4. **Add Alarm Timestamp Verification**
   - Verify `origWhen` is exactly 24h after initial time
   - Can extract from `dumpsys alarm` output

5. **Document Manual vs Automated Checks**
   - Clearly distinguish what script verifies vs what operator verifies
   - Align with golden run's manual verification steps

### Low Priority

6. **Add Screenshot Verification**
   - Golden run references screenshots
   - Could add automated screenshot comparison (future)

---

## Current Implementation vs Golden Run

### Test 0: Daily Rollover Verification

| Requirement | Golden Run | Current Implementation | Status |
|------------|------------|------------------------|--------|
| Prerequisites (5 items) | ✅ All verified | ✅ All verified | ✅ Aligned |
| Schedule notification | ✅ Manual | ✅ Manual | ✅ Aligned |
| Wait for fire/advance time | ✅ Manual | ✅ Manual | ✅ Aligned |
| Verify alarm count | ✅ 1 alarm | ✅ 1 alarm | ✅ Aligned |
| Verify rollover | ✅ Tomorrow scheduled | ✅ Tomorrow scheduled | ✅ Aligned |
| UI state checks | ✅ Before/after | ❌ Not checked | Gap |
| Logcat patterns | ✅ Sequences verified | ⚠️ Partial | Partial |
| Alarm timestamp | ✅ 24h verified | ❌ Not verified | Gap |

### Test 1: Force-Stop Recovery

| Requirement | Golden Run | Current Implementation | Status |
|------------|------------|------------------------|--------|
| Clean start | ✅ Auto-reset | ✅ Auto-reset | ✅ Aligned |
| Schedule notification | ✅ Manual | ✅ Manual | ✅ Aligned |
| Force-stop app | ✅ Manual | ✅ Manual | ✅ Aligned |
| Verify alarms cleared | ✅ 0 alarms | ✅ 0 alarms | ✅ Aligned |
| Relaunch app | ✅ Manual | ✅ Manual | ✅ Aligned |
| Verify recovery | ✅ 1 alarm restored | ✅ 1 alarm restored | ✅ Aligned |
| Recovery logs | ✅ FORCE_STOP scenario | ✅ FORCE_STOP scenario | ✅ Aligned |

---

## Next Steps

1. **Enhance Test 0 Prerequisites**
   - Add explicit checks for exact alarms and channel status
   - Use plugin API to verify all 5 items programmatically

2. **Add UI State Verification**
   - Check "Active Schedules" state via plugin API or UI inspection
   - Verify "Next Notification" time updates

3. **Add Logcat Pattern Checks**
   - Verify `source=TEST_NOTIFICATION` and `source=ROLLOVER_ON_FIRE` sequences
   - Check timing relationships

4. **Update Golden Run Documents**
   - Document which checks are automated vs manual
   - Clarify operator responsibilities

---

## Conclusion

**Current State:**
- Core test functionality: ✅ Well aligned with golden runs
- Detailed verification: ⚠️ Partial alignment
- Prerequisites: ⚠️ Need to verify all 5 items

**Key Insight:**
The golden run documents specify **what** should be verified, but don't always specify **how** (automated vs manual). Our implementation should:
1. Automate what can be automated
2. Clearly document what requires manual verification
3. Align with the golden run's verification sequence

**Priority:**
Focus on adding the missing prerequisite checks (exact alarms, channel status) to fully align with the golden run specification.
