f6df9e13fb
- Add TestEventsPlugin for receiving ADB broadcast intents - Create operator console UI (console/index.html, console.css, console.js) - Add test plan structure (plan.json) with phases, tests, and steps - Wire all test scripts (phase1, phase2, phase3) with step context and events - Add event emission helpers to alarm-test-lib.sh (step_start, step_pass, etc.) - Update test-phase1.sh with comprehensive prerequisite verification - Register TestEventsPlugin in capacitor.plugins.json - Add console documentation (CONSOLE-USAGE.md, CONSOLE-REMAINING-WORK.md) - Add test implementation alignment tracking (TEST-IMPLEMENTATION-ALIGNMENT.md) This enables real-time test progress tracking via structured events from shell scripts to the operator console UI.
216 lines
7.6 KiB
Markdown
216 lines
7.6 KiB
Markdown
# Test Implementation Alignment with Documentation
|
|
|
|
**Last Updated:** 2025-12-29
|
|
**Purpose:** Document how test scripts align with golden run specifications and runbook guidance
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
The test implementation is guided by three types of documentation:
|
|
|
|
1. **Golden Run Documents** (`PHASE1_TEST0_GOLDEN.md`, `PHASE1_TEST1_GOLDEN.md`)
|
|
- Define **what a successful test looks like**
|
|
- Specify expected outputs, UI states, logcat patterns
|
|
- Provide pass/fail checklists
|
|
- **These are the test specifications**
|
|
|
|
2. **Runbook** (`RUNBOOK-TESTING.md`)
|
|
- Provides **operator guidance**
|
|
- Documents how to run tests, interpret results
|
|
- Troubleshooting and common issues
|
|
|
|
3. **Console Documentation** (`CONSOLE-USAGE.md`, `CONSOLE-REMAINING-WORK.md`)
|
|
- Documents the operator console UI
|
|
- Event-driven test execution
|
|
|
|
---
|
|
|
|
## How Golden Runs Guide Implementation
|
|
|
|
### PHASE1_TEST0_GOLDEN.md Requirements
|
|
|
|
**Step 4 (lines 54-59) specifies prerequisites:**
|
|
```
|
|
4. Confirmed plugin status in the UI:
|
|
- ⚙️ Plugin Settings: ✅ Configured
|
|
- 🔌 Native Fetcher: ✅ Configured
|
|
- 🔔 Notifications: ✅ Granted
|
|
- ⏰ Exact Alarms: ✅ Granted
|
|
- 📢 Channel: ✅ Enabled (High)
|
|
```
|
|
|
|
**Current Implementation:**
|
|
- ✅ Checks notification permissions (`check_permissions()`)
|
|
- ✅ Checks plugin configuration (`check_plugin_configured()`)
|
|
- ✅ Verifies exact alarms permission (via `dumpsys package`)
|
|
- ✅ Verifies channel status (via logcat)
|
|
- ✅ Comprehensive `verify_all_prerequisites()` function added
|
|
- ✅ Final UI verification prompt for all 5 items (aligned with golden run step 4)
|
|
|
|
**Alignment Status:**
|
|
✅ **FULLY ALIGNED** - Test 0 now verifies all 5 prerequisites as specified in the golden run:
|
|
1. Plugin Settings: Configured (via `check_plugin_configured()`)
|
|
2. Native Fetcher: Configured (via logs + plugin config check)
|
|
3. Notifications: Granted (via `check_permissions()`)
|
|
4. Exact Alarms: Granted (via `dumpsys package`)
|
|
5. Channel: Enabled (High) (via logcat + UI verification)
|
|
|
|
The implementation includes both programmatic checks and a final UI verification prompt that matches the golden run's step 4.
|
|
|
|
### Pass/Fail Checklist Alignment
|
|
|
|
**Golden Run Checklist (lines 172-194):**
|
|
|
|
1. **Script Output:**
|
|
- ✅ "Found 1 notification alarm (expected: 1)" - **Implemented**
|
|
- ✅ "Notification alarms after rollover: 1" - **Implemented**
|
|
- ✅ "TEST 0 PASSED" verdict - **Implemented**
|
|
|
|
2. **UI State:**
|
|
- ⚠️ "Before scheduling: Active Schedules: No" - **Not explicitly checked**
|
|
- ⚠️ "After scheduling: Active Schedules: Yes" - **Not explicitly checked**
|
|
- ⚠️ "After rollover: Active Schedules: Yes" - **Not explicitly checked**
|
|
|
|
3. **dumpsys alarm:**
|
|
- ✅ Exactly one RTC_WAKEUP alarm - **Implemented**
|
|
- ✅ origWhen timestamp 24h later - **Not explicitly verified**
|
|
|
|
4. **logcat:**
|
|
- ⚠️ `source=TEST_NOTIFICATION` sequence - **Not explicitly checked**
|
|
- ⚠️ `source=ROLLOVER_ON_FIRE` sequence - **Not explicitly checked**
|
|
- ✅ No duplicate DNP-SCHEDULE entries - **Partially checked**
|
|
|
|
**Current Implementation Status:**
|
|
- Core functionality: ✅ Aligned
|
|
- Detailed verification: ⚠️ Partial alignment
|
|
- UI state checks: ❌ Not implemented
|
|
- Logcat pattern verification: ⚠️ Partial
|
|
|
|
---
|
|
|
|
## How Runbook Guides Implementation
|
|
|
|
### RUNBOOK-TESTING.md Structure
|
|
|
|
**Section 3: Phase 1: Daily Rollover & Recovery**
|
|
|
|
**Test Descriptions (lines 144-186):**
|
|
- Defines what each test should do
|
|
- Provides time estimates
|
|
- Lists key steps
|
|
|
|
**Current Implementation:**
|
|
- ✅ Test purposes match runbook descriptions
|
|
- ✅ Test steps align with runbook key steps
|
|
- ✅ Time estimates are documented
|
|
|
|
**Evidence Location (lines 187-194):**
|
|
- Specifies where evidence should be saved
|
|
- Current implementation: ✅ Aligned (`runs/<RUN_ID>/`)
|
|
|
|
---
|
|
|
|
## Alignment Recommendations
|
|
|
|
### High Priority
|
|
|
|
1. ✅ **Add Prerequisite Verification to Test 0** - **COMPLETED**
|
|
- ✅ Added `verify_all_prerequisites()` function
|
|
- ✅ Verifies all 5 UI status items (Plugin Settings, Native Fetcher, Notifications, Exact Alarms, Channel)
|
|
- ✅ Includes programmatic checks and final UI verification prompt
|
|
- ✅ Aligned with golden run step 4
|
|
|
|
2. **Add UI State Checks**
|
|
- Verify "Active Schedules" state before/after scheduling
|
|
- Verify "Next Notification" time updates correctly
|
|
- Can be done via UI inspection or plugin API
|
|
|
|
3. **Add Logcat Pattern Verification**
|
|
- Check for `source=TEST_NOTIFICATION` sequence
|
|
- Check for `source=ROLLOVER_ON_FIRE` sequence
|
|
- Verify timing relationships match golden run
|
|
|
|
### Medium Priority
|
|
|
|
4. **Add Alarm Timestamp Verification**
|
|
- Verify `origWhen` is exactly 24h after initial time
|
|
- Can extract from `dumpsys alarm` output
|
|
|
|
5. **Document Manual vs Automated Checks**
|
|
- Clearly distinguish what script verifies vs what operator verifies
|
|
- Align with golden run's manual verification steps
|
|
|
|
### Low Priority
|
|
|
|
6. **Add Screenshot Verification**
|
|
- Golden run references screenshots
|
|
- Could add automated screenshot comparison (future)
|
|
|
|
---
|
|
|
|
## Current Implementation vs Golden Run
|
|
|
|
### Test 0: Daily Rollover Verification
|
|
|
|
| Requirement | Golden Run | Current Implementation | Status |
|
|
|------------|------------|------------------------|--------|
|
|
| Prerequisites (5 items) | ✅ All verified | ✅ All verified | ✅ Aligned |
|
|
| Schedule notification | ✅ Manual | ✅ Manual | ✅ Aligned |
|
|
| Wait for fire/advance time | ✅ Manual | ✅ Manual | ✅ Aligned |
|
|
| Verify alarm count | ✅ 1 alarm | ✅ 1 alarm | ✅ Aligned |
|
|
| Verify rollover | ✅ Tomorrow scheduled | ✅ Tomorrow scheduled | ✅ Aligned |
|
|
| UI state checks | ✅ Before/after | ❌ Not checked | Gap |
|
|
| Logcat patterns | ✅ Sequences verified | ⚠️ Partial | Partial |
|
|
| Alarm timestamp | ✅ 24h verified | ❌ Not verified | Gap |
|
|
|
|
### Test 1: Force-Stop Recovery
|
|
|
|
| Requirement | Golden Run | Current Implementation | Status |
|
|
|------------|------------|------------------------|--------|
|
|
| Clean start | ✅ Auto-reset | ✅ Auto-reset | ✅ Aligned |
|
|
| Schedule notification | ✅ Manual | ✅ Manual | ✅ Aligned |
|
|
| Force-stop app | ✅ Manual | ✅ Manual | ✅ Aligned |
|
|
| Verify alarms cleared | ✅ 0 alarms | ✅ 0 alarms | ✅ Aligned |
|
|
| Relaunch app | ✅ Manual | ✅ Manual | ✅ Aligned |
|
|
| Verify recovery | ✅ 1 alarm restored | ✅ 1 alarm restored | ✅ Aligned |
|
|
| Recovery logs | ✅ FORCE_STOP scenario | ✅ FORCE_STOP scenario | ✅ Aligned |
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
1. **Enhance Test 0 Prerequisites**
|
|
- Add explicit checks for exact alarms and channel status
|
|
- Use plugin API to verify all 5 items programmatically
|
|
|
|
2. **Add UI State Verification**
|
|
- Check "Active Schedules" state via plugin API or UI inspection
|
|
- Verify "Next Notification" time updates
|
|
|
|
3. **Add Logcat Pattern Checks**
|
|
- Verify `source=TEST_NOTIFICATION` and `source=ROLLOVER_ON_FIRE` sequences
|
|
- Check timing relationships
|
|
|
|
4. **Update Golden Run Documents**
|
|
- Document which checks are automated vs manual
|
|
- Clarify operator responsibilities
|
|
|
|
---
|
|
|
|
## Conclusion
|
|
|
|
**Current State:**
|
|
- Core test functionality: ✅ Well aligned with golden runs
|
|
- Detailed verification: ⚠️ Partial alignment
|
|
- Prerequisites: ⚠️ Need to verify all 5 items
|
|
|
|
**Key Insight:**
|
|
The golden run documents specify **what** should be verified, but don't always specify **how** (automated vs manual). Our implementation should:
|
|
1. Automate what can be automated
|
|
2. Clearly document what requires manual verification
|
|
3. Align with the golden run's verification sequence
|
|
|
|
**Priority:**
|
|
Focus on adding the missing prerequisite checks (exact alarms, channel status) to fully align with the golden run specification.
|