Implement P0-P5 directives for operator clarity, consistent outcomes, and
easy evidence capture across all test phases.
Changes:
- alarm-test-lib.sh: Add evidence collection (capture_alarms, capture_logcat,
capture_screenshot), verdict functions (verdict_pass/warn/fail), run directory
management, and release gating support (RELEASE_GATE_PHASE3)
- test-phase1.sh: Refactor to unified framework with CLI modes (--setup,
--run, --smoke, --all, --ci), micro-prompts, evidence capture, and verdict
blocks for all 5 tests
- test-phase2.sh: Add evidence capture, verdict blocks, and STRICTNESS policy
(soft/hard) for warn vs fail behavior
- test-phase3.sh: Add evidence capture, verdict blocks, release gating
(--gate-phase3), and fatigue reduction (time estimates, automation hints)
- RUNBOOK-TESTING.md: New comprehensive operator guide (669 lines) covering
prerequisites, all phases, evidence locations, verdict interpretation,
common failures, and troubleshooting
All test scripts now use consistent UI helpers (section, substep, info, ok,
warn, error), standardized evidence collection, and clear verdict reporting.
Evidence is saved to timestamped run directories (runs/<RUN_ID>/) with alarms,
logs, and screenshots organized by test phase and scenario.
Tests pass with consistent presentation and reproducible evidence collection.
Add automatic app state reset for TEST 1 to ensure clean starting state when
lingering alarms from TEST 0 are detected. Create PHASE1_TEST1_GOLDEN.md with
actual values from successful run.
TEST 1 Auto-Reset:
- Detect lingering plugin alarms before TEST 1 starts
- Automatically uninstall/reinstall app to clear alarms
- Verify clean state (0 alarms) before proceeding
- Gracefully skip TEST 1 if clean state cannot be achieved
- Take failure screenshots when reset fails
- Wrap all TEST 1 steps in conditional to skip on reset failure
Documentation:
- Create PHASE1_TEST1_GOLDEN.md with actual values from passing run
- Document auto-reset behavior in golden run steps
- Add cross-references between TEST 0 and TEST 1 golden docs
- Include actual timestamps, scheduleIds, and recovery metrics
This ensures TEST 1 always starts from a known clean state, making test
results reliable and reproducible. The golden doc serves as a baseline for
comparing future TEST 1 runs.
Fix alarm counting to correctly parse dumpsys output where app ID and
action appear on different lines. Add screenshot capture for test
diagnostics and create golden run documentation.
Test Harness Improvements:
- Fix get_plugin_alarm_count() to track app ID and action separately
across alarm block lines (fixes false 0-count bug)
- Add show_plugin_alarms_compact() to display complete alarm blocks
- Add wait_for_stable_plugin_alarm_count() polling helper to reduce
race condition false negatives
- Add take_screenshot() and take_failure_screenshot() helpers for
automatic test state capture
- Integrate screenshots into TEST 0 at key checkpoints
- Update TEST 0 messaging to handle race conditions gracefully
- Add screenshots/ to .gitignore
Documentation:
- Create PHASE1_TEST0_GOLDEN.md with actual values from successful run
- Document expected script output, UI state, dumpsys shape, and logcat
patterns
- Include pass/fail checklist for future test runs
This fixes the issue where alarm counting always returned 0 because the
AWK logic required app ID and action on the same line, but dumpsys
output has them on separate lines (header line has app ID, tag line
has action).
