forked from trent_larson/crowd-funder-for-time-pwa
feat(harbor-pilot): add Playwright test investigation directive
- Create comprehensive MDC rule for systematic Playwright test failure investigation - Integrate rule into harbor_pilot_universal.mdc for team-wide access - Include investigation workflow, diagnostic commands, and evidence-based analysis - Document specific failure patterns (alert stacking, selector conflicts, timing issues) - Provide practical examples from recent test failure investigation - Add investigation commands for error context, trace files, and page snapshots This rule transforms ad-hoc test debugging into systematic investigation process, leveraging Playwright's built-in debugging tools for faster root cause identification.
This commit is contained in:
Matthew Raymer
@@ -203,4 +203,5 @@ Follow this exact order **after** the Base Contract’s **Objective → Result
|
||||
- Respect Base *Do-Not* (no filler, no invented facts, no censorship).
|
||||
- Prefer clarity over completeness when timeboxed; capture unknowns explicitly.
|
||||
- Apply historical comment management rules (see `.cursor/rules/historical_comment_management.mdc`)
|
||||
- Apply realistic time estimation rules (see `.cursor/rules/realistic_time_estimation.mdc`)
|
||||
- Apply realistic time estimation rules (see `.cursor/rules/realistic_time_estimation.mdc`)
|
||||
- Apply Playwright test investigation rules (see `.cursor/rules/playwright_test_investigation.mdc`)
|
||||
356
.cursor/rules/playwright-test-investigation.mdc
Normal file
356
.cursor/rules/playwright-test-investigation.mdc
Normal file
@@ -0,0 +1,356 @@
|
||||
---
|
||||
description: when working with playwright tests either generating them or using them to test code
|
||||
alwaysApply: false
|
||||
---
|
||||
# Playwright Test Investigation — Harbor Pilot Directive
|
||||
|
||||
**Author**: Matthew Raymer
|
||||
**Date**: 2025-08-21T14:22Z
|
||||
**Status**: 🎯 **ACTIVE** - Playwright test debugging guidelines
|
||||
|
||||
## Objective
|
||||
Provide systematic approach for investigating Playwright test failures with focus on UI element conflicts, timing issues, and selector ambiguity.
|
||||
|
||||
## Context & Scope
|
||||
- **Audience**: Developers debugging Playwright test failures
|
||||
- **In scope**: Test failure analysis, selector conflicts, UI state investigation, timing issues
|
||||
- **Out of scope**: Test writing best practices, CI/CD configuration
|
||||
|
||||
## Artifacts & Links
|
||||
- Test results: `test-results/` directory
|
||||
- Error context: `error-context.md` files with page snapshots
|
||||
- Trace files: `trace.zip` files for failed tests
|
||||
- HTML reports: Interactive test reports with screenshots
|
||||
|
||||
## Environment & Preconditions
|
||||
- OS/Runtime: Linux/Windows/macOS with Node.js
|
||||
- Versions: Playwright test framework, browser drivers
|
||||
- Services: Local test server (localhost:8080), test data setup
|
||||
- Auth mode: None required for test investigation
|
||||
|
||||
## Architecture / Process Overview
|
||||
Playwright test investigation follows a systematic diagnostic workflow that leverages built-in debugging tools and error context analysis.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Test Failure] --> B[Check Error Context]
|
||||
B --> C[Analyze Page Snapshot]
|
||||
C --> D[Identify UI Conflicts]
|
||||
D --> E[Check Trace Files]
|
||||
E --> F[Verify Selector Uniqueness]
|
||||
F --> G[Test Selector Fixes]
|
||||
G --> H[Document Root Cause]
|
||||
|
||||
B --> I[Check Test Results Directory]
|
||||
I --> J[Locate Failed Test Results]
|
||||
J --> K[Extract Error Details]
|
||||
|
||||
D --> L[Multiple Alerts?]
|
||||
L --> M[Button Text Conflicts?]
|
||||
M --> N[Timing Issues?]
|
||||
|
||||
E --> O[Use Trace Viewer]
|
||||
O --> P[Analyze Action Sequence]
|
||||
P --> Q[Identify Failure Point]
|
||||
```
|
||||
|
||||
## Interfaces & Contracts
|
||||
|
||||
### Test Results Structure
|
||||
| Component | Format | Content | Validation |
|
||||
|---|---|---|---|
|
||||
| Error Context | Markdown | Page snapshot in YAML | Verify DOM state matches test expectations |
|
||||
| Trace Files | ZIP archive | Detailed execution trace | Use `npx playwright show-trace` |
|
||||
| HTML Reports | Interactive HTML | Screenshots, traces, logs | Check browser for full report |
|
||||
| JSON Results | JSON | Machine-readable results | Parse for automated analysis |
|
||||
|
||||
### Investigation Commands
|
||||
| Step | Command | Expected Output | Notes |
|
||||
|---|---|---|---|
|
||||
| Locate failed tests | `find test-results -name "*test-name*"` | Test result directories | Use exact test name patterns |
|
||||
| Check error context | `cat test-results/*/error-context.md` | Page snapshots | Look for UI state conflicts |
|
||||
| View traces | `npx playwright show-trace trace.zip` | Interactive trace viewer | Analyze exact failure sequence |
|
||||
|
||||
## Repro: End-to-End Investigation Procedure
|
||||
|
||||
### 1. Locate Failed Test Results
|
||||
```bash
|
||||
# Find all results for a specific test
|
||||
find test-results -name "*test-name*" -type d
|
||||
|
||||
# Check for error context files
|
||||
find test-results -name "error-context.md" | head -5
|
||||
```
|
||||
|
||||
### 2. Analyze Error Context
|
||||
```bash
|
||||
# Read error context for specific test
|
||||
cat test-results/test-name-test-description-browser/error-context.md
|
||||
|
||||
# Look for UI conflicts in page snapshot
|
||||
grep -A 10 -B 5 "button.*Yes\|button.*No" test-results/*/error-context.md
|
||||
```
|
||||
|
||||
### 3. Check Trace Files
|
||||
```bash
|
||||
# List available trace files
|
||||
find test-results -name "*.zip" | grep trace
|
||||
|
||||
# View trace in browser
|
||||
npx playwright show-trace test-results/test-name/trace.zip
|
||||
```
|
||||
|
||||
### 4. Investigate Selector Issues
|
||||
```typescript
|
||||
// Check for multiple elements with same text
|
||||
await page.locator('button:has-text("Yes")').count(); // Should be 1
|
||||
|
||||
// Use more specific selectors
|
||||
await page.locator('div[role="alert"]:has-text("Register") button:has-text("Yes")').click();
|
||||
```
|
||||
|
||||
## What Works (Evidence)
|
||||
- ✅ **Error context files** provide page snapshots showing exact DOM state at failure
|
||||
- **Time**: 2025-08-21T14:22Z
|
||||
- **Evidence**: `test-results/60-new-activity-New-offers-for-another-user-chromium/error-context.md` shows both alerts visible
|
||||
- **Verify at**: Error context files in test results directory
|
||||
|
||||
- ✅ **Trace files** capture detailed execution sequence for failed tests
|
||||
- **Time**: 2025-08-21T14:22Z
|
||||
- **Evidence**: `trace.zip` files available for all failed tests
|
||||
- **Verify at**: Use `npx playwright show-trace <filename>`
|
||||
|
||||
- ✅ **Page snapshots** reveal UI conflicts like multiple alerts with duplicate button text
|
||||
- **Time**: 2025-08-21T14:22Z
|
||||
- **Evidence**: YAML snapshots show registration + export alerts simultaneously
|
||||
- **Verify at**: Error context markdown files
|
||||
|
||||
## What Doesn't (Evidence & Hypotheses)
|
||||
- ❌ **Generic selectors** fail with multiple similar elements at `test-playwright/testUtils.ts:161`
|
||||
- **Time**: 2025-08-21T14:22Z
|
||||
- **Evidence**: `button:has-text("Yes")` matches both "Yes" and "Yes, Export Data"
|
||||
- **Hypothesis**: Selector ambiguity due to multiple alerts with conflicting button text
|
||||
- **Next probe**: Use more specific selectors or dismiss alerts sequentially
|
||||
|
||||
- ❌ **Timing-dependent tests** fail due to alert stacking at `src/views/ContactsView.vue:860,1283`
|
||||
- **Time**: 2025-08-21T14:22Z
|
||||
- **Evidence**: Both alerts use identical 1000ms delays, ensuring simultaneous display
|
||||
- **Hypothesis**: Race condition between alert displays creates UI conflicts
|
||||
- **Next probe**: Implement alert queuing or prevent overlapping alerts
|
||||
|
||||
## Risks, Limits, Assumptions
|
||||
- **Trace file size**: Large trace files may impact storage and analysis time
|
||||
- **Browser compatibility**: Trace viewer requires specific browser support
|
||||
- **Test isolation**: Shared state between tests may affect investigation results
|
||||
- **Timing sensitivity**: Tests may pass/fail based on system performance
|
||||
|
||||
## Next Steps
|
||||
| Owner | Task | Exit Criteria | Target Date (UTC) |
|
||||
|---|---|---|---|
|
||||
| Development Team | Fix test selectors for multiple alerts | All tests pass consistently | 2025-08-22 |
|
||||
| Development Team | Implement alert queuing system | No overlapping alerts with conflicting buttons | 2025-08-25 |
|
||||
| Development Team | Add test IDs to alert buttons | Unique selectors for all UI elements | 2025-08-28 |
|
||||
|
||||
## References
|
||||
- [Playwright Trace Viewer Documentation](https://playwright.dev/docs/trace-viewer)
|
||||
- [Playwright Test Results](https://playwright.dev/docs/test-reporters)
|
||||
- [Test Investigation Workflow](./research_diagnostic.mdc)
|
||||
|
||||
## Competence Hooks
|
||||
- **Why this works**: Systematic investigation leverages Playwright's built-in debugging tools to identify root causes
|
||||
- **Common pitfalls**: Generic selectors fail with multiple similar elements; timing issues create race conditions; alert stacking causes UI conflicts
|
||||
- **Next skill unlock**: Implement unique test IDs and handle alert dismissal order in test flows
|
||||
- **Teach-back**: "How would you investigate a Playwright test failure using error context, trace files, and page snapshots?"
|
||||
|
||||
## Collaboration Hooks
|
||||
- **Reviewers**: QA team, test automation engineers
|
||||
- **Sign-off checklist**: Error context analyzed, trace files reviewed, root cause identified, fix implemented and tested
|
||||
|
||||
## Assumptions & Limits
|
||||
- Test results directory structure follows Playwright conventions
|
||||
- Trace files are enabled in configuration (`trace: "retain-on-failure"`)
|
||||
- Error context files contain valid YAML page snapshots
|
||||
- Browser environment supports trace viewer functionality
|
||||
|
||||
---
|
||||
|
||||
**Status**: Active investigation directive
|
||||
**Priority**: High
|
||||
**Maintainer**: Development team
|
||||
**Next Review**: 2025-09-21
|
||||
# Playwright Test Investigation — Harbor Pilot Directive
|
||||
|
||||
**Author**: Matthew Raymer
|
||||
**Date**: 2025-08-21T14:22Z
|
||||
**Status**: 🎯 **ACTIVE** - Playwright test debugging guidelines
|
||||
|
||||
## Objective
|
||||
Provide systematic approach for investigating Playwright test failures with focus on UI element conflicts, timing issues, and selector ambiguity.
|
||||
|
||||
## Context & Scope
|
||||
- **Audience**: Developers debugging Playwright test failures
|
||||
- **In scope**: Test failure analysis, selector conflicts, UI state investigation, timing issues
|
||||
- **Out of scope**: Test writing best practices, CI/CD configuration
|
||||
|
||||
## Artifacts & Links
|
||||
- Test results: `test-results/` directory
|
||||
- Error context: `error-context.md` files with page snapshots
|
||||
- Trace files: `trace.zip` files for failed tests
|
||||
- HTML reports: Interactive test reports with screenshots
|
||||
|
||||
## Environment & Preconditions
|
||||
- OS/Runtime: Linux/Windows/macOS with Node.js
|
||||
- Versions: Playwright test framework, browser drivers
|
||||
- Services: Local test server (localhost:8080), test data setup
|
||||
- Auth mode: None required for test investigation
|
||||
|
||||
## Architecture / Process Overview
|
||||
Playwright test investigation follows a systematic diagnostic workflow that leverages built-in debugging tools and error context analysis.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Test Failure] --> B[Check Error Context]
|
||||
B --> C[Analyze Page Snapshot]
|
||||
C --> D[Identify UI Conflicts]
|
||||
D --> E[Check Trace Files]
|
||||
E --> F[Verify Selector Uniqueness]
|
||||
F --> G[Test Selector Fixes]
|
||||
G --> H[Document Root Cause]
|
||||
|
||||
B --> I[Check Test Results Directory]
|
||||
I --> J[Locate Failed Test Results]
|
||||
J --> K[Extract Error Details]
|
||||
|
||||
D --> L[Multiple Alerts?]
|
||||
L --> M[Button Text Conflicts?]
|
||||
M --> N[Timing Issues?]
|
||||
|
||||
E --> O[Use Trace Viewer]
|
||||
O --> P[Analyze Action Sequence]
|
||||
P --> Q[Identify Failure Point]
|
||||
```
|
||||
|
||||
## Interfaces & Contracts
|
||||
|
||||
### Test Results Structure
|
||||
| Component | Format | Content | Validation |
|
||||
|---|---|---|---|
|
||||
| Error Context | Markdown | Page snapshot in YAML | Verify DOM state matches test expectations |
|
||||
| Trace Files | ZIP archive | Detailed execution trace | Use `npx playwright show-trace` |
|
||||
| HTML Reports | Interactive HTML | Screenshots, traces, logs | Check browser for full report |
|
||||
| JSON Results | JSON | Machine-readable results | Parse for automated analysis |
|
||||
|
||||
### Investigation Commands
|
||||
| Step | Command | Expected Output | Notes |
|
||||
|---|---|---|---|
|
||||
| Locate failed tests | `find test-results -name "*test-name*"` | Test result directories | Use exact test name patterns |
|
||||
| Check error context | `cat test-results/*/error-context.md` | Page snapshots | Look for UI state conflicts |
|
||||
| View traces | `npx playwright show-trace trace.zip` | Interactive trace viewer | Analyze exact failure sequence |
|
||||
|
||||
## Repro: End-to-End Investigation Procedure
|
||||
|
||||
### 1. Locate Failed Test Results
|
||||
```bash
|
||||
# Find all results for a specific test
|
||||
find test-results -name "*test-name*" -type d
|
||||
|
||||
# Check for error context files
|
||||
find test-results -name "error-context.md" | head -5
|
||||
```
|
||||
|
||||
### 2. Analyze Error Context
|
||||
```bash
|
||||
# Read error context for specific test
|
||||
cat test-results/test-name-test-description-browser/error-context.md
|
||||
|
||||
# Look for UI conflicts in page snapshot
|
||||
grep -A 10 -B 5 "button.*Yes\|button.*No" test-results/*/error-context.md
|
||||
```
|
||||
|
||||
### 3. Check Trace Files
|
||||
```bash
|
||||
# List available trace files
|
||||
find test-results -name "*.zip" | grep trace
|
||||
|
||||
# View trace in browser
|
||||
npx playwright show-trace test-results/test-name/trace.zip
|
||||
```
|
||||
|
||||
### 4. Investigate Selector Issues
|
||||
```typescript
|
||||
// Check for multiple elements with same text
|
||||
await page.locator('button:has-text("Yes")').count(); // Should be 1
|
||||
|
||||
// Use more specific selectors
|
||||
await page.locator('div[role="alert"]:has-text("Register") button:has-text("Yes")').click();
|
||||
```
|
||||
|
||||
## What Works (Evidence)
|
||||
- ✅ **Error context files** provide page snapshots showing exact DOM state at failure
|
||||
- **Time**: 2025-08-21T14:22Z
|
||||
- **Evidence**: `test-results/60-new-activity-New-offers-for-another-user-chromium/error-context.md` shows both alerts visible
|
||||
- **Verify at**: Error context files in test results directory
|
||||
|
||||
- ✅ **Trace files** capture detailed execution sequence for failed tests
|
||||
- **Time**: 2025-08-21T14:22Z
|
||||
- **Evidence**: `trace.zip` files available for all failed tests
|
||||
- **Verify at**: Use `npx playwright show-trace <filename>`
|
||||
|
||||
- ✅ **Page snapshots** reveal UI conflicts like multiple alerts with duplicate button text
|
||||
- **Time**: 2025-08-21T14:22Z
|
||||
- **Evidence**: YAML snapshots show registration + export alerts simultaneously
|
||||
- **Verify at**: Error context markdown files
|
||||
|
||||
## What Doesn't (Evidence & Hypotheses)
|
||||
- ❌ **Generic selectors** fail with multiple similar elements at `test-playwright/testUtils.ts:161`
|
||||
- **Time**: 2025-08-21T14:22Z
|
||||
- **Evidence**: `button:has-text("Yes")` matches both "Yes" and "Yes, Export Data"
|
||||
- **Hypothesis**: Selector ambiguity due to multiple alerts with conflicting button text
|
||||
- **Next probe**: Use more specific selectors or dismiss alerts sequentially
|
||||
|
||||
- ❌ **Timing-dependent tests** fail due to alert stacking at `src/views/ContactsView.vue:860,1283`
|
||||
- **Time**: 2025-08-21T14:22Z
|
||||
- **Evidence**: Both alerts use identical 1000ms delays, ensuring simultaneous display
|
||||
- **Hypothesis**: Race condition between alert displays creates UI conflicts
|
||||
- **Next probe**: Implement alert queuing or prevent overlapping alerts
|
||||
|
||||
## Risks, Limits, Assumptions
|
||||
- **Trace file size**: Large trace files may impact storage and analysis time
|
||||
- **Browser compatibility**: Trace viewer requires specific browser support
|
||||
- **Test isolation**: Shared state between tests may affect investigation results
|
||||
- **Timing sensitivity**: Tests may pass/fail based on system performance
|
||||
|
||||
## Next Steps
|
||||
| Owner | Task | Exit Criteria | Target Date (UTC) |
|
||||
|---|---|---|---|
|
||||
| Development Team | Fix test selectors for multiple alerts | All tests pass consistently | 2025-08-22 |
|
||||
| Development Team | Implement alert queuing system | No overlapping alerts with conflicting buttons | 2025-08-25 |
|
||||
| Development Team | Add test IDs to alert buttons | Unique selectors for all UI elements | 2025-08-28 |
|
||||
|
||||
## References
|
||||
- [Playwright Trace Viewer Documentation](https://playwright.dev/docs/trace-viewer)
|
||||
- [Playwright Test Results](https://playwright.dev/docs/test-reporters)
|
||||
- [Test Investigation Workflow](./research_diagnostic.mdc)
|
||||
|
||||
## Competence Hooks
|
||||
- **Why this works**: Systematic investigation leverages Playwright's built-in debugging tools to identify root causes
|
||||
- **Common pitfalls**: Generic selectors fail with multiple similar elements; timing issues create race conditions; alert stacking causes UI conflicts
|
||||
- **Next skill unlock**: Implement unique test IDs and handle alert dismissal order in test flows
|
||||
- **Teach-back**: "How would you investigate a Playwright test failure using error context, trace files, and page snapshots?"
|
||||
|
||||
## Collaboration Hooks
|
||||
- **Reviewers**: QA team, test automation engineers
|
||||
- **Sign-off checklist**: Error context analyzed, trace files reviewed, root cause identified, fix implemented and tested
|
||||
|
||||
## Assumptions & Limits
|
||||
- Test results directory structure follows Playwright conventions
|
||||
- Trace files are enabled in configuration (`trace: "retain-on-failure"`)
|
||||
- Error context files contain valid YAML page snapshots
|
||||
- Browser environment supports trace viewer functionality
|
||||
|
||||
---
|
||||
|
||||
**Status**: Active investigation directive
|
||||
**Priority**: High
|
||||
**Maintainer**: Development team
|
||||
**Next Review**: 2025-09-21
|
||||
Reference in New Issue
Block a user