Fix: account for new Export Data dialog #174
Open
jose
wants to merge 4 commits from playwright-test-60-fix
into master
4 changed files with 364 additions and 6 deletions
@ -0,0 +1,356 @@ |
|||
--- |
|||
description: when working with playwright tests either generating them or using them to test code |
|||
alwaysApply: false |
|||
--- |
|||
# Playwright Test Investigation — Harbor Pilot Directive |
|||
|
|||
**Author**: Matthew Raymer |
|||
**Date**: 2025-08-21T14:22Z |
|||
**Status**: 🎯 **ACTIVE** - Playwright test debugging guidelines |
|||
|
|||
## Objective |
|||
Provide systematic approach for investigating Playwright test failures with focus on UI element conflicts, timing issues, and selector ambiguity. |
|||
|
|||
## Context & Scope |
|||
- **Audience**: Developers debugging Playwright test failures |
|||
- **In scope**: Test failure analysis, selector conflicts, UI state investigation, timing issues |
|||
- **Out of scope**: Test writing best practices, CI/CD configuration |
|||
|
|||
## Artifacts & Links |
|||
- Test results: `test-results/` directory |
|||
- Error context: `error-context.md` files with page snapshots |
|||
- Trace files: `trace.zip` files for failed tests |
|||
- HTML reports: Interactive test reports with screenshots |
|||
|
|||
## Environment & Preconditions |
|||
- OS/Runtime: Linux/Windows/macOS with Node.js |
|||
- Versions: Playwright test framework, browser drivers |
|||
- Services: Local test server (localhost:8080), test data setup |
|||
- Auth mode: None required for test investigation |
|||
|
|||
## Architecture / Process Overview |
|||
Playwright test investigation follows a systematic diagnostic workflow that leverages built-in debugging tools and error context analysis. |
|||
|
|||
```mermaid |
|||
flowchart TD |
|||
A[Test Failure] --> B[Check Error Context] |
|||
B --> C[Analyze Page Snapshot] |
|||
C --> D[Identify UI Conflicts] |
|||
D --> E[Check Trace Files] |
|||
E --> F[Verify Selector Uniqueness] |
|||
F --> G[Test Selector Fixes] |
|||
G --> H[Document Root Cause] |
|||
|
|||
B --> I[Check Test Results Directory] |
|||
I --> J[Locate Failed Test Results] |
|||
J --> K[Extract Error Details] |
|||
|
|||
D --> L[Multiple Alerts?] |
|||
L --> M[Button Text Conflicts?] |
|||
M --> N[Timing Issues?] |
|||
|
|||
E --> O[Use Trace Viewer] |
|||
O --> P[Analyze Action Sequence] |
|||
P --> Q[Identify Failure Point] |
|||
``` |
|||
|
|||
## Interfaces & Contracts |
|||
|
|||
### Test Results Structure |
|||
| Component | Format | Content | Validation | |
|||
|---|---|---|---| |
|||
| Error Context | Markdown | Page snapshot in YAML | Verify DOM state matches test expectations | |
|||
| Trace Files | ZIP archive | Detailed execution trace | Use `npx playwright show-trace` | |
|||
| HTML Reports | Interactive HTML | Screenshots, traces, logs | Check browser for full report | |
|||
| JSON Results | JSON | Machine-readable results | Parse for automated analysis | |
|||
|
|||
### Investigation Commands |
|||
| Step | Command | Expected Output | Notes | |
|||
|---|---|---|---| |
|||
| Locate failed tests | `find test-results -name "*test-name*"` | Test result directories | Use exact test name patterns | |
|||
| Check error context | `cat test-results/*/error-context.md` | Page snapshots | Look for UI state conflicts | |
|||
| View traces | `npx playwright show-trace trace.zip` | Interactive trace viewer | Analyze exact failure sequence | |
|||
|
|||
## Repro: End-to-End Investigation Procedure |
|||
|
|||
### 1. Locate Failed Test Results |
|||
```bash |
|||
# Find all results for a specific test |
|||
find test-results -name "*test-name*" -type d |
|||
|
|||
# Check for error context files |
|||
find test-results -name "error-context.md" | head -5 |
|||
``` |
|||
|
|||
### 2. Analyze Error Context |
|||
```bash |
|||
# Read error context for specific test |
|||
cat test-results/test-name-test-description-browser/error-context.md |
|||
|
|||
# Look for UI conflicts in page snapshot |
|||
grep -A 10 -B 5 "button.*Yes\|button.*No" test-results/*/error-context.md |
|||
``` |
|||
|
|||
### 3. Check Trace Files |
|||
```bash |
|||
# List available trace files |
|||
find test-results -name "*.zip" | grep trace |
|||
|
|||
# View trace in browser |
|||
npx playwright show-trace test-results/test-name/trace.zip |
|||
``` |
|||
|
|||
### 4. Investigate Selector Issues |
|||
```typescript |
|||
// Check for multiple elements with same text |
|||
await page.locator('button:has-text("Yes")').count(); // Should be 1 |
|||
|
|||
// Use more specific selectors |
|||
await page.locator('div[role="alert"]:has-text("Register") button:has-text("Yes")').click(); |
|||
``` |
|||
|
|||
## What Works (Evidence) |
|||
- ✅ **Error context files** provide page snapshots showing exact DOM state at failure |
|||
- **Time**: 2025-08-21T14:22Z |
|||
- **Evidence**: `test-results/60-new-activity-New-offers-for-another-user-chromium/error-context.md` shows both alerts visible |
|||
- **Verify at**: Error context files in test results directory |
|||
|
|||
- ✅ **Trace files** capture detailed execution sequence for failed tests |
|||
- **Time**: 2025-08-21T14:22Z |
|||
- **Evidence**: `trace.zip` files available for all failed tests |
|||
- **Verify at**: Use `npx playwright show-trace <filename>` |
|||
|
|||
- ✅ **Page snapshots** reveal UI conflicts like multiple alerts with duplicate button text |
|||
- **Time**: 2025-08-21T14:22Z |
|||
- **Evidence**: YAML snapshots show registration + export alerts simultaneously |
|||
- **Verify at**: Error context markdown files |
|||
|
|||
## What Doesn't (Evidence & Hypotheses) |
|||
- ❌ **Generic selectors** fail with multiple similar elements at `test-playwright/testUtils.ts:161` |
|||
- **Time**: 2025-08-21T14:22Z |
|||
- **Evidence**: `button:has-text("Yes")` matches both "Yes" and "Yes, Export Data" |
|||
- **Hypothesis**: Selector ambiguity due to multiple alerts with conflicting button text |
|||
- **Next probe**: Use more specific selectors or dismiss alerts sequentially |
|||
|
|||
- ❌ **Timing-dependent tests** fail due to alert stacking at `src/views/ContactsView.vue:860,1283` |
|||
- **Time**: 2025-08-21T14:22Z |
|||
- **Evidence**: Both alerts use identical 1000ms delays, ensuring simultaneous display |
|||
- **Hypothesis**: Race condition between alert displays creates UI conflicts |
|||
- **Next probe**: Implement alert queuing or prevent overlapping alerts |
|||
|
|||
## Risks, Limits, Assumptions |
|||
- **Trace file size**: Large trace files may impact storage and analysis time |
|||
- **Browser compatibility**: Trace viewer requires specific browser support |
|||
- **Test isolation**: Shared state between tests may affect investigation results |
|||
- **Timing sensitivity**: Tests may pass/fail based on system performance |
|||
|
|||
## Next Steps |
|||
| Owner | Task | Exit Criteria | Target Date (UTC) | |
|||
|---|---|---|---| |
|||
| Development Team | Fix test selectors for multiple alerts | All tests pass consistently | 2025-08-22 | |
|||
| Development Team | Implement alert queuing system | No overlapping alerts with conflicting buttons | 2025-08-25 | |
|||
| Development Team | Add test IDs to alert buttons | Unique selectors for all UI elements | 2025-08-28 | |
|||
|
|||
## References |
|||
- [Playwright Trace Viewer Documentation](https://playwright.dev/docs/trace-viewer) |
|||
- [Playwright Test Results](https://playwright.dev/docs/test-reporters) |
|||
- [Test Investigation Workflow](./research_diagnostic.mdc) |
|||
|
|||
## Competence Hooks |
|||
- **Why this works**: Systematic investigation leverages Playwright's built-in debugging tools to identify root causes |
|||
- **Common pitfalls**: Generic selectors fail with multiple similar elements; timing issues create race conditions; alert stacking causes UI conflicts |
|||
- **Next skill unlock**: Implement unique test IDs and handle alert dismissal order in test flows |
|||
- **Teach-back**: "How would you investigate a Playwright test failure using error context, trace files, and page snapshots?" |
|||
|
|||
## Collaboration Hooks |
|||
- **Reviewers**: QA team, test automation engineers |
|||
- **Sign-off checklist**: Error context analyzed, trace files reviewed, root cause identified, fix implemented and tested |
|||
|
|||
## Assumptions & Limits |
|||
- Test results directory structure follows Playwright conventions |
|||
- Trace files are enabled in configuration (`trace: "retain-on-failure"`) |
|||
- Error context files contain valid YAML page snapshots |
|||
- Browser environment supports trace viewer functionality |
|||
|
|||
--- |
|||
|
|||
**Status**: Active investigation directive |
|||
**Priority**: High |
|||
**Maintainer**: Development team |
|||
**Next Review**: 2025-09-21 |
|||
# Playwright Test Investigation — Harbor Pilot Directive |
|||
|
|||
**Author**: Matthew Raymer |
|||
**Date**: 2025-08-21T14:22Z |
|||
**Status**: 🎯 **ACTIVE** - Playwright test debugging guidelines |
|||
|
|||
## Objective |
|||
Provide systematic approach for investigating Playwright test failures with focus on UI element conflicts, timing issues, and selector ambiguity. |
|||
|
|||
## Context & Scope |
|||
- **Audience**: Developers debugging Playwright test failures |
|||
- **In scope**: Test failure analysis, selector conflicts, UI state investigation, timing issues |
|||
- **Out of scope**: Test writing best practices, CI/CD configuration |
|||
|
|||
## Artifacts & Links |
|||
- Test results: `test-results/` directory |
|||
- Error context: `error-context.md` files with page snapshots |
|||
- Trace files: `trace.zip` files for failed tests |
|||
- HTML reports: Interactive test reports with screenshots |
|||
|
|||
## Environment & Preconditions |
|||
- OS/Runtime: Linux/Windows/macOS with Node.js |
|||
- Versions: Playwright test framework, browser drivers |
|||
- Services: Local test server (localhost:8080), test data setup |
|||
- Auth mode: None required for test investigation |
|||
|
|||
## Architecture / Process Overview |
|||
Playwright test investigation follows a systematic diagnostic workflow that leverages built-in debugging tools and error context analysis. |
|||
|
|||
```mermaid |
|||
flowchart TD |
|||
A[Test Failure] --> B[Check Error Context] |
|||
B --> C[Analyze Page Snapshot] |
|||
C --> D[Identify UI Conflicts] |
|||
D --> E[Check Trace Files] |
|||
E --> F[Verify Selector Uniqueness] |
|||
F --> G[Test Selector Fixes] |
|||
G --> H[Document Root Cause] |
|||
|
|||
B --> I[Check Test Results Directory] |
|||
I --> J[Locate Failed Test Results] |
|||
J --> K[Extract Error Details] |
|||
|
|||
D --> L[Multiple Alerts?] |
|||
L --> M[Button Text Conflicts?] |
|||
M --> N[Timing Issues?] |
|||
|
|||
E --> O[Use Trace Viewer] |
|||
O --> P[Analyze Action Sequence] |
|||
P --> Q[Identify Failure Point] |
|||
``` |
|||
|
|||
## Interfaces & Contracts |
|||
|
|||
### Test Results Structure |
|||
| Component | Format | Content | Validation | |
|||
|---|---|---|---| |
|||
| Error Context | Markdown | Page snapshot in YAML | Verify DOM state matches test expectations | |
|||
| Trace Files | ZIP archive | Detailed execution trace | Use `npx playwright show-trace` | |
|||
| HTML Reports | Interactive HTML | Screenshots, traces, logs | Check browser for full report | |
|||
| JSON Results | JSON | Machine-readable results | Parse for automated analysis | |
|||
|
|||
### Investigation Commands |
|||
| Step | Command | Expected Output | Notes | |
|||
|---|---|---|---| |
|||
| Locate failed tests | `find test-results -name "*test-name*"` | Test result directories | Use exact test name patterns | |
|||
| Check error context | `cat test-results/*/error-context.md` | Page snapshots | Look for UI state conflicts | |
|||
| View traces | `npx playwright show-trace trace.zip` | Interactive trace viewer | Analyze exact failure sequence | |
|||
|
|||
## Repro: End-to-End Investigation Procedure |
|||
|
|||
### 1. Locate Failed Test Results |
|||
```bash |
|||
# Find all results for a specific test |
|||
find test-results -name "*test-name*" -type d |
|||
|
|||
# Check for error context files |
|||
find test-results -name "error-context.md" | head -5 |
|||
``` |
|||
|
|||
### 2. Analyze Error Context |
|||
```bash |
|||
# Read error context for specific test |
|||
cat test-results/test-name-test-description-browser/error-context.md |
|||
|
|||
# Look for UI conflicts in page snapshot |
|||
grep -A 10 -B 5 "button.*Yes\|button.*No" test-results/*/error-context.md |
|||
``` |
|||
|
|||
### 3. Check Trace Files |
|||
```bash |
|||
# List available trace files |
|||
find test-results -name "*.zip" | grep trace |
|||
|
|||
# View trace in browser |
|||
npx playwright show-trace test-results/test-name/trace.zip |
|||
``` |
|||
|
|||
### 4. Investigate Selector Issues |
|||
```typescript |
|||
// Check for multiple elements with same text |
|||
await page.locator('button:has-text("Yes")').count(); // Should be 1 |
|||
|
|||
// Use more specific selectors |
|||
await page.locator('div[role="alert"]:has-text("Register") button:has-text("Yes")').click(); |
|||
``` |
|||
|
|||
## What Works (Evidence) |
|||
- ✅ **Error context files** provide page snapshots showing exact DOM state at failure |
|||
- **Time**: 2025-08-21T14:22Z |
|||
- **Evidence**: `test-results/60-new-activity-New-offers-for-another-user-chromium/error-context.md` shows both alerts visible |
|||
- **Verify at**: Error context files in test results directory |
|||
|
|||
- ✅ **Trace files** capture detailed execution sequence for failed tests |
|||
- **Time**: 2025-08-21T14:22Z |
|||
- **Evidence**: `trace.zip` files available for all failed tests |
|||
- **Verify at**: Use `npx playwright show-trace <filename>` |
|||
|
|||
- ✅ **Page snapshots** reveal UI conflicts like multiple alerts with duplicate button text |
|||
- **Time**: 2025-08-21T14:22Z |
|||
- **Evidence**: YAML snapshots show registration + export alerts simultaneously |
|||
- **Verify at**: Error context markdown files |
|||
|
|||
## What Doesn't (Evidence & Hypotheses) |
|||
- ❌ **Generic selectors** fail with multiple similar elements at `test-playwright/testUtils.ts:161` |
|||
- **Time**: 2025-08-21T14:22Z |
|||
- **Evidence**: `button:has-text("Yes")` matches both "Yes" and "Yes, Export Data" |
|||
- **Hypothesis**: Selector ambiguity due to multiple alerts with conflicting button text |
|||
- **Next probe**: Use more specific selectors or dismiss alerts sequentially |
|||
|
|||
- ❌ **Timing-dependent tests** fail due to alert stacking at `src/views/ContactsView.vue:860,1283` |
|||
- **Time**: 2025-08-21T14:22Z |
|||
- **Evidence**: Both alerts use identical 1000ms delays, ensuring simultaneous display |
|||
- **Hypothesis**: Race condition between alert displays creates UI conflicts |
|||
- **Next probe**: Implement alert queuing or prevent overlapping alerts |
|||
|
|||
## Risks, Limits, Assumptions |
|||
- **Trace file size**: Large trace files may impact storage and analysis time |
|||
- **Browser compatibility**: Trace viewer requires specific browser support |
|||
- **Test isolation**: Shared state between tests may affect investigation results |
|||
- **Timing sensitivity**: Tests may pass/fail based on system performance |
|||
|
|||
## Next Steps |
|||
| Owner | Task | Exit Criteria | Target Date (UTC) | |
|||
|---|---|---|---| |
|||
| Development Team | Fix test selectors for multiple alerts | All tests pass consistently | 2025-08-22 | |
|||
| Development Team | Implement alert queuing system | No overlapping alerts with conflicting buttons | 2025-08-25 | |
|||
| Development Team | Add test IDs to alert buttons | Unique selectors for all UI elements | 2025-08-28 | |
|||
|
|||
## References |
|||
- [Playwright Trace Viewer Documentation](https://playwright.dev/docs/trace-viewer) |
|||
- [Playwright Test Results](https://playwright.dev/docs/test-reporters) |
|||
- [Test Investigation Workflow](./research_diagnostic.mdc) |
|||
|
|||
## Competence Hooks |
|||
- **Why this works**: Systematic investigation leverages Playwright's built-in debugging tools to identify root causes |
|||
- **Common pitfalls**: Generic selectors fail with multiple similar elements; timing issues create race conditions; alert stacking causes UI conflicts |
|||
- **Next skill unlock**: Implement unique test IDs and handle alert dismissal order in test flows |
|||
- **Teach-back**: "How would you investigate a Playwright test failure using error context, trace files, and page snapshots?" |
|||
|
|||
## Collaboration Hooks |
|||
- **Reviewers**: QA team, test automation engineers |
|||
- **Sign-off checklist**: Error context analyzed, trace files reviewed, root cause identified, fix implemented and tested |
|||
|
|||
## Assumptions & Limits |
|||
- Test results directory structure follows Playwright conventions |
|||
- Trace files are enabled in configuration (`trace: "retain-on-failure"`) |
|||
- Error context files contain valid YAML page snapshots |
|||
- Browser environment supports trace viewer functionality |
|||
|
|||
--- |
|||
|
|||
**Status**: Active investigation directive |
|||
**Priority**: High |
|||
**Maintainer**: Development team |
|||
**Next Review**: 2025-09-21 |
Loading…
Reference in new issue