verify-tests-fail-without-fix

Verify Tests Fail Without Fix

Verifies UI tests actually catch the issue. Supports two workflow modes:

Mode 1: Verify Failure Only (Test Creation)

Use when creating tests before writing a fix:

• Runs tests to verify they FAIL (proving they catch the bug)
• No fix files required
• Perfect for test-first development

# Auto-detect test filter from changed test files
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android

# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356"

Mode 2: Full Verification (Fix Validation)

Use when validating both tests and fix:

1. Without fix - tests should FAIL (bug is present)
2. With fix - tests should PASS (bug is fixed)

# Auto-detect everything (recommended)
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -RequireFullVerification

# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356" -RequireFullVerification

Note: -RequireFullVerification ensures the script errors if no fix files are detected, preventing silent fallback to failure-only mode.

Requirements

Verify Failure Only Mode:

• Test files in the PR (or working directory)

Full Verification Mode:

• Test files in the PR
• Fix files in the PR (non-test code changes)

The script auto-detects which mode to use based on whether fix files are present.

Expected Output

Verify Failure Only Mode:

╔═══════════════════════════════════════════════════════════╗
║              VERIFICATION PASSED ✅                       ║
╠═══════════════════════════════════════════════════════════╣
║  Tests FAILED as expected!                                ║
║  This proves the tests correctly reproduce the bug.       ║
╚═══════════════════════════════════════════════════════════╝

Full Verification Mode:

╔═══════════════════════════════════════════════════════════╗
║              VERIFICATION PASSED ✅                       ║
╠═══════════════════════════════════════════════════════════╣
║  - FAIL without fix (as expected)                         ║
║  - PASS with fix (as expected)                            ║
╚═══════════════════════════════════════════════════════════╝

What It Does

Verify Failure Only Mode (no fix files):

1. Fetches base branch from origin (if available)
2. Auto-detects test classes from changed test files
3. Runs tests (should FAIL to prove they catch the bug)
4. Reports result

Full Verification Mode (fix files detected):

1. Fetches base branch from origin to ensure accurate diff
2. Auto-detects fix files (non-test code) from git diff
3. Auto-detects test classes from TestCases.Shared.Tests/*.cs
4. Reverts fix files to base branch
5. Runs tests (should FAIL without fix)
6. Restores fix files
7. Runs tests (should PASS with fix)
8. Generates markdown reports:
   • CustomAgentLogsTmp/TestValidation/verification-report.md - Full detailed report
   • CustomAgentLogsTmp/PRState/verification-report.md - Gate section for PR agent
9. Reports result

Output Files

The skill generates output files under CustomAgentLogsTmp/PRState/<PRNumber>/verify-tests-fail/:

Plus UI test logs in CustomAgentLogsTmp/UITests/:

• android-device.log or ios-device.log - Device logs
• test-output.log - NUnit test output

Example structure:

CustomAgentLogsTmp/
├── UITests/                           # Shared UI test logs
│   ├── android-device.log
│   └── test-output.log
└── PRState/
    └── 27847/
        └── verify-tests-fail/
            ├── verification-report.md  # Full detailed report
            ├── verification-log.txt
            ├── test-without-fix.log
            └── test-with-fix.log

PR Number Detection:

• Auto-detected from branch name (e.g., pr-27847)
• Falls back to gh pr view command
• Uses "unknown" if detection fails
• Can be manually specified with -PRNumber parameter

Troubleshooting

Optional Parameters

# Require full verification (fail if no fix files detected) - recommended
-RequireFullVerification

# Explicit test filter
-TestFilter "Issue32030|ButtonUITests"

# Explicit fix files  
-FixFiles @("src/Core/src/File.cs")

# Explicit base branch
-BaseBranch "main"