4.0 KiB

Raw Blame History

Bisecting a Regression Across Published Playwright Versions

When a user reports a regression between two published Playwright versions (e.g. "works in 1.58, broken in 1.59.1"), reproduce both side by side from npm — do not try to bisect against the monorepo source. Reading the compiled JS in node_modules/playwright/lib/** is faster and avoids build/branch confusion.

Setup (two side-by-side installs)

Use ~/tmp/<version-tag>/ (NOT /tmp/) — the user's shell sessions live in ~/tmp.

mkdir -p ~/tmp/<good>/tests ~/tmp/<bad>/tests

# Skip `npm init playwright@latest` — it's interactive and the scaffold
# pulls in 3 projects (chromium/firefox/webkit) which produces 6 test runs
# from a single spec and is confusing. Do this instead:
( cd ~/tmp/<good> && npm init -y && npm install @playwright/test@<good-ver> && npx playwright install chromium)
( cd ~/tmp/<bad>  && npm init -y && npm install @playwright/test@<bad-ver> && npx playwright install chromium )

Write a minimal playwright.config.ts with a single chromium project — the default scaffold's 3-project config will run the same spec 6 times and obscure output:

import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
  testDir: './tests',
  projects: [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }],
});

Drop the repro spec (and any helper files) into both folders identically. Run:

( cd ~/tmp/<good> && npx playwright test )
( cd ~/tmp/<bad>  && npx playwright test )

Confirm the difference is real before investigating.

Investigating the diff in node_modules

The compiled JS in node_modules/playwright-core/lib/ and node_modules/playwright/lib/ is the source of truth for what shipped.

In recent versions of Playwright these are bundled, so you can't compare on per-file basis. You can extract files from bundles via grep though and compare.

Once you've found a candidate function, diff it across the two versions. Patches are usually 1–3 lines.

Verifying the hypothesis

Edit the compiled JS in ~/tmp/<bad>/node_modules/playwright/lib/... directly and re-run the test. No build step is needed — Node loads the JS as-is. Revert when done (or just delete the folder).

For stack-trace bugs in particular, a console.log(new Error().stack) inserted at the capture site (e.g. inside expect.js's captureRawStack) instantly shows whether the issue is microtask-boundary related vs. a stack-filter regression vs. something else.

Reporting

When the root cause is confirmed:

Quote the offending lines from node_modules/.../lib/... of the bad version, with file path.
Show the equivalent code from the good version for contrast.
Explain why the change breaks the user's case (don't just point at the diff).
Propose and verify a minimal fix by patching the bad install in place.

Post the writeup as a comment on the original issue with gh issue comment <number> --repo microsoft/playwright --body "$(cat <<'EOF' ... EOF)".

Pitfalls

Don't run npm init playwright@latest — it's interactive and --quiet does not skip the prompts. npm init -y + npm install @playwright/test@<ver> is faster and deterministic.
Don't use the scaffold's default config — the 3 browser projects multiply test runs by 3 and confuse the output. One chromium project is enough for 99% of repros.
Don't cd between commands in a single Bash call without && — the shell cwd resets between tool invocations.
/tmp/ is not ~/tmp/ — pick one and stay consistent. The user's interactive shells default to ~/tmp/, so prefer that.
Don't rm -rf an existing ~/tmp/<ver>/ without checking — it may be the user's prior work. Edit in place instead.
Don't try to map the bug to monorepo source first. The shipped JS is what the user is running; source may have already been refactored or fixed on main. Investigate node_modules/ first, then map the fix back to source only when proposing the upstream patch.

4.0 KiB Raw Blame History Unescape Escape