"It Passed in Testing, Yet..." — My Journey Battling Flaky E2E Tests with Playwright
"The unit tests are all green, and the logic has been verified. Yet, immediately after deployment, users start reporting that they can't log in."
A few months ago, I faced this exact nightmare. I thought I was safe—using Vitest to lock down function behavior, TypeScript to ensure type safety, and a robust suite to prevent regressions. However, unit tests couldn't catch the "mismatches" caused by timing issues in UI rendering or gaps during API response waits.
Ultimately, the root cause lay in complex frontend state transitions and slight desynchronizations in asynchronous processes. Realizing this, I knew that unit tests alone weren't enough; I needed E2E testing. I decided to implement Playwright, but that led me into a different battle: not "tests failing," but "flaky" tests—the frustrating inconsistency of tests that pass sometimes and fail others.
Initially, my only goal was to get the tests to pass. I resorted to "quick fixes" like hardcoded sleeps—using `page.waitForTimeout(3000)` to wait for elements to appear. This worked perfectly on my high-performance local machine. However, as soon as I integrated them into GitHub Actions for CI, everything fell apart.
CI environments are much more resource-constrained than local machines. Due to network latency or CPU load, elements wouldn't always appear within three seconds. This resulted in "flaky tests"—the kind that pass one moment and fail the next. This undermined the entire purpose of testing; every time a build failed, I wasted precious time investigating whether it was an actual bug or just a timing issue.
The solution was learning to properly utilize Playwright's "Web-first Assertions." Playwright has a built-in mechanism that automatically retries until an element reaches a specific state (such as `Visible` or `Enabled`). By adopting this approach, I could eliminate those problematic `sleep` calls.
// Bad practice: Relying on forced waits
test('bad-example: unstable test using sleep', async ({ page }) => {
await page.goto('/dashboard');
await page.click('#submit-button');
// Just waiting for a set amount of time. This will fail in CI environments.
await page.waitForTimeout(3000);
const successMessage = await page.textContent('.success-msg');
expect(successMessage).toBe('保存完了');
});
// Recommended practice: Using Web-first Assertions
test('good-example: auto-retrying assertions', async ({ page }) => {
await page.goto('/dashboard');
await page.click('#submit-button');
// expect(locator).toBeVisible() automatically retries until the condition is met.
// Even with network delays or rendering lags, it will succeed if it happens within the timeout.
const successMessage = page.locator('.success-msg');
await expect(successMessage).toBeVisible();
await expect(successMessage).toHaveText('保存完了');
});
Simply focusing on these "auto-retrying" assertions dramatically improved test stability. Additionally, while I initially struggled with browser dependencies in GitHub Actions, I resolved the issue by updating the workflow to run `npx playwright install --with-deps`, which cleared the hurdles of environment setup.
E2E tests are not meant to verify "logic correctness" in the same way unit tests do; they are designed to ensure that the "User Flow" remains intact. To achieve this, you must write assertions focused on "state transitions"—verifying how elements appear and whether they are interactable.
At first, writing test code felt like a "cost" to the project. Now, however, every time I see that green checkmark from Playwright in my CI pipeline, I feel the mental burden of deployment lift. It was a long road of trial and error, but it ultimately led to stable automation.