Design philosophy to avoid crying late at night during E2E test: Escape from Playwright hell
The next morning, after I closed my PC and pumped my fists in joy, saying, "CI passed!", I received a merciless pile of alerts on Slack. I thought my heart would stop. I want to punch myself back in the day when I thought ``Playwright is super fast and all you have to do is write code'' when it came to automating E2E testing.
After all, automated tests are 100 times harder to maintain than to write. This is a survival strategy ``to avoid being woken up in the middle of the night'' that I finally arrived at after sipping muddy water and struggling with E2E tests for five years.
1. Free yourself from certification hell: Hack the “premises” of your tests
Running login tests in every case is insane. In the early days, I used `page.fill()` to type in my email address and password every time I tested. As a result, when the number of tests exceeded 100, the execution time exceeded 20 minutes, and in the end, the development environment DB was overloaded due to overload on the login API.
The solution is "reusing authentication states". Use Playwright's `storageState` to save cookies after logging in and start the test in the "logged in" state.
# auth.setup.py
from playwright.sync_api import sync_playwright
def run():
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://myapp.com/login")
page.fill('input[name="email"]', "[email protected]")
page.fill('input[name="password"]', "password")
page.click('button[type="submit"]')
# Save the state after login to a file
page.context.storage_state(path="auth.json")
browser.close()
if __name__ == "__main__":
run()
Run this only once before running the test and load `storage_state="auth.json"` in the `pytest` settings. This alone reduces the waiting time for login processing to zero. "The login test only needs to test the login function". This division dramatically reduces the weight of the test suite.
2. Conquer asynchronous "waiting": `waitForTimeout` is the devil's whisper
When I get stuck loading dynamic content, I want to write `page.wait_for_timeout(5000)`. However, this is the same as accumulating debt. On days when the network is slow, it crashes, and on days when it's fast, you wait in vain.
Trust the Web-First Assertion provided by Playwright. It monitors the state of the element and advances the test the moment it appears.
# Bad example: hard code (never do this)
page.wait_for_timeout(3000)
assert page.locator('.result-item').is_visible()
# Good example: Web-First assertion
# Automatically retry internally until elements are aligned (default 5 seconds)
expect(page.locator('.result-item')).to_have_count(5, timeout=10000)
If you ask me, "Why is this assertion the best?" the answer is simple. "Because the tests don't depend on how fast the code runs, they depend on changes in the DOM". This is the only way to eliminate flaky tests.
3. The iframe abyss: why elements are not found
The most confusing thing about Playwright is `iframe`. Even if I search with normal `page.locator()`, no matter how much I examine the selector, I get a `TimeoutError`. This is because the contents of an iframe are a world apart from the main DOM tree. **It is necessary to use `frame_locator` to clearly declare ``which frame to search for.''
# Correctly identify elements within an iframe
frame = page.frame_locator('iframe[name="payment-form"]')
frame.locator('input[name="card-number"]').fill('1234-5678-9012-3456')
If you have a crappy specification where the iframe is dynamically generated, make sure to have `page.frame_locator` as a variable and design it to be reused. In addition to increasing the resolution of the selector, you can dramatically improve the stability of your tests by simply defining ``what is the parent'' structurally.
4. Problem that only occurs in CI environment: Debugging guidelines
It works perfectly locally, but only crashes on GitHub Actions. 80% of this is due to "environmental differences."
1. Fixed viewport: If the local and CI screen sizes are different, the mobile menu may be hidden or elements may overlap. Specify `viewport` strictly in `playwright.config.py`.
2. Switching API endpoint: Make sure to use the environment variable `BASE_URL` to hit the backend of the test environment.
3. Timeout adjustment: CI often has lower specs than local. In addition to setting the default timeout for `expect` a bit longer, make sure to save screenshots and trace files of failures as artifacts to determine why it's taking so long. **
Finally: Don't aim for perfection, aim for the place you need to protect
The conclusion I've learned over the past five years is that if you try to test everything, you'll kill yourself by maintaining the tests.
- Login, payment and data storage.
- Protect only these critical flows related to "money and trust" with E2E.
- Leave the rest to unit tests and integration tests.