Killing flaky tests with hermetic Playwright
Flaky end-to-end tests almost always come from depending on something you do not control. The fix is to make each test own its entire world.
A flaky test is worse than no test. It trains the team to ignore red, and once people start re-running until green, the suite has stopped meaning anything. Almost every flaky end-to-end test I have chased came down to the same root cause: the test depended on something it did not control. The cure is to make tests hermetic, so each one owns its entire world.
Mock the network, do not hit the real backend
The biggest source of flakiness is a live backend: latency, shared data that other tests mutate, an environment that is occasionally down. So the end-to-end tests never call the real API. I intercept requests at the network layer and serve deterministic responses (for us, canned GraphQL payloads). The test controls exactly what the app receives, so the same run produces the same result every time.
Seed deterministic data
When a test needs data, it sets that data up itself and assumes nothing left over from a previous run. No “the first user in the list,” no reliance on whatever state a shared database happens to be in. Given the same inputs, the app renders the same output. Tests that share mutable state are tests that fail in random order.
Isolate every test
Each test runs in its own fresh browser context: clean storage, clean cookies, no session leaking in from the test before it. Playwright makes this the default and I lean into it. No test depends on another having run first, which also means the tests can run in parallel and in any order without interfering with each other.
Wait for state, never for time
A hard-coded sleep(2000) is the most common flaky-test smell there is. It is too
short on a slow CI machine and too long everywhere else. Instead I use
Playwright’s web-first assertions, which wait for a condition (an element visible,
text present, a request settled) and only fail after a real timeout. The test
proceeds the moment the app is actually ready, not on a guess.
Remove nondeterminism at the source
Animations, transitions, and real timestamps all introduce timing that varies run to run. I disable animations in the test environment and control anything time-dependent so “now” is fixed. The less a test leaves to chance, the less chance has to break it.
The payoff
A hermetic suite fails for exactly one reason: something is actually broken. That is the whole point. When red always means a real problem, people trust it, they fix it immediately, and the tests keep doing their job instead of getting muted. Speed and parallelism come for free, because tests that depend on nothing external can run anywhere, all at once.