Flaky tests: six causes and how to tell them apart

A flaky test is one whose verdict flips although the code stayed put. Nothing changed between the green run and the red one, so the difference must come from something the test depends on without saying so: the speed of the machine, the calendar, the sequence of other tests, a random number, a server somewhere else. That is good news, because the list of hidden inputs is short. This guide goes through six causes that cover most cases, shows what each looks like when it fails, and gives a way to confirm it before you change anything. The aim is to replace "re-run it until it is green" with a named cause and a fix.

Start by naming the hidden input

Re-running a failing test and hoping tells you nothing about the cause. Ask instead what the test reads that the code under test does not control. The pytest documentation puts the general idea plainly: a flaky test relies on some system state, which means the test environment is not isolated enough, and tests at a higher level are more likely to be flaky because they depend on more of that state (pytest, flaky tests). Each cause below is one kind of hidden state. Collect the failure message, the order the tests ran in, and whether the failure happened locally or only on a shared runner. Those three facts separate most of the causes.

Cause one: a fixed sleep that waits for background work

The test starts something asynchronous, sleeps for a chosen time, then checks the result. On a quick machine the work finishes and the test passes; on a busy runner it does not finish and the test fails, with no change in the code. Fowler calls the bare sleep a common mistake and recommends a callback or polling instead (Fowler, non-determinism).

How to recognise it: the failure is an assertion that the work is not done yet, it appears more often under load or in parallel runs, and a longer sleep makes it rarer but never removes it. The fix is to wait for a condition. Poll the status in a loop with a timeout, or move a fake timer forward by hand so no real time passes.

One test writes something and another reads it. A module-level list, a file left on disk, a row left in a table, a cached object: any of them can carry data from one test into the next. The suite passes when the tests run in the written order and fails when a runner shuffles them or splits them across workers. Fowler's advice is isolation, so that one test cannot affect another, and properly isolated tests can run in any sequence (Fowler, non-determinism). The pytest page adds that flaky results under parallel runs can point to a reliance on test order, and that a different test may be leaving data behind (pytest, flaky tests).

How to recognise it: the test passes alone and fails in the full suite, or the reverse. Run it alone, then run the suite in a shuffled order; the pytest page names plugins that randomise order for exactly this purpose. The fix is to rebuild the fixtures for every single test, so each one creates the state it needs and removes it afterwards.

Cause three: the real clock

Anything that reads the current time or date into an expectation is true only for a while. A test that compares a formatted date to a hard-coded string passes today and fails after midnight, at the close of a month, or when the machine is in another time zone. A test that asserts a duration with a tight margin fails when the runner stalls for a moment; the pytest page warns that overly strict assertions bring timing issues (pytest, flaky tests). Fowler singles out the system clock as a prime source of non-determinism, and suggests wrapping the clock so a test can substitute one frozen at a chosen moment (Fowler, non-determinism).

How to recognise it: failures cluster around particular dates or hours, or the expected value contains something that looks like today. The fix is a fake clock, set ahead of the call under test, with the chosen date written into the test itself.

Cause four: random values with no fixed seed

If a random number feeds an expectation, the test is a lottery. Most of the draws pass, and now and then one lands on an edge the code mishandles. Python's documentation says that a generator started without a seed takes its start from the system time or from the operating system's randomness, and that reusing a seed should reproduce the same sequence from run to run (Python, random).

How to recognise it: the failure message contains a value that you cannot find in the test file, and a second run does not show it. Log the seed on every run. The fix is to build the generator from a fixed seed inside the test, and to write the awkward values you found as ordinary cases, so the edge is tested every time rather than by chance.

Cause five: a request to a live server

A test that reaches a real server inherits everything about that server: its uptime, its speed, its rate limits, its data. It fails when the network is slow, when the service is down, or when the answer changes. Fowler's advice is that, to keep a test deterministic, you reach for a test double in place of the live system (Fowler, non-determinism).

How to recognise it: timeouts, connection errors and refused requests in the failure output, often only on the shared runner. The fix is to replace the service with a fake, or with a tiny server of your own that the test starts and stops. Keep a separate, clearly named check for the real service, run on its own schedule, so an outage there does not redden the whole suite.

Cause six: leaked resources and parallel overlap

The last cause is the least visible. A test opens a connection, a port, a temporary directory or a thread and does not release it. Later tests then run short of that resource, or two tests that run at once fight over the same fixed port or file name. Fowler points to resource leaks as the next thing to investigate when no single test is at fault, and suggests handling such resources through a pool (Fowler, non-determinism). The pytest page notes that tests which change global state typically cannot run in parallel (pytest, flaky tests).

How to recognise it: errors about addresses already in use, too many open files, or a missing directory, and failures that rise with how many workers. The fix is to let the operating system choose a free port, to give every test its own temporary directory, and to close what it opens in a teardown step that runs even when the test fails.

A procedure that can fail

You can check a suspected cause instead of arguing about it. Pick one flaky test and run it alone many times in a loop. If it still fails, the cause is inside the test: time, randomness, a sleep or the network. If it passes alone and fails in the suite, suspect shared state or leaked resources, and confirm by shuffling the order. Then change only the one thing you suspect, for example freeze the clock, and repeat the loop. A fix that does not change the failure rate was not the cause. Keep a note of each flaky test and its cause; Fowler warns that tests thrown into quarantine and forgotten mean the bug detection system is eroding (Fowler, non-determinism). Our own habit against that is to give every quarantined test an owner and a date.

When this does not apply

Only causes visible inside the test file itself are covered. A genuine bug in the code is a different matter: one that only appears under rare timing, such as a race condition in production code, makes the failure real, and removing the flakiness would hide it. This guide also says little about problems in the environment: a full disk, a faulty runner, a dependency that changed between builds, or a fixture defined in another file that this review cannot see. Browser and end-to-end tests have extra causes of their own, such as animations and page load order. No fresh experiment of ours sits behind this text. It is built on public documentation and familiar patterns, so read the six causes as a starting list, not a complete one.

Where a review can help

These causes repeat so often that a reader can scan a test file for them. Trusting a green run because it looked fine once is the habit behind many flaky suites, and our write-up on a small model praising a lossy sync job describes the same habit during code review.

For a second reader on a single test file before merging, our flaky-test-review skill takes one test file and reports five of the causes above: a fixed sleep, a dependence on the real date or time, state shared between tests, random values without a fixed seed, and an outgoing request to a live service. Every finding is a single line: a short label, the test, the reason for the flip and the change to make. Safe forms such as a frozen clock or a seeded generator are left alone, and a clean file is simply reported as clean. Code under test, other files and CI settings are outside what it reads.