Comment on page
Managing flaky tests
In parallel mode, queued PRs are tested in a batch.
Parallel mode with two PRs
For example, when there are two PRs (
PR #2) enqueued, we create two draft PRs (
Draft PR #3and
Draft PR #4) for testing. These two draft PRs are based on the latest
mainbranch and run tests to check tests won't fail after merge.
Draft PR #3includes
Draft PR #4includes
PR #2. This way, the tests of these PRs can run in parallel, making sure that the post-merge state still pass the tests.
When a CI run fails in one of those parallel build draft PRs, that failing a PR gets dequeued. This causes a queue reset. Imagine that
Draft PR #3fails the CI run. This includes
PR #1and this PR gets dequeued.
Draft PR #4also includes
PR #1and the CI run for this PR is cancelled. Because all the PRs created after
PR #1includes it, this makes a cascading effect and effectively the entire queue is reset.
If your test is flaky, this queue reset happens frequently. This slows down the entire queue and affects the efficacy of the team.
Optimistic validation is a feature to address this situation. When a CI run fails for a draft PR, instead of immediately dequeue the PR and causing a queue reset, it waits for other PRs to finish.
Continue from the example above. When a CI run for
Draft PR #3fails, without optimistic validation, it dequeues the
PR #1and resets the entire queue. With optimistic validation, it waits for other draft PRs (in this case
Draft PR #4) to finish the CI runs. When they pass the CI, it assumes that the CI run would have been passed and merges the PRs.
There are two settings to configure optimistic validation. Both are specified within the parallel mode section of the configuration.
optimistic_validation- boolean value that configures only the look-ahead of CI. If the CI of the top of the PR fails, it will still immediately dequeue the PR. Defaults to
optimistic_validation_failure_depth- integer value that how many PRs, including the failing PR itself, it will check for the optimistic validation. Setting
1effectively means that it won't use optimistic validation. This is capped to 3 in order to cap the GitHub API calls. Requires
optimistic_validationto be enabled.
By introducing optimistic validation, real errors will take longer to be kicked out of the queue. This is because when there is a CI run failure, it keeps waiting for other batches to finish. Therefore, the failure depth should generally be kept small.
Most of the Aviator users find this side-effect is acceptable as the number of real failures in the queue are relatively small. However, this depends on how flaky your tests are.