LogoLogo
HomeAbout usSign up
  • Introduction
  • AttentionSet
    • AttentionSet Best Practices
    • How to View AttentionSet for Others
    • How to Manually Change Attention
    • AttentionSet Chrome extension
    • Attention reasons
    • AttentionSet Slack Home Page
  • Releases
    • Getting Started with Releases
    • Concepts for Releases
      • Terminology for Releases
      • Two-step delivery
      • Rollbacks
      • Cherry-picks
      • Dogfood, Canary and Rollout
      • Release notes
    • How-to Guides for Releases
      • How to Create a Release Project
      • How to Configure Environments
      • How to Create a Scheduled Release
      • Create Custom Workflow Parameters
      • How to Manage Cherry-Picks
      • How to Resolve a Cherry-Pick Failure
      • Working with your CI / CD
        • GitHub Actions workflow
        • Buildkite workflow
    • API Reference for Releases
  • FlexReview
    • Getting Started with FlexReview
    • How to Onboard a Large Org
    • Concepts for FlexReview
      • Read-Only Mode in FlexReview
      • Recursive Ownership in FlexReview
      • Reviewer suggestion and assignment
      • FlexReview Teams and SLO Management
      • Validation in FlexReview
    • How-to Guides for FlexReview
      • How to Get a Reviewer Suggestion
      • How to Exclude Reviewers
      • How to Set Up Team Rules
      • Whitelist Teams for Review Assignment
      • Troubleshoot Reviewer Assignment
      • PagerDuty Integration for Reviewers
      • How to Set Up FlexReview Validation
      • Recommended Slack Notification Settings
      • How to Exclude OOO Reviewers
    • FlexReview Reference
      • Configuration
      • Slash commands
      • Expert scoring algorithms
      • Slack Notifications
      • Out of Office User Exclusion
    • FlexReview Roadmap
  • MergeQueue
    • Getting Started with MergeQueue
    • Merge Rules
    • How-to Guides for MergeQueue
      • How to Configure Base Branches
      • How to Customize Required Checks
      • How to Set Up Fast-Forwarding
      • How to Set Up Pre-Queue Conditions
      • How to Queue / Dequeue via API
      • Pause / Unpause Queues via API
      • Slash Commands Using GitHub Comments
      • How to Customize Sticky Comments
      • Require an Aviator Status Check
      • Backport a PR
      • How to Configure ChangeSets
      • Custom Integrations
        • GitHub Integration
      • How to Create Personal Access Tokens
      • How to Set Up SAML Configuration
        • Microsoft Active Directory
      • How to Merge Stacked PRs
      • How to Block Pull Request Mergeing with Slash Commands
    • Concepts for MergeQueue
      • Queue Modes
      • Pull Request Lifecycle
      • Analytics
      • Parallel Mode
      • CI Status Requirements
      • MQ Created Branches
      • Batching
      • Managing flaky tests
      • Fast-forwarding
      • Pre-Queue Conditions
      • Sticky Comments
      • Backporting
      • Paused Queues
      • Affected Targets
        • Directory-Based Affected Targets
        • nx based affected targets
        • GitHub Actions based Affected Targets
      • ChangeSets
        • Global CI Validation
        • ChangeSets FAQs
      • Priority Merges
        • Instant Merges
      • Merge Rules Audit Trail
      • Timeline
      • Ready Hook
      • Reduce Queue Failures From Staleness
    • MergeQueue References
      • Configuration Schema
      • Configuration Reference MergeQueue
      • GitHub Slash Commands
      • Status Codes
  • Stacked PRs CLI
    • Quickstart for Stacked PRs CLI
    • CLI Installation
    • How-to Guides for Stacked PRs CLI
      • How to Create Stacked PRs in CLI
      • How to Navigate & Modify Stacked PRs
      • Add Commits in the Stack
      • How Split a Commit in CLI
      • How to Split and Fold Pull Requests
      • How to Rename a Branch in CLI
      • How to Adopt a Branch in CLI
      • Orphan a Branch with Aviator CLI
      • How to Do Git Subcommand Aliasing
      • How to Create an Access Token
      • How to Set Up Auto Completion in CLI
      • How to Use Editor Plugins in CLI
    • Concepts for StackedPRs CLI
    • How to Rebase and Sync with GitHub
    • Configuration for StackedPRs CLI
    • Stacked PRs FAQs and Troubleshooting
      • Working with Aviator CLI
      • Default Branch Update Master to Main
    • Manpages for Stacked PRs CLI
      • av(1)
      • av-adopt Command Guide
      • av-auth-status(1) in CLI
      • av-stack-branch(1) in CLI
      • av-commit-create(1) in CL
      • av-stack-diff(1) in CLI
      • av-fetch(1) in CLI
      • av-git-interaction Command Guide
      • av-init(1) in CLI
      • av-stack-next(1) in CLI
      • av-orphan Command Guide
      • av-pr-status(1) in CLI
      • av-pr-create(1) in CLI
      • av-stack-prev(1) in CLI
      • av-stack-reorder(1) in CLI
      • av-reparent Command Guide
      • av-restack Command Guide
      • av-commit-split(1) in CLI
      • av-switch Command Guide
      • av-stack-sync(1) in CLI
      • av-stack-tidy(1) in CLI
      • av-stack-tree(1) in CLI
    • Aviator CLI Major Releases
      • Aviator CLI v0.1.0 Release Notes
  • Aviator's Chrome Extension
  • Pilot Automated Actions
    • Scheduled Events
    • JavaScript Execution
    • Pilot Automated Actions Reference
      • GitHub Reference
      • MergeQueue Reference
      • Slack Reference
  • API and Integrations
    • Slack Integration Guide
    • GraphQL API Quickstart
    • Prometheus Metrics Setup for GCP
    • Reference
      • JSON API
      • GraphQL
      • Webhooks
      • Monitoring Metrics
  • Manage
    • Access Management
    • GitHub App Permissions
    • Security
      • Aviator Agents Data Usage & Retention Policy
    • On-Premise Installation
      • GitHub App for On-Prem
      • GitHub OAuth for On-Prem
      • Use Helm Instructions
      • Use Docker Compose Instructions
      • Prometheus endpoint
      • Slack Integration for On-Premise
      • Google SSO Login for On-Prem
    • FAQs
      • Troubleshooting GitHub app connection
      • MergeQueue FAQs
      • Billing FAQs
Powered by GitBook
On this page
  • Flaky test problem in parallel mode
  • Optimistic validation
  • Configuring optimistic validation
  • Side-effect of optimistic validation

Was this helpful?

  1. MergeQueue
  2. Concepts for MergeQueue

Managing flaky tests

Learn about the flaky test problem in parallel mode and how to manage flaky tests with optimistic validation. Optimistic validation configuration instructions.

PreviousBatchingNextFast-forwarding

Last updated 8 months ago

Was this helpful?

Optimistic validation is a feature to mitigate the queue reset caused by flaky tests in . When a test fails in a batch, it waits for other batches to see if a test passes.

Flaky test problem in parallel mode

In parallel mode, queued PRs are tested in a batch.

For example, when there are two PRs (PR #1 and PR #2) enqueued, we create two draft PRs (Draft PR #3 and Draft PR #4) for testing. These two draft PRs are based on the latest main branch and run tests to check tests won't fail after merge. Draft PR #3 includes PR #1 and Draft PR #4 includes PR #1 and PR #2. This way, the tests of these PRs can run in parallel, making sure that the post-merge state still pass the tests.

When a CI run fails in one of those parallel build draft PRs, that failing a PR gets dequeued. This causes a queue reset. Imagine that Draft PR #3 fails the CI run. This includes PR #1 and this PR gets dequeued. Draft PR #4 also includes PR #1 and the CI run for this PR is cancelled. Because all the PRs created after PR #1 includes it, this makes a cascading effect and effectively the entire queue is reset.

If your test is flaky, this queue reset happens frequently. This slows down the entire queue and affects the efficacy of the team.

Optimistic validation

Optimistic validation is a feature to address this situation. When a CI run fails for a draft PR, instead of immediately dequeue the PR and causing a queue reset, it waits for other PRs to finish.

Continue from the example above. When a CI run for Draft PR #3 fails, without optimistic validation, it dequeues the PR #1 and resets the entire queue. With optimistic validation, it waits for other draft PRs (in this case Draft PR #4) to finish the CI runs. When they pass the CI, it assumes that the CI run would have been passed and merges the PRs.

Configuring optimistic validation

There are two settings to configure optimistic validation. Both are specified within the parallel mode section of the configuration.

use_optimistic_validation - boolean value that configures only the look-ahead of CI. If the CI of the top of the PR fails, it will still immediately dequeue the PR. Defaults to true.

optimistic_validation_failure_depth - integer value that how many PRs, including the failing PR itself, it will check for the optimistic validation. Setting 1 effectively means that it won't use optimistic validation. This is capped to 3 in order to cap the GitHub API calls. Requires use_optimistic_validation to be enabled.

merge_rules:
  labels:
    trigger: "mergequeue"
  merge_mode:
    type: "parallel"
  parallel_mode:
    max_parallel_builds: 10
    use_optimistic_validation: true
    optimistic_validation_failure_depth: 2

Side-effect of optimistic validation

By introducing optimistic validation, real errors will take longer to be kicked out of the queue. This is because when there is a CI run failure, it keeps waiting for other batches to finish. Therefore, the failure depth should generally be kept small.

Most of the Aviator users find this side-effect is acceptable as the number of real failures in the queue are relatively small. However, this depends on how flaky your tests are.

parallel mode
Parallel mode with two PRs