LogoLogo
HomeAbout usSign up
  • Introduction
  • AttentionSet
    • AttentionSet Best Practices
    • How to View AttentionSet for Others
    • How to Manually Change Attention
    • AttentionSet Chrome extension
    • Attention reasons
    • AttentionSet Slack Home Page
  • Releases
    • Getting Started with Releases
    • Concepts for Releases
      • Terminology for Releases
      • Two-step delivery
      • Rollbacks
      • Cherry-picks
      • Dogfood, Canary and Rollout
      • Release notes
    • How-to Guides for Releases
      • How to Create a Release Project
      • How to Configure Environments
      • How to Create a Scheduled Release
      • Create Custom Workflow Parameters
      • How to Manage Cherry-Picks
      • How to Resolve a Cherry-Pick Failure
      • Working with your CI / CD
        • GitHub Actions workflow
        • Buildkite workflow
    • API Reference for Releases
  • FlexReview
    • Getting Started with FlexReview
    • How to Onboard a Large Org
    • Concepts for FlexReview
      • Read-Only Mode in FlexReview
      • Recursive Ownership in FlexReview
      • Reviewer suggestion and assignment
      • FlexReview Teams and SLO Management
      • Validation in FlexReview
    • How-to Guides for FlexReview
      • How to Get a Reviewer Suggestion
      • How to Exclude Reviewers
      • How to Set Up Team Rules
      • Whitelist Teams for Review Assignment
      • Troubleshoot Reviewer Assignment
      • PagerDuty Integration for Reviewers
      • How to Set Up FlexReview Validation
      • Recommended Slack Notification Settings
      • How to Exclude OOO Reviewers
    • FlexReview Reference
      • Configuration
      • Slash commands
      • Expert scoring algorithms
      • Slack Notifications
      • Out of Office User Exclusion
    • FlexReview Roadmap
  • MergeQueue
    • Getting Started with MergeQueue
    • Merge Rules
    • How-to Guides for MergeQueue
      • How to Configure Base Branches
      • How to Customize Required Checks
      • How to Set Up Fast-Forwarding
      • How to Set Up Pre-Queue Conditions
      • How to Queue / Dequeue via API
      • Pause / Unpause Queues via API
      • Slash Commands Using GitHub Comments
      • How to Customize Sticky Comments
      • Require an Aviator Status Check
      • Backport a PR
      • How to Configure ChangeSets
      • Custom Integrations
        • GitHub Integration
      • How to Create Personal Access Tokens
      • How to Set Up SAML Configuration
        • Microsoft Active Directory
      • How to Merge Stacked PRs
      • How to Block Pull Request Mergeing with Slash Commands
    • Concepts for MergeQueue
      • Queue Modes
      • Pull Request Lifecycle
      • Analytics
      • Parallel Mode
      • CI Status Requirements
      • MQ Created Branches
      • Batching
      • Managing flaky tests
      • Fast-forwarding
      • Pre-Queue Conditions
      • Sticky Comments
      • Backporting
      • Paused Queues
      • Affected Targets
        • Directory-Based Affected Targets
        • nx based affected targets
        • GitHub Actions based Affected Targets
      • ChangeSets
        • Global CI Validation
        • ChangeSets FAQs
      • Priority Merges
        • Instant Merges
      • Merge Rules Audit Trail
      • Timeline
      • Ready Hook
      • Reduce Queue Failures From Staleness
    • MergeQueue References
      • Configuration Schema
      • Configuration Reference MergeQueue
      • GitHub Slash Commands
      • Status Codes
  • Stacked PRs CLI
    • Quickstart for Stacked PRs CLI
    • CLI Installation
    • How-to Guides for Stacked PRs CLI
      • How to Create Stacked PRs in CLI
      • How to Navigate & Modify Stacked PRs
      • Add Commits in the Stack
      • How Split a Commit in CLI
      • How to Split and Fold Pull Requests
      • How to Rename a Branch in CLI
      • How to Adopt a Branch in CLI
      • Orphan a Branch with Aviator CLI
      • How to Do Git Subcommand Aliasing
      • How to Create an Access Token
      • How to Set Up Auto Completion in CLI
      • How to Use Editor Plugins in CLI
    • Concepts for StackedPRs CLI
    • How to Rebase and Sync with GitHub
    • Configuration for StackedPRs CLI
    • Stacked PRs FAQs and Troubleshooting
      • Working with Aviator CLI
      • Default Branch Update Master to Main
    • Manpages for Stacked PRs CLI
      • av(1)
      • av-adopt Command Guide
      • av-auth-status(1) in CLI
      • av-stack-branch(1) in CLI
      • av-commit-create(1) in CL
      • av-stack-diff(1) in CLI
      • av-fetch(1) in CLI
      • av-git-interaction Command Guide
      • av-init(1) in CLI
      • av-stack-next(1) in CLI
      • av-orphan Command Guide
      • av-pr-status(1) in CLI
      • av-pr-create(1) in CLI
      • av-stack-prev(1) in CLI
      • av-stack-reorder(1) in CLI
      • av-reparent Command Guide
      • av-restack Command Guide
      • av-commit-split(1) in CLI
      • av-switch Command Guide
      • av-stack-sync(1) in CLI
      • av-stack-tidy(1) in CLI
      • av-stack-tree(1) in CLI
    • Aviator CLI Major Releases
      • Aviator CLI v0.1.0 Release Notes
  • Aviator's Chrome Extension
  • Pilot Automated Actions
    • Scheduled Events
    • JavaScript Execution
    • Pilot Automated Actions Reference
      • GitHub Reference
      • MergeQueue Reference
      • Slack Reference
  • API and Integrations
    • Slack Integration Guide
    • GraphQL API Quickstart
    • Prometheus Metrics Setup for GCP
    • Reference
      • JSON API
      • GraphQL
      • Webhooks
      • Monitoring Metrics
  • Manage
    • Access Management
    • GitHub App Permissions
    • Security
      • Aviator Agents Data Usage & Retention Policy
    • On-Premise Installation
      • GitHub App for On-Prem
      • GitHub OAuth for On-Prem
      • Use Helm Instructions
      • Use Docker Compose Instructions
      • Prometheus endpoint
      • Slack Integration for On-Premise
      • Google SSO Login for On-Prem
    • FAQs
      • Troubleshooting GitHub app connection
      • MergeQueue FAQs
      • Billing FAQs
Powered by GitBook
On this page
  • Goals
  • Analyzing past data
  • Calculating the suggestions
  • Reviewer Assignment
  • Unowned file paths
  • Fill the gap in CODEOWNERS
  • Multiple reviewer assignment

Was this helpful?

  1. FlexReview
  2. Concepts for FlexReview

Reviewer suggestion and assignment

Learn about reviewer suggestions and assignments in FlexReview. Reviewer suggestions reduce code review response time and help improve the developer workflow.

PreviousRecursive Ownership in FlexReviewNextFlexReview Teams and SLO Management

Last updated 4 months ago

Was this helpful?

Goals

The ultimate goal with the reviewer suggestions is to reduce code review response time. FlexReview also has additional goals to help improve the developer workflow:

  • Reduce the total number of reviewers needed for each code review, and therefore, reduce iteration time

  • Knowledge sharing - expand the suggested pool of reviewers when applicable

  • Ensure that the code owners requirement is satisfied, if a code owners file exist

Analyzing past data

When you enable FlexReview on a repository, it starts in and indexes the past pull requests data to calculate the domain expertise score of each author and reviewer for various code paths. It also captures the team memberships of the developers, and reads the existing CODEOWNERS file, or in a custom .aviator/OWNERSfile.

Calculating the suggestions

For detailed understanding of the calculations, please refer to the . FlexReview takes into account a few factors to identify the right reviewers:

  • Complexity of the code change - the complexity is calculated separately for each path in the given PR.

  • Domain expertise score - this is calculated based on the past data of the code reviews. A user can gain domain expertise both by authoring the code as well as reviewing it. Read the algorithm for a detailed explanation of this calculation. Domain expertise score is calculated both for the author and the reviewer.

  • Current review load of all valid reviewer candidates.

  • Availability (timezone, OOO) - Not implemented yet.

  • Ownership defined in the CODEOWNERS file - the suggester service will choose the least amount of reviewers possible for each code change in order to satisfy the CODEOWNERS requirement if the file exists.

Based on these heuristics, reviewers are suggested for a pull request.

  • When the code complexity is high and the domain expertise of the author is low, a reviewer with high domain expertise will be chosen.

  • When the code complexity is low and the domain expertise of the author is high, the pool of reviewers is increased so that non-experts may also be chosen based on the current review load, ownership, and availability.

Reviewer Assignment

As part of reviewer assigned using Expert mode, Aviator also posts a comment on the GitHub pull request explaining the breakdown and reasoning for why that particular reviewer was assigned.

The suggested reviewers are automatically assigned to the PR at the time of suggestion. If GitHub assigns reviewers based on CODEOWNERS, FlexReview will replace the assigned reviewers based on the suggested reviewers.

Note that if you are using the CODEOWNERS, then GitHub will automatically assign teams or specific reviewers. In this case, FlexReview can override the add or remove reviewers.

Unowned file paths

If there are certain file paths that are unowned, that is, there is no rule in the CODEOWNERS file that matches the file. In such cases:

  • If the pull request only contains the unowned file paths, no reviewers will be assigned.

  • If an unowned file is modified along with some owned files, the reviewers of those owned files will be responsible for reviewing the unowned files.

Fill the gap in CODEOWNERS

In general, we recommend relaxing CODEOWNERSand moving teams to .aviator/OWNERS unless you need a review strictly from that specific team. However, there are cases you do need such configuration.

Depending on the configured assignment method on FlexReview, it's possible that the assigned reviewer won't satisfy GitHub CODEOWNERSrequirements. In that case, FlexReview adds extra reviewers to satisfy CODEOWNERS. This applies only for the teams that have FlexReview enabled. For the FlexReview disabled teams, a reviewer is not automatically assigned and you will need to assign a reviewer manually or other means.

Multiple reviewer assignment

If a pull request touches multiple files that are owned by different teams, in some cases it is better to have multiple reviewers when they can collectively review the changes with the right context. FlexReview will generally choose the least amount of reviewers, but it assigns multiple reviewers in such situation.

For example, let's say we have following .aviator/OWNERSfile:

# .aviator/OWNERS
src             @acme-corp/engineering
src/ios         @acme-corp/ios-eng
src/ios/auth    @acme-corp/ios-auth-eng
src/ios/net     @acme-corp/ios-net-eng

and, we have users named @alice, who is in ios-engand ios-auth-eng teams, and @bob, who is in ios-engand ios-net-eng teams. When we have a PR that modifies files under src/ios/auth and src/ios/net, FlexReview will try to find a candidate from ios-eng team. @alice would know well about the auth directory, but not about the net directory. @bob would be the opposite. If we pick one reviewer from them, they might not know well on the other side of the directory.

FlexReview addresses this case by choosing multiple reviewers if one person would not cover the expertise well. It compares a case where one reviewer reviews all files and a case where multiple reviewers review individual files, and chooses the one that has a good score.

read-only mode
calculations algorithm