# Introduction Building blocks for shipping AI-generated code quickly and reliably. ![](/files/15STeyw0mrH5c6QFrblB) AI writes code faster than teams can review, merge, and release it. The bottleneck moved — from typing the change to everything that happens after. Aviator is a suite of building blocks for that *everything-after*: capturing intent, verifying behavior, queuing safe merges, coordinating releases, and routing reviewer attention to where it matters. Some pieces are open and free, designed to drop in beside what you already use. The rest are the power tools we build for teams pushing thousands of PRs a day. ## Getting started — free tools Two entry points. Adopt them in isolation; combine when you're ready.

Inbox

One feed for every change that needs your attention — across every repo. Stops the polling-Slack-and-GitHub-notifications habit and replaces it with a single ranked queue./pages/cdRRYKA7T3Mz8STnKHQI

Stacked PRs CLI

An open-source CLI for managing stacked PRs natively in GitHub. Smaller, focused reviews; cleaner history; no extra hosting./pages/ylZefdDmSp7omt3SgJoZ
## Power tools Four products that compound on each other once you're shipping at scale.

Verify

Capture the intent for every change and verify the running code against it before merge. Deterministic guardrails for AI-generated code — catches the slop that diff review misses./pages/JjM4YfnStrk7xzBdxUrR

MergeQueue

The most popular and most scalable merge queue in the industry. In production at Figma, DoorDash, Notion, Meta, and others — tens of thousands of PRs through the queue every day./pages/g5wtKxOBFem6y5iZEBGZ

Releases

One dashboard for deployments, rollbacks, and cherry-picks across every environment. Replaces the spreadsheet-and-Slack release ritual with a single source of truth./pages/P0eU9BLUbW60pnfDTwK6

Runbooks

Multiplayer AI coding with standardized playbooks. Your team's repeatable work — refactors, migrations, upgrades — turned into agentic runbooks anyone can launch./pages/GUH1AzgC7aRs0QtEoSB2
# Inbox Read documentation for Aviator Inbox, a dashboard for developers to manage pull request reviews and track code review workflows from GitHub. Inbox is your personalized dashboard for managing pull request reviews from GitHub. It organizes your PRs into meaningful sections so you can quickly identify what needs your attention and take action. {% hint style="info" %} Inbox replaces the previous AttentionSet feature with a more powerful, organized experience. {% endhint %}

Inbox dashboard

## Sections Inbox organizes your pull requests into predefined sections based on your relationship to the PR and its current state: ### Requested review PRs where you've been requested as a reviewer and need to take action. These are PRs authored by others that are waiting on your review. ### Returned PRs you authored where the attention has come back to you — typically because a reviewer has commented, requested changes, or approved your PR. ### Approved PRs you authored that have received all required approvals and are ready for the next step (e.g., merging or queuing). ### Approved by me PRs where you've submitted an approval as a reviewer. These are no longer waiting on your action, but you can track their progress here. ### Waiting for others PRs you authored that are waiting for action from reviewers. You don't need to do anything here — the ball is in someone else's court. ### Drafts Draft PRs you've authored. These are work-in-progress changes that aren't yet ready for review. ### Queued PRs you authored that are currently in the merge queue, waiting to be merged. ### Snoozed PRs you've temporarily snoozed to remove them from your active sections. See [Snooze](#snooze) below. ## Section filters Each predefined section has a filter icon that lets you apply exclusion filters to fine-tune what appears in that section. You can exclude specific authors, labels, or PRs with failing CI, and set a freshness threshold to hide stale PRs.

Section exclusion filters

## Attention The core concept behind Inbox is **attention** — identifying who should be acting on a PR at any given moment. When the author sends a PR for review, reviewers have the attention. When reviewers respond, the author gets the attention back. Every PR can have one or more people who have the attention. A merged PR will not have anyone's attention. Attention is typically managed automatically based on GitHub events, but it can also be toggled [manually](/inbox/manually-change-attention) as needed. See [Attention reasons](/inbox/attention-reasons) for a full list of events that trigger attention changes. ## Real-time updates Inbox updates automatically in real time as PR activity happens. When a reviewer comments, a CI check fails, or a PR is approved, your Inbox sections update immediately without needing to refresh the page. ## Snooze Sometimes a PR doesn't need your immediate attention. The snooze feature lets you temporarily hide a PR from your active Inbox sections until a specified time. Snoozed PRs are removed from all predefined sections and moved to a dedicated **Snoozed** view, accessible from the Inbox sidebar. When the snooze period expires, the PR automatically returns to the appropriate section. To snooze a PR, click the snooze icon on any PR row in your Inbox. You can choose from preset durations (30 minutes, 1 hour, 3 hours, tomorrow morning, next week) or select **Custom** to pick a specific date and time.

Snooze duration options

When choosing a custom snooze duration, a calendar picker lets you select the exact date and time to unsnooze.

Custom snooze date and time picker

## Repository filtering If you work across multiple repositories, you can filter your entire Inbox by repository using the dropdown in the top right. The repository filter applies across all sections, letting you focus on PRs from a specific project.

Repository filter

## Custom sections In addition to the predefined sections, you can create your own custom sections with personalized filters. This is useful for tracking specific types of PRs — for example, PRs from a particular team, PRs with a specific label, or failing CI checks. See [Custom sections](/inbox/custom-sections) for details on creating and managing custom sections. ## Getting started If you are a new Aviator user, you will first need to install the Aviator app on your GitHub repository. This is typically done as part of the initial setup wizard. You can choose to start with any of MergeQueue or FlexReview workflows to begin the setup.

Welcome to Inbox

Once the initial setup is complete, you will be prompted to link your GitHub account. This step is necessary so we can verify your GitHub username that is used to track your attention. After linking your account, navigate to the **Inbox** page in the Aviator dashboard to see your pull requests organized by section. # Custom Sections Learn how to create custom sections in Aviator Inbox to organize pull requests with personalized filters based on authors, labels, repositories, and more. Custom sections let you create personalized views of pull requests in your Inbox. Each custom section is defined by a set of filters and appears alongside the predefined sections in your Inbox sidebar. ## Creating a custom section To create a custom section: 1. Click **New section** at the bottom of the Inbox sidebar. 2. Give your section a name (e.g., "Dependabot PRs" or "Frontend team"). 3. Configure the filters for the section (see [Available filters](#available-filters) below). 4. Save the section.

Custom section editor

## Available filters Custom sections support the following filters: | Filter | Description | | ---------------- | --------------------------------------------------------------------------------------------------- | | **Authors** | Show only PRs authored by specific GitHub users. | | **Labels** | Show only PRs with specific GitHub labels. | | **Repositories** | Limit the section to PRs from specific repositories. Leave empty to search across all repositories. | | **Draft status** | Filter by whether the PR is a draft or not. | | **PR status** | Filter by pull request status (e.g., pending, queued). | | **CI status** | Filter by CI status — passing or failing. | You can combine multiple filters to create precise views. For example, you could create a section that shows all PRs with the "dependencies" label to track Dependabot updates across your repositories. ## Managing custom sections ### Editing To edit a custom section, click the edit icon next to the section name in the sidebar. You can update the section name, filters, and badge count setting. ### Reordering Custom sections can be reordered in the sidebar by dragging them to the desired position. ### Deleting To delete a custom section, click the edit icon and then select **Delete**. This action cannot be undone. ## Badge counts Each custom section can optionally show a badge count in the sidebar, displaying the number of PRs matching the section's filters. This is useful for at-a-glance visibility into sections that need attention. To toggle the badge count, edit the section and check or uncheck the **Show badge count** option. ## Examples Here are some useful custom section configurations: * **Dependabot PRs**: Filter by the author "dependabot" to track automated dependency updates. * **Urgent reviews**: Filter by a "urgent" or "priority" label to surface high-priority PRs. * **Team PRs**: Filter by authors who are members of your team. * **Failing CI**: Filter by CI status "failing" to quickly find PRs that need fixing. * **Release PRs**: Filter by a "release" label to track release-related changes. # Inbox Best Practices Check out best practices for Inbox to learn standard operating principles for maximum benefits on developer productivity. Inbox works best when everyone agrees to some standard principles of operating: * Defining a rough response time helps the author manage the right expectations without getting to the point of constant reminders, or escalations. * If you think that you will not get to the review within a reasonable time frame, it's recommended to remove attention from yourself and inform the author. * As an author who has the attention, if your change requires a bit of rework before it can be sent back for review, it's recommended to move the PR to a draft or remove the attention from the reviewers. * As an author, if you assign an optional reviewer just as an FYI and you don't expect a proper review from the reviewer, you should remove attention from that reviewer. * Likewise, as a reviewer, if you are assigned an optional review that you do not intend to get to, you should remove attention from yourself. * Use the **snooze** feature for PRs that are blocked or not immediately actionable, rather than leaving them cluttering your active sections. * Set up [**custom sections**](/inbox/custom-sections) for recurring workflows (e.g., tracking PRs from your team or PRs with specific labels) to reduce manual searching. * Use **section filters** to exclude noisy authors (like bots) or labels from specific sections, keeping your Inbox focused on what matters. # How to Manually Change Attention Learn how to manually add or remove attention on a PR in Inbox. Get detailed instructions and examples on using the Aviator dashboard. Sometimes it's helpful to add or remove your own attention on a PR. For instance, if the reviewer has responded back to the changes, but it'll take a while to refactor and resubmit for review, it might be useful to remove the attention from yourself as the author. Likewise, if you are not able to review the change within a reasonable time frame, you may want to let the author know and remove the attention from yourself. Similarly, it might be sometimes useful to bring a PR to someone's attention. Let's say a reviewer replied back with a partial review as they are still reviewing through the rest of the changes, or a reviewer dismissed the attention from themselves and now needs to be reassigned. In such cases, you can assign the reviewer's attention. Attention can be assigned or unassigned to an individual from the PR details view within the Aviator dashboard. From any PR in your Inbox, navigate to the PR details view to manage attention. On the PR details view, you can toggle attention for the authors and reviewers using the flag icon next to each subscriber's name.

PR details view with attention toggles for subscribers

# Attention Reasons Understand the reasons for attention assignment in Inbox: code review assignment, reviewer comments, PR approval, required status check failures, and more. Inbox automatically assigns attention based on PR activity. The following reasons may be assigned for getting attention: * A code review has been assigned to you * A reviewer commented on a PR authored by you * A reviewer approved a PR authored by you * Someone manually assigned the attention to you * A required status check is failing on a PR authored by you And likewise there may be PRs associated with you but are waiting on actions from others: * A PR is waiting for the reviewer's comments * A PR you reviewed is waiting on the author to take an action * You manually removed attention from this PR * The PR is in the merge queue waiting to be merged # Inbox Chrome Extension Access the Aviator AttentionSet Chrome extension and learn how it works. See PRs waiting for your review, indicated by the orange flag on GitHub's PR list view. Aviator’s [Chrome extension](/aviator-chrome-extension) also lets you view AttentionSet directly from the GitHub UI. Similar to the Aviator dashboard workflow, you can see all the PRs that are waiting for your attention, indicated by the orange flag on the GitHub’s PR list view:
And toggle the attention on and off for the authors and reviewers from the PR details view page, right under the Reviewers section.

Assign or remove attention

# Inbox Slack Home Page Access the Aviator AttentionSet in Slack. See PRs waiting for your review, and the review status of your PRs. You can view your AttentionSet inside Slack, giving you quick access to your assigned PRs and the status of any of your created PRs. To open your AttentionSet in Slack, navigate over to the Aviator app in Slack:
Then select the Home tab at the top, where you'll have access to your AttentionSet.
For On-Premise installations, your admins may need to enable this, instructions are available [here](/manage/on-premise-installation/slack-integration#add-in-slack-home-page). # Runbooks Runbooks uses Claude Code agents to plan and execute coding tasks in isolated sandboxes. Describe what you want, review the generated spec, and let agents implement it. ## Why Runbooks Individual AI tools like Claude Code are powerful but create challenges at team scale: conversations disappear when you close the session, each developer gets different solutions to the same problem, and implementation knowledge stays siloed. Runbooks solves this with three capabilities: **Multiplayer AI**: Multiple developers collaborate in shared sessions. Experienced AI engineers guide others on using better prompts, teams align on approaches together, and everyone sees the AI's reasoning. No more inconsistent patterns from isolated AI conversations. **Persistent knowledge**: Every solution becomes an organizational asset. Context files teach agents your codebase patterns. Successful runbooks become templates that capture proven workflows. New team members leverage accumulated knowledge immediately. **Spec-driven execution**: Review and approve plans before any code changes. Step-by-step execution with clear milestones replaces ad-hoc prompting. Standardized templates ensure consistent approaches across teams. ## Quick start 1. [Create your first Runbook](/runbooks/getting-started) 2. Review and refine the generated plan 3. Execute steps and merge the resulting PRs ## How it works **Planning**: Describe your task in the chat interface. Agents analyze your codebase, gather context, and generate a step-by-step plan. **Execution**: Run the plan [step-by-step](/runbooks/how-to-guides/step-by-step-execution) with review at each stage, or use [one-shot mode](/runbooks/concepts/one-shot-mode) for simpler tasks. **Collaboration**: [Invite team members](/runbooks/concepts/collaborating-with-the-team) to review plans, provide feedback, and assign steps. ## Sandboxes Agents execute in isolated environments with your codebase: * [Cloud sandboxes](/runbooks/concepts/cloud-sandboxes) - Managed by Aviator, no setup required * [SSH sandboxes](/runbooks/concepts/ssh-sandboxes) - Self-hosted on your infrastructure ## Use cases * [Code migrations](/runbooks/concepts/use-cases/code-migrations) - Framework upgrades, API changes * [Bug fixes](/runbooks/concepts/use-cases/bug-fixes) - Investigate and fix issues * [Refactoring](/runbooks/concepts/use-cases/code-refactoring) - Improve code structure * [Test coverage](/runbooks/concepts/use-cases/improving-code-coverage) - Add missing tests * [Flaky tests](/runbooks/concepts/use-cases/flaky-test-resolution) - Fix unreliable tests * [Build optimization](/runbooks/concepts/use-cases/improve-build-times) - Speed up CI ## Configuration * [Context files](/runbooks/how-to-guides/context-management) - Help agents understand your codebase * [MCP servers](/runbooks/how-to-guides/mcp-servers) - Add custom tool integrations * [Tool permissions](/runbooks/how-to-guides/claude-code-tools) - Control agent capabilities * [Personas](/runbooks/how-to-guides/persona-management) - Customize agent behavior * [Templates](/runbooks/concepts/templates) - Reuse runbooks across tasks ## Prerequisites * GitHub account with repository access * Runbooks access ([cloud](https://app.aviator.co) or [on-premise](/manage/on-premise-installation)) {% embed url="" %} # Getting started Runbooks is a multiplayer AI-coding platform that helps you perform consistent, reliable code changes across your repositories. Whether you're managing product backlog, upgrading dependencies, refactoring code patterns, or implementing organization-wide changes, Runbooks streamlines the process with intelligent automation. ### 1. Installation This setup guide will walk you through the initial set up for Runbooks. If you plan to use the self-hosted option, please contact . 1. Create an account: 2. Follow the onboarding flow to connect to the Aviator GitHub app, authorize one or more repositories that you want to use with Runbooks. You can always add more repositories later. {% hint style="info" %} If you have trouble connecting the app, please read the [troubleshooting doc](/manage/faqs/troubleshooting-github-app-connection). {% endhint %} 3. Go to Runbooks Config to customize the [Sandbox](https://app.aviator.co/runbooks/settings/sandbox) and [Claude code rules](https://app.aviator.co/runbooks/settings) as needed. By default, Aviator uses cloud sandboxes. ### 2. Access Runbooks Navigate to the **Runbooks** section in your Aviator dashboard. You'll see two options: * [**Chat Interface**](https://app.aviator.co/runbooks): Where you describe what you want to accomplish * [**Runbooks Library**](https://app.aviator.co/runbooks/templates): Your collection of published runbooks ### 3. Create your first Runbook #### Option A: Start from Scratch
1. Click on the chat interface at the top 2. Describe your migration goal in natural language: ``` I want to upgrade all React components from class-based to functional components with hooks. ``` 3. The AI agents will analyze your request and create a structured runbook 4. You can also choose “**One shot**” for smaller and descriptive task for agents to perform uninterrupted. #### Option B: Use a Template 1. Navigate to [**Library**](https://app.aviator.co/runbooks/templates) in the runbooks section 2. Browse available templates by category. For example: * **Framework Upgrades** (React, Angular, Vue.js) * **Language Migrations** (Python 2→3, JavaScript→TypeScript) * **Security Updates** (Dependency vulnerabilities, security patterns) * **Code Quality** (Linting rules, formatting standards) 3. Select a template and customize it for your needs #### Option C: Using Slack
1. Make sure [Slack integration is enabled](/api/personal-integrations) in settings 2. Describe your task in a single line or use a thread for longer description. The Aviator bot reads the entire thread. Simply say `@Aviator oneshot ` 3. The AI agents will analyze your request, and run it in oneshot mode. You will also get a Runbooks link to access the plan and progress. ### 4. Execute and monitor While generating the plan, the agents may ask some clarifying questions to gather enough requirements. Once the plan steps are generated: 1. Review the planned changes before execution 2. [Invite your team members](/runbooks/concepts/collaborating-with-the-team) and other stakeholders to provide feedback 3. Run the runbook step by step or all at once ### 5. Review the code Once execution is complete for one or more steps, the PRs will be generated. At this point you can review the code like you would a normal PR directly through GitHub. 1. Submit a review 2. Post `/aviator revise` either on a thread, or on the review for Aviator to pick revise the PR based on the comments. 3. You can go back and forth with the agents requesting multiple iterations to improve the code. Once you are satisfied with the changes, you can merge the PR manually or using [MergeQueue](https://docs.aviator.co/mergequeue). ### 6. Publishing a template If this is a common workflow that can be applied to other parts of the repository or a different repository (e.g. a new log format, or upgrading a react component), you can publish this Runbook as a template. Once published, this will now be available for other developers to use. You can also perform batch operations using this template. Learn more about templates. To publish a template, simply go to the actions menu on top right, and click “Publish as template”. Alternatively, if this Runbooks is complete and will not be used again, you can also archive it. ### Advanced Features #### Custom Personas Create specialized [personas](/runbooks/concepts/personas) for different types of work: * **Security Expert**: Focuses on security implications * **Performance Optimizer**: Emphasizes performance improvements * **Accessibility Specialist**: Ensures accessibility compliance #### MCP Server Integration Connect external tools and services: * **Code analysis tools** (SonarQube, CodeClimate) * **Testing frameworks** (Jest, Cypress, Playwright) * **Documentation systems** (Confluence, Notion) #### Batch Processing Scale your operations: * **Template-based batches**: Apply the same template to multiple repos * **Progressive rollouts**: Gradually deploy changes across teams * **Dependency management**: Handle complex repository relationships ### Getting Help * **Community**: [Ask help on Discord](https://discord.gg/MmQWrY9xrA) * **Support**: Email us for assistance: [*howto@aviator.co*](mailto:howto@aviator.co) ### Security and Compliance Runbooks includes several security features: * **Audit trails**: Complete history of all changes * **Access controls**: Role-based permissions * **Secure execution**: Isolated sandbox environments * **Data protection**: Encrypted storage and transmission Email **** for any security questions or to request our SOC2 report. ### Billing and Usage Runbooks usage is tracked and billed based on: * **Execution time**: Runtime in sandbox environments * **LLM usage**: How many tokens are used by the LLM provider Monitor your usage in the billing section of your workspace settings. ### Next Steps Ready to get started? Here are some recommended first steps: 1. **Explore Templates**: Browse the template library for inspiration 2. **Try a Simple Migration**: Start with a low-risk change like updating documentation 3. **Join the Community**: Share your templates and learn from others 4. **Invite Your Team**: Runbooks are more useful as a multiplayer tool ### See also * [One shot mode](/runbooks/concepts/one-shot-mode) * [Runbook Format](/runbooks/concepts/runbook-format) * [Templates](/runbooks/concepts/templates) # Concepts # Architecture Overview Runbooks consists of several components that work together to enable collaborative AI-driven code changes.
## Core Components ### Web Dashboard The primary interface for interacting with Runbooks: * **Chat interface**: Conversational UI for describing tasks and reviewing plans * **Runbook editor**: View and modify generated execution steps * **Session management**: Create, clone, and manage runbook sessions * **Collaboration**: Invite team members, assign steps, share sessions * **Configuration**: Manage personas, context files, MCP servers, and tool permissions ### API Layer GraphQL API that handles: * Authentication via GitHub OAuth, SAML, or API tokens * Session state and user permissions * Routing requests to background workers * Real-time updates to the dashboard via WebSocket ### Orchestration Layer Background task queues manage the execution lifecycle: * **Task distribution**: Distributes work across available workers * **Sandbox lifecycle**: Creates, monitors, and cleans up sandbox instances * **Concurrency control**: Limits concurrent sandboxes per account * **Retry logic**: Handles transient failures with exponential backoff ### Sandboxes Isolated execution environments where Claude Code runs. All LLM interactions happen exclusively within sandboxes—your code and prompts never leave the sandbox boundary, preventing any possibility of prompt injection or data exfiltration from affecting your main systems. **Cloud sandboxes**: * Managed containers with pre-configured environments * Auto-pause after inactivity, resume on demand * Custom templates via Dockerfile * Configurable timeout (1-240 minutes) **SSH sandboxes**: * Self-hosted Linux servers you control * SSH key-based authentication * Connection pooling and load balancing across instances * Repository caching for faster subsequent runs Both sandbox types: * Clone your repository via authenticated Git URLs * Run Claude Code with your configured permissions * Execute pre-execution scripts (`.aviator/scripts/pre-execution.sh`) * Stream output back to the dashboard in real-time via custom transport layers ## Session Management Sessions are the core unit of work in Runbooks. Each session maintains: **Conversation history**: All messages between users and agents are preserved. When you resume a session, Claude Code receives the full context of previous interactions, enabling coherent multi-turn conversations across days or weeks. **Execution state**: Sessions track which steps are pending, in progress, or completed. You can pause execution, switch to a different step, or re-run failed steps without losing context. **Collaborator access**: Multiple team members can join a session. Everyone sees the same conversation, can send messages, and trigger execution. This enables real-time collaboration where senior engineers can guide juniors through complex tasks. **Branch and PR tracking**: Sessions maintain links to working branches and pull requests created during execution. When you provide feedback on a PR, the session context helps Claude Code understand what was attempted and what needs to change. **Resume capability**: Claude Code sessions can be paused and resumed. The system preserves the agent's internal state, so resuming a session continues exactly where it left off rather than starting fresh. ## Tool Permissions Control what Claude Code can do during planning and execution phases separately. ### Allowlists and Denylists **Allowlist**: Only listed tools are permitted. Unlisted tools are blocked. **Denylist**: Listed tools are blocked. All other tools remain available. You can configure different permissions for each phase: * **Planning phase**: Typically read-only tools for code analysis * **Execution phase**: Write access for implementing changes ### Tool Patterns **Basic tools**: `Read`, `Write`, `Edit`, `Grep`, `Glob`, `WebSearch` **Bash commands with wildcards**: ``` Bash(npm run build) # Exact command only Bash(npm run:*) # Any npm run subcommand Bash(pytest:*) # pytest with any arguments ``` **MCP tools**: `mcp__servername` or `mcp__servername__toolname` ### System Restrictions Certain operations are always blocked regardless of configuration: * Git commit, push, and other write operations (Runbooks manages git automatically) * This ensures consistent commit messages, proper attribution, and controlled PR workflows ## MCP Servers Extend Claude Code's capabilities with custom tool integrations via the Model Context Protocol. ### Configuration MCP servers are configured per-account in JSON format: ```json { "mcpServers": { "github": { "type": "stdio", "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } } } } ``` ### Server Types **stdio**: Local command-based servers. Requires `command`, optional `args` and `env`. **sse/http**: Remote servers. Requires `url`, optional `headers`. ### Using MCP Tools Once configured, MCP tools are automatically available to Claude Code with the `mcp__` prefix: * `mcp__github` - All tools from the github server * `mcp__github__create_issue` - Specific tool You can control access to MCP tools using the same allowlist/denylist mechanism as built-in tools. ### Secrets Reference secrets in MCP configuration using `${SECRET_NAME}` syntax. Secrets are: * Encrypted at rest * Injected at runtime into the sandbox * Never exposed in logs or UI after creation ## Execution Flow ### Planning Phase 1. User submits task description via chat 2. Background worker acquires a sandbox 3. Sandbox clones repository and checks out target branch 4. Claude Code receives system prompt (persona + context files) and user prompt 5. Agent analyzes codebase with planning-phase tool permissions 6. Generates step-by-step execution plan 7. Plan streams back to dashboard for review ### Execution Phase 1. User approves plan or triggers step execution 2. Background worker checks out appropriate branch 3. Runs pre-execution script if present (`.aviator/scripts/pre-execution.sh`) 4. Claude Code implements the step with execution-phase permissions 5. On completion: * Commits changes with structured message * Creates or updates pull request * Links PR to the step 6. Results stream back to dashboard 7. User can provide feedback, triggering iteration ### Feedback Loop When you comment on a PR or provide feedback in the chat: 1. Claude Code receives context of previous work plus your feedback 2. Generates updated implementation 3. Pushes new commits to the existing PR 4. CI runs again, and the cycle continues until you're satisfied ## GitHub Integration ### Repository Access * Authenticated via GitHub App installation * Supports private repositories within the installation scope * Git credentials refreshed automatically in sandboxes ### Pull Request Workflow * PRs created per-step or per-runbook (configurable) * Draft PRs for work-in-progress * Automatic undraft when step completes * PR body includes step description and runbook context * Co-authored-by tags for attribution ### Branch Management * Working branches created automatically * Target branch configurable per session * Clean state before each execution (uncommitted changes discarded) ## Integrations ### Slack * **One-shot mode**: Execute simple tasks directly from Slack threads * **Notifications**: Step completion, PR creation, failures * **Links**: Deep links back to dashboard for full context ### Context Files Persistent knowledge that helps agents understand your codebase: * Markdown documents stored per-account * Automatically injected into Claude Code prompts * Reference important files for prioritized context * See [Context management](/runbooks/how-to-guides/context-management) for details # Runbook Format Runbooks are the core abstraction for defining and executing code transformations. They encapsulate the complete plan for a code change, from analysis to implementation. ## Format Runbooks use a custom format designed for flexibility and clarity. Typical components include: * **Metadata**: Version, author, collaborators, creation date * **Objective**: High-level description of the transformation * **Analysis**: Code patterns to identify and transform * **Steps**: Ordered list of actions to perform * **Validation**: Tests and checks to verify success The Runbooks in Aviator system must be structured in a specific way, so that these can be processed by the agents: * **Steps and sub-steps -** Each step should be contained that either requires a specific action (like creating a branch), or otherwise a code change that can be done in a pull request. * **Top-level context** - context associated with the entire task that can be used for every step in the LLM * **Step-level context** - context associated with a specific step or a sub-step. ### Steps Each Runbook is broken down into steps and sub-steps, the goal of each step is to define a very small clearly scoped task that the agents can perform with accuracy. Aviator agents understands these steps and it’s boundaries well, and have proper naming convention with defined rules: * Each step is a number increment starting from `1`. * Each substep is prefixed with the step name of the parent, followed by the number. E.g. `1.1` , `1.2`, ,.. * The sequence must be contiguous * Each step name must be unique Runbook is validated on edits based on the formatting rules above. Agents uniquely understands each step, that way you can ask the agents to modify, reorder or execute a specific step or a sub-step. ## Context There are several ways to provide context to the agents. Please read the Context Management guidelines for the deep dive. Along with some of the global context that helps agents understand the code base, some additional context may also be provided within runbooks. This would be specific to the task at hand. The context can either be provided before any of the steps in the Runbook, or may be provided within the Runbook specific step or sub-step. ## Sharing and editing Any active Runbook with a chat session can be shared with other users within your organization to collaborate together. This expands the chat experience inviting other developers to provide feedback and make edits. There are two ways of editing the Runbook. ### **Manually** Either you can manually update each Runbook by clicking on edit on the right panel within the Runbook view. This opens the editor with raw Markdown to edit. Click Save when done editing. Aviator Runbooks follow a specific pattern and is validated on manual edits. ### **Using Agents** You can also ask the agents to update specific part of all of the Runbook directly in the chat session. ## Versioning Aviator maintains the historic versioning of all Runbooks. The versioning is sequential in increasing order and you can rollback to an older version of Runbook at any time. Even after rolling back to the previous version, a subsequent version would still represent the next incremental number than the highest sequential version. # Templates Runbook Templates are reusable automation patterns that standardize common coding tasks such as code migration, refactoring, and maintenance work. They provide structured, step-by-step guidance for complex development workflows and can be customized for your specific project needs. Aviator offers a library full of prebuilt templates, and you can also publish your own templates to be used within your organization. ## What is a Runbook Template? A Runbook Template is a predefined workflow that contains: * **Structured markdown content** with hierarchical execution steps * **Categorization** for easy discovery and organization * **Reusable patterns** for common development tasks * **Usage tracking** to identify popular templates Templates serve as starting points for creating new runbooks, allowing teams to standardize their approach to routine but complex tasks like framework migrations, code modernization, and systematic refactoring. Each Runbook Template follows the same format as a standard Runbook. ## Prebuilt templates We offer prebuilt templates for some common coding tasks patterns such as code migration, refactoring, maintenance work, etc. These prebuilt templates are agnostic of your code base but has some standard learnings based on the tasks. When you use these prebuilt templates, Runbooks agents will modify the Runbook template to adapt to your code base. This provides better quality results because it blends the standard process to perform an action with your code context. You ca ## Custom templates Custom templates are the Runbooks published by your team manually. These templates are shared across the organization. Anyone can request to publish a template, and these can be available in the library once approved by an admin. ### Creating a template To create a template, you must have edit access to the Runbook. Simply go to your Runbook and from the actions menu select: `Publish ...` . This will be send a request to the admins, and will show up in pending approval in the library section. Once approved by admins, this will now be available to be reused. {% hint style="info" %} Runbooks can be shared and reused even without them being published. Publishing Runbooks help create the templates that may have long term wider usability. {% endhint %} ## Template categories Categories makes it easy to organize the templates, and make them searchable. Think of categories as labels that can be associated with the Runbook Templates. Categories can also be added or removed as needed. Prebuilt templates are organized into the following default categories: | Category | Description | Examples | | ------------- | ---------------------------------------------------- | ------------------------------------------------------------- | | react | React framework related templates | Class to hooks migration, state management updates | | typescript | TypeScript migration and type-related templates | JavaScript to TypeScript conversion, type safety improvements | | migration | General migration templates | Language upgrades, framework migrations | | refactor | Code refactoring and systematic improvements | Code organization, pattern standardization | | setup | Project setup and configuration templates | Development environment setup, tooling configuration | | tooling | Development tooling and build system templates | Build optimization, CI/CD improvements | | test\_quality | Test quality, coverage, and testing best practices | Test coverage improvement, flaky test fixes | | upgrade | Framework and dependency upgrade templates | Package updates, breaking change management | | performance | Performance optimization and build time improvements | Bundle optimization, runtime performance | | flaky\_test | Flaky test resolution and test stability | Test reliability improvements | | readability | Code readability, formatting, and documentation | Code style standardization, documentation updates | | misc | Miscellaneous and other templates | Custom workflows, specialized tasks | You cannot modify the categories for pre-built templates but the categories can be modified for all custom templates. ## See also * [Managing Templates](/runbooks/how-to-guides/managing-templates) # Personas Runbook Personas are customizable AI behavior profiles that define how the system interacts with your code during runbook generation, step execution, and Q\&A sessions. They allow you to tailor the AI's approach, expertise level, and communication style to match your team's specific needs and project requirements. ## **What Are Runbook Personas?** A Runbook Persona is a configuration that defines a system prompt that influences AI behavior and decision making. You can use these personas for specialized expertise for different domains, for example, planning, development, security, etc. Personas can also be used to modify the communication style - make it more casual or strict, or be creative “answer like Yoda”. Personas can also be used to provide context awareness for specific project types or methodologies. Personas ensure consistent, context-appropriate assistance across all runbook interactions, whether you're generating migration plans, executing code changes, or discussing implementation details. The system prompt provided in the personas are directly passed to the background agents that are interacting with the LLMs. ## **Persona Types** ### **Built-in Persona Types** * **Planner**: used for strategic planning and high-level architecture. Focuses on comprehensive planning, risk analysis, and step sequencing. * **Developer**: used for code implementation and technical execution. Emphasizes practical implementation, code quality, and technical details. * **Custom**: user-defined specialized roles. Completely customizable behavior based on your specific needs ### **Default System Personas** The system includes default personas that provide baseline functionality: * **Default Planner**: Comprehensive planning with emphasis on safety and thoroughness. * **Default Developer**: Practical code implementation focused on quality and maintainability These personas cannot be deleted but can be customized by admins. ## **Sharing and Permissions** Personas can be private or shared. A private persona is created by an individual and can only be used by them. These personas exist for personal preferences, experimental configurations. The creator can view, edit and delete the persona themselves. A shared persona on the other hand can only be created by admins and maintainers. These are visible to all users in the organization. Some use cases for this include team standards, organization-wide best practices. Only admins and maintainers can created, edit or delete these personas. All the users can read the system prompts of these personas. ## See also * [Persona management](/runbooks/how-to-guides/persona-management) # Collaborating with the team The platform supports team collaboration through shared [Runbooks](/runbooks/concepts/runbook-format) and coordinated execution. ## Adding Collaborators Collaborators receive same access as the primary user. You can add the collaborator from the chat interface of the Runbook. Simply navigate to the "Share" button, search for a team member and Invite. The invited member will receive an email with the link to the Runbook. You can also copy the link and share with your team member directly after adding them as a collaborator.
### Collaborator's access All collaborators see the same Runbook version. All the changes are immediately visible to all team members. The collaborators can also view the execution history, edit the Runbook or execute any step. ## Best Practices for Teams * Establish clear ownership for Runbooks * Use descriptive names and documentation * Regular reviews of generated changes * Coordinate execution timing * Share learnings through global context # Context and learnings Runbooks automatically learn from your coding sessions to provide better assistance over time. This document explains how context works and how it benefits your workflow. ## Types of context Runbooks use two types of context: | Type | Source | Purpose | | ----------------- | ------------------------------------ | ------------------------------------- | | **Learnings** | Automatically captured from sessions | Remember solutions to problems | | **Context Files** | Manually created by you | Document architecture and conventions | Both types are shared across your account, so knowledge captured by one team member benefits everyone. ## What are learnings? Learnings are reusable insights captured from your Runbook sessions. When an agent encounters a problem and finds a solution, it remembers that pattern so it can apply the same fix in future sessions. **Example:** ``` Pattern: "pytest fails with ModuleNotFoundError for local packages" Solution: "Run `pip install -e .` to install the package in editable mode" Applies to: pytest, Python projects ``` The next time anyone on your team hits a similar issue, the agent already knows how to fix it. ## When are learnings captured? Learnings are captured during high-signal moments — times when something notable happens that's worth remembering: | Moment | What's Captured | | ----------------------------- | --------------------------------------------------------------------- | | **Errors resolved** | Problems encountered during execution and how they were fixed | | **CI failures fixed** | Build/test failures and the changes that resolved them | | **Review feedback addressed** | Code improvements made based on reviewer comments | | **User corrections** | When you modify a Runbook or provide feedback to improve the approach | Routine operations that complete without issues don't generate learnings — only moments where something was learned. ## How are learnings used? When you start a new Runbook session, agents automatically retrieve relevant learnings based on: * **Files you're working with** — Learnings about specific file types or paths * **Frameworks in use** — Learnings related to React, pytest, Docker, etc. * **Commands being run** — Learnings about specific build tools or scripts Relevant learnings appear as helpful hints when planning your task, helping you avoid known pitfalls. ## Learnings lifecycle ``` ┌─────────────────────────────────────────────────────────────┐ │ 1. CAPTURE │ │ During execution, notable problems and solutions │ │ are identified and saved │ └─────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────┐ │ 2. STORE │ │ Learnings are stored per-account, deduplicated, │ │ and tagged with relevant context (files, frameworks) │ └─────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────┐ │ 3. RETRIEVE │ │ When starting a new task, relevant learnings are │ │ automatically matched based on your current context │ └─────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────┐ │ 4. APPLY │ │ Learnings surface as guidance during planning, │ │ helping avoid known issues before they occur │ └─────────────────────────────────────────────────────────────┘ ``` ## What makes a good learning? Learnings are most valuable when they're: * **Specific** — Clear problem description with exact error messages * **Actionable** — Concrete solution that can be applied * **Reusable** — Applicable beyond the original task * **Project-relevant** — Specific to your codebase patterns and conventions Generic knowledge (like "run npm install to install dependencies") is not captured — only insights specific to your project. ## Improving over time The learning system improves with use: * **Frequency tracking** — Learnings that help repeatedly are prioritized * **Recency weighting** — Recent learnings are more relevant than old ones * **Pattern matching** — Similar problems are grouped together The more you use Runbooks, the smarter they become for your specific codebase. ## What are context files? Context files are markdown documents you create to describe your project. Unlike learnings (which are captured automatically), context files are authored by you to provide foundational knowledge about: * Project architecture and conventions * API patterns and data models * Testing requirements * Deployment procedures * Team-specific coding standards See [Context Management](/runbooks/how-to-guides/context-management) for how to create and manage context files. ## Privacy * Learnings and context files are scoped to your account — never shared across organizations * Learnings don't contain sensitive data like secrets or credentials * You can view and manage all learnings in the Context tab *** **Next:** [Managing Context](/runbooks/how-to-guides/context-management) — Learn how to browse, create, and manage learnings and context files. # Working with large codebases Handling enterprise-scale repositories requires specific strategies and optimizations. ## Code Analysis Strategy ### **Intelligent Searching** * Agents download code to remote servers * Use reference-based searching for efficiency * Progressive analysis from entry points ### **Token Optimization** * Focus on relevant code sections * Incremental context building * Caching of analysis results ## Performance Considerations ``` Repository Size Recommended Approach ────────────── ─────────────────── < 100K LOC Standard analysis 100K-1M LOC Targeted search patterns > 1M LOC Modular execution strategy ``` ## Large Codebase Best Practices 1. **Divide and Conquer** * Break transformations into smaller Runbooks * Focus on specific modules or packages * Use incremental rollout strategy 2. **Context Management** * Carefully curate relevant documentation * Use MCP for external context * Limit scope to necessary files 3. **Resource Planning** * Monitor token usage trends * Scale agent containers as needed * Use caching effectively # One shot mode [One-shot mode](/runbooks/how-to-guides/one-shot-mode) is an automated execution mode for Runbooks that streamlines the entire code modification workflow into a single, uninterrupted process. When enabled, Runbooks generates a complete execution plan and immediately begins implementing all steps without requiring manual approval or intervention between steps. One-shot can be triggered from the browser window or from Slack directly.
one-shot mode
#### When to Use One-Shot Mode One-shot mode is ideal for: * **Well-defined tasks** where requirements are clear and unambiguous * **Automated migrations** such as dependency upgrades, API version updates, or batch refactoring * **Repetitive operations** that follow established patterns in your codebase * **Quick fixes** where the scope is limited and the approach is straightforward * **Time-sensitive changes** that need to be implemented quickly without multi-step review cycles #### When NOT to Use One-Shot Mode Avoid one-shot mode for: * **Complex refactoring** that may require architectural decisions or mid-execution adjustments * **Exploratory tasks** where you're unsure of the full scope or implementation approach * **High-risk changes** to critical systems that require careful review at each stage * **Tasks with unclear requirements** that might benefit from clarifying questions #### Key Differences from Standard Mode | Aspect | Standard Mode | One-Shot Mode | | -------------------------- | ----------------------------------------------- | ---------------------------------------------------- | | **Requirements gathering** | Asks clarifying questions | Skips clarification, makes reasonable assumptions | | **Plan approval** | Waits for user to approve the runbook | Automatically begins execution after generation | | **Execution control** | User chooses when to execute each step | All steps execute automatically in sequence | | **Pull request strategy** | Creates separate PRs for each major step | Creates a single draft PR, updates it with each step | | **User involvement** | Interactive, allows modifications between steps | Fully automated, monitoring only | ### See also [How to use one-shot mode](/runbooks/how-to-guides/one-shot-mode) # Cloud sandboxes The Cloud based sandboxes offer easy to setup environments managed by us. This setup is ideal for individual developers, small teams who want to get started quickly. If you’d like to manage your own sandboxes, please contact . Each cloud runner uses 4 core CPU with 4GB memory. ## Cloud runners The cloud based runners are currently available in US and EU regions. Each runner consist of a baseline image that contains `git` binary and `claude code` CLI along with a few simple tools to manage the lifecycle of these runners. All our cloud runners are single tenant and do not share permissions or environment with any other task. These act as background agents running claude code and performing operations as directed by the user, while streaming the output back to the controller. ## Lifecycle management Every time a new task is created, a new runner will be spawned in the cloud. The runner will automatically select one of the environments suitable for this task. Cloning an existing Runbook creates a new task and a new runner environment. A runner will automatically pause after 60 mins of inactivity by default. This timeout is configurable per repository (see [Timeout configuration](#timeout-configuration)). You may also choose to pause the runner manually anytime during the task. To unpause an existing runner, simply post a message on the chat associated with that Runbook. This will reload the runner from the previous saved state and resume the conversation. All the context is loaded back when resuming the runner. All paused runners are permanently deleted after 7 days of inactivity. Resuming a runner resets that window. ## Environment management Environments control what tools and permissions are enabled for the runners. These environments can be preset for each repository in the Runbooks settings. ### Prebuilt environments For quick setup, we recommend using one of our prebuilt environments: * **Python** - Python 3.13, pip, virtualenv, uv, pytest * **Node.js** - Node 18 LTS, npm, yarn * **Go** - Go 1.21, go mod * **Java** - OpenJDK 17, Maven, Gradle * **Ruby** - Ruby 3.2, RubyGems, Bundler * **Rust** - Rust stable, Cargo * **PHP** - PHP 8.2, Composer * **React/TypeScript** - Node.js, TypeScript, React, Vite, ESLint, Prettier * **Vue.js** - Node.js, Vue 3, Vite, TypeScript support * **Angular** - Node.js, Angular CLI, TypeScript * **Next.js** - Node.js, Next.js 14, TypeScript, Tailwind CSS * **MEAN Stack** - MongoDB, Express, Angular, Node.js, TypeScript * **MERN Stack** - MongoDB, Express, React, Node.js, TypeScript * **Django + React** - Python, Django, Node.js, React, PostgreSQL * **Rails + Vue** - Ruby on Rails, Node.js, Vue.js, PostgreSQL * **Spring Boot + React** - Java, Spring Boot, Node.js, React, Maven * **Go + Next.js** - Go, Node.js, Next.js, TypeScript * **FastAPI + React** - Python, FastAPI, Node.js, React, TypeScript ### Custom templates Create custom sandbox templates when prebuilt environments don't meet your requirements. #### Creating a custom template Navigate to **Runbooks Settings > Custom Templates** and click **Create Template**. Provide: * **Display name**: A descriptive name for your template * **Dockerfile content**: Instructions to build your environment #### Dockerfile requirements Custom templates use Dockerfiles with some restrictions: * `ADD` and `COPY` commands are not allowed (no local file access during build) * Use `RUN` commands to install dependencies via package managers **Example Dockerfile**: ```dockerfile FROM ubuntu:22.04 # Install system dependencies RUN apt-get update && apt-get install -y \ git \ curl \ build-essential \ && rm -rf /var/lib/apt/lists/* # Install Node.js 20 RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \ && apt-get install -y nodejs # Install Python 3.11 with uv RUN apt-get update && apt-get install -y python3.11 python3.11-venv \ && curl -LsSf https://astral.sh/uv/install.sh | sh # Install project-specific tools RUN npm install -g pnpm turbo ``` #### Build process After submitting your Dockerfile: 1. The system validates the Dockerfile syntax 2. A build job starts (typically takes 2-10 minutes) 3. Once complete, the template appears in your sandbox configuration options #### Using custom templates Select your custom template in **Runbooks Settings > Sandbox Configuration**: 1. Set **Sandbox Type** to `cloud` 2. Choose your template from the **Sandbox Template** dropdown New Runbook sessions will use your custom template. #### Timeout configuration Cloud sandboxes have a configurable timeout (1-240 minutes). Set this in **Runbooks Settings > Sandbox Configuration** to control how long inactive sandboxes remain running before auto-pausing. ## Secrets management Secrets are credentials that are stored encrypted and can be injected into the runners for each environment. These secrets can be used for two purposes: * providing access to MCP configuration * using as environment variables to perform specific actions within the runners These secrets can be created by anyone, either for the entire organization or just for personal use. Admins can manage the permissions of organization wide secrets. Secrets cannot be retrieved in clear text once saved. # SSH sandboxes SSH sandboxes provide a self-managed execution environment for Runbooks, allowing organizations to run Claude Code on their own infrastructure instead of using Aviator's managed cloud sandboxes. This approach offers enhanced security, control, and customization for enterprise environments. ### Architecture overview SSH sandboxes operate as remote Linux environments that Aviator connects to via SSH transport. Each sandbox maintains a working directory where Git repositories are cloned and managed. The system supports multiple sandbox instances per account for load balancing and redundancy. The implementation uses SSH key-based authentication with encrypted private key storage. Connection management includes pooling, locking mechanisms to prevent conflicts, and automatic retry logic for robust operation. ### Configuration requirements #### Account-level setup Organizations configure SSH sandboxes through the Runbooks settings interface. The configuration requires selecting SSH as the sandbox type and providing an SSH private key for authentication. Private keys support OpenSSH, RSA, and EC formats. Keys are encrypted using Fernet encryption before storage and automatically decrypted during connection establishment.
#### SSH sandbox instances Individual sandbox instances require configuration of hostname or IP address, SSH port (default 22), username (default root), and working directory path for repository operations. Each sandbox can be individually enabled or disabled through the active status flag. The system enforces unique hostnames per account to prevent configuration conflicts. #### Infrastructure requirements SSH sandboxes require Linux servers with specific software installations: Git with LFS support, Node.js for npm packages, Python with uv package manager, and Claude Code CLI. It also optionally can include build tools including curl and build-essential. ### Managing Sandboxes To learn more about managing the Sandboxes, checkout [SSH Sandboxes Configuration Guide](/runbooks/how-to-guides/ssh-sandboxes-configuration-guide) ### Execution workflow #### Sandbox selection and initialization When executing Runbooks with SSH sandboxes configured, the system automatically selects an available sandbox instance from the pool. Redis-based locking prevents concurrent usage of the same sandbox instance. Repository cloning occurs automatically using authenticated Git URLs from the GitHub integration. The system detects existing repositories to avoid unnecessary re-cloning and supports Git submodules and LFS. #### Command execution Claude Code commands execute remotely via SSH transport with JSON message streaming between Aviator and the remote instance. Environment variables are injected during execution, and stdout/stderr streams are captured and forwarded to the Runbooks interface. Working directories are created uniquely for concurrent executions, with automatic cleanup of temporary files after completion. Existing repositories persist across sessions for improved performance. #### Resource management Connection pooling manages SSH connections efficiently with automatic reuse where possible. Load balancing distributes workload across available sandbox instances when multiple sandboxes are configured. The system implements comprehensive cleanup procedures, removing temporary directories and releasing locks after execution completion. ### Authentication and security #### SSH key management SSH authentication relies on public-private key pairs configured between Aviator and the sandbox instances. Organizations generate key pairs using their preferred format and configure the public key in the sandbox's authorized\_keys file. Private keys are stored encrypted in Aviator's database and undergo automatic validation during configuration. The system supports key rotation through the settings interface. #### Git repository access Git authentication leverages existing GitHub App permissions and authenticated URLs. Sandbox environments receive automatic credential configuration enabling access to private repositories based on the organization's GitHub integration settings. ### Performance considerations #### Connection optimization The SSH transport implementation includes connection pooling to minimize establishment overhead. Repository caching reduces cloning time for subsequent executions of the same codebase. Multiple sandbox instances enable parallel execution when workload demands exceed single-instance capacity. Lock coordination ensures safe concurrent operations without resource conflicts. #### Scaling patterns Organizations can provision additional sandbox instances to handle increased execution volume. The system automatically distributes load across available instances without requiring manual intervention. Sandbox instances can be scaled independently based on usage patterns, with the ability to temporarily disable instances for maintenance without disrupting service. ### Integration benefits #### Enterprise compliance SSH sandboxes enable organizations to maintain code execution within their own infrastructure boundaries, supporting compliance requirements that restrict external code processing. Security policies can be implemented at the infrastructure level, including network isolation, monitoring, and access controls beyond what cloud sandboxes provide. #### Customization capabilities Organizations can install custom tools, configure specific environment settings, and implement specialized build processes within their sandbox environments. Development team preferences for specific tool versions, security configurations, or monitoring integrations can be accommodated through direct sandbox customization. #### Cost management SSH sandboxes eliminate per-execution charges associated with cloud sandbox usage, particularly beneficial for organizations with high Runbooks usage volumes. Infrastructure costs become predictable through fixed sandbox provisioning rather than variable usage-based pricing models. ### See also * [SSH Sandboxes Configuration Guide](/runbooks/how-to-guides/ssh-sandboxes-configuration-guide) * [Cloud sandboxes](/runbooks/concepts/cloud-sandboxes) # Use Cases Take a look at some common use cases with examples. Runbooks are meant to solve most of the use cases that involve improving and modifying an existing codebase, but not everything is worth using Runbooks for. ### When to use Runbooks Runbooks are great for most well defined or well scoped tasks. It is meant to absorb business context and get smarter over time. Some examples include: * [Product backlog](/runbooks/concepts/use-cases/product-backlog) * [Refactoring](/runbooks/concepts/use-cases/code-refactoring) * [Bug fixes](/runbooks/concepts/use-cases/bug-fixes) * [Improving build times](/runbooks/concepts/use-cases/improve-build-times) * [Flaky test resolution](/runbooks/concepts/use-cases/flaky-test-resolution) * [Code migration](/runbooks/concepts/use-cases/code-migrations) * [Improving readability](/runbooks/concepts/use-cases/improving-readability) * [Improving code coverage](/runbooks/concepts/use-cases/improving-code-coverage) ### When not to use Runbooks Runbooks are not great for brainstorming ideas or working on greenfield projects. If requires a lot of back-and-forth with your agents, you might be better off using Claude code or similar coding tools in your terminal. ### Creating templates When you use Runbooks for a specific use case that occurs often (e.g. a bug fix or a flaky test resolution), create templates. [Template](/runbooks/concepts/templates) can be created from any new or existing Runbook. These are great ways to share your process with the team and enhance collective development. ### Getting Started 1. **Choose a simple feature**: Start with a well-defined, isolated feature 2. **Write clear requirements**: Provide detailed specifications and examples 3. **Review and refine**: Work with Runbooks to refine the implementation plan 4. **Execute step by step**: Run the implementation in stages with review points 5. **Learn and iterate**: Use the experience to improve future automation # Product backlog Transform your product backlog from a list of wishes into automated implementations. Runbooks excels at taking feature requests and systematically implementing them across your codebase with AI-powered automation. ### When to use #### Ideal scenarios * **Feature Rollouts**: Implementing new features across multiple components * **UI/UX Improvements**: Adding new interface elements or improving user experience * **Cross-cutting Features**: Features that need to be implemented in multiple places * **Standardization**: Implementing company-wide features or standards * **Infrastructure Features**: Adding monitoring, logging, or security features #### Perfect for teams that * Have consistent architectural patterns * Need to implement similar features across multiple services * Want to maintain consistency in feature implementation * Have clear feature specifications and requirements ### Available templates From the runbooks library, these templates help implement user-facing features: **Dark mode implementation** * Automatically adds dark mode support to applications * Implements theme switching logic * Updates color schemes and styling consistently * Adds user preference storage **Responsive design conversion** * Converts legacy layouts to modern responsive design * Updates CSS and layout components * Adds interactive drag and drop features **Feature infrastructure templates** * Implements modern loading states * Adds skeleton components for better UX ### Without using templates #### 1. Feature analysis and planning Start by clearly defining your feature requirements: ``` "I want to add dark mode support to our React application. The app should: - Remember user's theme preference - Support system theme detection - Apply consistent theming across all components - Include a theme toggle in the header" ``` **What runbooks does:** * Analyzes your current styling architecture (CSS-in-JS, SCSS, etc.) * Identifies all components that need theme support * Plans the implementation strategy * Creates a step-by-step execution plan #### 2. Architecture assessment Runbooks examines your codebase to understand: * Current styling patterns and conventions * State management approach (Redux, Context, etc.) * Component structure and hierarchy * Existing theme or styling infrastructure #### 3. Implementation planning The AI creates a detailed runbook covering: * Theme system architecture * Component updates needed * State management changes * Testing requirements * Documentation updates #### 4. Automated implementation Runbooks executes the plan by: * Creating theme provider components * Updating existing components with theme support * Implementing theme toggle functionality * Adding user preference persistence * Updating styling across the application ### Real-world examples #### Example 1: Adding search functionality **User request:** ``` "Add search functionality to our e-commerce platform. Use existing Elasticsaerch integration. Users should be able to: - Search products by name, category, and tags - See real-time search suggestions - Filter results by price, rating, and availability - Save search history" ``` **Runbook implementation:** 1. **Backend API development** * Explores existing Elasticsearch integration * Implement autocomplete and suggestion APIs * Add search analytics tracking 2. **Frontend search components** * Build search input with autocomplete * Create search results page with filtering * Implement search history storage 3. **Database updates** * Add search indexes * Create search analytics tables * Update product schema for search optimization #### Example 2: Implementing user notifications **User request:** ``` "Implement a notification system that shows: - Real-time alerts for important events - Toast notifications for user actions - Email digest for weekly summaries - Push notifications for mobile users" ``` **Runbook implementation:** 1. **Notification infrastructure** * Set up WebSocket connections for real-time updates * Create notification storage and management * Implement notification templates 2. **UI components** * Build notification center component * Create toast notification system * Add notification preferences page 3. **External integrations** * Configure email service integration * Set up push notification services * Add notification analytics *** ### See also * [Bug fixes](/runbooks/concepts/use-cases/bug-fixes) * [Code refactoring](/runbooks/concepts/use-cases/code-refactoring) # Code refactoring Code refactoring with Runbooks involves automatically analyzing your codebase, identifying improvement opportunities, and systematically updating code to modern patterns and best practices. Unlike manual refactoring that happens piecemeal, Runbooks can refactor entire codebases consistently and safely. ### When to use #### Ideal scenarios * **Legacy Code Modernization**: Converting old patterns to modern best practices * **Framework Upgrades**: Refactoring code to use new framework features * **Performance Optimization**: Replacing inefficient patterns with optimized ones * **Consistency Improvements**: Standardizing code patterns across the codebase * **Architecture Evolution**: Moving from one architectural pattern to another #### Perfect for teams that * Have large codebases with inconsistent patterns * Need to modernize legacy applications * Want to improve code maintainability and readability * Need to standardize development practices across teams ### Available refactoring templates #### React modernization templates **React class components to hooks** * Converts class-based components to functional components with hooks * Migrates lifecycle methods to useEffect hooks * Transforms state management to useState hooks * Updates prop types and component patterns **Legacy Layout to Responsive Design** * Converts fixed layouts to responsive, mobile-first designs * Updates CSS from table-based to flexbox/grid layouts * Implements modern responsive breakpoints * Adds accessibility improvements #### JavaScript modernization templates **jQuery to modern JavaScript** * Replaces jQuery selectors with modern DOM methods * Converts jQuery AJAX to fetch API * Updates event handling to modern patterns * Removes jQuery dependency and improves performance **Legacy JavaScript to ES6+** * Converts var to let/const * Updates function declarations to arrow functions * Implements destructuring and template literals * Adds modern module imports/exports ### Without using templates #### 1. Codebase analysis Start by describing what you want to refactor: ``` "Refactor our React application to use modern patterns: - Convert all class components to functional components with hooks - Replace lifecycle methods with appropriate hooks - Update state management from this.setState to useState - Maintain all existing functionality and prop interfaces" ``` **What runbooks does:** * Scans your entire codebase for refactoring opportunities * Identifies all class components and their complexity * Analyzes lifecycle method usage and state patterns * Creates a comprehensive refactoring plan #### 2. Safety analysis Before refactoring, Runbooks performs safety checks: * **Dependency Analysis**: Identifies which components depend on others * **Test Coverage Review**: Ensures adequate testing for safe refactoring * **Breaking Change Detection**: Identifies potential breaking changes * **Risk Assessment**: Evaluates complexity and potential issues #### 3. Incremental refactoring plan Runbooks creates a step-by-step plan: 1. **Start with Leaf Components**: Begin with components that have no dependencies 2. **Work Up the Tree**: Gradually refactor parent components 3. **Preserve Interfaces**: Maintain existing prop and callback interfaces 4. **Validate Each Step**: Run tests after each component refactoring #### 4. Automated implementation The AI executes refactoring systematically: * Updates one component at a time * Maintains git history for easy rollback * Runs tests after each change * Creates detailed pull requests with explanations ### Real-world refactoring examples #### Example 1: Component modernization **Before refactoring:** ```javascript class UserProfile extends React.Component { constructor(props) { super(props); this.state = { loading: true, user: null, error: null }; } componentDidMount() { this.fetchUser(); } componentDidUpdate(prevProps) { if (prevProps.userId !== this.props.userId) { this.fetchUser(); } } fetchUser = async () => { try { this.setState({ loading: true }); const user = await api.getUser(this.props.userId); this.setState({ user, loading: false }); } catch (error) { this.setState({ error, loading: false }); } } render() { const { loading, user, error } = this.state; // render logic... } } ``` **After runbooks refactoring:** ```javascript const UserProfile = ({ userId }) => { const [loading, setLoading] = useState(true); const [user, setUser] = useState(null); const [error, setError] = useState(null); const fetchUser = useCallback(async () => { try { setLoading(true); const userData = await api.getUser(userId); setUser(userData); } catch (err) { setError(err); } finally { setLoading(false); } }, [userId]); useEffect(() => { fetchUser(); }, [fetchUser]); // render logic... }; ``` #### Example 2: State management refactoring **Legacy Redux Pattern:** ``` "Refactor our Redux implementation to use Redux Toolkit: - Convert action creators to createSlice - Update reducers to use Immer - Implement RTK Query for API calls - Maintain existing component interfaces" ``` **Runbook Implementation:** 1. **Slice Creation**: Convert traditional Redux files to RTK slices 2. **Store Migration**: Update store configuration to use RTK 3. **Component Updates**: Update components to use new hooks 4. **API Integration**: Replace manual API calls with RTK Query 5. **Type Safety**: Add TypeScript types for better developer experience #### Example 3: CSS architecture refactoring **From global CSS to CSS modules:** ``` "Refactor our CSS architecture: - Convert global styles to CSS modules - Implement consistent naming conventions - Remove unused CSS rules - Add CSS custom properties for theming" ``` **Implementation steps:** 1. **Analysis**: Identify all CSS files and their usage patterns 2. **Modularization**: Convert global styles to component-specific modules 3. **Cleanup**: Remove unused styles and consolidate duplicates 4. **Theming**: Extract colors and spacing to CSS custom properties 5. **Documentation**: Update style guides and component documentation *** ### See also * [Improving readability](/runbooks/concepts/use-cases/improving-readability) * [Code migrations](/runbooks/concepts/use-cases/code-migrations) # Bug fixes Bug fixing with Runbooks involves using AI to analyze error patterns, understand root causes, and implement comprehensive fixes across your codebase. Unlike manual bug fixing that addresses issues one at a time, Runbooks can identify patterns and fix similar bugs systematically across multiple files and repositories. ### When to use #### Ideal scenarios * **Systematic bug patterns**: When the same bug appears in multiple places * **Legacy vode issues**: Fixing bugs in poorly documented legacy code * **Security vulnerabilities**: Addressing security issues consistently across the codebase * **Performance bugs**: Identifying and fixing performance bottlenecks * **Cross-repository issues**: Bugs that span multiple services or repositories #### Perfect for teams that * Need to fix bugs consistently across large codebases * Want to prevent similar bugs from reoccurring * Have limited time for manual bug investigation * Need to ensure comprehensive fixes without missing edge cases ### Two approaches #### 1. Template-based bug fixing Use pre-built templates for common bug patterns from the [runbooks library](/runbooks/how-to-guides/managing-templates). **Common Bug Fix Templates Available:** * **Memory leak prevention**: Fixes event listener cleanup and reference management * **SQL injection prevention**: Updates database queries to use parameterized statements * **XSS vulnerability fixes**: Sanitizes user input and output across applications * **Race condition resolution**: Implements proper synchronization and state management * **Null pointer exception fixes**: Adds defensive programming patterns #### 2. One-Shot Bug Fixing (without Templates) For unique or specific bugs, use one-shot mode to get immediate diagnosis and fixes. ### One-Shot Bug Fixing Deep Dive [One-shot mode](/runbooks/how-to-guides/one-shot-mode) is perfect for urgent bug fixes and unique issues that don't fit standard templates. Here's how it works: #### How One-Shot Mode Works **Immediate Analysis and Fix:** 1. **Direct Problem Description**: Describe the bug you're experiencing 2. **Instant AI Analysis**: AI immediately analyzes your codebase for the issue 3. **Root Cause Identification**: Identifies the underlying cause and related issues 4. **Comprehensive Fix**: Implements a complete solution including edge cases 5. **Automatic Execution**: Runs the fix immediately without requiring step-by-step approval #### When to Use One-Shot Mode **Production Issues:** ``` "Our checkout process is failing with a 500 error when users have more than 10 items in their cart. The error logs show 'Cannot read property length of undefined' in the order calculation service." ``` **Performance Emergencies:** ``` "Our application memory usage keeps growing until it crashes. It seems related to WebSocket connections not being properly cleaned up when users disconnect." ``` **Security Incidents:** ``` "We discovered that user input in our comment system isn't being properly sanitized, potentially allowing XSS attacks. We need to fix this immediately across all user input points." ``` #### One-Shot Example: Memory Leak Fix **Problem Description:** ``` "Our React application has a memory leak. Components are not cleaning up event listeners and timers when they unmount, causing memory usage to grow continuously." ``` **One-Shot AI Analysis:** 1. **Codebase scan**: Scans all React components for event listeners and timers 2. **Pattern detection**: Identifies components missing cleanup in useEffect 3. **Impact assessment**: Determines which components are most problematic 4. **Fix generation**: Creates comprehensive cleanup implementation **Immediate Fix Implementation:** ```javascript // Before: Memory leak useEffect(() => { window.addEventListener('resize', handleResize); const timer = setInterval(updateData, 1000); }, []); // After: Proper cleanup useEffect(() => { const handleResize = () => { /* handler logic */ }; window.addEventListener('resize', handleResize); const timer = setInterval(updateData, 1000); return () => { window.removeEventListener('resize', handleResize); clearInterval(timer); }; }, []); ``` **One-Shot benefits:** * **Speed**: Immediate diagnosis and fix without needing any user interactions * **Comprehensive**: Fixes all instances of the pattern, not just the reported case * **Prevention**: Adds safeguards to prevent similar issues in the future * **In Slack context:** Can be triggered from Slack in-context ### Real-World Bug Fixing Examples #### Example 1: Cross-Browser Compatibility Bug **Problem report:** ``` "Our drag and drop functionality works in Chrome but fails in Safari and Firefox. Users report that items don't drop correctly and sometimes disappear." ``` **Root cause analysis:** 1. **Browser API differences**: Identifies Safari-specific event handling requirements 2. **Polyfill requirements**: Determines which polyfills are needed 3. **Event model differences**: Finds inconsistencies in drag/drop event handling 4. **Testing strategy**: Creates cross-browser testing approach **Systematic fix:** ```javascript // Before: Chrome-only implementation element.addEventListener('drop', (e) => { const data = e.dataTransfer.getData('text'); // Chrome-specific handling }); // After: Cross-browser compatible element.addEventListener('drop', (e) => { e.preventDefault(); const data = e.dataTransfer.getData('text/plain') || e.dataTransfer.getData('text'); // Safari fallback handling if (!data && e.dataTransfer.files.length > 0) { // Handle file drop for Safari } // Cross-browser drop handling }); ``` #### Example 2: State Management Bug **Problem report:** ``` "Users are experiencing stale data in our dashboard. When they update settings in one component, other components don't reflect the changes until they refresh the page." ``` **Root cause analysis:** 1. **State flow mapping**: Maps how data flows through the application 2. **Update pattern analysis**: Identifies where state updates aren't propagating 3. **Cache invalidation issues**: Finds stale caching problems 4. **Race condition detection**: Identifies timing-related state issues **Systematic fix:** * Implements proper state invalidation * Adds reactive data updates * Fixes cache management * Includes state debugging tools #### Example 3: API error handling bug **Problem report:** ``` "Our application crashes when the API returns unexpected error formats. Some services return errors differently, and our error handling doesn't account for all variations." ``` **Systematic fix:** 1. **Error Format Analysis**: Catalogs all possible error response formats 2. **Handler Standardization**: Creates consistent error handling patterns 3. **Fallback Implementation**: Adds graceful degradation for unknown errors 4. **Monitoring Integration**: Adds error tracking and alerting *** ### See also * [Improving readability](/runbooks/concepts/use-cases/improving-readability) * [Improving code coverage](/runbooks/concepts/use-cases/improving-code-coverage) # Code migrations Code migrations can be of a few flavors - upgrading a library (e.g. SQLAlchemy 1.4 to 2.0 in python), upgrading a language framework (Java 8 to Java 17) or migrating to a completely new framework (Ember to React). Typically large migrations can require multiple teams collaboration and may be spread over a huge amount of codebase. Using Runbooks these migrations can be planned out with all teams involved and executed step by step while ensuring compatibility. Remote Agents framework also comes pre-bundled with a [library of Runbook templates](mailto:undefined) that can be used as a starting point while adding more context of the nuances associated with your own codebase. ### When to use #### Ideal scenarios * **Framework upgrades**: Moving to newer versions of React, Angular, Vue, or other frameworks * **Language migrations**: Converting from one programming language to another * **Architecture evolution**: Moving from monolith to microservices or updating patterns * **Technology stack modernization**: Updating to modern tools and best practices * **Platform migrations**: Moving between cloud platforms or infrastructure patterns #### Perfect for teams that * Need to upgrade legacy applications to modern frameworks * Want to adopt new technologies without rewriting everything from scratch * Have tight deadlines for technology updates * Need to maintain functionality during migration * Want to minimize migration risks and ensure consistency ### Pre-built migration templates #### Frontend framework migrations examples **React Migration Templates** * **React 16 to 18**: Updates React APIs, lifecycle methods, and new features * **Class components to hooks**: Converts class-based components to functional components * **Enzyme to react testing library**: Updating the rendering and quering method * **PropTypes to TypeScript**: Adds static type checking with TypeScript **Angular Migration Templates** * **Angular 12 to 15**: Handles breaking changes and new API adoptions * **AngularJS to Angular**: Complete framework migration with modern patterns * **Angular Forms migration**: Updates to reactive forms and validation patterns * **Angular Router updates**: Migrates routing configurations and guards **Vue Migration Templates** * **Vue 2 to Vue 3**: Comprehensive migration including Composition API * **Options API to Composition API**: Modernizes component structure * **Vuex to Pinia**: State management migration to modern patterns * **Vue Router updates**: Updates routing for Vue 3 compatibility #### Backend and language migrations **Java migration templates** * **Java 8 to 11**: Updates language features and deprecated APIs * **Java 11 to 17**: Adopts new language features and performance improvements * **Java 17 to 21**: Latest language features and virtual threads * **Spring Boot 2.x to 3.x**: Framework upgrade with configuration updates **Node.js migration templates** * **Node.js 14 to 18**: Runtime updates and new API adoption * **Node.js 16 to 20**: Performance improvements and feature updates * **Express.js upgrades**: Framework version migrations with middleware updates * **CommonJS to ES modules**: Module system modernization **Python migration templates** * **Python 2.7 to 3.x**: Complete language migration with syntax updates * **Django upgrades**: Framework version migrations with ORM updates * **Flask modernization**: Updates to modern Flask patterns and features * **Async/Await migration**: Converts callback patterns to modern async syntax #### Infrastructure and platform migrations **Cloud platform migrations** * **File storage migration**: Migration to AWS, Azure, or Google Cloud * **Docker Optimization**: Container best practices and multi-stage builds * **CI/CD Platform Migration**: Jenkins to GitHub Actions, Travis to GitLab CI **Database migration templates** * **MySQL 5.7 to 8.0**: Database version upgrades with compatibility fixes * **PostgreSQL Upgrades**: Version migrations with feature adoption * **MongoDB Updates**: NoSQL database version and API migrations ### Migration without a template #### 1. Migration assessment and planning Start by describing your migration goals: ``` "We need to migrate our React 16 application to React 18. The application includes: - 150+ components using class-based patterns - Legacy Context API implementation - Deprecated lifecycle methods - PropTypes for type checking - We want to adopt React 18 features while maintaining all functionality" ``` **What Runbooks does:** * Analyzes your entire codebase to understand current patterns * Identifies all components that need migration * Maps dependencies and potential breaking changes * Creates a comprehensive migration plan with risk assessment #### 2. Compatibility analysis Runbooks performs deep compatibility analysis: * **Breaking changes**: Identifies APIs that have changed or been removed * **Dependency updates**: Analyzes third-party library compatibility * **Feature adoption**: Recommends new features to adopt during migration * **Risk assessment**: Evaluates potential issues and mitigation strategies #### 3. Incremental migration strategy The AI creates a step-by-step migration plan: * **Infrastructure Updates**: Update build tools, dependencies, and configuration * **Core API Migration**: Update fundamental framework APIs and patterns * **Component Migration**: Convert components in dependency order * **Feature Enhancement**: Adopt new framework features and improvements * **Testing and Validation**: Comprehensive testing at each stage #### 4. Automated implementation Runbooks executes migration systematically: * Updates one component or module at a time * Maintains functionality throughout the process * Runs tests after each change to ensure stability * Creates detailed pull requests explaining each migration step ### Real-world migration examples #### Example 1: React Class to Hooks Migration **Before Migration:** ```javascript class UserProfile extends React.Component { constructor(props) { super(props); this.state = { user: null, loading: true, error: null }; } async componentDidMount() { try { const user = await api.getUser(this.props.userId); this.setState({ user, loading: false }); } catch (error) { this.setState({ error, loading: false }); } } componentDidUpdate(prevProps) { if (prevProps.userId !== this.props.userId) { this.fetchUser(); } } fetchUser = async () => { this.setState({ loading: true }); try { const user = await api.getUser(this.props.userId); this.setState({ user, loading: false }); } catch (error) { this.setState({ error, loading: false }); } } render() { const { user, loading, error } = this.state; if (loading) return ; if (error) return ; return ; } } ``` **After Runbooks Migration:** ```javascript import { useState, useEffect, useCallback } from 'react'; const UserProfile = ({ userId }) => { const [user, setUser] = useState(null); const [loading, setLoading] = useState(true); const [error, setError] = useState(null); const fetchUser = useCallback(async () => { setLoading(true); setError(null); try { const userData = await api.getUser(userId); setUser(userData); } catch (err) { setError(err); } finally { setLoading(false); } }, [userId]); useEffect(() => { fetchUser(); }, [fetchUser]); if (loading) return ; if (error) return ; return ; }; export default UserProfile; ``` #### Example 2: Python 2 to Python 3 Migration **Before Migration:** ```python # Python 2 syntax and patterns import urllib2 import ConfigParser class DataProcessor: def __init__(self, config_file): self.config = ConfigParser.ConfigParser() self.config.read(config_file) def fetch_data(self, url): response = urllib2.urlopen(url) data = response.read() return unicode(data, 'utf-8') def process_items(self, items): for item in items: print "Processing:", item yield item.upper() def get_keys(self, data_dict): return data_dict.keys() ``` **After Runbooks Migration:** ```python # Python 3 syntax and patterns import urllib.request import configparser class DataProcessor: def __init__(self, config_file): self.config = configparser.ConfigParser() self.config.read(config_file) def fetch_data(self, url): with urllib.request.urlopen(url) as response: data = response.read() return data.decode('utf-8') def process_items(self, items): for item in items: print("Processing:", item) yield item.upper() def get_keys(self, data_dict): return list(data_dict.keys()) ``` #### Example 3: Angular 12 to 15 Migration **Before Migration:** ```typescript // Angular 12 patterns import { Component, OnInit } from '@angular/core'; import { FormBuilder, FormGroup, Validators } from '@angular/forms'; @Component({ selector: 'app-user-form', template: `
` }) export class UserFormComponent implements OnInit { userForm: FormGroup; constructor(private fb: FormBuilder) {} ngOnInit() { this.userForm = this.fb.group({ name: ['', Validators.required], email: ['', [Validators.required, Validators.email]] }); } onSubmit() { if (this.userForm.valid) { console.log(this.userForm.value); } } } ``` **After Runbooks Migration:** ```typescript // Angular 15 with modern patterns import { Component, inject } from '@angular/core'; import { FormBuilder, Validators, ReactiveFormsModule } from '@angular/forms'; import { CommonModule } from '@angular/common'; @Component({ selector: 'app-user-form', standalone: true, imports: [CommonModule, ReactiveFormsModule], template: `
` }) export class UserFormComponent { private fb = inject(FormBuilder); userForm = this.fb.nonNullable.group({ name: ['', Validators.required], email: ['', [Validators.required, Validators.email]] }); onSubmit() { if (this.userForm.valid) { const formValue = this.userForm.getRawValue(); console.log(formValue); } } } ``` ### Advanced migration scenarios #### Monolith to microservices migration **Large-scale architecture migration:** ``` "Migrate our monolithic application to microservices architecture: - Extract user management into a separate service - Create API gateways for service communication - Implement distributed tracing and monitoring - Maintain data consistency across services" ``` **Migration strategy:** 1. **Service boundary analysis**: Identify logical service boundaries 2. **API design**: Create clean interfaces between services 3. **Data migration**: Handle database splitting and consistency 4. **Infrastructure setup**: Container orchestration and service discovery 5. **Gradual cutover**: Incremental migration with feature flags #### Cross-platform migration **Technology stack migration:** ``` "Migrate our .NET Framework application to .NET Core for cloud deployment: - Update framework-specific APIs to .NET Core equivalents - Migrate Entity Framework to EF Core - Update deployment configuration for containerization - Maintain Windows compatibility while adding Linux support" ``` **Comprehensive approach:** 1. **Compatibility assessment**: Identify .NET Core compatibility issues 2. **API migration**: Update framework-specific code 3. **Configuration updates**: Modernize configuration patterns 4. **Testing strategy**: Ensure cross-platform compatibility 5. **Deployment modernization**: Container and cloud-ready deployment #### Legacy system modernization **Complete technology refresh:** ``` "Modernize our legacy jQuery application to React: - Convert jQuery DOM manipulation to React components - Replace Ajax calls with modern fetch API and state management - Implement modern routing and navigation - Add TypeScript for type safety" ``` **Modernization steps:** 1. **Component identification**: Map jQuery code to React components 2. **State management**: Implement modern state patterns 3. **API integration**: Replace jQuery Ajax with modern patterns 4. **UI modernization**: Update styling and user experience 5. **Testing implementation**: Add comprehensive test coverage *** ### See also * [Code refactoring](/runbooks/concepts/use-cases/code-refactoring) * [Product backlog](/runbooks/concepts/use-cases/product-backlog) # Improving code coverage Code coverage improvement with Runbooks involves using AI to analyze your existing test suite, identify coverage gaps, and automatically generate comprehensive tests to improve overall code coverage. Unlike manual test writing that can miss edge cases and be time-consuming, Runbooks can systematically ensure all code paths are tested with meaningful, maintainable test cases. ### When to use #### Ideal scenarios * Low test coverage * Legacy code testing * Critical path validation * Regression prevention * Ci/cd quality gates #### Perfect for teams that * Need to meet specific code coverage targets for compliance or quality * Have legacy codebases with minimal test coverage * Want to improve confidence in deployments and releases * Need to add tests quickly without sacrificing quality * Struggle with writing comprehensive edge case tests ### Code coverage improvement strategies #### Available coverage enhancement templates **Unit test generation** * Generates comprehensive unit tests for untested functions and methods * Creates tests for edge cases and boundary conditions * Implements proper mocking and isolation patterns * Adds parameterized tests for multiple input scenarios **Integration test creation** * Builds integration tests for component interactions * Tests API endpoints with various input combinations * Validates database operations and data flow * Creates end-to-end workflow testing **Edge case and error handling tests** * Identifies and tests error conditions and exception paths * Creates tests for boundary values and invalid inputs * Validates error messages and recovery mechanisms * Tests timeout and resource constraint scenarios **Regression test suite development** * Converts bug reports into permanent regression tests * Creates tests that prevent specific issues from reoccurring * Builds comprehensive test suites for critical business logic * Implements property-based testing for complex scenarios ### Without using templates #### 1. Coverage analysis and gap identification Start by describing your coverage improvement goals: ``` "Our application has only 45% test coverage and we need to reach 80% before our next release. Key areas that need coverage include: - Payment processing logic (currently 20% covered) - User authentication and authorization (currently 60% covered) - Data validation and sanitization functions (currently 30% covered) - Error handling and recovery mechanisms (currently 15% covered) - API endpoints and their various response scenarios" ``` **What Runbooks does:** * Analyzes your current test suite and coverage reports * Identifies specific functions, classes, and code paths lacking coverage * Maps critical business logic that requires comprehensive testing * Creates a prioritized plan focusing on high-risk, low-coverage areas #### 2. Coverage gap analysis Runbooks performs comprehensive analysis: * **Line coverage** * **Branch coverage** * **Function coverage** * **Edge case coverage** * **Integration points** #### 3. Test generation strategy The AI creates a systematic test generation plan: 1. **Critical path coverage** 2. **Edge case generation** 3. **Integration testing** 4. **Regression prevention** 5. **Maintenance strategy** #### 4. Implementation and validation Runbooks implements coverage improvements systematically: * Generates tests that follow existing project patterns and conventions * Creates meaningful test names and clear assertions * Implements proper setup and teardown procedures * Validates that new tests actually improve meaningful coverage ### Real-world code coverage examples #### Example 1: Payment processing coverage **Uncovered payment function:** ```javascript // Original function with no test coverage function processPayment(amount, paymentMethod, userAccount) { if (amount <= 0) { throw new Error('Amount must be positive'); } if (!paymentMethod || !paymentMethod.isValid) { throw new Error('Invalid payment method'); } if (userAccount.balance < amount && paymentMethod.type !== 'credit') { throw new Error('Insufficient funds'); } const transaction = { id: generateTransactionId(), amount: amount, method: paymentMethod.type, timestamp: new Date(), status: 'pending' }; if (amount > 10000) { transaction.requiresApproval = true; notifyAdministrators(transaction); } const result = chargePaymentMethod(paymentMethod, amount); if (result.success) { transaction.status = 'completed'; updateUserBalance(userAccount, amount, paymentMethod.type); } else { transaction.status = 'failed'; transaction.errorCode = result.errorCode; } saveTransaction(transaction); return transaction; } ``` **Generated comprehensive test suite:** ```javascript describe('processPayment', () => { let mockPaymentMethod; let mockUserAccount; beforeEach(() => { mockPaymentMethod = { type: 'credit', isValid: true }; mockUserAccount = { id: 'user123', balance: 1000 }; // Mock external dependencies jest.spyOn(global, 'generateTransactionId').mockReturnValue('tx123'); jest.spyOn(global, 'chargePaymentMethod').mockResolvedValue({ success: true }); jest.spyOn(global, 'updateUserBalance').mockImplementation(); jest.spyOn(global, 'saveTransaction').mockImplementation(); jest.spyOn(global, 'notifyAdministrators').mockImplementation(); }); afterEach(() => { jest.restoreAllMocks(); }); describe('input validation', () => { test('throws error for zero amount', () => { expect(() => processPayment(0, mockPaymentMethod, mockUserAccount)) .toThrow('Amount must be positive'); }); test('throws error for negative amount', () => { expect(() => processPayment(-100, mockPaymentMethod, mockUserAccount)) .toThrow('Amount must be positive'); }); test('throws error for null payment method', () => { expect(() => processPayment(100, null, mockUserAccount)) .toThrow('Invalid payment method'); }); test('throws error for invalid payment method', () => { mockPaymentMethod.isValid = false; expect(() => processPayment(100, mockPaymentMethod, mockUserAccount)) .toThrow('Invalid payment method'); }); }); describe('balance validation', () => { test('throws error for insufficient debit card funds', () => { mockPaymentMethod.type = 'debit'; mockUserAccount.balance = 50; expect(() => processPayment(100, mockPaymentMethod, mockUserAccount)) .toThrow('Insufficient funds'); }); test('allows credit card payment regardless of balance', () => { mockPaymentMethod.type = 'credit'; mockUserAccount.balance = 50; expect(() => processPayment(100, mockPaymentMethod, mockUserAccount)) .not.toThrow(); }); test('allows debit payment when balance is sufficient', () => { mockPaymentMethod.type = 'debit'; mockUserAccount.balance = 150; expect(() => processPayment(100, mockPaymentMethod, mockUserAccount)) .not.toThrow(); }); }); describe('high-value transaction handling', () => { test('requires approval for transactions over $10,000', () => { const result = processPayment(15000, mockPaymentMethod, mockUserAccount); expect(result.requiresApproval).toBe(true); expect(notifyAdministrators).toHaveBeenCalledWith( expect.objectContaining({ amount: 15000, requiresApproval: true }) ); }); test('does not require approval for transactions under $10,000', () => { const result = processPayment(5000, mockPaymentMethod, mockUserAccount); expect(result.requiresApproval).toBeUndefined(); expect(notifyAdministrators).not.toHaveBeenCalled(); }); test('requires approval for exactly $10,001', () => { const result = processPayment(10001, mockPaymentMethod, mockUserAccount); expect(result.requiresApproval).toBe(true); expect(notifyAdministrators).toHaveBeenCalled(); }); }); describe('payment processing', () => { test('creates completed transaction for successful payment', () => { chargePaymentMethod.mockResolvedValue({ success: true }); const result = processPayment(100, mockPaymentMethod, mockUserAccount); expect(result).toEqual({ id: 'tx123', amount: 100, method: 'credit', timestamp: expect.any(Date), status: 'completed' }); expect(updateUserBalance).toHaveBeenCalledWith( mockUserAccount, 100, 'credit' ); expect(saveTransaction).toHaveBeenCalledWith(result); }); test('creates failed transaction for unsuccessful payment', () => { chargePaymentMethod.mockResolvedValue({ success: false, errorCode: 'CARD_DECLINED' }); const result = processPayment(100, mockPaymentMethod, mockUserAccount); expect(result).toEqual({ id: 'tx123', amount: 100, method: 'credit', timestamp: expect.any(Date), status: 'failed', errorCode: 'CARD_DECLINED' }); expect(updateUserBalance).not.toHaveBeenCalled(); expect(saveTransaction).toHaveBeenCalledWith(result); }); }); describe('edge cases', () => { test('handles very small amounts correctly', () => { const result = processPayment(0.01, mockPaymentMethod, mockUserAccount); expect(result.amount).toBe(0.01); expect(result.status).toBe('completed'); }); test('handles maximum safe integer amount', () => { const maxAmount = Number.MAX_SAFE_INTEGER; const result = processPayment(maxAmount, mockPaymentMethod, mockUserAccount); expect(result.amount).toBe(maxAmount); expect(result.requiresApproval).toBe(true); }); test('preserves payment method type in transaction', () => { mockPaymentMethod.type = 'paypal'; const result = processPayment(100, mockPaymentMethod, mockUserAccount); expect(result.method).toBe('paypal'); }); }); }); ``` #### Example 2: Error handling coverage **Uncovered error handling logic:** ```python # API endpoint with poor error handling coverage def create_user_account(username, email, password): try: # Validate input if not username or len(username) < 3: return {'error': 'Username must be at least 3 characters'}, 400 if not email or '@' not in email: return {'error': 'Invalid email address'}, 400 if not password or len(password) < 8: return {'error': 'Password must be at least 8 characters'}, 400 # Check if user exists existing_user = database.find_user(email=email) if existing_user: return {'error': 'Email already registered'}, 409 # Create user user_data = { 'username': username, 'email': email, 'password': hash_password(password), 'created_at': datetime.utcnow(), 'active': True } user_id = database.create_user(user_data) send_welcome_email(email, username) return {'user_id': user_id, 'message': 'Account created successfully'}, 201 except DatabaseConnectionError: logger.error('Database connection failed during user creation') return {'error': 'Service temporarily unavailable'}, 503 except EmailServiceError: # User was created but email failed - that's ok logger.warning(f'Welcome email failed for user {username}') return {'user_id': user_id, 'message': 'Account created successfully'}, 201 except Exception as e: logger.error(f'Unexpected error in create_user_account: {str(e)}') return {'error': 'Internal server error'}, 500 ``` **Generated error handling test suite:** ```python import pytest from unittest.mock import patch, MagicMock from datetime import datetime class TestCreateUserAccount: def setup_method(self): """Setup test fixtures before each test.""" self.valid_username = "testuser" self.valid_email = "test@example.com" self.valid_password = "password123" def test_successful_user_creation(self): """Test successful user creation with all valid inputs.""" with patch('database.find_user', return_value=None), \ patch('database.create_user', return_value='user123'), \ patch('send_welcome_email') as mock_email: result, status = create_user_account( self.valid_username, self.valid_email, self.valid_password ) assert status == 201 assert result['user_id'] == 'user123' assert result['message'] == 'Account created successfully' mock_email.assert_called_once_with(self.valid_email, self.valid_username) # Username validation tests def test_empty_username_returns_error(self): """Test that empty username returns validation error.""" result, status = create_user_account("", self.valid_email, self.valid_password) assert status == 400 assert result['error'] == 'Username must be at least 3 characters' def test_short_username_returns_error(self): """Test that username shorter than 3 characters returns error.""" result, status = create_user_account("ab", self.valid_email, self.valid_password) assert status == 400 assert result['error'] == 'Username must be at least 3 characters' def test_minimum_valid_username_length(self): """Test that 3-character username is accepted.""" with patch('database.find_user', return_value=None), \ patch('database.create_user', return_value='user123'), \ patch('send_welcome_email'): result, status = create_user_account("abc", self.valid_email, self.valid_password) assert status == 201 # Email validation tests def test_empty_email_returns_error(self): """Test that empty email returns validation error.""" result, status = create_user_account(self.valid_username, "", self.valid_password) assert status == 400 assert result['error'] == 'Invalid email address' def test_email_without_at_symbol_returns_error(self): """Test that email without @ symbol returns validation error.""" result, status = create_user_account( self.valid_username, "invalid-email", self.valid_password ) assert status == 400 assert result['error'] == 'Invalid email address' def test_none_email_returns_error(self): """Test that None email returns validation error.""" result, status = create_user_account(self.valid_username, None, self.valid_password) assert status == 400 assert result['error'] == 'Invalid email address' # Password validation tests def test_empty_password_returns_error(self): """Test that empty password returns validation error.""" result, status = create_user_account(self.valid_username, self.valid_email, "") assert status == 400 assert result['error'] == 'Password must be at least 8 characters' def test_short_password_returns_error(self): """Test that password shorter than 8 characters returns error.""" result, status = create_user_account(self.valid_username, self.valid_email, "short") assert status == 400 assert result['error'] == 'Password must be at least 8 characters' def test_minimum_valid_password_length(self): """Test that 8-character password is accepted.""" with patch('database.find_user', return_value=None), \ patch('database.create_user', return_value='user123'), \ patch('send_welcome_email'): result, status = create_user_account( self.valid_username, self.valid_email, "12345678" ) assert status == 201 # Duplicate email tests def test_existing_email_returns_conflict_error(self): """Test that existing email returns 409 conflict error.""" existing_user = {'id': 'existing123', 'email': self.valid_email} with patch('database.find_user', return_value=existing_user): result, status = create_user_account( self.valid_username, self.valid_email, self.valid_password ) assert status == 409 assert result['error'] == 'Email already registered' # Database error handling tests def test_database_connection_error_returns_503(self): """Test that database connection error returns 503 service unavailable.""" with patch('database.find_user', side_effect=DatabaseConnectionError()): result, status = create_user_account( self.valid_username, self.valid_email, self.valid_password ) assert status == 503 assert result['error'] == 'Service temporarily unavailable' def test_database_error_during_creation_returns_503(self): """Test database error during user creation returns 503.""" with patch('database.find_user', return_value=None), \ patch('database.create_user', side_effect=DatabaseConnectionError()): result, status = create_user_account( self.valid_username, self.valid_email, self.valid_password ) assert status == 503 assert result['error'] == 'Service temporarily unavailable' # Email service error handling tests def test_email_service_error_still_creates_user(self): """Test that email service error doesn't prevent user creation.""" with patch('database.find_user', return_value=None), \ patch('database.create_user', return_value='user123'), \ patch('send_welcome_email', side_effect=EmailServiceError()): result, status = create_user_account( self.valid_username, self.valid_email, self.valid_password ) assert status == 201 assert result['user_id'] == 'user123' assert result['message'] == 'Account created successfully' # General exception handling tests def test_unexpected_exception_returns_500(self): """Test that unexpected exceptions return 500 internal server error.""" with patch('database.find_user', side_effect=ValueError("Unexpected error")): result, status = create_user_account( self.valid_username, self.valid_email, self.valid_password ) assert status == 500 assert result['error'] == 'Internal server error' # Edge case tests def test_unicode_username_handling(self): """Test that unicode characters in username are handled correctly.""" unicode_username = "tëst_üser" with patch('database.find_user', return_value=None), \ patch('database.create_user', return_value='user123'), \ patch('send_welcome_email'): result, status = create_user_account( unicode_username, self.valid_email, self.valid_password ) assert status == 201 def test_very_long_inputs_handling(self): """Test handling of very long input strings.""" long_username = "a" * 1000 long_password = "b" * 1000 with patch('database.find_user', return_value=None), \ patch('database.create_user', return_value='user123'), \ patch('send_welcome_email'): result, status = create_user_account( long_username, self.valid_email, long_password ) assert status == 201 ``` *** ### See also * [Improve build times](/runbooks/concepts/use-cases/improve-build-times) * [Bug fixes](/runbooks/concepts/use-cases/bug-fixes) # Improving readability Code readability improvement with Runbooks involves using AI to analyze your codebase for readability issues and systematically applying improvements that make code easier to understand, maintain, and extend. Unlike manual code cleanup that happens inconsistently, Runbooks can apply readability improvements across your entire codebase following consistent standards and best practices. ### When to use #### Ideal scenarios * Legacy code cleanup * Code standardization * Onboarding optimization * Documentation debt * Refactoring for clarity #### Perfect for teams that * Struggle with code maintenance due to poor readability * Have inconsistent coding standards across the codebase * Need to onboard new developers quickly * Want to reduce time spent understanding existing code * Have accumulated technical debt in code quality ### Available readability improvement templates #### Code structure and organization templates **Function and method decomposition** * Breaks large, complex functions into smaller, focused units * Implements single responsibility principle * Adds clear function naming and documentation * Improves parameter organization and defaults **Class and module organization** * Restructures classes for better separation of concerns * Organizes methods and properties logically * Implements clear interface definitions * Adds comprehensive class documentation **File and directory structure** * Reorganizes files into logical groupings * Implements consistent naming conventions * Creates clear module boundaries * Adds index files for better imports #### Naming and documentation templates **Variable and function naming** * Converts cryptic variable names to descriptive ones * Implements consistent naming conventions * Removes abbreviated and unclear names * Adds type hints and documentation **API documentation generation** * Generates comprehensive API documentation * Adds docstrings and comments to all public methods * Creates usage examples and parameter descriptions * Implements documentation standards (JSDoc, Sphinx, etc.) **Code comments and explanations** * Adds explanatory comments for complex logic * Documents business rules and requirements * Explains algorithmic choices and trade-offs * Removes outdated or incorrect comments #### Formatting and style templates **Code formatting standardization** * Applies consistent indentation and spacing * Implements standard bracket and parentheses styles * Organizes imports and dependencies consistently * Removes trailing whitespace and empty lines **Modern language feature adoption** * Updates to modern syntax for better readability * Implements destructuring and template literals * Uses modern iteration and functional methods * Adopts cleaner async/await patterns ### Without using templates #### 1. Readability assessment Start by describing your readability challenges: ``` "Our codebase has grown organically over 5 years and has become difficult to maintain. Issues include: - Functions that are 200+ lines long with unclear purposes - Variable names like 'data', 'item', 'temp' throughout the code - Missing documentation for complex business logic - Inconsistent formatting and coding styles across files - Deep nesting that makes control flow hard to follow" ``` **What Runbooks does:** * Analyzes your codebase for complexity metrics and readability issues * Identifies functions and classes that need decomposition * Maps inconsistent naming patterns and style violations * Creates a prioritized improvement plan focusing on high-impact areas #### 2. Complexity analysis Runbooks performs comprehensive analysis: * Cyclomatic complexity * Nesting depth * Function length * Naming consistency * Documentation coverage #### 3. Systematic improvement strategy The AI creates a structured improvement plan: 1. Critical path cleanup 2. Naming standardization 3. Function decomposition 4. Documentation addition 5. Style standardization #### 4. Implementation and validation Runbooks implements improvements systematically: * Makes one type of improvement at a time for easy review * Preserves all functionality while improving readability * Runs tests to ensure behavior remains unchanged * Creates detailed documentation of improvements made ### Real-world readability improvement examples #### Example 1: Function decomposition and naming **Before improvement:** ```javascript // Cryptic function with unclear purpose and poor structure function processData(data) { let result = []; for (let i = 0; i < data.length; i++) { if (data[i].status === 'active' && data[i].balance > 0) { let item = { id: data[i].id, name: data[i].firstName + ' ' + data[i].lastName, total: data[i].balance * 1.08, category: data[i].balance > 1000 ? 'premium' : 'standard' }; if (data[i].lastLoginDate) { let daysSince = (Date.now() - new Date(data[i].lastLoginDate)) / (1000 * 60 * 60 * 24); if (daysSince < 30) { item.engagement = 'high'; } else if (daysSince < 90) { item.engagement = 'medium'; } else { item.engagement = 'low'; } } result.push(item); } } return result; } ``` **After Runbooks improvement:** ```javascript /** * Transforms raw user data into enriched customer profiles for reporting * @param {Array} users - Raw user data from the database * @returns {Array} Enriched customer profiles with calculated fields */ function createCustomerProfiles(users) { const activeUsersWithBalance = filterActiveUsers(users); return activeUsersWithBalance.map(user => createCustomerProfile(user)); } /** * Filters users to only include active users with positive balance * @param {Array} users - All users from the database * @returns {Array} Filtered list of active users with balance */ function filterActiveUsers(users) { return users.filter(user => user.status === 'active' && user.balance > 0 ); } /** * Creates an enriched customer profile from a user record * @param {User} user - Individual user record * @returns {CustomerProfile} Enriched profile with calculated fields */ function createCustomerProfile(user) { return { id: user.id, name: formatFullName(user), total: calculateTotalWithTax(user.balance), category: determineCustomerCategory(user.balance), engagement: calculateEngagementLevel(user.lastLoginDate) }; } /** * Formats user's first and last name into a full name * @param {User} user - User with firstName and lastName * @returns {string} Formatted full name */ function formatFullName(user) { return `${user.firstName} ${user.lastName}`; } /** * Calculates total amount including 8% tax * @param {number} balance - User's current balance * @returns {number} Balance with tax included */ function calculateTotalWithTax(balance) { const TAX_RATE = 0.08; return balance * (1 + TAX_RATE); } /** * Determines customer category based on balance * @param {number} balance - User's current balance * @returns {string} 'premium' for high-value customers, 'standard' otherwise */ function determineCustomerCategory(balance) { const PREMIUM_THRESHOLD = 1000; return balance > PREMIUM_THRESHOLD ? 'premium' : 'standard'; } /** * Calculates user engagement level based on last login date * @param {string|null} lastLoginDate - ISO date string of last login * @returns {string} Engagement level: 'high', 'medium', 'low', or 'unknown' */ function calculateEngagementLevel(lastLoginDate) { if (!lastLoginDate) { return 'unknown'; } const daysSinceLogin = calculateDaysSince(lastLoginDate); if (daysSinceLogin < 30) return 'high'; if (daysSinceLogin < 90) return 'medium'; return 'low'; } /** * Calculates the number of days between a date and now * @param {string} dateString - ISO date string * @returns {number} Number of days since the given date */ function calculateDaysSince(dateString) { const date = new Date(dateString); const now = Date.now(); const millisecondsPerDay = 1000 * 60 * 60 * 24; return (now - date.getTime()) / millisecondsPerDay; } ``` #### Example 2: Class structure and documentation **Before improvement:** ```python # Poorly organized class with unclear responsibilities class DataManager: def __init__(self, db_conn): self.db = db_conn self.cache = {} def get_data(self, id, type): if type == 'user': if id in self.cache: return self.cache[id] result = self.db.execute("SELECT * FROM users WHERE id = ?", (id,)) self.cache[id] = result return result elif type == 'order': result = self.db.execute("SELECT * FROM orders WHERE id = ?", (id,)) if result: user_data = self.get_data(result['user_id'], 'user') result['user_name'] = user_data['name'] return result def save_data(self, data, type): if type == 'user': self.db.execute("INSERT INTO users VALUES (?, ?, ?)", (data['id'], data['name'], data['email'])) self.cache[data['id']] = data elif type == 'order': self.db.execute("INSERT INTO orders VALUES (?, ?, ?)", (data['id'], data['user_id'], data['amount'])) ``` **After Runbooks improvement:** ```python from typing import Dict, Optional, Any from abc import ABC, abstractmethod class DatabaseRepository(ABC): """Abstract base class for database repositories with caching support.""" def __init__(self, database_connection): """Initialize repository with database connection.""" self._database = database_connection self._cache: Dict[Any, Dict] = {} @abstractmethod def _get_table_name(self) -> str: """Return the database table name for this repository.""" pass @abstractmethod def _create_insert_query(self, data: Dict) -> tuple: """Create INSERT query and parameters for the given data.""" pass class UserRepository(DatabaseRepository): """Repository for managing user data with caching support.""" def _get_table_name(self) -> str: return "users" def _create_insert_query(self, user_data: Dict) -> tuple: """Create INSERT query for user data.""" query = "INSERT INTO users (id, name, email) VALUES (?, ?, ?)" params = (user_data['id'], user_data['name'], user_data['email']) return query, params def get_user_by_id(self, user_id: int) -> Optional[Dict]: """ Retrieve user by ID with caching support. Args: user_id: The unique identifier for the user Returns: User data dictionary or None if not found """ if user_id in self._cache: return self._cache[user_id] query = "SELECT * FROM users WHERE id = ?" user_data = self._database.execute(query, (user_id,)) if user_data: self._cache[user_id] = user_data return user_data def save_user(self, user_data: Dict) -> None: """ Save user data to database and update cache. Args: user_data: Dictionary containing user information Must include 'id', 'name', and 'email' keys """ query, params = self._create_insert_query(user_data) self._database.execute(query, params) self._cache[user_data['id']] = user_data class OrderRepository(DatabaseRepository): """Repository for managing order data with user enrichment.""" def __init__(self, database_connection, user_repository: UserRepository): """Initialize order repository with user repository dependency.""" super().__init__(database_connection) self._user_repository = user_repository def _get_table_name(self) -> str: return "orders" def _create_insert_query(self, order_data: Dict) -> tuple: """Create INSERT query for order data.""" query = "INSERT INTO orders (id, user_id, amount) VALUES (?, ?, ?)" params = (order_data['id'], order_data['user_id'], order_data['amount']) return query, params def get_order_by_id(self, order_id: int) -> Optional[Dict]: """ Retrieve order by ID with user information enrichment. Args: order_id: The unique identifier for the order Returns: Order data dictionary with user_name field added, or None if not found """ query = "SELECT * FROM orders WHERE id = ?" order_data = self._database.execute(query, (order_id,)) if not order_data: return None return self._enrich_order_with_user_data(order_data) def _enrich_order_with_user_data(self, order_data: Dict) -> Dict: """Add user information to order data.""" user_data = self._user_repository.get_user_by_id(order_data['user_id']) if user_data: order_data['user_name'] = user_data['name'] return order_data def save_order(self, order_data: Dict) -> None: """ Save order data to database. Args: order_data: Dictionary containing order information Must include 'id', 'user_id', and 'amount' keys """ query, params = self._create_insert_query(order_data) self._database.execute(query, params) ``` #### Example 3: Complex logic simplification **Before improvement:** ```javascript // Complex nested logic that's hard to follow function calculatePrice(item, user, discount, shipping) { let price = item.basePrice; if (user.type === 'premium') { if (user.yearsActive > 5) { price = price * 0.85; } else if (user.yearsActive > 2) { price = price * 0.9; } else { price = price * 0.95; } } else if (user.type === 'regular') { if (user.yearsActive > 3) { price = price * 0.95; } } if (discount) { if (discount.type === 'percentage') { price = price * (1 - discount.value / 100); } else if (discount.type === 'fixed') { price = Math.max(0, price - discount.value); } } if (shipping.expedited && price > 100) { price += shipping.cost * 0.5; } else if (shipping.expedited) { price += shipping.cost; } else if (price < 50) { price += shipping.cost; } return Math.round(price * 100) / 100; } ``` **After Runbooks improvement:** ```javascript /** * Calculates the final price for an item including all discounts and shipping * @param {Item} item - Product item with base pricing * @param {User} user - Customer information for loyalty discounts * @param {Discount|null} discount - Optional discount to apply * @param {Shipping} shipping - Shipping options and costs * @returns {number} Final price rounded to 2 decimal places */ function calculateFinalPrice(item, user, discount, shipping) { let price = item.basePrice; price = applyLoyaltyDiscount(price, user); price = applyPromoDiscount(price, discount); price = addShippingCost(price, shipping); return roundToTwoDecimals(price); } /** * Applies loyalty discount based on user type and years active * @param {number} price - Current price * @param {User} user - User with type and yearsActive properties * @returns {number} Price after loyalty discount */ function applyLoyaltyDiscount(price, user) { const discountRate = getLoyaltyDiscountRate(user); return price * (1 - discountRate); } /** * Determines loyalty discount rate based on user type and tenure * @param {User} user - User information * @returns {number} Discount rate as decimal (0.15 = 15% discount) */ function getLoyaltyDiscountRate(user) { if (user.type === 'premium') { return getPremiumUserDiscountRate(user.yearsActive); } if (user.type === 'regular' && user.yearsActive > 3) { return 0.05; // 5% discount for long-term regular users } return 0; // No loyalty discount } /** * Calculates discount rate for premium users based on tenure * @param {number} yearsActive - Number of years user has been active * @returns {number} Discount rate as decimal */ function getPremiumUserDiscountRate(yearsActive) { if (yearsActive > 5) return 0.15; // 15% discount if (yearsActive > 2) return 0.10; // 10% discount return 0.05; // 5% discount } /** * Applies promotional discount if available * @param {number} price - Current price * @param {Discount|null} discount - Discount object or null * @returns {number} Price after promotional discount */ function applyPromoDiscount(price, discount) { if (!discount) { return price; } if (discount.type === 'percentage') { return price * (1 - discount.value / 100); } if (discount.type === 'fixed') { return Math.max(0, price - discount.value); } return price; // Unknown discount type, no change } /** * Adds shipping cost based on order value and shipping options * @param {number} price - Current order total * @param {Shipping} shipping - Shipping configuration * @returns {number} Price including shipping */ function addShippingCost(price, shipping) { const qualifiesForFreeShipping = price >= 50 && !shipping.expedited; if (qualifiesForFreeShipping) { return price; } const shippingCost = calculateShippingCost(price, shipping); return price + shippingCost; } /** * Calculates shipping cost based on order value and expedited option * @param {number} price - Order total * @param {Shipping} shipping - Shipping options * @returns {number} Shipping cost */ function calculateShippingCost(price, shipping) { if (!shipping.expedited) { return shipping.cost; } // Expedited shipping discount for orders over $100 const expeditedDiscount = price > 100 ? 0.5 : 1.0; return shipping.cost * expeditedDiscount; } /** * Rounds a number to two decimal places * @param {number} value - Number to round * @returns {number} Number rounded to 2 decimal places */ function roundToTwoDecimals(value) { return Math.round(value * 100) / 100; } ``` *** ### See also * [Product backlog](/runbooks/concepts/use-cases/product-backlog) * [Code refactoring](/runbooks/concepts/use-cases/code-refactoring) # Flaky test resolution Flaky test resolution with Runbooks involves using AI to analyze test failure patterns, identify the underlying causes of intermittent failures, and implement systematic fixes across your test suite. Unlike manual debugging that addresses tests one by one, Runbooks can identify common patterns and fix entire categories of flaky tests simultaneously. ### When to use #### Ideal scenarios * Intermittent ci/cd failures * Timing-dependent test failures * Environment-specific failures * Large test suite instability * Developer productivity impact #### Perfect for teams that * Experience frequent "retry CI" requests due to flaky tests * Struggle with inconsistent test results across environments * Need to improve CI/CD pipeline reliability * Want to restore confidence in their test suite * Have limited time to manually debug intermittent test failures ### Common flaky test patterns and solutions #### Available flaky test resolution templates **Async operation timing issues** * Fixes tests that fail due to inadequate waiting for async operations * Implements proper wait conditions and timeout handling * Replaces arbitrary timeouts with condition-based waiting * Adds retry logic for network-dependent operations **Test data isolation problems** * Resolves tests that interfere with each other due to shared state * Implements proper test setup and teardown procedures * Adds database transaction isolation for database tests * Creates independent test data for each test case **UI test stabilization** * Fixes browser-based tests with timing and rendering issues * Implements proper element waiting strategies * Adds stable locator strategies resistant to UI changes * Handles dynamic content and loading states **Network and external service mocking** * Replaces unreliable external service calls with stable mocks * Implements proper network timeout and retry handling * Adds fallback strategies for service unavailability * Creates deterministic test environments ### Without using template #### 1. Flaky test analysis Start by describing the flaky test behavior: ``` "Our test suite has become unreliable with about 20% of CI runs failing due to flaky tests. The failures seem random and include: - API tests that sometimes timeout - UI tests that fail to find elements - Database tests that occasionally fail with constraint violations - Integration tests that pass locally but fail in CI" ``` **What Runbooks does:** * Analyzes test failure logs and patterns across multiple CI runs * Identifies common failure modes and their frequency * Maps failures to specific test categories and underlying causes * Creates a prioritized remediation plan based on impact and frequency #### 2. Root cause identification Runbooks performs deep analysis to identify: * Timing issues * State pollution * Environment dependencies * External dependencies * Resource constraints #### 3. Systematic fix implementation The AI creates and executes a comprehensive fix strategy: 1. Immediate stabilization 2. Pattern-based fixes 3. Infrastructure improvements 4. Prevention measures #### 4. Validation and monitoring Runbooks implements validation and monitoring: * Runs tests multiple times to verify stability improvements * Implements test reliability monitoring * Creates alerts for new flaky test patterns * Documents fix patterns for future reference ### Real-world flaky test resolution examples #### Example 1: async operation timing issues **Flaky test pattern:** ```javascript // Flaky: Sometimes fails due to timing test('user data loads correctly', async () => { render(); // ❌ Flaky: May not be enough time await waitFor(() => { expect(screen.getByText('John Doe')).toBeInTheDocument(); }, { timeout: 1000 }); }); ``` **After Runbooks fix:** ```javascript // Stable: Proper async handling test('user data loads correctly', async () => { // Mock API response for deterministic testing const mockUser = { id: '123', name: 'John Doe' }; jest.spyOn(api, 'getUser').mockResolvedValue(mockUser); render(); // ✅ Stable: Wait for specific condition, not arbitrary timeout await waitFor(() => { expect(screen.getByText('John Doe')).toBeInTheDocument(); }, { timeout: 10000, interval: 100 }); // Verify the API was called correctly expect(api.getUser).toHaveBeenCalledWith('123'); }); ``` #### Example 2: database test isolation **Flaky test pattern:** ```javascript // Flaky: Tests interfere with each other describe('User Management', () => { test('creates new user', async () => { const user = await createUser({ email: 'test@example.com' }); expect(user.email).toBe('test@example.com'); }); test('finds existing user', async () => { // ❌ Flaky: Depends on previous test state const user = await findUser('test@example.com'); expect(user).toBeDefined(); }); }); ``` **After Runbooks fix:** ```javascript // Stable: Proper test isolation describe('User Management', () => { let testUserId; beforeEach(async () => { // Clean slate for each test await cleanupTestData(); }); afterEach(async () => { // Cleanup after each test if (testUserId) { await deleteUser(testUserId); testUserId = null; } }); test('creates new user', async () => { const uniqueEmail = `test-${Date.now()}@example.com`; const user = await createUser({ email: uniqueEmail }); testUserId = user.id; expect(user.email).toBe(uniqueEmail); }); test('finds existing user', async () => { // ✅ Stable: Create own test data const uniqueEmail = `test-${Date.now()}@example.com`; const createdUser = await createUser({ email: uniqueEmail }); testUserId = createdUser.id; const foundUser = await findUser(uniqueEmail); expect(foundUser.id).toBe(createdUser.id); }); }); ``` #### Example 3: UI test stabilization **Flaky test pattern:** ```javascript // Flaky: Element may not be ready test('submits form successfully', async () => { await page.goto('/contact'); // ❌ Flaky: Button might not be clickable yet await page.click('#submit-button'); // ❌ Flaky: Success message might not appear immediately const successMessage = await page.textContent('.success-message'); expect(successMessage).toContain('Form submitted'); }); ``` **After Runbooks fix:** ```javascript // Stable: Proper waiting and error handling test('submits form successfully', async () => { await page.goto('/contact'); // ✅ Stable: Wait for page to be fully loaded await page.waitForLoadState('networkidle'); // Fill form fields with proper waiting await page.fill('[data-testid="name-input"]', 'John Doe'); await page.fill('[data-testid="email-input"]', 'john@example.com'); await page.fill('[data-testid="message-input"]', 'Test message'); // ✅ Stable: Wait for button to be enabled and visible const submitButton = page.locator('[data-testid="submit-button"]'); await submitButton.waitFor({ state: 'visible' }); await expect(submitButton).toBeEnabled(); // Submit form and wait for response await submitButton.click(); // ✅ Stable: Wait for success message with retry const successMessage = page.locator('[data-testid="success-message"]'); await expect(successMessage).toBeVisible({ timeout: 10000 }); await expect(successMessage).toContainText('Form submitted successfully'); }); ``` ### Advanced flaky test scenarios #### Network-dependent test stabilization **API integration test issues:** ``` "Our API integration tests are flaky due to: - Network timeouts in CI environment - External service availability issues - Rate limiting causing intermittent failures - Different response times in different environments" ``` **Comprehensive solution:** 1. Service mocking 2. Retry logic 3. Circuit breakers 4. Environment parity **Implementation example:** ```javascript // Stable API test with proper mocking and retry test('fetches user data with retry logic', async () => { // Mock external API for reliability const mockApiResponse = { id: 1, name: 'John Doe' }; nock('https://api.external-service.com') .get('/users/1') .reply(200, mockApiResponse); const user = await retryOperation( () => apiClient.getUser(1), { maxRetries: 3, retryDelay: 100 } ); expect(user).toEqual(mockApiResponse); }); ``` #### Parallel test execution issues **Race condition resolution:** ``` "Our tests fail when run in parallel due to: - Shared database records being modified simultaneously - Port conflicts for test servers - File system conflicts for temporary files - Shared cache or session state" ``` **Systematic fix approach:** 1. Resource isolation 2. Database partitioning 3. Port management 4. Temporary file isolation #### Browser test stabilization **Cross-browser flakiness:** ``` "Our browser tests exhibit different flaky behaviors across browsers: - Chrome: Timing issues with animations - Firefox: Element interaction problems - Safari: Different event handling behavior - Edge: Font rendering affecting layout tests" ``` **Browser-specific optimization:** 1. Browser configuration 2. Wait strategies 3. Event handling 4. Visual testing *** ### See also * [Improving code coverage](/runbooks/concepts/use-cases/improving-code-coverage) * [Bug fixes](/runbooks/concepts/use-cases/bug-fixes) # Improve build times Build time optimization with Runbooks involves systematically analyzing your build processes, identifying performance bottlenecks, and implementing proven optimization strategies. Unlike manual optimization that requires deep expertise in build tools, Runbooks can automatically detect inefficiencies and apply industry best practices across your entire build pipeline. ### When to use ### Ideal scenarios * **Slow Development Builds**: Local development builds taking too long * **CI/CD Pipeline Bottlenecks**: Deployment pipelines causing development delays * **Large Codebase Builds**: Monorepo or large applications with complex build requirements * **Multi-Environment Builds**: Different optimization needs for dev, staging, and production * **Team Productivity Issues**: Build times impacting developer workflow #### Perfect for teams that * Experience frequent build delays in development * Have complex build pipelines with multiple steps * Need to optimize CI/CD costs and execution time * Want to improve developer experience and productivity * Manage large codebases with growing build complexity ### Available build optimization templates #### Frontend build optimization templates **Webpack bundle optimization** * Implements code splitting at route and component levels * Adds dynamic imports for large dependencies * Optimizes vendor bundle splitting and caching * Implements tree shaking and dead code elimination **Docker build optimization** * Multi-stage build implementation for smaller images * Layer caching optimization for faster rebuilds * Build context optimization to reduce transfer times * Parallel build strategies for complex applications **TypeScript compilation optimization** * Incremental compilation setup * Project references for monorepo optimization * Build cache implementation and management * Parallel type checking and compilation #### CI/CD pipeline templates **Build artifact sharing** * Implements build artifact caching across pipeline stages * Optimizes artifact storage and retrieval * Reduces duplicate work in multi-stage pipelines * Implements smart invalidation strategies **Distributed build systems** * Sets up distributed compilation for large codebases * Implements build result sharing across developers * Optimizes resource allocation for build workers * Adds build time monitoring and optimization **Conditional build execution** * Implements change detection for selective builds * Adds path-based build triggers * Optimizes test execution based on code changes * Implements smart dependency analysis ### Without using template #### 1. Build performance analysis Start by describing your build performance issues: ``` "Our React application build takes 15 minutes in CI and 8 minutes locally. The webpack build seems to be the bottleneck, and our developers are frustrated with long feedback cycles. We need to optimize both local development and production builds." ``` **What Runbooks does:** * Analyzes your webpack configuration and build output * Identifies bundle size and dependency issues * Reviews build pipeline stages and timing * Creates a comprehensive optimization strategy #### 2. Bottleneck identification Runbooks performs deep analysis to identify: * **Bundle analysis**: Large dependencies and duplicate code * **Build stage timing**: Which stages take the most time * **Cache utilization**: Opportunities for improved caching * **Parallelization**: Tasks that can run concurrently * **Resource usage**: CPU, memory, and I/O bottlenecks #### 3. Optimization strategy planning The AI creates a prioritized optimization plan: * **Quick Wins**: Immediate improvements with minimal risk * **Medium-term Improvements**: More complex optimizations * **Long-term Architecture**: Fundamental build system improvements * **Monitoring Setup**: Tracking and maintaining optimization gains #### 4. Implementation and validation Runbooks implements optimizations systematically: * Makes one optimization at a time for easy rollback * Measures performance improvement after each change * Validates that functionality remains intact * Documents optimization techniques for future reference ### Real-world build optimization examples #### Example 1: Webpack bundle optimization **Before optimization:** ```javascript // Single large bundle taking 8 minutes to build module.exports = { entry: './src/index.js', output: { filename: 'bundle.js', path: path.resolve(__dirname, 'dist'), }, // No optimization configuration }; ``` **After runbooks optimization:** ```javascript module.exports = { entry: './src/index.js', output: { filename: '[name].[contenthash].js', path: path.resolve(__dirname, 'dist'), clean: true, }, optimization: { splitChunks: { chunks: 'all', cacheGroups: { vendor: { test: /[\\/]node_modules[\\/]/, name: 'vendors', chunks: 'all', }, common: { name: 'common', minChunks: 2, chunks: 'all', enforce: true, }, }, }, }, cache: { type: 'filesystem', buildDependencies: { config: [__filename], }, }, }; ``` **Results:** Build time reduced from 8 minutes to 2 minutes locally, 15 minutes to 4 minutes in CI. #### Example 2: Docker build optimization **Before optimization:** ```dockerfile FROM node:16 COPY . /app WORKDIR /app RUN npm install RUN npm run build CMD ["npm", "start"] ``` **After Runbooks optimization:** ```dockerfile # Multi-stage build for optimization FROM node:16-alpine AS dependencies WORKDIR /app COPY package*.json ./ RUN npm ci --only=production && npm cache clean --force FROM node:16-alpine AS build WORKDIR /app COPY package*.json ./ RUN npm ci COPY . . RUN npm run build FROM node:16-alpine AS runtime WORKDIR /app COPY --from=dependencies /app/node_modules ./node_modules COPY --from=build /app/dist ./dist COPY package*.json ./ EXPOSE 3000 CMD ["npm", "start"] ``` **Results:** Docker build time reduced from 12 minutes to 3 minutes, image size reduced by 60%. #### Example 3: CI/CD pipeline optimization **Before optimization:** ```yaml # Sequential pipeline taking 25 minutes jobs: test: runs-on: ubuntu-latest steps: - checkout: v2 - run: npm install - run: npm run test - run: npm run lint - run: npm run build - run: npm run e2e-tests ``` **After runbooks optimization:** ```yaml # Parallel pipeline with caching jobs: setup: runs-on: ubuntu-latest steps: - checkout: v2 - uses: actions/cache@v2 with: path: ~/.npm key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }} - run: npm ci - run: npm run build - uses: actions/upload-artifact@v2 with: name: build-files path: dist/ test: needs: setup runs-on: ubuntu-latest strategy: matrix: test-type: [unit, integration, lint] steps: - checkout: v2 - uses: actions/download-artifact@v2 - run: npm ci - run: npm run ${{ matrix.test-type }} e2e: needs: setup runs-on: ubuntu-latest steps: - checkout: v2 - uses: actions/download-artifact@v2 - run: npm ci - run: npm run e2e-tests ``` **Results:** Pipeline time reduced from 25 minutes to 8 minutes through parallelization and caching. ### Technology-specific optimizations #### React/javascript optimizations **Bundle size optimization:** * Code splitting implementation at route and component levels * Dynamic imports for large third-party libraries * Tree shaking configuration for unused code elimination * Webpack bundle analyzer integration for ongoing monitoring **Development server optimization:** * Hot module replacement optimization * Development build caching * Faster refresh configuration * Memory usage optimization #### TypeScript project optimization **Compilation speed improvements:** * Project references for large codebases * Incremental compilation setup * Build tool integration optimization * Type checking parallelization #### Docker and container optimization **Multi-stage build optimization:** * Dependency caching layers * Build context optimization * Layer ordering for maximum cache reuse * Image size optimization techniques *** ### See also * [Flaky test resolution](/runbooks/concepts/use-cases/flaky-test-resolution) * [Improving code coverage](/runbooks/concepts/use-cases/improving-code-coverage) # How to guides # One shot mode [One shot mode](/runbooks/concepts/one-shot-mode) is an automated execution mode for Runbooks that streamlines the entire code modification workflow into a single, uninterrupted process. ### Starting a One Shot Session #### **Via Web Interface**
1. Navigate to the Runbooks dashboard 2. Click "New Runbook" 3. Select your repository 4. Enable the **One shot mode** toggle from the dropdown 5. Describe your task in the input field 6. Click "Create" to start execution #### **Via Slack**
one-shot mode with Slack
Send a message to the Aviator bot using this format: ``` @Aviator oneshot ``` **Example:** ``` @Aviator oneshot my-app/frontend Add an alert status bar when user is out of credits ``` You can also skip the org-name when specifying the repo: ``` @Aviator oneshot frontend Add an alert status bar when user is out of credits ``` ### Execution Flow Once a one-shot session is initiated, the following happens automatically: #### **1. Runbook Generation** * Aviator analyzes your codebase to understand the current state * Generates a comprehensive execution plan with numbered steps * Documents assumptions made based on code analysis * No clarifying questions are asked; reasonable defaults are used #### **2. Automatic Execution** * All steps begin executing immediately after plan generation * Steps execute sequentially in dependency order * Each step generates code changes and commits them #### **3. Pull Request Management** * A single **draft PR** is created with the first code-generating step * Subsequent steps add commits to the same PR * The PR title reflects the overall task (e.g., `[WIP] Upgrade React from 17 to 18`) * PR description includes: runbook URL, executor, and progress indication #### **4. Completion** * When all steps complete successfully, the draft PR is automatically marked as **ready for review** * PR title is updated to remove `[WIP]` prefix * You receive a notification (via chat or Slack) that execution is complete * The PR is ready for team review and merge ### Monitoring Progress While one-shot mode runs automatically, you can monitor progress in real-time: * **Chat interface**: Shows step-by-step execution logs and status updates * **Pull request**: View incremental commits as each step completes * **Runbook page**: Displays current step status (queued, in progress, completed) ### Handling Failures If a step fails during execution: * Execution **stops** at the failed step * Remaining queued steps return to "not started" status * The draft PR remains open with all completed work * Error details are shown in the chat interface * You can: * Manually fix the issue and re-run the failed step * Modify the runbook plan and continue * Close the PR if the approach needs reconsideration ### Best Practices 1. **Start with clear task descriptions**: Since one-shot mode skips clarification, provide detailed requirements upfront * ✅ Good: "Add TypeScript types to the UserProfile.tsx file" * ❌ Vague: "Update components" 2. **Review the generated plan**: Although execution starts automatically, you can view the plan in the chat interface. If it's not what you expected, you can cancel and restart in standard mode. 3. **Use for incremental changes**: One-shot mode works best for tasks that are additive or isolated. For changes that might conflict with ongoing work, consider standard mode with step-by-step review. 4. **Leverage for repetitive tasks**: Once you've validated an approach with standard mode, use one-shot mode for similar tasks across different parts of your codebase. 5. **Monitor the PR**: Even though execution is automatic, review the PR commits to ensure changes align with expectations before merging. *** ### Example Workflows #### Example 1: Add Type Safety to a Component **Task**: Add TypeScript types to the Button.tsx component **Command** (Slack): ``` @Aviator oneshot my-app Add proper TypeScript interfaces to Button.tsx ``` **What happens**: 1. Runbook analyzes the `Button.tsx` file 2. Generates steps: define props interface, add return type, update exports 3. Executes all steps automatically 4. Creates draft PR with the updated component 5. Marks PR ready when complete #### Example 2: Add Error Handling **Task**: Add try-catch blocks to the authentication service **Steps** (via web interface): 1. Enable one-shot mode 2. Enter task: "Add error handling to all methods in auth.service.ts" 3. Aviator generates plan: wrap each method in try-catch, add logging 4. Execution proceeds automatically 5. Single PR contains all error handling changes #### Example 3: Update Documentation **Task**: Add JSDoc comments to a utility file **Command** (Slack): ``` @Aviator oneshot my-repo Add JSDoc comments to utils/formatters.js ``` **What happens**: 1. Runbook analyzes all functions in `formatters.js` 2. Generates plan: add JSDoc for each function with parameters and return types 3. Executes automatically 4. Creates PR with documented code 5. Marks PR ready when complete ### See also * [One-shot mode concepts](/runbooks/concepts/one-shot-mode) # Editing Runbooks Runbooks can be edited in three main ways: 1. **Chat-based editing** - Use natural language prompts to modify Runbooks through the AI agent 2. **Manual full Runbook editing** - Edit the entire [Runbook structure](/runbooks/concepts/runbook-format) and content directly 3. **Individual step editing** - Modify specific steps within a Runbook Each approach has its own advantages depending on your needs and the complexity of changes you want to make. ### Chat-based editing with prompts The easiest way to modify a runbook is by providing natural language instructions to the AI agent through the chat interface. This method is ideal for making conceptual changes or adding new functionality without diving into the technical details. #### How to use chat-based editing 1. Navigate to your runbook in the Aviator interface 2. Open the chat panel or find the "Edit with AI" option 3. Describe the changes you want to make in natural language 4. Review the proposed changes and approve them
Chat prompt to update Runbook
#### Example prompts for editing Here are some effective ways to request runbook modifications: **Adding new steps:** * "Add a step to run unit tests before deployment" * "Include a code review reminder at the beginning" * "Add validation to check if all environment variables are set" **Modifying existing logic:** * "Change the deployment target from staging to production" * "Update the notification channel to #releases instead of #general" * "Modify the approval process to require two reviewers instead of one" **Conditional modifications:** * "Only run the deployment step if all tests pass" * "Add a rollback step that triggers if deployment fails" * "Skip the documentation update for hotfix branches" **Integration changes:** * "Add GitHub status checks integration" * "Include Jira ticket updates in the workflow" #### Best practices for prompts * **Be specific**: Instead of "make it better," explain exactly what you want to achieve * **Provide context**: Mention the current behavior and desired outcome * **Use examples**: If you have specific commands or configurations in mind, include them * **Test incrementally**: Make one change at a time to ensure each modification works as expected ### Editing individual steps When you need to make targeted changes to specific parts of your Runbook, individual step editing provides a focused approach without affecting the rest of the workflow. #### Selecting a step to edit 1. Open your Runbook in the main view 2. Find the step you want to modify in the steps list 3. Click the edit icon (pencil) next to the step name 4. The step editor will open in a focused view
Editing a step
#### Step editor interface Editing a step gives you the raw markdown that you can edit. Please remember the first line in the raw markdown represents the title of the step.
#### Common step modifications **Updating commands:** * Modify shell commands or scripts * Change command-line arguments * Update file paths or resource references **Adjusting conditions:** * Modify when the step should execute * Update success/failure criteria * Change timeout values **Managing dependencies:** * Add or remove dependencies on other steps * Modify input requirements * Update output specifications **Configuring notifications:** * Change notification recipients * Modify message templates * Update delivery channels ### Manual full Runbook editing (not recommended) For complex modifications or when you need precise control over the runbook structure, manual editing gives you complete access to the Runbook's markdown to edit. {% hint style="info" %} Note that full raw markdown is not recommended as it can cause parsing errors and may leave your Runbook non-executable. {% endhint %} #### Accessing the runbook editor 1. Open your Runbook from the main dashboard 2. Click the "Markdown", tab and click "Edit". 3. The full editor will open, showing the complete Runbook structure as raw text.
Please familiarize yourself with [Runbooks format](/runbooks/concepts/runbook-format) before editing the full markdown. ### Versioning Aviator maintains the historic versioning of all Runbooks. The versioning is sequential in increasing order and you can rollback to an older version of Runbook at any time. Even after rolling back to the previous version, a subsequent version would still represent the next incremental number than the highest sequential version. ### See more * [Runbook Format](/runbooks/concepts/runbook-format) * [Best practices for planning](/runbooks/how-to-guides/best-practices-for-planning) # PR Lifecycle management Runbooks provides three distinct workflows for managing pull request creation during code migrations. Organizations can configure a default workflow, and users can override this setting per chat session. ### Workflow configuration #### Organization defaults Administrators configure the default PR workflow in the Runbooks settings. This applies to all new chat sessions unless overridden by individual users. #### Per-session overrides Users can modify the PR workflow for their current chat session through the settings popup on the main chat screen. This setting persists for the duration of the session and overrides the organization default. ### Available workflows #### Stacked PRs (default) The system automatically creates a separate pull request for each completed step. Each PR builds upon the previous one, creating a stack of dependent changes. **Behavior**: Step execution triggers immediate PR creation without user intervention. The next step begins automatically on a new branch that includes the previous step's changes. **Use case**: Complex migrations where each step represents a logical unit that benefits from individual review and testing. #### Single PR All changes from multiple steps are consolidated into a single pull request. The initial step creates the PR, and subsequent steps add commits to the same branch. **Behavior**: After the first step completes, a PR is created. All following steps commit directly to this branch, updating the existing PR with new changes. The PR description automatically updates to reflect step progress. **Use case**: Cohesive migrations where the complete change set should be reviewed and tested together. #### Manual approval The system pauses after each step completion and requests user approval before creating a PR. Users control exactly when and how PRs are created. **Approval dialog options**: When subsequent steps are queued, users can create a stacked PR, continue on the same branch, continue on a new branch, or cancel remaining steps. When no steps remain queued, users can create a PR or dismiss the dialog. **Branch selection**: When manually executing a new step without creating a PR for the previous step, users choose whether to continue on the current branch or start a new one. **Use case**: Teams requiring explicit approval for PR creation or complex workflows that need case-by-case decision making. ### Workflow execution details #### Stacked PRs execution Each step creates a new branch based on the previous step's branch. The system generates a PR immediately upon step completion. Subsequent steps automatically begin on branches that include all previous changes, maintaining the dependency chain. #### Single PR execution The first step creates both a feature branch and an initial PR. Subsequent steps execute on the same branch, adding commits that appear in the existing PR. The PR description updates automatically to document which steps have completed and their associated changes. #### Manual approval execution After each step completes, execution halts and presents a review dialog. The dialog includes a link to view the diff in GitHub. User decisions determine whether to create a PR, how to handle branching for the next step, or whether to continue the migration at all. For steps executed without prior PR creation, the system prompts for branch selection before beginning execution. Users specify whether the new step should build on existing changes or start fresh. ### Branch management #### Naming conventions Stacked PRs use sequential branch names that reflect the step hierarchy. Single PR workflows use a single descriptive branch name for the entire migration. Manual workflows allow users to influence branch creation through their approval decisions. #### Dependency tracking The system maintains awareness of which changes depend on others, ensuring that stacked PRs preserve the correct order and that single PRs include all necessary context. Manual workflows rely on user decisions to maintain logical change groupings. #### Conflict resolution When conflicts arise between branches, the system provides clear indication of the conflict location and affected files. Users can resolve conflicts through their normal Git workflow before continuing with the migration. ### Integration considerations #### CI/CD pipelines Each workflow type triggers CI/CD differently. Stacked PRs generate separate pipeline runs for each PR, allowing incremental validation. Single PRs run the complete test suite against the consolidated changes. Manual workflows depend on when users choose to create PRs. #### Code review processes Stacked PRs enable reviewers to examine changes incrementally, with each PR building understanding for the next. Single PRs require reviewers to understand the complete migration in one review cycle. Manual workflows allow teams to align PR creation with their specific review practices. #### Deployment strategies Teams using feature flags or gradual rollouts may prefer stacked PRs for incremental deployment. Teams requiring atomic deployments typically choose single PRs. Manual workflows provide flexibility to align with existing deployment processes. ### See also * [Using Stacked PRs](/runbooks/how-to-guides/using-stacked-prs) * [Editing Runbooks](/runbooks/how-to-guides/editing-runbooks) # Step-by-step execution Runbooks in Aviator execute through a carefully orchestrated step-by-step process that ensures reliable automation while providing visibility into each stage of your workflow. Understanding how this execution model works helps you design more effective Runbooks and troubleshoot issues when they arise. ### Sequential step processing Every Runbook creates [one or more steps](/runbooks/concepts/runbook-format), with larger tasks that may go into hundreds of steps and substeps. These steps are designed to be executed in-order as a subsequent step in the Runbook may have dependency on a previous step completed. By default, runbook steps execute sequentially, with each step waiting for the previous one to complete before beginning. This sequential approach provides predictable behavior and makes it easier to understand the workflow logic, especially when steps have dependencies on outputs from earlier stages. ### Executing a single step To execute a single step, simply click on "Execute Next" button on the steps view.
You can alternatively also click on "Run" button next to a Step or a Substep. Keep in mind when clicking "Run" button that: * when running the next substep, only that substep is executed, * when running the parent step, it will enqueue and execute all substeps within that step one by one.
### Step queuing While a step is running, you can still enqueue more steps. This can be done by clicking on the "+" sign next to the steps. These steps will not immediately run and will be executed when the previous steps have been completed. When enqueuing or running a step further down the order, Runbooks will automatically enqueue all steps in the middle that are not executed yet.
### Stopping execution Execution of any step can be perfomed by either clicking "Stop" on the Claude code message bar, or by clicking "Stop execution" button at the top right on the Steps view.
In both cases, the execution is stopped and the step is marked as failed. Any subsequent steps that were queued will be automatically dequeued.
*** # Managing Templates ## **Using Templates** [Templates](/runbooks/concepts/templates) are a great way to share and run Runbooks with prebuilt context. ### **Discovery and Selection** Templates can be browsed and filtered by: * **Categories**: Find templates by functional area * **Keywords**: Search by name or description * **Usage popularity**: See which templates are most commonly used * **Recency**: Find recently created or updated templates ## **Creating Runbooks from Templates** When using a template, you choose a particular repository and provide additional context. This information is then used to modify the Runbook to adapt to the required needs. Once a Runbook is created, it can be modified or executed like any other Runbook. ![Using an existing template](/files/uatkfnXj7RVVxFAmb3p4) ## **Template Management** ### **Usage Tracking** The system automatically tracks how many times a template has been used, when the template was most recently selected, and which templates lead to successful completions. ### **Template Evolution** Templates can be updated with improved steps based on usage feedback. Every update is also internally maintained with version tracking. This way, you can track which Runbook was created using which version of the template. ### **Best Practices** #### **Writing Effective Templates** 1. **Be Specific**: Each action should be clear and unambiguous 2. **Include Context**: Explain why steps are necessary 3. **Add Verification**: Include testing and validation steps 4. **Consider Edge Cases**: Address common complications 5. **Provide Examples**: Include code snippets where helpful #### **Template Organization** 1. **Use Clear Categories**: Choose the most specific applicable category 2. **Write Descriptive Names:** Make templates easily discoverable 3. **Maintain Consistency**: Follow established patterns within your organization 4. **Document Assumptions**: Note prerequisites and constraints #### **Step Design Principles** 1. **Single Responsibility**: Each step should do one thing well 2. **Logical Grouping**: Related actions should be grouped together 3. **Dependency Awareness**: Order steps to respect dependencies 4. **Error Recovery**: Consider rollback and error handling ## **Advanced Features** The following features are still experimental and not available in the public beta release. Please reach out to us to early acccess: ****. ### **Template Variables** Support for parameterized templates that can be customized at runtime: ```markdown #### 1.1: Update {{FRAMEWORK_NAME}} Configuration - 1. Modify {{CONFIG_FILE}} to include new settings - 2. Update imports from {{OLD_PACKAGE}} to {{NEW_PACKAGE}} ``` #### **Conditional Steps** Support for optional or conditional execution paths: ```markdown #### 1.2: [CONDITIONAL] Update TypeScript Configuration **Only execute if project uses TypeScript** - 1. Update tsconfig.json for new syntax - 2. Fix type compatibility issues ``` #### **Template Inheritance** Support for template hierarchies where specialized templates extend base templates: ```python **# Extended React Migration Template** - *Extends: Base Migration Template** **## Additional React-Specific Steps** ``` ### **Troubleshooting** #### **Common Template Issues** 1. **Parsing Errors**: Ensure proper markdown formatting and section headers 2. **Missing Steps**: Verify all required phases and tasks are included 3. **Invalid Categories**: Check that categories match predefined values 4. **Execution Failures**: Review step dependencies and prerequisites # Persona management ## **Using Personas** By default Aviator will choose the default personas for planning and execution. To switch A custom persona can be selected directly from the Rubook chat interface when interacting with the agents. To switch the persona, select the “Switch persona” from the actions menu on the top right. ## Creating Personas Personas can be managed from the Runbook Settings. [Personas](/runbooks/concepts/personas) are system prompts that are passed over to LLM agents to perform specific actions. ### **System prompt designing** A well-designed system prompt should: 1. **Define the role clearly** ```bash You are a React migration specialist with expertise in modernizing legacy React applications. ``` 2. **Specify the approach** ```bash When analyzing code, prioritize component reusability and performance optimization. ``` 3. **Set quality standards** ```bash All code changes should include comprehensive error handling and maintain backward compatibility. ``` 4. **Include context awareness** ```bash Consider the existing codebase patterns and team coding standards when making recommendations. ``` ### **Example System Prompts** #### **Security Expert Persona** ```bash You are a cybersecurity expert specializing in secure code development. Your primary focus is identifying and preventing security vulnerabilities in code changes. When reviewing or generating code: - Always validate and sanitize user inputs - Follow the principle of least privilege - Implement proper authentication and authorization checks - Use parameterized queries to prevent SQL injection - Ensure sensitive data is properly encrypted - Consider potential attack vectors and edge cases - Recommend security testing strategies Communicate security risks clearly and provide actionable mitigation strategies. ``` #### **Performance Optimization Persona** ```bash You are a performance optimization specialist focused on creating efficient, scalable code solutions. When analyzing and implementing changes: - Identify potential performance bottlenecks - Optimize algorithms and data structures - Minimize unnecessary computations and memory usage - Consider caching strategies where appropriate - Evaluate database query performance - Suggest profiling and monitoring approaches - Balance performance with code readability Provide specific metrics and benchmarking recommendations when possible. ``` #### **Legacy Modernization Persona** ```bash You are a legacy system modernization expert with deep experience in gradual migration strategies. Your approach emphasizes: - Incremental changes that minimize risk - Maintaining system stability during transitions - Creating bridge patterns between old and new code - Comprehensive testing at each migration step - Clear rollback strategies for each change - Documentation of migration decisions and rationale Always consider the business impact and provide timeline estimates for complex changes. ``` ## **Persona creating strategies** ### **Team Standardization** Create shared personas that embody your team's best practices: * **Code Review Persona**: Emphasizes thorough review practices and common pitfalls * **Testing Persona**: Focuses on comprehensive test coverage and quality * **Documentation Persona**: Ensures all changes include proper documentation #### **Project-Specific Personas** Develop personas tailored to specific projects or technologies: * **Microservices Persona**: Specialized in distributed system patterns * **Mobile App Persona**: Focuses on mobile-specific considerations * **Data Pipeline Persona**: Emphasizes data quality and processing efficiency #### **Role-Based Personas** Create personas that match different team roles: * **Senior Developer Persona**: Provides mentorship and architectural guidance * **Junior Developer Persona**: Offers detailed explanations and learning opportunities * **DevOps Persona**: Focuses on deployment, monitoring, and infrastructure concerns ### **Best Practices** #### **Persona Design** 1. **Start Simple**: Begin with clear, focused personas before adding complexity 2. T**est Iteratively**: Refine prompts based on actual usage and feedback 3. **Document Intent**: Clearly describe what each persona is designed to accomplish 4. **Version Control**: Keep track of persona changes and their impact #### **Team Adoption** 1. **Establish Standards**: Define when to use specific personas 2. **Share Knowledge**: Document successful persona patterns 3. **Regular Review**: Periodically evaluate and update personas based on team needs 4. **Training:** Ensure team members understand how to effectively use personas # Context management This guide covers how to view, create, and manage learnings and context files for your Runbooks account. For an overview of how context works, see [Context and Learnings](/runbooks/concepts/context-and-learnings). ## Managing learnings View and manage learnings from the **Context** tab in your Runbooks dashboard. ### Browsing learnings The Context tab shows all learnings captured across your account: * **Search and filter** — Find learnings by keyword, framework, or file pattern * **Sort by relevance** — See most frequently used or most recent learnings first * **View details** — Click any learning to see its full pattern, solution, and usage history * **See source sessions** — Track which Runbook sessions contributed each learning ### Creating custom learnings You can manually add learnings to share project knowledge with your team: 1. Go to the **Context** tab 2. Click **Add Learning** 3. Fill in the learning details: * **Pattern** — Describe the problem or situation (e.g., "pytest fails with import errors in monorepo") * **Solution** — What works to resolve it (e.g., "Add conftest.py with sys.path configuration") * **Applies to** — Optional filters for files, frameworks, or commands 4. Click **Save** Custom learnings are treated the same as auto-captured ones — they'll surface in future Runbook sessions when relevant. ### Editing and removing learnings * **Edit** — Update a learning's pattern, solution, or applicability filters * **Delete** — Remove learnings that are no longer relevant or were captured incorrectly *** ## Managing context files Context files are markdown documents that describe your project architecture and conventions. ### Creating a context file 1. Navigate to **Runbooks Settings > Context Files** 2. Click **Add Context File** 3. Write your markdown content with a `# Heading` as the title ### Structure Use a clear heading as your title: ```markdown # Authentication System ## Overview Our app uses JWT-based authentication with refresh tokens. ## Key Files - `src/auth/jwt.py` - Token generation and validation - `src/auth/middleware.py` - Request authentication - `src/models/user.py` - User model with password hashing ## Patterns - All protected routes use the `@require_auth` decorator - Tokens expire after 15 minutes - Refresh tokens are stored in HTTP-only cookies ``` ### Referencing important files Reference files in your codebase using code formatting. The system extracts these paths and uses them to prioritize context during planning: * Inline: `` `src/auth/jwt.py` `` * In lists or code blocks ### Updating context files Edit context files from **Runbooks Settings > Context Files**. Changes apply immediately to new Runbook sessions. For existing sessions, agents pick up changes when: * You start a new planning phase * The agent re-reads context during execution *** ## Best practices **Keep context files focused** — One file per major system or domain. Avoid monolithic documents. **Document patterns, not code** — Explain conventions and "why" rather than duplicating code. **Include file paths** — Reference actual files so agents can locate implementations. **Let learnings accumulate** — The more you use Runbooks, the smarter they become. **Review learnings periodically** — Check the Context tab to remove outdated patterns or refine solutions. *** ## FAQ **Can I see what learnings have been captured?** Yes. Browse all learnings in the Context tab. You can also see session-specific learnings in the details section of each Runbook session. **Do learnings slow down my workflow?** No. Learning capture happens in the background and doesn't block your work. **Can I disable learning capture?** Contact support if you need to disable learning capture for your account. **Are learnings used in all Runbook sessions?** Yes, relevant learnings are automatically retrieved for all Runbook sessions in your account. # Claude Code tool permissions Configure which Claude Code tools agents can use during planning and execution phases. This gives you control over agent capabilities and enforces security policies. ## Overview Runbooks uses Claude Code to execute tasks. Claude Code has access to various tools like file operations, shell commands, and web searches. You can restrict these tools using allowlists and denylists. **Allowlist**: Only listed tools are permitted. If set, unlisted tools are blocked. **Denylist**: Listed tools are blocked. All other tools remain available. ## Configuration Navigate to **Runbooks Settings > Claude Code Tools** to configure tool permissions. You can set separate configurations for: * **Planning phase**: When agents analyze code and create plans * **Execution phase**: When agents implement changes ## Available tools | Tool | Description | | -------------- | ------------------------- | | `Bash` | Execute shell commands | | `Read` | Read file contents | | `Write` | Create or overwrite files | | `Edit` | Modify existing files | | `MultiEdit` | Batch file edits | | `Glob` | Find files by pattern | | `Grep` | Search file contents | | `Task` | Spawn sub-agents | | `WebSearch` | Search the web | | `WebFetch` | Fetch URL contents | | `NotebookEdit` | Edit Jupyter notebooks | | `TodoWrite` | Manage task lists | | `KillShell` | Terminate running shells | | `ExitPlanMode` | Exit planning mode | ## Tool patterns ### Basic tools Specify tool names directly: ``` Read Write Grep WebSearch ``` ### Bash command patterns Restrict Bash to specific commands using patterns: ``` Bash(npm run build) Bash(npm run test:*) Bash(pytest:*) ``` The `:*` suffix acts as a wildcard, matching any arguments after the prefix. **Examples**: * `Bash(npm run build)` - Only allows exact command `npm run build` * `Bash(npm run:*)` - Allows `npm run build`, `npm run test`, `npm run lint`, etc. * `Bash(pytest:*)` - Allows `pytest` with any arguments ### MCP tools If you have MCP servers configured, their tools appear with the `mcp__` prefix: ``` mcp__github mcp__github__create_issue mcp__slack__post_message ``` **Format**: `mcp__` or `mcp____` Note: Wildcards are not supported for MCP tools. Each tool must be listed explicitly. ## Example configurations ### Restrict planning to read-only Allow agents to explore but not modify during planning: **Planning allowlist**: ``` Read Grep Glob WebSearch Task ``` ### Restrict execution to specific commands Limit what agents can execute: **Execution allowlist**: ``` Read Write Edit Bash(npm run build) Bash(npm run test:*) Bash(npm run lint) ``` ### Block dangerous operations Prevent specific actions: **Execution denylist**: ``` Bash(rm -rf:*) Bash(git push:*) Bash(curl:*) WebFetch ``` ## Precedence If both allowlist and denylist are set: 1. Allowlist is applied first (only listed tools are considered) 2. Denylist is applied second (removes tools from the allowed set) ## Default behavior With no configuration: * All standard Claude Code tools are available * Git operations (`git commit`, `git push`) are blocked by the system * Runbooks manages git operations automatically for consistent PR workflows # MCP servers Configure Model Context Protocol (MCP) servers to extend agent capabilities with custom tools and integrations. ## Overview MCP servers provide additional tools that agents can use during Runbook execution. Common use cases include: * Database access * API integrations (GitHub, Slack, Jira) * Custom internal tools * File system operations on remote systems ## Configuration Navigate to **Runbooks Settings > MCP Servers** to configure servers. The configuration uses JSON format: ```json { "mcpServers": { "server-name": { // server configuration } } } ``` ## Server types ### stdio Local command-based servers that communicate via stdin/stdout. ```json { "mcpServers": { "github": { "type": "stdio", "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } } } } ``` **Required fields**: * `command`: The executable to run **Optional fields**: * `type`: Set to `"stdio"` (default if omitted) * `args`: Array of command arguments * `env`: Environment variables (object of key-value strings) ### sse Remote servers using Server-Sent Events transport. ```json { "mcpServers": { "remote-tools": { "type": "sse", "url": "https://mcp.example.com/sse", "headers": { "Authorization": "Bearer ${API_TOKEN}" } } } } ``` **Required fields**: * `type`: Must be `"sse"` * `url`: Server endpoint URL **Optional fields**: * `headers`: HTTP headers (object of key-value strings) ### http Remote servers using HTTP transport. ```json { "mcpServers": { "api-tools": { "type": "http", "url": "https://mcp.example.com/api", "headers": { "X-API-Key": "${API_KEY}" } } } } ``` **Required fields**: * `type`: Must be `"http"` * `url`: Server endpoint URL **Optional fields**: * `headers`: HTTP headers (object of key-value strings) ## Using secrets Reference secrets in your configuration using `${SECRET_NAME}` syntax: ```json { "mcpServers": { "database": { "type": "stdio", "command": "mcp-postgres", "env": { "DATABASE_URL": "${DATABASE_URL}" } } } } ``` Create secrets in **Runbooks Settings > Secrets**. Secrets are encrypted at rest and injected at runtime. ## Example configurations ### GitHub integration ```json { "mcpServers": { "github": { "type": "stdio", "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_TOKEN}" } } } } ``` ### Slack integration ```json { "mcpServers": { "slack": { "type": "stdio", "command": "npx", "args": ["-y", "@modelcontextprotocol/server-slack"], "env": { "SLACK_BOT_TOKEN": "${SLACK_TOKEN}" } } } } ``` ### PostgreSQL access ```json { "mcpServers": { "postgres": { "type": "stdio", "command": "npx", "args": ["-y", "@modelcontextprotocol/server-postgres"], "env": { "DATABASE_URL": "${DATABASE_URL}" } } } } ``` ### Multiple servers ```json { "mcpServers": { "github": { "type": "stdio", "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_TOKEN}" } }, "filesystem": { "type": "stdio", "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/workspace"] } } } ``` ## Using MCP tools in Runbooks Once configured, MCP tools appear with the `mcp__` prefix in Claude Code: * `mcp__github__create_issue` * `mcp__slack__post_message` * `mcp__postgres__query` You can control access to these tools using [Claude Code tool permissions](/runbooks/how-to-guides/claude-code-tools). ## Restrictions * The `sdk` server type is not available in user configurations * Server names must be alphanumeric with hyphens and underscores only * Each server name must be unique within your configuration # Best practices for planning #### 1. Start with Clear Requirements Before Runbooks generates the plan, provide comprehensive context: ``` "I want to migrate our authentication from OAuth1 to OAuth2. Context: - Current implementation: src/auth/oauth1.ts - 50K active users, zero downtime required - Must support both flows during 2-week migration window - Need to preserve all existing session data - Compliance requirement: audit trail for all auth changes Success criteria: - All existing users can continue logging in - New users use OAuth2 automatically - Complete migration data report for compliance - No performance degradation" ``` #### 2. Reference Your Codebase Point to specific files, patterns, and conventions: ``` "Follow the repository pattern established in src/repositories/ Each repository should: - Extend BaseRepository - Use dependency injection for the database client - Include CRUD operations and custom queries - Have corresponding tests in tests/repositories/" ``` #### 3. Iterate in Small Steps Don't wait until everything is wrong to provide feedback: ``` "Step 1 looks good - please proceed with that. For Step 2, let's discuss the database schema changes before implementation." ``` #### 4. Provide Examples from Your Codebase ``` "The error handling should match our pattern in src/services/PaymentService.ts: - Custom error classes for different failure modes - Structured logging with correlation IDs - Retry logic with exponential backoff - Circuit breaker for external services" ``` #### 5. Set Expectations for Testing ``` "After executing step 3: 1. Run the full test suite: npm test 2. Check for TypeScript errors: npm run type-check 3. Test manually in staging environment 4. Verify API response format with Postman collection in /docs/api 5. Run the migration script in a database dump I'll review the PR and provide feedback on the implementation." ``` #### 6. Use Step Editing for Precision When you need exact changes, edit the step directly rather than describing changes in chat. This ensures: * No ambiguity in requirements * Exact specifications are captured * Changes are tracked in the runbook version #### 7. Provide Feedback Continuously Don't wait for all steps to execute. Review and comment as you go: * After runbook generation: Review the overall plan * During execution: Check each PR as it's created * After testing: Report results and issues immediately #### 8. Share Test Results ``` "Executed Step 4.1 - test results: ✅ Unit tests: All 45 tests passing ✅ Integration tests: Passing (3 min runtime) ❌ E2E tests: 2 failures in authentication flow - LoginTest.tsx: Expected redirect to /dashboard, got /login - SessionTest.tsx: Session cookie not being set Logs show that the JWT token is generated but not included in the response headers. Can you update Step 4.1 to include the token in the Authorization header?" ``` #### 9. Request Explanations When Needed ``` "Can you explain the approach in Step 3.2? I'm not sure why we're using a WeakMap instead of a regular Map. What's the benefit for our use case?" ``` #### 10. Collaborate on Complex Decisions ``` "For Step 6 (database schema migration), I see three options: 1. Add new columns, migrate data, drop old columns (zero downtime) 2. Create new table, migrate data, swap tables (requires brief downtime) 3. Use database views for backward compatibility (complex but safe) Our constraints: - 24/7 uptime requirement - 500GB database - Must complete within maintenance window Which approach do you recommend and why?" ``` # Batch Operations with Runbook Templates Batch operations allow you to apply a single Runbook template across multiple targets simultaneously, saving time when you need to perform the same changes across many repositories or directories. Instead of manually creating individual Runbooks one at a time, you can create dozens or even hundreds in a single operation. ### Overview Runbook templates support two types of batch operations: 1. **Multi-Repository Batches**: Apply a template to multiple repositories, creating one Runbook per repository 2. **Multi-Directory Batches**: Apply a template to multiple directories within a single repository, creating one Runbook per directory Each batch operation can target up to 200 repositories or directories. When you create a batch, the system creates individual “fanouts” for each target, which are processed concurrently with automatic retry and timeout handling. ### Use Cases Batch operations are particularly useful for: * **Dependency upgrades**: Update a library version across all microservices in your organization * **Code migrations**: Apply framework migrations or API changes to multiple services * **Security patches**: Roll out security fixes across your entire repository portfolio * **Style consistency**: Enforce code style or linting rule changes across a monorepo’s directories * **Configuration updates**: Update configuration files or environment settings across multiple projects ### Creating a Batch Operation #### Step 1: Navigate to Your Template Batch operations can only be created for Templates. If you don’t have an existing template, you can [publish](https://docs.aviator.co/runbooks/getting-started#id-6.-publishing-a-template) any Runbook to a Template from the chat session. From the Runbooks templates page, select the template you want to apply in batch mode. Click on the template to view its details. #### Step 2: Open the Batch Modal On the template detail page, click the dropdown arrow next to the “Use this plan” button and select **“Create Runbook Batch”**. ![Create Runbook Batch](/files/bXjUBM1QSJnOZ1pxEnVC) #### Step 3: Choose Your Batch Type The batch creation modal presents two tabs: #### Per Repository Use this option to apply the template to multiple repositories. Each repository will get its own Runbook instance. **How to select repositories:** * Use the repository multi-select dropdown to choose target repositories * You can search and select multiple repositories from your organization * Selected repositories appear as chips that can be individually removed **Example use case:** Upgrading React from v17 to v18 across all frontend repositories in your organization. #### Per Directory Use this option to apply the template to multiple directories within a single repository. Each directory will get its own Runbook instance. **How to specify directories:** 1. Select a single repository from the dropdown 2. Enter directory paths in the text area, one per line 3. Alternatively, upload a CSV or TXT file containing directory paths 4. The system automatically deduplicates directory paths **Example use case:** Migrating all service modules in a monorepo from CommonJS to ES modules, targeting `services/auth`, `services/payments`, `services/notifications`, etc. ![Per directory or per repository](/files/ILhTQ3CSXeQ1oCK2MQoJ) #### Step 4: Add Customization Instructions (Optional) Customization instructions allow you to tailor the template execution for your specific use case. These instructions are applied to all Runbooks in the batch and help Claude Code understand context-specific requirements. **Example customization instructions:** 1. For a dependency upgrade: ``` We're upgrading from Lodash v4.17.15 to v4.17.21. Pay special attention to the deprecated methods in our codebase. We use lodash/fp extensively in our data transformation pipelines. ``` 2. For a code migration: ``` This is a Next.js Pages Router to App Router migration. Preserve all existing SEO metadata and ensure dynamic routes continue to work. We use ISR (Incremental Static Regeneration) heavily, so convert those patterns to the new revalidate approach. ``` The more specific your instructions, the better Claude Code can adapt the template to your needs. #### Step 5: Configure One Shot Mode (Optional) Enable the “One Shot Mode” toggle if you want all generated Runbooks to execute immediately after creation. When disabled the Runbooks are created and available to review before execution. For more information about One Shot Mode, see the [One Shot Mode documentation](https://docs.aviator.co/runbooks/how-to-guides/one-shot-mode). #### Step 6: Review and Submit Click **“Review”** to see a summary of your batch configuration: * Selected repositories or directories * Custom instructions (if provided) * One Shot Mode setting Verify everything is correct, then click **“Submit”** to create the batch. ![Review and Submit](/files/uRgF0ldQNES5tEIkk1bn) ### Monitoring Batch Progress After submitting, you’ll be redirected to the template’s **“Batch Jobs”** tab where you can monitor progress. ![Monitor batch progress](/files/chc5eZwlmXT54o9ZgDsE) #### Understanding Batch Status Each batch displays: * **Triggered by and timestamp**: Who created the batch and when * **Overall status**: * **In Progress**: At least one fanout is still processing * **Completed**: All fanouts have completed successfully * **Failed**: All fanouts have reached a terminal state with at least one failure * **Fanout counts**: Breakdown of how many fanouts are completed, in progress, pending, or failed #### Viewing Individual Fanouts Click on a batch row to expand it and see detailed information: **Batch Configuration:** * One Shot Mode status * Custom instructions (if any were provided) **Fanout List:** Each fanout represents one repository or directory and shows: * Target repository, and directory if applicable * Status indicator with color coding: * 🟢 **Green**: Completed successfully * 🔵 **Blue**: In progress * ⚪ **Gray**: Pending (waiting to start) * 🔴 **Red**: Failed * Link to the created Runbook ![Viewing individual fanouts](/files/Z4ZDgSsBqDI4ouxHjkYG) #### Interacting with Generated Runbooks Once a fanout completes successfully, you can: * Click on the runbook to view its progress and results * Share the runbook with team members using the share button * Review the changes and pull requests created by the runbook ### Technical Details #### Concurrency and Processing Batch operations process multiple fanouts concurrently to optimize execution time. The system: * Processes fanouts in parallel according to configured concurrency limits, default **5** simultaneous fanouts * Automatically retries stalled operations * Times out fanouts that exceed configured duration limits #### Batch Size Limits * Maximum of **200 targets** per batch (repositories or directories) * This limit ensures reliable processing and reasonable execution times #### Monitoring and Reliability The platform includes automated monitoring that: * Detects and restarts stalled batches every 30 minutes * Cancels hanging fanouts that exceed timeout thresholds * Provides real-time status updates in the UI ### Best Practices 1. **Start small**: Test your template on a few repositories first before creating a large batch 2. **Use descriptive customization instructions**: Provide context about your codebase’s specific patterns and requirements 3. **Monitor progress actively**: Keep an eye on the Batch Jobs tab during execution, especially for large batches 4. **Check for failures**: If fanouts fail, review the error messages and adjust your template or customization instructions accordingly ### Troubleshooting #### Batch is stuck in “In Progress” The monitoring system checks for stalled batches every 30 minutes and automatically restarts them. If a batch appears stuck for longer, it may indicate an issue with your account credits or configuration. #### Multiple fanouts are failing Review the error messages on failed fanouts. Common issues include: * Insufficient Runbook credits * Repository access permissions * Invalid directory paths * Template incompatibility with target codebase structure #### Want to cancel a batch Individual fanouts can be cancelled by cancelling their associated Runbooks. The batch will update its status as fanouts complete or are cancelled. ### See also * [Templates](/runbooks/concepts/templates) * [Managing Templates](/runbooks/how-to-guides/managing-templates) * [One shot mode](/runbooks/how-to-guides/one-shot-mode) # Pre-Execution Scripts ### Overview Pre-execution scripts allow you to run custom setup commands automatically before each Runbook step is executed. This feature is useful for environment configuration, dependency installation, cache warming, or any other preparatory tasks that your code changes require. ### How It Works When you execute a Runbook step, Aviator automatically checks for a pre-execution script at a specific location in your repository. If found, the script is executed in the sandbox environment before Claude Code begins working on the step. #### Execution Flow 1. **Step begins** - A Runbook step is queued for execution 2. **Sandbox setup** - The appropriate branch is checked out in the sandbox 3. **Context injection** - Any existing context files are loaded 4. **Pre-execution script** - If `.aviator/scripts/pre-execution.sh` exists, it runs 5. **Step execution** - Claude Code proceeds with the actual step implementation 6. **PR creation** - Changes are committed and pushed to a pull request ### Configuration #### Script Location Place your pre-execution script at the following path in your repository: ``` .aviator/scripts/pre-execution.sh ``` This path is checked at the start of every step execution. If the file doesn't exist, execution proceeds normally without any errors. #### Script Requirements * **Format**: Bash shell script * **Permissions**: The script will be automatically made executable (`chmod +x`) * **Execution**: Runs with `bash pre-execution.sh` * **Working Directory**: Executes in the root of your repository ### Examples #### Example 1: Install Dependencies ```bash #!/bin/bash # .aviator/scripts/pre-execution.sh echo "Installing npm dependencies..." npm install echo "Building project..." npm run build exit 0 ``` #### Example 2: Environment Setup ```bash #!/bin/bash # .aviator/scripts/pre-execution.sh echo "Setting up environment..." # Copy environment configuration cp .env.example .env # Install Python dependencies pip install -r requirements.txt # Run database migrations python manage.py migrate echo "Environment setup complete!" exit 0 ``` #### Example 3: Conditional Setup Based on Files ```bash #!/bin/bash # .aviator/scripts/pre-execution.sh echo "Running pre-execution setup..." # Check if package.json exists if [ -f "package.json" ]; then echo "Found package.json, installing Node dependencies..." npm install fi # Check if requirements.txt exists if [ -f "requirements.txt" ]; then echo "Found requirements.txt, installing Python dependencies..." pip install -r requirements.txt fi exit 0 ``` #### Example 4: Cache Warming ```bash #!/bin/bash # .aviator/scripts/pre-execution.sh echo "Warming up caches..." # Pre-compile assets npm run build:assets # Warm up application cache curl -s http://localhost:3000/api/warmup > /dev/null echo "Cache warming complete!" exit 0 ``` ### Output and Feedback #### Successful Execution When your pre-execution script runs successfully (exits with code 0), you'll see: * ✅ A success message in the Runbook chat * The script's stdout output (first 2000 characters) * Indication that step execution is proceeding **Example output:** ``` ✅ Pre-execution script completed successfully Output: Installing npm dependencies... added 234 packages in 5s Building project... Build completed successfully ``` #### Failed Execution If your pre-execution script fails (exits with non-zero code), you'll see: * ⚠️ A warning message with the exit code * Both stdout and stderr output (first 1000 characters each) * Step execution continues despite the failure **Example output:** ``` ⚠️ Pre-execution script failed with exit code 1 Stdout: Installing npm dependencies... Stderr: npm ERR! missing package.json npm ERR! enoent ENOENT: no such file or directory ``` **Important**: A failed pre-execution script does not stop step execution. The step will continue, but may fail later if required setup was incomplete. ### Best Practices #### 1. Keep Scripts Idempotent Your script should be safe to run multiple times: ```bash #!/bin/bash # Good: Check before installing if [ ! -d "node_modules" ]; then npm install fi # Good: Use --force-reinstall only when needed pip install -r requirements.txt --quiet ``` #### 2. Provide Clear Output Help users understand what's happening: ```bash #!/bin/bash echo "=== Pre-execution Setup ===" echo "1. Installing dependencies..." npm install echo "2. Running database migrations..." python manage.py migrate echo "3. Setup complete!" ``` #### 3. Handle Errors Gracefully Exit with appropriate codes and messages: ```bash #!/bin/bash set -e # Exit on first error echo "Installing dependencies..." if ! npm install; then echo "ERROR: Failed to install npm dependencies" exit 1 fi echo "Success!" exit 0 ``` #### 4. Keep Execution Time Reasonable The script runs on every step, so keep it fast: * Cache dependencies when possible * Avoid unnecessary rebuilds * Skip steps that are already complete ```bash #!/bin/bash # Check if already built if [ -f ".build_cache" ]; then echo "Build cache found, skipping rebuild" exit 0 fi echo "Building project..." npm run build touch .build_cache ``` #### 5. Use Environment Variables Configure behavior without code changes: ```bash #!/bin/bash # Allow disabling via environment variable if [ "$SKIP_PRE_EXECUTION" = "true" ]; then echo "Pre-execution skipped (SKIP_PRE_EXECUTION=true)" exit 0 fi echo "Running setup..." npm install ``` ### Troubleshooting #### Script Not Running If your pre-execution script isn't being executed: 1. **Check the path**: Ensure the file is at `.aviator/scripts/pre-execution.sh` 2. **Check the branch**: The script must exist in the branch being checked out 3. **Check file type**: Ensure it's a regular file, not a symlink 4. **Commit the script**: The file must be committed to the repository #### Script Fails Silently If your script fails without clear output: 1. Add `set -x` at the top to enable debug output 2. Add explicit error messages with `echo` 3. Ensure stdout and stderr are not redirected to `/dev/null` 4. Check for permission issues with file access ```bash #!/bin/bash set -x # Enable debug mode set -e # Exit on error echo "Starting pre-execution..." # Your commands here ``` #### Slow Execution If your pre-execution script takes too long: 1. Profile which commands are slow 2. Add caching for expensive operations 3. Consider if all steps are necessary 4. Use parallel execution when possible ```bash #!/bin/bash # Run independent tasks in parallel npm install & pip install -r requirements.txt & wait echo "All dependencies installed" ``` ### Limitations * **Timeout**: Scripts should complete within a reasonable timeframe. Timeout is 120 seconds. * **One script per repository**: Only `.aviator/scripts/pre-execution.sh` is checked * **Execution environment**: Runs in the sandbox environment with the checked-out code * **No step-specific scripts**: The same script runs for all steps * **No conditional skipping**: The script runs if it exists, regardless of step content ### Security Considerations * Scripts run with the same permissions as Runbook step execution * Be cautious with sensitive data in script output (it appears in chat) * Avoid downloading untrusted dependencies or executing remote code * Review scripts in PRs before merging ### FAQ **Q: Does the pre-execution script run for every step?** A: Yes, if the script exists, it runs before every Runbook step execution. **Q: What happens if the script fails?** A: The failure is logged and shown to the user, but step execution continues. The step may fail later if required setup was incomplete. **Q: Can I have different scripts for different steps?** A: Not currently. The same `.aviator/scripts/pre-execution.sh` runs for all steps. **Q: Can I use other scripting languages?** A: The script is executed with `bash`, but you can invoke other interpreters: ```bash #!/bin/bash python3 my_setup_script.py node my_setup_script.js ``` **Q: Will the script run in one-shot mode?** A: Yes, the pre-execution script runs in both regular and one-shot Runbook execution modes. **Q: Can I see detailed logs of what the script did?** A: The script's stdout is shown in the chat (first 2000 characters). For full logs, the script runs in the same sandbox where Claude Code operates. ### Related Documentation * [Single step](/runbooks/how-to-guides/step-by-step-execution) * [Cloud Sandbox Configuration](/runbooks/concepts/cloud-sandboxes) * [Custom Personas](/runbooks/how-to-guides/persona-management) # Using Stacked PRs Remote agents create all changes as stacked PRs that makes it easy to update incrementally without needing to redo the whole stack. These remote agents leverage the [stacked PR CLI](/aviator-cli) to perform these operations, you can also checkout an entire stack and make updates using the same CLI. ``` PR #1: Foundation Changes └─ PR #2: Core Implementation └─ PR #3: Test Updates └─ PR #4: Documentation ``` ### How stacked PRs work in Runbooks By default, each step in the Runbook creates a separate pull request that builds upon the previous one. All the substeps with the step add commits to the parent step within the same PR. So for instance, if your Runbooks looks like the folllowing: ``` ## Step 1 #### 1.1 #### 1.2 ### Step 2 #### 2.1 ``` then your will have two separate PRs such that: ``` main <- step-1-pr <- step-2-pr ``` Here `step-1-pr` contains two commits (`1.1` and `1.2`), and `step-2-pr` contains one commit (`2.1`). Any revisions through GitHub comments are appended as additional commits to existing branches. This dependency chain ensures that changes are applied in the correct order and that each step has access to the modifications made by previous steps. ### Branch management strategy Runbooks automatically manage the Git branching strategy for stacked PRs. Each step creates a uniquely named branch following the pattern `av/rb-{session-id}-{step-number]-{hash}` to avoid conflicts and provide clear traceability back to the specific Runbook execution. The Runbook execution engine handles all Git operations including branch creation, commits, and PR creation. Developers don't need to manually manage branches or worry about merge conflicts between steps, as the system ensures each step starts from a clean state based on the previous step's completed changes. ### Step isolation and dependency management Each step in a stacked PR workflow operates in isolation while having access to the cumulative changes from all previous steps. This isolation ensures that step failures don't corrupt the state of other steps and that individual steps can be retried or modified without affecting the entire workflow. For complex workflows with conditional logic, the system can create branching stacks where certain PRs are only created if specific conditions are met. This allows Runbooks to implement sophisticated automation logic while maintaining the reviewability of each individual change. ### Review and approval workflow [Stacked PRs](https://www.aviator.co/blog/stacked-prs-code-changes-as-narrative/) enable a granular review process where each step can be reviewed and approved independently. Reviewers can focus on the specific changes introduced by each step without being overwhelmed by the entire automation workflow at once. The Runbook execution engine coordinates the review process by ensuring that PRs are presented to reviewers in the correct order. Automated checks and CI processes run independently for each PR in the stack, providing step-level validation and ensuring that each modification maintains code quality and doesn't introduce regressions. ### Merge coordination PRs should be merged in dependency order, starting from the base of the stack and progressing through each step until all changes are integrated into the target branch. Stacked PRs from Runbook executions integrate seamlessly with [Aviator's MergeQueue](/mergequeue). The merge queue ensures that stacked PRs are processed in the correct order and that the final integration doesn't conflict with other concurrent changes to the codebase. The merge queue validation runs comprehensive checks across the entire stack before allowing any PRs to merge, ensuring that the complete automation workflow will integrate cleanly with the current state of the target branch. This integration provides additional safety guarantees for automated code modifications while maintaining the development team's existing merge queue workflows and policies. ### Tracking progress The Runbook execution dashboard provides real-time visibility into the stacked PR workflow, showing the status of each PR, review progress, and merge coordination. Developers can track the progress of complex automation workflows and intervene when necessary. Each PR in the stack includes metadata linking it back to the specific runbook execution and step, making it easy to understand the context and purpose of automated changes. This traceability is essential for debugging issues and understanding the impact of runbook executions on the codebase.
### Inbox The system also provides notifications and alerts when stacked PRs require attention, such as when reviews are needed or when merge conflicts need manual resolution. # Providing feedback This guide helps you provide feedback on code generated by Runbooks to ensure high-quality results and iterative improvements. ### **Slash command `/aviator revise`** You can review the generated PR and use the `/aviator revise` [slash command](/mergequeue/how-to-guides/slash-commands) to trigger automatic code revisions based on your feedback. There are a few ways this command can be submitted: #### As a PR Comment The command can be posted as a top level PR comment to address all unresolved review comments on the PR. This can be helpful for scenarios where multiple reviewers have left comments. ![/aviator revise as a PR comment](/files/CDpzsWhiJNlqlMiTwchm) If you want to exclude certain comments to be addressed, please mark them as resolved manually. Runbooks agents will ignore any resolved comments in the PR. {% hint style="info" %} Runbooks agents reacts with a 👍 emoji to acknowledge the command. {% endhint %} This comment can also be posted as part of the review summary: ![/aviator revise as a review summary](/files/QgFByUyZGaHvCxCaVIfA) If posted as the review summary, Runbooks agents will accept all the unresolved comments (not just the ones part of this review). ### Posting with inline instructions You can optionally also provide some inline instructions when posting the `/aviator revise` comment. In such case, the Runbooks agents will also take those instructions into account when revising the PR. In case there are no unresolved comments, `/aviator revise ` can also be used to be provide a single inline instruction to revise the PR. {% hint style="info" %} If there are no unresolved comments and no inline instructions are provided, Runbooks agents will return an error, posted as a comment. {% endhint %} ### As a review comment If you want Runbooks agents to only address a single review comment, you can post `/aviator revise` as a review comment itself. In such case, the Runbooks agents will ignore all the other open review comments in the PR. ![/aviator revise as a review comment](/files/EnNgPsynUNVxtLaGIWlC) Similarly, you can also revise an existing comment using the same strategy by posting `/aviator revise` as a reply thread on a existing review comment. ![/aviator revise as a review thread](/files/rN5HL5igJgTwQj9nuSzq) If there are multiple replies in a review thread, Runbooks agents will pick all the replies together to address the change requested. ### Troubleshooting **I posted /aviator revise but it didn't do anything.** Ensure that the comment was not posted as pending comment. And make sure that it's a Runbooks generated PR. Currently `/aviator revise` only works on PRs generated by Runbooks. If there is an emoji reaction, the Runbooks agents might be still processing the request. Check the Runbooks URL posted in the PR description for real-time status. Make sure no other tasks is being run for that particular Runbook. To avoid conflicts, Runbooks agents only perform one action at a time for a given Runbook. If the problem persists, please contact us ****. ### Best Practices #### Be Specific and Concrete ❌ **Vague**: "This doesn't look right." ✅ **Specific**: "Preserve the existing error handling logic in `handleAuth()`. Currently, it removes the try-catch block which we need for logging." #### Provide Context ❌ **Missing context**: "Add tests." ✅ **With context**: "Add unit tests using Jest, following the pattern in `tests/auth/oauth.test.ts`. Focus on testing the error paths since this code handles sensitive authentication logic." #### Reference Existing Code ✅ **Good feedback**: ``` "The new UserService class should follow the same pattern as OrderService in src/services/OrderService.ts, including: - Constructor dependency injection - Private helper methods for validation - Consistent error handling with ServiceError class" ``` #### Address One Issue at a Time When providing feedback on multiple issues, break them into separate, numbered points: ``` Feedback on Step 4: 1. The database query needs pagination - we have 10M+ users 2. Add an index on the email column for performance 3. Use prepared statements to prevent SQL injection 4. Return a consistent error format matching our API spec ``` #### Include Error Messages When something fails, include the actual error: ``` "Step 3.2 execution failed with this error: TypeError: Cannot read property 'map' of undefined at line 45 The issue is that the code assumes `users` is always an array, but it can be undefined when the API returns an error." ``` ### Feedback Examples #### Example 1: Using `/aviator revise` for PR Feedback **Step 1: Review the Generated PR** Runbook executes Step 3.1 and creates PR #456. You review and find several issues. **Step 2: Add Specific Line Comments** ``` [Comment on line 23 in UserService.ts] This should handle the case where email is null or undefined. The current code will throw an error. [Comment on line 45 in UserService.ts] We need to log this error to our monitoring system using logger.error() before throwing. [Comment on line 78 in UserService.test.ts] Missing test case for when the API returns a 429 rate limit error. ``` **Step 3: Choose Your Revision Strategy** **Option A - Process All Comments Together**: ``` [Add as top-level PR comment] /aviator revise ``` This will: * Pick up all three unresolved comments * Generate code changes addressing each issue * Push a new commit with all fixes **Option B - Fix Issues One at a Time**: ``` [Reply in the thread for line 23] /aviator revise [After reviewing that fix, reply in the thread for line 45] /aviator revise [After reviewing that fix, reply in the thread for line 78] /aviator revise ``` **Option C - Provide Inline Feedback**: ``` /aviator revise Add null check for email field at line 23: if (!email) throw new ValidationError('Email is required') ``` **Step 4: Review and Iterate** After Aviator pushes the changes: 1. Review the new commit 2. If satisfied, resolve the conversation 3. If more changes needed, add another comment and use `/aviator revise` again #### Example 2: Refining Requirements **Initial request**: "Migrate from Redux to React Context" **Runbook generates**: Basic context migration **Your feedback**: ``` "Good start, but we need to: 1. Keep Redux for server state (API calls) 2. Use Context only for UI state (theme, sidebar) 3. Maintain the same action creator pattern for consistency 4. Add TypeScript types for all context values" ``` #### Example 3: Correcting Technical Details **Step 3.1**: "Update imports to use ES modules" **Your feedback**: ``` "We're using CommonJS (require/module.exports) because we need to support Node 14. Please keep the CommonJS syntax and instead focus on organizing the imports alphabetically and grouping internal vs external dependencies." ``` #### Example 4: Adding Implementation Details **Step 2**: "Add error handling to API calls" **Your feedback**: ``` "Please implement error handling following this pattern: 1. Use our custom ApiError class from src/errors/ApiError.ts 2. Catch network errors separately from API errors 3. Log errors using our structured logger (src/utils/logger.ts) 4. Return user-friendly messages from src/constants/errorMessages.ts 5. Preserve the original error in the log for debugging Example from src/services/UserService.ts lines 45-60." ``` #### Example 5: Providing Code Examples **Step 4.2**: "Update tests to match new API" **Your feedback**: ```` "Here's the pattern for testing async API calls in our codebase: ```typescript describe('fetchUserData', () => { it('should handle success response', async () => { mockApiClient.get.mockResolvedValue({ data: mockUser }); const result = await fetchUserData(123); expect(result).toEqual(mockUser); }); it('should throw ApiError on failure', async () => { mockApiClient.get.mockRejectedValue(new Error('Network error')); await expect(fetchUserData(123)).rejects.toThrow(ApiError); }); }); ``` Please follow this pattern for all the new tests." ```` #### Quick Reference: `/aviator revise` Commands | Command | Where to Use | What It Does | | ---------------------------- | -------------------- | ---------------------------------------------- | | `/aviator revise` | Top-level PR comment | Processes all unresolved comments in the PR | | `/aviator revise` | Comment thread reply | Processes only that specific thread's feedback | | `/aviator revise ` | Anywhere in PR | Applies the inline feedback immediately | #### Key Takeaways ✅ **Do**: * Use `/aviator revise` for code-level feedback on PRs * Add specific line comments before triggering revisions * Provide context and examples in your review comments * Iterate: review changes, add more feedback, use `/aviator revise` again * Edit runbook steps directly for plan-level changes ❌ **Don't**: * Don't wait until everything is wrong to provide feedback * Don't provide vague feedback like "this doesn't work" * Don't forget to test changes after they're applied * Don't skip adding comments before using top-level `/aviator revise` Remember: Runbooks learns from your feedback. The more detailed and constructive your input, the better it becomes at understanding your codebase and requirements. # Handling CI failure This guide gives an overview of how Runbooks agents auto-fix CI failures. Runbooks agents currently can only fix failure in pull requests generated by Runbooks. ### Detecting CI failure Runbooks framework automatically track any CI failures on the pull requests that are generated by Runbooks. Once the in-flight and queued steps are completed, Runbooks agents start monitoring the CI status checks of the commit SHA. A few things to note: * If there are multiple PRs generated by Runbooks, all PRs are monitored for CI status check. * Runbooks only tracks the latest head commit SHA for each PR. * If latest head commit SHA is added manually (not be Runbooks), CI is not monitored. * If a new commit gets added in middle of the CI validation workflow, the CI rework workflow is terminated. * Runbooks will monitor all CI checks that are reported in GitHub and will work through only the first failed status check among one of the supported CI (see below). ### Rework config By default, Runbooks agents will attempt fixing the CI 2 times. This property can be updated via Runbooks Config > Pull Requests > **CI Failure Auto-Rework.** For every attempt, Runbooks agents will also publish a summary as a GitHub comment. ![Screenshot 2025-12-11 at 11.49.54 AM.png](/files/6el1pUEmKn1mBuzptgmM) ### Supported CI To understand the CI failure reason, Runbooks agents analyze the CI logs #### GitHub workfows GitHub workflows are natively supported and no additional permissions are required. #### Buildkite To access Buildkite logs, you must provide the Buildkite API token through the [Workspace integrations page](https://app.aviator.co/settings/workspace/integrations): **Step 1**: Go to “Personal Settings” from the top-right corner
**Step 2**: Click “API Access Tokens”
**Step 3**: Click “New API Access Token” Name the token and pick your organization
Choose “**Read Artifacts**”, “**Read Builds**”, “**Read Build logs**”, and “**Read Pipelines”**. ![Screenshot 2025-12-11 at 11.44.17 AM.png](/files/llOLARl5fENQZG8YlWIy) Click “Create New API Access Token” **Step 4**: Copy the created access token, and go to , paste the created access token to the Buildkite integration.
# SSH Sandboxes Configuration Guide [SSH Sandboxes](/runbooks/concepts/ssh-sandboxes) allow you to run Claude Code runbooks on your own infrastructure instead of using Aviator’s managed cloud sandboxes. This gives you more control over the execution environment and allows you to work with private resources that aren’t accessible from the cloud. ### Overview SSH Sandboxes connect to your own servers or virtual machines where Claude Code can execute runbook steps. This is useful when: * You need access to private internal resources * You have specific security or compliance requirements * You want to control the execution environment and dependencies * You need to work with on-premises infrastructure ### Prerequisites Before configuring SSH sandboxes, ensure you have: 1. **SSH Server Access**: A Linux/Unix server with SSH access 2. **SSH Key Pair**: An SSH private key for authentication 3. **Claude Code**: Claude Code CLI is installed on this sandbox 4. **Working Directory**: A designated directory where Claude Code can create and modify files. This is the root level directory that will host GitHub repositories. 5. **Network Access**: The server should be reachable from Aviator’s infrastructure ### Configuration Steps #### 1. Choose Sandbox Type 1. Navigate to **Runbooks** → **Config** → **Sandbox** 2. Select **SSH Sandbox (Self-managed)** from the sandbox type options 3. You’ll see requirements and configuration options appear #### 2. Configure SSH Private Key When you select SSH sandbox, you’ll need to provide an SSH private key: 1. **If no key is configured**: A textarea will appear for you to paste your SSH private key 2. **If a key is already stored**: You’ll see a “SSH private key is securely stored” message with an option to update it **Supported Key Formats:** * OpenSSH format (recommended) * RSA keys * EC (Elliptic Curve) keys **Example OpenSSH key format:** ``` -----BEGIN OPENSSH PRIVATE KEY----- b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAA... -----END OPENSSH PRIVATE KEY----- ``` #### 3. Set Up SSH Sandbox Instances After configuring your SSH key, you need to register your sandbox instances using the REST API: ```bash # Add a new SSH sandbox instance curl -X POST \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "hostname": "your-server.example.com", "port": 22, "username": "ubuntu", "working_dir": "/home/ubuntu/runbooks", "active": true }' ``` **Required Fields:** * `hostname`: Server hostname or IP address * `port`: SSH port (usually 22) * `username`: SSH username for authentication * `working_dir`: Absolute path where Claude Code can work * `active`: Whether this sandbox is available for use ### Managing SSH Sandbox Instances #### List Sandbox Instances ```bash curl -X GET \ -H "Authorization: Bearer YOUR_API_TOKEN" ``` #### Update a Sandbox Instance ```bash curl -X PUT \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "hostname": "updated-server.example.com", "port": 2222, "username": "deploy", "working_dir": "/opt/runbooks", "active": true }' ``` #### Delete a Sandbox Instance ```bash curl -X DELETE \ -H "Authorization: Bearer YOUR_API_TOKEN" ``` ### Best Practices #### Security 1. **Use dedicated SSH keys**: Create a separate SSH key pair specifically for Aviator 2. **Limit user permissions**: Use a dedicated user account with minimal required permissions 3. **Restrict network access**: Configure firewall rules to allow connections only from Aviator’s IP ranges (`34.127.109.72`, `35.247.14.14`) 4. **Regular key rotation**: Periodically update your SSH private keys #### Server Setup 1. **Working directory**: Ensure the working directory has sufficient disk space and proper permissions. You can also preload the repositories in this directory for faster load time. For instance, if you specific working directory as: `/home/ubuntu/dev/` then, Aviator will search or create repositories in `/home/ubuntu/dev//`. If you preload the repository here, Aviator will reuse it and fetch only required branch. This can help save load time. 2. **Dependencies**: Install any tools or dependencies your runbooks might need 3. **Monitoring**: Set up monitoring to track resource usage during runbook execution 4. **Backup**: Regularly backup important data, as runbooks may modify files #### Performance 1. **Server resources**: Ensure adequate CPU, memory, and disk space for your runbooks 2. **Network latency**: Choose servers with low latency to Aviator’s infrastructure (we are hosted in US-Oregon region) 3. **Multiple instances**: Configure multiple sandbox instances for redundancy and load distribution ### Troubleshooting #### Common Issues **SSH Connection Failed** * Verify hostname and port are correct * Check if SSH service is running on the target server * Ensure firewall allows connections from Aviator’s infrastructure * Verify the SSH private key format and permissions **Authentication Failed** * Confirm the SSH private key matches the public key on the server * Check username is correct * Verify the user has SSH access permissions **Working Directory Issues** * Ensure the directory exists and is accessible * Check that the SSH user has read/write permissions * Verify sufficient disk space is available **Runbook Execution Errors** * Check server logs for detailed error messages * Verify required dependencies are installed * Ensure the working directory has proper permissions #### Getting Help If you encounter issues: 1. Check the server’s SSH logs (`/var/log/auth.log` or `/var/log/secure`) 2. Verify network connectivity from your local machine to the server 3. Test SSH connection manually with the same credentials 4. Contact Aviator support with relevant error messages and configuration details ### Security Considerations * SSH private keys are encrypted at rest using industry-standard encryption * Keys are never logged or exposed in plain text * All SSH connections use secure authentication * Consider using certificate-based authentication for enhanced security * Regularly audit SSH access logs on your servers ### See also * [SSH sandboxes](/runbooks/concepts/ssh-sandboxes) * [Cloud sandboxes](/runbooks/concepts/cloud-sandboxes) # SSH Sandboxes setup for onprem Onprem setup is very similar to the [cloud SSH sandbox setup](/runbooks/how-to-guides/ssh-sandboxes-configuration-guide), only a few things differ. ### Overview SSH Sandboxes connect to your own servers or virtual machines where Claude Code can execute runbook steps. This is useful when: * You need access to private internal resources * You have specific security or compliance requirements * You want to control the execution environment and dependencies * You need to work with on-premises infrastructure ### Prerequisites Before configuring SSH sandboxes, ensure you have: 1. **SSH Server Access**: A Linux/Unix server with SSH access 2. **SSH Key Pair**: An SSH private key for authentication 3. **Claude Code**: Claude Code CLI is installed on this sandbox 4. **Working Directory**: A designated directory where Claude Code can create and modify files. This is the root level directory that will host GitHub repositories. 5. **Network Access**: The server should be reachable from Aviator’s infrastructure ### Configuration Steps #### 1. Choose Sandbox Type 1. Navigate to **Runbooks** → **Config** → **Sandbox** 2. Select **SSH Sandbox (Self-managed)** from the sandbox type options 3. You’ll see requirements and configuration options appear #### 2. Configure SSH Private Key When you select SSH sandbox, you’ll need to provide an SSH private key: 1. **If no key is configured**: A textarea will appear for you to paste your SSH private key 2. **If a key is already stored**: You’ll see a “SSH private key is securely stored” message with an option to update it **Supported Key Formats:** * OpenSSH format (recommended) * RSA keys * EC (Elliptic Curve) keys **Example OpenSSH key format:** ``` -----BEGIN OPENSSH PRIVATE KEY----- b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAA... -----END OPENSSH PRIVATE KEY----- ``` #### 3. Set Up SSH Sandbox Instances After configuring your SSH key, you need to register your sandbox instances using the REST API: ```bash # Add a new SSH sandbox instance curl -X POST \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "hostname": "your-server.example.com", "port": 22, "username": "ubuntu", "working_dir": "/home/ubuntu/runbooks", "active": true }' ``` **Required Fields:** * `hostname`: Server hostname or IP address * `port`: SSH port (usually 22) * `username`: SSH username for authentication * `working_dir`: Absolute path where Claude Code can work * `active`: Whether this sandbox is available for use ### Managing SSH Sandbox Instances #### List Sandbox Instances ```bash curl -X GET \ -H "Authorization: Bearer YOUR_API_TOKEN" ``` #### Update a Sandbox Instance ```bash curl -X PUT \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "hostname": "updated-server.example.com", "port": 2222, "username": "deploy", "working_dir": "/opt/runbooks", "active": true }' ``` #### Delete a Sandbox Instance ```bash curl -X DELETE \ -H "Authorization: Bearer YOUR_API_TOKEN" ``` ### Using Sidecar container for sandboxes In a kubernetes self-hosted configuration, it's possible to use Sidecar containers within the agent-worker pod such that each agent-worker pod is running a SSH sandbox alongside the primary agent-worker container. To configure such container, specify the following environment variables on your pods: * `SSH_SANDBOX_SIDECAR_URL`: This would likely be `localhost` but it can be different depending on your setup. Also include the port if applicable, defaults to `22`, example: `localhost:5454`. * `SSH_SANDBOX_SIDECAR_ROOT_DIR`: This is same as `working_dir` in the API above. When using these environment variables, agent-workers will ignore all Sandboxes specified via the API. It will assume each worker has it's own sandbox. This approach could be useful to incorporate ephemeral sandboxes within your kubernetes cluster running Aviator server, and be able to scale up or scale down the instances. ### Best Practices #### Security 1. **Use dedicated SSH keys**: Create a separate SSH key pair specifically for Aviator 2. **Limit user permissions**: Use a dedicated user account with minimal required permissions 3. **Restrict network access**: Configure firewall rules to restrict ingress connections only from the Aviator's background worker instances (the main onprem server). 4. **Regular key rotation**: Periodically update your SSH private keys #### Preloading instances 1. **Working directory**: Ensure the working directory has sufficient disk space and proper permissions. You can also preload the repositories in this directory for faster load time. For instance, if you specific working directory as: `/home/ubuntu/dev/` then, Aviator will search or create repositories in `/home/ubuntu/dev//`. If you preload the repository here, Aviator will reuse it and fetch only required branch. This can help save load time. 2. **Dependencies**: Install any tools or dependencies your runbooks might need 3. **Monitoring**: Set up monitoring to track resource usage during runbook execution 4. **Backup**: Regularly backup important data, as runbooks may modify files #### Performance 1. **Server resources**: Ensure adequate CPU, memory, and disk space for your runbooks 2. **Network latency**: Choose servers with low latency to Aviator’s infrastructure (we are hosted in US-Oregon region) 3. **Multiple instances**: Configure multiple sandbox instances for redundancy and load distribution ### Troubleshooting # FAQ & Troubleshooting #### Frequently Asked Questions **Q: What programming languages are supported?** A: The system is language and framework agnostic. Agents can work with any codebase accessible via GitHub. **Q: What agents are supported?** A: Currently Runbooks supports Claude code agents, the support for Gemini and Codex is coming soon. **Q: Are there repository size limitations?** A: There are no hard limits on repository size, though larger codebases may consume more LLM tokens. **Q: How do agents handle build failures?** A: Agents automatically analyze build and test results, then iteratively recreate changes to fix issues. **Q: Can I use my own LLM API keys?** A: Yes, you can provide your own Claude API keys. Aviator also support Bedrock. #### Common Issues **Agent Connection Failures** * Verify GitHub App permissions are correctly configured * Check network connectivity from agent containers * Ensure API gateway is accessible **High Token Usage** * Review context management settings * Consider breaking large tasks into smaller Runbooks * Optimize search patterns for large codebases **PR Creation Errors** * Confirm write permissions for target repository * Check branch protection rules * Verify agent has latest repository state # API Reference for Runbooks This reference page describes the usage of REST API for Runbooks. Get instructions on how to create a runbook, check its status, and more. This document describes the usage of REST API for Runbooks. You can also use [GraphQL API](/api/reference/graphql) for composite queries. ## Create a Runbook `POST` `/api/v1/runbook` This creates a new Runbook and triggers its execution asynchronously. The runbook is created using the provided prompt and repository, and the response includes the runbook number and a URL to track its progress. **Headers** | Name | Value | | ------------- | ------------------ | | Content-Type | `application/json` | | Authorization | `Bearer ` | **Body** | Name | Type | Description | | --------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `repository` | Object | Repository to run the runbook against. Must contain `org` (GitHub organization name) and `name` (repository name). | | `prompt` | String | Task description for the runbook. | | `title` | String. *Optional* | Title for the runbook. If not provided, a title will be auto-generated. | | `oneshot` | Boolean. *Optional* | Whether to run in [one-shot mode](/runbooks/concepts/one-shot-mode). Default is `true`. | | `draft` | Boolean. *Optional* | Whether to create pull requests as drafts. If not provided, defaults to the project configuration. | | `pr_mode` | String. *Optional* | PR creation mode. Possible values: `manual`, `single_pr`, `stacked_pr`. If not provided, defaults to the project configuration. | | `target_branch` | String. *Optional* | Base branch for the runbook. If not provided, defaults to the repository's default branch. | | `description` | String. *Optional* | Text to prepend to the body of the first pull request opened by the runbook. Persisted on the runbook so subsequent body refreshes keep the prefix. | | `labels` | Array of strings. *Optional* | Labels applied to every pull request opened by the runbook. Persisted on the runbook (de-duplicated, order preserved) and re-applied on each PR. | | `author_email` | String. *Optional* | Email of the user to attribute the runbook to. Must resolve to a user in this account; otherwise the request returns `400`. Defaults to the user associated with the API token. | **Request body example** ```json { "repository": { "org": "acme-corp", "name": "backend" }, "prompt": "Upgrade all React dependencies to v19", "title": "React v19 upgrade", "oneshot": true, "draft": true, "pr_mode": "single_pr", "target_branch": "develop", "description": "Tracks the org-wide React v19 migration. See RFC-123 for context.", "labels": ["migration", "react-v19"], "author_email": "engineer@acme-corp.com" } ``` **Response** If successful, HTTP 202 response is returned back since the runbook runs asynchronously. The response uses the same schema as the [Get Runbook Status](#get-runbook-status) endpoint. At creation time, step counts are all zero and `pull_requests` is empty. {% tabs %} {% tab title="202" %} ```json { "runbook_number": 42, "url": "https://app.aviator.co/r/42", "status": "generating_steps", "steps": { "total": 0, "not_started": 0, "in_progress": 0, "completed": 0, "failed": 0, "queued": 0 }, "pull_requests": [] } ``` {% endtab %} {% tab title="400" %} ```json { "error": "bad-request", "message": "Invalid request body" } ``` {% endtab %} {% tab title="402" %} ```json { "error": "insufficient-credits", "message": "Account has no remaining runbook credits." } ``` {% endtab %} {% tab title="404" %} ```json { "error": "not-found", "message": "Repository not found." } ``` {% endtab %} {% endtabs %} ## Update a Runbook `PATCH` `/api/v1/runbook/` Append a new prompt to an existing Runbook and optionally update the PR description and labels. The new prompt is delivered as a follow-up chat message to the runbook's agent loop, which resumes execution and pushes additional commits to the same pull request branch. Description and label updates are persisted on the runbook and applied to its pull request after the agent's next commit. If the runbook has not opened a PR yet at the time of the call, the overrides still apply to the first PR it opens. **Headers** | Name | Value | | ------------- | ------------------ | | Content-Type | `application/json` | | Authorization | `Bearer ` | **Parameters** | Name | Description | | ---------------- | ----------------------------------- | | `runbook_number` | The number identifying the runbook. | **Body** | Name | Type | Description | | ------------- | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `prompt` | String | Follow-up prompt for the runbook. Delivered to the agent as a new user chat message; the agent resumes and pushes additional commits to the existing PR branch. | | `description` | String. *Optional* | Replaces the previously stored PR body prefix. If omitted, the existing prefix is preserved. | | `labels` | Array of strings. *Optional* | Labels to add to the runbook's pull request. Merged with previously stored labels (de-duplicated, order preserved); existing labels are not removed. | **Request body example** ```json { "prompt": "Also migrate the test files to use the new React v19 testing utilities.", "description": "Tracks the org-wide React v19 migration, including tests. See RFC-123.", "labels": ["tests"] } ``` **Response** If successful, HTTP 202 response is returned back since the additional work runs asynchronously. The response uses the same schema as the [Get Runbook Status](#get-runbook-status) endpoint and reflects the runbook's current state at the moment of the call (step counts and pull requests from work that's already been done). {% tabs %} {% tab title="202" %} ```json { "runbook_number": 42, "url": "https://app.aviator.co/r/42", "status": "in_progress", "steps": { "total": 5, "not_started": 0, "in_progress": 1, "completed": 4, "failed": 0, "queued": 0 }, "pull_requests": [ { "number": 1234, "url": "https://github.com/acme-corp/backend/pull/1234", "step_numbers": ["1.1", "1.2", "2.1"] } ] } ``` {% endtab %} {% tab title="400" %} ```json { "error": "bad-request", "message": "Invalid request body" } ``` {% endtab %} {% tab title="404" %} ```json { "error": "not-found", "message": "Runbook not found." } ``` {% endtab %} {% endtabs %} ## Get Runbook Status `GET` `/api/v1/runbook/` Retrieve the current status of a Runbook by its number. The response includes the aggregate status, step counts, and any associated pull requests. **Headers** | Name | Value | | ------------- | ---------------- | | Authorization | `Bearer ` | **Parameters** | Name | Description | | ---------------- | ----------------------------------- | | `runbook_number` | The number identifying the runbook. | **Response** If successful, HTTP 200 response is returned back. {% tabs %} {% tab title="200" %} ```json { "runbook_number": 42, "url": "https://app.aviator.co/r/42", "status": "in_progress", "steps": { "total": 5, "not_started": 1, "in_progress": 2, "completed": 1, "failed": 0, "queued": 1 }, "pull_requests": [ { "number": 1234, "url": "https://github.com/acme-corp/backend/pull/1234", "step_numbers": ["1.1", "1.2"] }, { "number": 1235, "url": "https://github.com/acme-corp/backend/pull/1235", "step_numbers": ["2.1"] } ] } ``` {% endtab %} {% tab title="404" %} ```json { "error": "not-found", "message": "Runbook not found." } ``` {% endtab %} {% endtabs %} ### Response schema The Create, Update, and Get endpoints all return the same response schema. | Name | Type | Description | | ------------------------------ | ------- | ------------------------------------------------------------------------------------ | | `runbook_number` | Integer | The unique number identifying the runbook. | | `url` | String | URL to view the runbook in the Aviator dashboard. | | `status` | String | Aggregate status of the runbook. See [Status values](#status-values) below. | | `steps` | Object | Breakdown of step counts by status. Only counts leaf steps (actual execution units). | | `steps.total` | Integer | Total number of leaf steps. | | `steps.not_started` | Integer | Steps that have not started. | | `steps.in_progress` | Integer | Steps currently executing. | | `steps.completed` | Integer | Steps that completed successfully. | | `steps.failed` | Integer | Steps that failed. | | `steps.queued` | Integer | Steps queued for execution. | | `pull_requests` | Array | Pull requests created by the runbook, deduplicated by PR number. | | `pull_requests[].number` | Integer | GitHub pull request number. | | `pull_requests[].url` | String | URL of the GitHub pull request. | | `pull_requests[].step_numbers` | Array | List of step numbers (e.g., `"1.1"`, `"2.3"`) associated with this pull request. | ### Status values The `status` field is computed from leaf step statuses at query time. | Status | Condition | | --------------------- | ------------------------------------------------------------------------------------- | | `generating_steps` | No steps exist yet (the runbook is still generating its execution plan). | | `not_started` | All steps are `not_started`. | | `in_progress` | At least one step is `in_progress`. | | `partially_completed` | Mix of `completed`/`queued` and `not_started` steps, with none in-progress or failed. | | `failed` | At least one step has `failed` and none are `in_progress`. | | `completed` | All steps are `completed`. | # Verify Aviator Verify replaces line-by-line code review with verification of *what the change is supposed to do*. You work with your agent locally. Aviator captures your intent through a single CLI call, runs every acceptance criterion against the running code, and produces a review document with verdicts and evidence. The reviewer judges behavior — not the diff. You can start using Verify with code-scan alone — no preview, no infrastructure setup. A preview is what unlocks **runtime verification**: scenarios driven against your running code. Most teams get value from day one with code-scan, then add a preview when behavioral criteria start mattering. ### The flow 1. **Build with your agent.** Implement the task with Cursor, Claude, Copilot — anything you use today. No new tooling, no behavior change. 2. **Submit intent with the Aviator CLI.** When you're done, the agent runs `/verify-submit`, which captures the intent and the acceptance criteria from what was built and submits them. Set up [agent hooks](/verify/how-to-guides/set-up-agent-hooks) and it gets prompted before every PR. 3. **Aviator verifies end to end.** Scenarios run in your preview, invariants apply automatically, code-scan handles structural checks. 4. **Review the behavior.** The reviewer sees the intent, the verdict per criterion, and the evidence. They approve, waive with reason, or ask for another scenario on the spot. → [How it works](/verify/how-it-works) — the full narrative, with diagrams. ### Invariants — the standing rules Every code review has a long tail of repeated comments. *Use the structured logger. No direct user-table writes. Errors emit a metrics counter. Cap external dependencies. Stripe amounts use the Money type.* Reviewers say the same thing on PR after PR, and any one of them missed in any one review becomes a future bug or migration. Verify turns those into **invariants** — team-defined rules in an account-level catalog that apply to every matching change automatically. When a runbook is created, an LLM selector picks which invariants legitimately apply to this change, and they're materialized as acceptance criteria alongside the user-supplied ones. From there they flow through the same verifier pipeline — code-scan or runtime — and produce verdicts on the same review surface. How invariants get into your catalog: * **Mine your PR history.** Aviator's AI reads your team's actual PR review comments and proposes invariants from the patterns it sees. The highest-leverage onboarding path — your team is already enforcing these rules in review, just not encoding anywhere. * **Extract from your docs.** `CONTRIBUTING.md`, `LLM.md`, and similar files become drafts. * **Adopt from templates.** Aviator's starter library covers the common categories: security baseline, observability, data access, backwards compatibility. * **Author manually.** Admins write the rule directly. All AI-drafted invariants land in draft status. An admin reviews each before it goes active. Failed invariant verdicts can be waived by the reviewer with a category (`false_positive`, `doesnt_apply`, `accepted_risk`, `fix_in_followup`) — every waiver is recorded in the audit trail. → [Concepts: Invariants](/verify/concepts/invariants) ### Why this approach Code review was designed when humans wrote code slowly. AI changes that. Code generates in minutes; reviewing it line by line still takes the same hour as before. The shift: * **Traditional review asks:** *Does this code look okay?* * **Verify asks:** *Does this code do what we agreed it should do?* Three things follow: * **Faster reviews.** A reviewer judges intent and evidence, not 800 lines of diff. * **Systematic checking.** Every criterion is verified, every time — no sampling, no fatigue. * **Complete audit trail.** Every change links intent → verifier verdict → evidence → reviewer decision. → [Why intent-driven verification](/verify/concepts/why-intent-driven-verification) ### Core concepts | Concept | What it is | | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | [Intent and acceptance criteria](/verify/reference/spec-format) | What the agent submits — what the change is for, and the verifiable assertions it must satisfy. | | [Invariants](/verify/concepts/invariants) | Team-defined rules applied to every matching change. Security baseline, data access, conventions. | | [Previews](/verify/concepts/previews) | Ephemeral environments scenarios run against. Per-run, torn down after. | | [Verification layers](/verify/concepts/verification-layers) | How criteria, invariants, and domain contracts stack. | | [Audit and compliance](/verify/concepts/audit-trails-and-compliance) | Immutable trail of every step — submission, verdicts, evidence, decisions. | ### Getting started 1. [Connect a repository](/verify/how-to-guides/connect-a-repository) — wire up GitHub. 2. [Your first verification](/verify/your-first-spec) — a 15-minute hands-on. Install the CLI, ship a small change, watch verification run. No preview needed. 3. [Setting up org invariants](/verify/setting-up-org-invariants) — codify the rules every change must respect. 4. [Creating a preview](/verify/how-to-guides/creating-a-preview) — *optional.* Unlocks runtime verification when your criteria start asking about behavior the diff alone can't answer. ### Quick links * **Submitting:** [Aviator CLI](/verify/reference/cli) · [Set up agent hooks](/verify/how-to-guides/set-up-agent-hooks) * **Writing for the agent:** [SKILL.md guide](/verify/how-to-guides/writing-a-skill-md) · [Effective acceptance criteria](/verify/how-to-guides/writing-effective-acceptance-criteria) * **Configuration:** [Preview YAML](/verify/reference/preview-yaml) · [Spec format](/verify/reference/spec-format) * **Operations:** [Managing previews](/verify/how-to-guides/managing-previews) · [Seed data for previews](/verify/how-to-guides/seed-data-for-previews) * **Reviewing:** [Review verification on the pull request](/verify/how-to-guides/verify-on-github) · [Understanding verification results](/verify/reference/understanding-verification-results) * **When things go wrong:** [Fixing verification failures](/verify/how-to-guides/fixing-verification-failures) # How it works Verify replaces line-by-line code review with verification of *what the change is supposed to do*. You work with your agent locally. Aviator captures your intent through a CLI call, runs every acceptance criterion against the running code, and produces a review document with verdicts and evidence.
Four-stage Verify workflow: build, submit intent, verify, review

The end-to-end Verify loop

The full loop: 1. **Build with your agent.** Implement the task with Cursor, Claude, Copilot, or anything else you use today. No new tooling, no behavior change. 2. **Submit intent with the Aviator CLI.** When you're done, the agent runs `/verify-submit`, which reads the change and submits it through the [Aviator CLI](/verify/reference/cli). It captures the intent you and the agent agreed on and the acceptance criteria from what was built. 3. **Aviator verifies end to end.** Every criterion runs through the verification pipeline. Code-scan handles structural checks against the diff — this works out of the box. Runtime checks run against a preview environment if you've configured one (optional; see [Concepts: Previews](/verify/concepts/previews)). Matching invariants apply automatically. 4. **Review the behavior.** The reviewer sees the intent, the verdict per criterion, and the evidence behind each verdict. They can approve, waive with reason, or ask for another scenario on the spot. ### What gets submitted The submission carries two things: * **Intent** — a short statement of *what* this change is for. It carries the constraints that aren't visible from the diff alone. * **Acceptance criteria** — verifiable assertions the change must satisfy. Each criterion is independent and has a clear pass/fail shape. The agent generates both from what was actually built. You submit *after* implementing, not before — there is no separate "approve the spec first" step. ### The verification pipeline Every criterion runs through exactly one verifier. A classifier picks the verifier based on the criterion text and the files the change touched.
Verification pipeline: criterion is classified and routed to one of two verifier paths

Each criterion is classified and routed to a single verifier path

| Verifier | Used for | Evidence captured | | ------------- | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------- | | **Code-scan** | Structural assertions — file scope, dependency surface, function signatures | Diff or AST snippets that demonstrate the assertion | | **Runtime** | Behavioral assertions — endpoint contracts, error shapes, side effects, UI behavior | Screenshots, console logs, DOM snapshots, API responses, full traces | ### Invariants in the loop Acceptance criteria are what *this* change should do. **Invariants** are what *every* change should respect — your team's standing rules. They're a major part of the flow once you start using Verify seriously. Four phases: 1. **Catalog.** Invariants live in an account-level catalog. Admins can author them manually, adopt from templates, or accept AI-drafted ones. Three of the AI sources worth knowing about: * **PR-comment mining.** AI reads your team's actual PR review comments and proposes invariants from the patterns. The highest-leverage source — you're already enforcing these in review, this just encodes them. * **Docs extraction.** Your `CONTRIBUTING.md`, `LLM.md`, or similar guidance files are read and proposed as invariants. * **Repo-signal synthesis.** AI looks at the shape of the codebase and proposes rules that fit. All AI drafts wait in pending status. An admin promotes drafts to active before they start producing verdicts. 2. **Selection.** When a runbook is submitted, an LLM **selector** reads the intent, the acceptance criteria, and the change set, and picks which eligible invariants legitimately apply to this change. Eligibility is gated by optional conditions on the invariant (e.g. `file_path_glob: src/**/*.py`); among the eligible, the selector decides what actually fits. 3. **Materialization.** Selected invariants are materialized as acceptance criteria on the runbook, tagged with `source: baseline_invariant`. From here they flow through the same verifier pipeline as user criteria — code-scan or runtime — and produce verdicts on the same review surface. 4. **Review and waivers.** Reviewers see invariant verdicts alongside user criteria. Invariant-sourced criteria can't be edited per-runbook (they're catalog-managed), but they can be **waived** with a category: | Waiver category | When to use it | | ----------------- | ----------------------------------------------------------- | | `false_positive` | The invariant fired but misjudged this case. | | `doesnt_apply` | The rule is valid but isn't relevant to this PR. | | `accepted_risk` | The failure is real but the author accepts the trade-off. | | `fix_in_followup` | The failure is real and will be addressed in a separate PR. | Every waiver is attributed and reasoned, and shows up in the audit trail. The first invariant you add changes the value of every future change going through Verify — because that rule now applies automatically, forever, without anyone having to remember it. → [Concepts: Invariants](/verify/concepts/invariants) ### Why previews matter (and why they're optional) Previews are optional. Verify works on day one with code-scan alone — the classifier routes every criterion to code-scan when no preview is configured, and you get verdicts on structural assertions from the start. A preview unlocks **runtime verification**: behavioral criteria — endpoint contracts, UI flows, error shapes — need the code to actually run. Aviator builds an ephemeral preview environment per run: it loads your image, injects secrets, runs a setup script, and exposes the port the scenarios hit. The reviewer can also open the preview from the review document to explore it manually. Most teams start without a preview and add one when behavioral criteria start mattering. A repo can declare multiple previews — staging, sandbox, prod-mirror — and tag scenarios to the one they need. ### The review document The review document is the surface reviewers actually use. Three regions: 1. **Intent** — what you and the agent agreed to build. 2. **Evidence** — every criterion with its verdict and the proof. 3. **Decisions** — request another scenario, open the preview, approve, or waive with reason. The reviewer is judging behavior and intent — not the diff. The diff is still there if you want it, but it's not the primary surface. The same verdicts and decisions are available on the pull request itself. With the [Aviator Chrome extension](/aviator-chrome-extension), a **Verify** tab on the PR mirrors the review document and carries the rerun, waive and remove actions. See [Review verification on the pull request](/verify/how-to-guides/verify-on-github). ### Audit and compliance Every step is recorded as an immutable event: intent submitted, who submitted it, criteria generated, verifier and verdict per criterion, evidence captured, reviewer decisions, exceptions and waivers. Compliance teams export this log as a single audit trail per change. → [Audit trails and compliance](/verify/concepts/audit-trails-and-compliance) ### Running with remote agents If you'd rather not run the agent locally, Aviator Runbooks runs the implementing agent inside a sandbox, submits the same way, and waits for verification. The verification path is identical — only the agent's location changes. Useful for batch work, off-hours runs, or non-developer-driven changes. ### See also * [Reference: Aviator CLI](/verify/reference/cli) * [How to: Set up agent hooks](/verify/how-to-guides/set-up-agent-hooks) * [Setting up org invariants](/verify/setting-up-org-invariants) * [Concepts: Invariants](/verify/concepts/invariants) * [How to: Writing a SKILL.md](/verify/how-to-guides/writing-a-skill-md) * [Concepts: Verification layers](/verify/concepts/verification-layers) * [Reference: Spec format](/verify/reference/spec-format) # Your first verification In this tutorial, you'll ship a small change end to end through Verify: implement it with your agent, submit the intent with the Aviator CLI, watch verification run, and read the review document. By the end, you'll have a working loop and a sense of where each piece fits. **Time:** \~15 minutes **Prerequisites:** * An Aviator org with a connected GitHub repo — see [Connect a repository](/verify/how-to-guides/connect-a-repository) * A coding agent — Claude Code or Codex * Write access to the repo You **don't need a preview** to do this tutorial. Without one, Verify runs code-scan only — every criterion is checked against the diff. You'll see a fully working loop end to end. Add a preview later when you want behavioral (runtime) verdicts; see [Creating a preview](/verify/how-to-guides/creating-a-preview). ### Step 1: Install the Aviator CLI and the agent plugin The CLI is how your agent talks to Aviator. Install it once per machine: ```bash brew install aviator-co/tap/aviator export AVIATOR_API_TOKEN= ``` Create a token at [app.aviator.co/settings/personal/api\_token](https://app.aviator.co/settings/personal/api_token). To keep it out of your shell profile, put it in `~/.config/aviator/config.yaml` instead — see [Authentication](/verify/reference/cli#authentication). Next, install the `/verify-submit` skill from the [Aviator agent plugins](https://github.com/aviator-co/agent-plugins). In Claude Code: ``` /plugin marketplace add aviator-co/agent-plugins /plugin install aviator@aviator-plugins ``` Finally, from inside the repo, set up the pre-PR reminder so your agent doesn't need to be told each time: ```bash aviator init ``` See [Set up agent hooks](/verify/how-to-guides/set-up-agent-hooks) for what this writes and how to scope it to your team or just yourself. It's optional for this tutorial — you can prompt the agent directly instead. ### Step 2: Pick a small change For the tutorial, pick something small and concrete. Examples that work well for a first run: * Add a `GET /health` endpoint that returns `{"status": "ok"}` with a 200. * Add a `/version` endpoint that returns the current commit SHA. * Add a feature flag check to an existing endpoint. Keep the scope to one or two files. You're learning the loop, not stress-testing the verifier. ### Step 3: Implement the change Open your agent in the repo. Describe the change normally: > Add a `GET /health` endpoint to the API. It should return `{"status": "ok"}` with a 200 response and not require authentication. Let the agent implement it. Iterate until you're aligned on the details — same flow you use today. Don't worry about Verify yet. When you're happy with the change, tell the agent it's ready. ### Step 4: Submit the intent Tell the agent to submit the intent: > /verify-submit The skill reads the change, drafts the submission with you, and calls `aviator verify`. It carries: * **Intent** — what the change is for, captured from your conversation. * **Acceptance criteria** — the verifiable assertions, generated from what was built. * **Working branch** — the branch the work lives on, so the PR you open from it is verified against these criteria. The CLI prints a session URL like `https://app.aviator.co/r/218`. Open the URL in your browser. You'll see the runbook with the generated plan, the acceptance criteria, and (once verification starts) a streaming verdict per criterion. ### Step 5: Watch verification run The runbook page updates as the verification pipeline progresses. The phases: 1. **Runbook generation.** Aviator turns the submission into a structured plan + acceptance criteria. Any matching invariants from your account catalog are materialized as additional criteria. 2. **Preview is built and booted** (only if you've configured a preview). Your image is loaded, secrets injected, setup script runs. 3. **Criteria run.** Each criterion is routed to one of two verifier paths — code-scan (static analysis of the diff) or runtime (executed against the preview). Without a preview, every criterion routes to code-scan. For a small change, the whole run completes in 30–90 seconds. ### Step 6: Read the review document When the run finishes, each criterion shows: * **Verdict** — `pass`, `fail`, `warn`, or `error`. * **Verifier** — which path ran it (Code-scan or Runtime). * **Evidence** — the diff snippet (code-scan) or runtime output (request/response, screenshot, log line) that backs the verdict. For the health endpoint without a preview, you should see something like: | Criterion | Verifier | Verdict | | ----------------------------------- | --------- | ------- | | Endpoint `GET /health` exists | Code-scan | ✓ Pass | | Returns 200 | Code-scan | ✓ Pass | | Response body is `{"status": "ok"}` | Code-scan | ✓ Pass | | Does not require authentication | Code-scan | ✓ Pass | With a preview configured, the behavioral criteria would route to Runtime instead — and you'd see the actual request + response as evidence. Click any verdict to see the evidence. If something failed, the verdict is annotated with the file and line that caused it. Fix the issue, then start another run from the runbook UI — a plain push doesn't re-run Verify on its own (see [When a run is triggered](/verify/concepts/how-verification-works#when-a-run-is-triggered)). ### Step 7: Approve (or send back) From the review document you can: * **Approve** — sign off and continue your normal merge flow. * **Waive a failed verdict with a category** — `false_positive`, `doesnt_apply`, `accepted_risk`, or `fix_in_followup`. Recorded in the audit trail. * **Edit the acceptance criteria** — use `aviator edit` to replace them, then re-verify. Re-running `/verify-submit` creates a new session, so edit rather than resubmit. * **Open the preview** — if you have one configured, poke at the running code yourself before approving. If you'd rather stay on GitHub, the [Verify tab on the pull request](/verify/how-to-guides/verify-on-github) carries the verdicts and the rerun, waive and remove actions. Approve to close the loop. The audit trail now has a complete record: runbook submission, verdicts per criterion with evidence, your decision. ### What you just learned * Verify runs *after* you build, not before. There's no separate "approve the spec first" step. * The CLI is the only handoff between you and Aviator. One call carries everything. * Acceptance criteria come from what was built, not what you planned upfront. * The review document is the surface — verdicts and evidence per criterion, not a 500-line diff. * Reviewer decisions, scenario evidence, and verdicts are all recorded as one immutable audit trail per change. ### Next steps * [Creating a preview](/verify/how-to-guides/creating-a-preview) — unlock runtime verdicts. This is usually the next step once you have a feel for the loop. * [How Verify works](/verify/how-it-works) — the full picture * [Concepts: Invariants](/verify/concepts/invariants) — encode team rules so they apply automatically * [Writing a SKILL.md](/verify/how-to-guides/writing-a-skill-md) — give the scenario runner the context it needs * [Fixing verification failures](/verify/how-to-guides/fixing-verification-failures) — what to do when a verdict goes red # Setting up org invariants Invariants are rules that apply to every matching change. Instead of asking the agent to assert "requires authentication" on every submission, you encode it once. Every matching runbook is then checked automatically. In this tutorial, you'll create a security invariant by hand, watch the selector pick it up, and learn how it appears in a real verification run. **Time:** \~10 minutes **Prerequisites:** * Admin access to your Aviator account. * At least one repository connected to Verify. * Familiar with [How Verify works](/verify/how-it-works) and [Concepts: Invariants](/verify/concepts/invariants). ### Step 1: Open the invariant editor Go to **Verify → Settings → Invariants**. You'll see the invariant catalog — every entry your account has, grouped by category. The list may already include AI-drafted entries (from PR-comment mining or docs extraction) and template-derived ones; those wait in draft status until an admin promotes them. For this tutorial, you'll create a manual entry. ### Step 2: Create a new invariant Click **New invariant** and give it: * **Title:** `auth-required-on-handlers` * **Category:** `security` * **Source:** `manual` (set automatically when you create from the UI) Pick a title you'll be okay with seeing in the audit trail — every verdict references it. ### Step 3: Write the body The body is the assertion the verifier will check. Keep it specific about the assertion, vague about the implementation: ``` All HTTP handlers must call an authentication middleware before any business logic. Endpoints that intentionally accept anonymous traffic must declare themselves as exceptions (see Conditions below). ``` A few rules (covered in depth in [Invariants — Writing a good invariant](/verify/concepts/invariants#writing-a-good-invariant)): * Don't name specific functions or modules — the rule should survive renames. * Don't write the *fix* — the verifier will explain what's wrong. * If you need "do X *except* when Y," use conditions or accept that the selector will pass on Y-shaped runbooks. ### Step 4: Add conditions (optional) Conditions gate when the invariant is *eligible* to apply. Most invariants don't need them — leave them empty and the invariant is eligible for every runbook. For this security rule, add a condition if you want to scope by language: * **Condition type:** `file_path_glob` * **Pattern:** `src/**/*.go` This makes the invariant eligible only when a runbook touches Go files under `src/`. Eligibility doesn't mean the invariant applies — the selector still decides per-runbook whether it's a defensible match. Conditions are useful for hard exclusions, not fine-grained scoping. ### Step 5: Save as draft Click **Save**. New invariants land in **draft** status. Draft invariants are visible in the catalog but the selector ignores them when building runbook criterion lists. Use the draft state to review the rule's wording — both with yourself and with the rest of your team — before it starts producing verdicts. ### Step 6: Promote to active Once you're happy with the wording, edit the invariant and toggle **Status → Active**. From this point on, the selector considers this invariant for every new runbook. When it picks the invariant for a runbook, it materializes it as an acceptance criterion (tagged with `source: baseline_invariant`) and the criterion is verified through the standard pipeline. The audit trail records the promotion: who promoted, when, and from which draft state. ### Step 7: Watch it in a real runbook Trigger a verification on a PR that touches Go files under `src/`. Open the review document. If the selector picked your invariant for this change, you'll see a criterion in the criterion list tagged as an invariant. It runs alongside the user-supplied criteria. Verdict + evidence are produced the same way. If the selector *didn't* pick it, that's fine — the LLM judged it didn't apply to this change. Look at the runbook's spec and scope: does the rule actually apply here? If yes and the selector missed it, the rule body may be too vague. ### Step 8: See a violation Push a change that intentionally violates the rule — for example, a handler that skips the middleware: ```go func (h *Handler) PublicEndpoint(w http.ResponseWriter, r *http.Request) { // No auth middleware called w.Write([]byte("hello")) } ``` The next verification run shows the invariant criterion as failed: ``` ✗ auth-required-on-handlers (invariant) Handler PublicEndpoint at src/handlers/public.go:23 does not call an authentication middleware before responding. ``` The verdict explains the failure but doesn't prescribe the fix — that's deliberate. ### Step 9: Waive when appropriate If a verdict is wrong or doesn't apply, the reviewer can waive it with a categorized reason: `false_positive`, `doesnt_apply`, `accepted_risk`, or `fix_in_followup`. Waive from the review document, or from the [Verify tab on the pull request](/verify/how-to-guides/verify-on-github). Every waiver is recorded in the audit trail. If you find yourself waiving the same invariant repeatedly, the rule is wrong. See [Concepts: Invariants — Waivers](/verify/concepts/invariants#waivers). ### What you just did * Created a manual invariant in the account catalog. * Wrote a rule body that's specific about the assertion but doesn't prescribe implementation. * Added a glob condition to scope eligibility by language. * Promoted it from draft to active, at which point the selector started considering it. * Saw it materialize as a criterion in a real runbook and produce a verdict. ### Next steps * [Concepts: Invariants](/verify/concepts/invariants) — sources (including AI-from-PR-comments), conditions, categories, waivers. * [Verification layers](/verify/concepts/verification-layers) — how invariants compose with user criteria. * [Fixing verification failures](/verify/how-to-guides/fixing-verification-failures) — what reviewers do when an invariant verdict goes red. # Concepts # Why intent-driven verification This document explains the reasoning behind Verify's approach to code changes. ### Code review was designed for human-written code A developer would spend hours or days writing a feature, then submit it for review. A reviewer could read the diff, understand the changes, and catch issues. This worked because: * Code was written slowly * Diffs were human-sized * Reviewers had time to think deeply AI-generated code breaks all three. ### AI changes the math AI tools can generate code in minutes. A developer describes what they want; the AI produces an implementation. The code is often correct, but it's also: * Produced faster than it can be reviewed line by line * Harder to evaluate — *is this what I actually asked for?* * Easy to accumulate without careful inspection Teams face a bad set of choices: 1. Review everything carefully → reviewers become a bottleneck. 2. Review quickly → catch fewer issues. 3. Skip review for AI code → lose oversight. None of these are good. They're all symptoms of the same problem: the *thing being reviewed* is wrong. ### The wrong question Traditional review asks: **"Does this code look okay?"** A reviewer reads the diff and makes a judgment. Problems with this: * "Okay" is subjective. * Reviewers have different standards. * The reviewer often doesn't know what the code was *supposed* to do. * Fatigue produces rubber-stamping. For AI-generated code, the question gets harder, not easier. The code is often clean and idiomatic — AI is good at that. But is it *correct*? Does it match what was intended? ### The right question Verify asks: **"Does this code do what we agreed it should do?"** Answering that requires two things: 1. **A clear statement of what the change is for** — captured as the intent and the acceptance criteria the change must satisfy. 2. **Systematic checking that the running code matches** — verdicts and evidence for every criterion. The intent is the contract. Verification is the enforcement. The reviewer judges *both* — was this the right intent, and did verification actually demonstrate it. ### Where reviewers actually spend their time In the Verify flow, the agent implements the change, then submits the intent and acceptance criteria through the MCP. Verification runs. The reviewer opens a review document that contains: * The intent. * Every criterion with a verdict. * The evidence behind each verdict — scenario output, matched invariants, code-scan diffs. That's what the reviewer reads. They don't trawl the diff. They check that the intent reflects what was actually needed, and that the evidence convinces them the criteria were really met. This is faster than diff review because: * The intent is short. * The evidence is concrete, not interpreted. * The reviewer isn't second-guessing implementation choices the agent already made and the verifier already approved. * There's a clear checklist — the criteria themselves. A reviewer answers: *"If this evidence is convincing, should this merge?"* They don't answer: *"Is this the best way to implement this? Did the developer forget a case?"* The first verifiers caught what they could; the reviewer focuses on the residue — intent fit and judgment calls the verifier can't make. ### Verification is systematic Human review is sampling. A reviewer reads the diff and uses judgment. They might catch issues, they might not. Different reviewers catch different things. Verification is systematic. Every criterion runs. Every invariant that matches the change runs. If an invariant says "no hardcoded credentials," it checks — not "probably checks," *checks*. Same code + same intent → same verdicts. This doesn't mean verification is perfect. If the agent missed a criterion in the intent, the verifier won't check it. But for what's submitted, verification is thorough — and the missed criterion is itself a reviewable surface: did the agent capture the right set of assertions? ### Audit trails come free Because intent is explicit and verification is recorded, you get audit trails automatically: * What was the submitted intent? * Which criteria did the agent generate? * Which verifier handled each, and what evidence did it produce? * Who reviewed it, and what did they approve, waive, or send back? Traditional code review produces a comment thread. Intent-driven verification produces a structured trail — the same record reviewers see is what auditors export. → [Audit trails and compliance](/verify/concepts/audit-trails-and-compliance) ### When verification doesn't make sense Verification adds overhead. For some changes, that overhead isn't worth it: * Typo fixes * Documentation updates * Trivial config changes Verification is enabled per repository, and draft PRs are left alone until they're marked ready for review. Not everything needs this level of rigor — and forcing it everywhere produces noise that erodes trust in the verdicts. ### The mental model Think of verification like a building inspection. A building isn't approved by an architect watching every nail. It's approved by an inspector evaluating the finished structure against the blueprints — with measurements, load tests, and photographic evidence to back the verdict. The architect signs off based on the inspector's report. Intent-driven verification follows the same shape: * **Blueprint:** the submitted intent and acceptance criteria. * **Construction:** the agent's implementation. * **Inspection:** the verifier pipeline, producing verdicts + evidence per criterion. * **Sign-off:** the reviewer approves based on intent fit and evidence quality. Reviewers don't watch every line of code get written. They evaluate what was built against what was agreed, with proof in the room. ### See also * [How verification works](/verify/concepts/how-verification-works) — the inspection pipeline * [How Verify works](/verify/how-it-works) — the end-to-end loop * [Audit trails and compliance](/verify/concepts/audit-trails-and-compliance) # Verification layers A verification run is built from two sources of acceptance criteria: criteria the **submitter** (you or your agent) supplied for *this* change, and criteria the **team** encoded as invariants that apply to *every* matching change. Both flow through the same pipeline. This page is about how the two layers interact at run time. For the deeper concept of invariants themselves, see [Invariants](/verify/concepts/invariants). For the pipeline, see [How verification works](/verify/concepts/how-verification-works). ### The two layers | Layer | Where it comes from | Scope | Materialized as | | ---------------------- | -------------------------------------------------------------------------- | ---------------------- | ---------------------------------------------------------------- | | **User criteria** | The MCP submission. Generated by the agent from the spec, or hand-written. | This change only. | Acceptance criteria, source: `generated`, `parsed`, or `manual`. | | **Invariant criteria** | The team's invariant catalog. Picked per-runbook by the selector. | Every matching change. | Acceptance criteria, source: `baseline_invariant`. | The selector is what makes invariants *apply*. When a runbook is created, the selector reads the intent, the user criteria, and the change set, and picks which active invariants to materialize as criteria on this runbook. See [Invariants — How invariants apply to a runbook](/verify/concepts/invariants#how-invariants-apply-to-a-runbook). The result: a single list of acceptance criteria with mixed sources. Verify doesn't distinguish at run time — every criterion goes through the same classifier and verifier path. ### How the layers combine When verification starts, Aviator assembles the criterion set: 1. **Every user criterion** runs. Each one is routed through the verifier pipeline. 2. **Every invariant-materialized criterion** runs. Same pipeline, same verdict shape. The two sets are concatenated. A typical run has 3–7 user criteria and a handful of invariant criteria. Both produce verdicts with evidence in the same review document. A run passes only when every criterion passes (or is waived). Failures don't collapse — three failing criteria produce three verdicts, not one. ### Why two layers instead of one You could push everything into user criteria: have the agent emit "requires authentication" on every submission. It would technically work, but it's the wrong factoring: * The agent would have to remember the team's full ruleset on every submission. * A bad submission could silently omit a rule. * Changes to the ruleset would require re-instructing the agent. Invariants live outside the per-submission loop. The team encodes them once. They apply automatically. The submitter doesn't need to know — the selector loads the matching set itself. The mirror argument applies to user criteria: not everything can be an invariant. "This endpoint returns these fields" is specific to a change. Forcing it into an invariant would either bloat the catalog or push too much per-change content into team policy. The split is: invariants for what's true across changes, user criteria for what's specific to this one. ### What the reviewer sees In the review document, every criterion shows the same things: text, verdict, verifier, evidence. The only visible difference is a small tag indicating whether the criterion came from user input or from an invariant. Reviewers can: * **Approve** when every criterion passes. * **Waive an invariant verdict** with a reason and a category (see [Invariants — Waivers](/verify/concepts/invariants#waivers)). * **Edit user criteria** through the agent (via `editRunbook` on the MCP) when the wrong criteria were generated. Invariant criteria can't be edited per-runbook. They're auto-managed by the catalog; if a rule is wrong, fix the invariant itself. The same verdicts and the same waive action are available on the pull request itself through the [Verify tab](/verify/how-to-guides/verify-on-github). ### When to put a check in each layer | Use… | When the check is… | | ------------- | -------------------------------------------------------------------------- | | User criteria | Specific to this change. Won't apply elsewhere. | | Invariant | Recurring. You'd ask the agent to assert it on multiple unrelated changes. | The test for promoting a check to an invariant is recurrence: if you'd write the same criterion text on the next ten submissions, it should be an invariant. ### See also * [Invariants](/verify/concepts/invariants) — deep concept page * [How verification works](/verify/concepts/how-verification-works) — the per-check pipeline * [Setting up org invariants](/verify/setting-up-org-invariants) # How verification works This document goes deeper than the [How it works](/verify/how-it-works) narrative. It covers what actually happens inside a verification run — how criteria are classified, how each verifier produces evidence, and why the same code always yields the same verdict. ### A verification run, end to end Once a verification auto-run trigger fires (or you start a run by hand), Aviator does three things in order: 1. **Constructs the run.** Pulls the change set (the diff against the target branch), the acceptance criteria, and the invariants whose conditions match the change. Allocates a preview if any criterion will need one. 2. **Routes each check through the pipeline.** Every criterion runs through one of two verifier paths. Verdicts are produced in parallel where possible. 3. **Compiles the results.** Verdicts, evidence, and links to the preview are assembled into the review document. The run is observable in real time — the review document streams updates as verdicts land. ### When a run is triggered Verify runs against an open PR. Aside from manual runs you start yourself, it runs on its own at a few specific moments in the PR's life rather than on every push: | Moment | Runs when | Run label | | ------------------- | ---------------------------------------------------------------------------- | -------------------------- | | **PR ready** | A PR with acceptance criteria first becomes ready for review (leaves draft). | `on PR ready` | | **Approval** | The PR receives code-owner approval. | `on approval` | | **Queued** | The PR is added to the merge queue. | `on queue` | | **PR linked** | An externally-opened PR is linked to a runbook. | `on PR link` | | **Criteria edited** | Acceptance criteria are edited through the `editRunbook` MCP tool. | replaces the in-flight run | A plain push to the branch does **not** start a run. Pushing keeps the change set fresh and pre-computes baselines so the next triggered run is fast, but the verdict you already have stands until one of the moments above fires. You can also start a run by hand from the runbook UI — manual runs bypass every rule below and always execute. Verify de-duplicates so the same change isn't verified twice: * **PR ready** fires once per PR. Flipping it back to draft and ready again, or amending the commit, won't re-run it. * **Approval**, **queued**, and **PR linked** each run once per commit + criteria set. Duplicate webhooks, repeated approvals, or re-queues on the same commit are no-ops. * **Criteria edited** supersedes an in-flight run that was using the old criteria — the stale run is cancelled and a fresh one starts on the new criteria. A failed run doesn't count against de-duplication, so re-triggering after a pipeline issue works. And if baseline invariants for a commit haven't been selected yet, the run is **deferred** and starts automatically once that selection finishes — see [Understanding verification results](/verify/reference/understanding-verification-results) for the `deferred` status. ### The criterion pipeline
Each criterion is classified and routed to a verifier path

One criterion → one verifier path → one verdict with evidence

Each criterion goes through exactly one verifier. A classifier picks the path based on the criterion text and the files the change touched. | Verifier | Picked when… | Produces | | ------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | | **Code-scan** | The criterion is a structural assertion — file scope, dependency surface, function signature, type. | The diff or AST snippets that demonstrate the claim. | | **Runtime** | The criterion is behavioral — endpoint contract, error shape, side effect, UI behavior. | Screenshots, console logs, DOM snapshots, API responses, full trace. | If you read a verdict and disagree with the path the classifier picked, that's a signal — usually the criterion was written ambiguously enough to land in the wrong path. Tightening the criterion text usually moves it back. ### How runtime verification actually runs Runtime verdicts are the most expensive. They need: 1. A **preview** to run against (see [Concepts: Previews](/verify/concepts/previews)). 2. A **skill set** that tells the agent how to operate the preview — base URL, test users, fixtures (see [Writing a SKILL.md](/verify/how-to-guides/writing-a-skill-md)). 3. The **criterion text** — the specific claim being verified. The runtime runner is an agent that drives the preview through a small program tailored to the criterion: set up preconditions, exercise the endpoint or click through the flow, capture evidence, compare to the expected shape. It records what it did and what came back. Evidence shapes for runtime verdicts: | Evidence type | When | | ------------- | -------------------------------------------------------------- | | Screenshot | UI scenarios, visual assertions. | | DOM snapshot | UI state assertions, "the modal closed after X." | | Console log | Client-side error/behavior assertions. | | API response | Backend assertions — status, headers, body shape, timing. | | Trace | Full agent transcript of the run, captured for every scenario. | This is the part most teams underinvest in early — a thin SKILL.md and stale seed data produce runtime verdicts you can't trust. The pipeline runs fast, but the verdict is only as good as the preview it ran against. ### How invariants compose with criteria Invariants are your team's standing rules. When the runbook is created, the invariant selector picks which active invariants apply to this change based on the intent, the criteria, and the change set — and materializes each pick as an acceptance criterion (tagged with `source: baseline_invariant`). The composition rule: a run passes only if every criterion — user-authored or invariant-materialized — passes. Failures stack: one failed user criterion plus two failed invariants produces three verdicts on the same review document, not one merged failure. See [Verification layers](/verify/concepts/verification-layers) for how invariants and user criteria interact, and [Invariants](/verify/concepts/invariants) for where invariants come from and how the selector works. ### Determinism Code-scan verdicts are deterministic. Run the same change against the same criterion twice — same verdict, same evidence. Runtime verdicts depend on an agent driving the preview, which uses an LLM. They're stable for the same input but not strictly reproducible across re-runs. The system mitigates this by caching evidence per criterion + change set so re-runs of the same content don't re-execute. Why determinism matters where you can get it: * You can reproduce failures. * Re-runs after no code changes don't flip results. * Audit records hold up across time. ### Performance A typical run completes in 30–120 seconds: | Factor | Effect | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------ | | Diff size | Larger diffs → more code-scan + invariant matches. Sub-linear. | | Number of criteria | One verifier per criterion. Linear; parallelized where possible. | | Runtime criteria | Each runtime criterion drives the preview. Dominates total time once present. | | Preview cold-start | Most of the variance. Bake heavy work into the image — see [Managing previews](/verify/how-to-guides/managing-previews). | | Matching invariant count | Linear; usually cheap. Most invariants resolve from the diff. | If a run is slow, look at the runtime-criterion count and the preview boot time. Those two dominate. ### Comparison with other approaches **vs. static analysis.** Static analysis tools (linters, type checkers) verify syntax and patterns. Verify checks declared intent against the running behavior. Static analysis says "this variable is unused." Verify says "this endpoint doesn't return what the intent said it should." Use both — they catch different things. **vs. tests.** Tests assert that specific inputs produce specific outputs, written by the developer. Verify runs runtime checks driven by the submitted intent against an ephemeral preview. Tests check what you remembered to write a test for. Verify checks what you said the change was supposed to do. Tests catch regressions; verification catches intent drift. **vs. AI code review.** AI code review tools leave comments on diffs. They're faster than human review but inherit the same limitations: no record of intent, no systematic coverage, sampling-based, no audit trail. Verify starts from declared intent and produces structured verdicts with evidence. ### See also * [How Verify works](/verify/how-it-works) — the end-to-end narrative * [Invariants](/verify/concepts/invariants) * [Previews](/verify/concepts/previews) * [Verification layers](/verify/concepts/verification-layers) # Invariants An **invariant** is a team-defined rule that Verify applies to every matching change. Where user-supplied acceptance criteria describe what *this* change should do, invariants describe what *every* change should respect. They live in your Aviator account and update once for everyone. A good invariant captures something your team learned the hard way — usually as a recurring review comment — so reviewers don't have to flag it again. ### Where invariants come from Invariants live in a per-account catalog. Each invariant has a source: | Source | What it is | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | **Manual** | Authored by an admin in **Settings → Invariants**. | | **Template** | Instantiated from Aviator's starter library — common patterns you can adopt as-is or tweak. | | **AI-generated** | AI-drafted from signals about the repo, awaiting admin approval. | | **AI from docs** | Extracted from your repo's `CONTRIBUTING.md`, `LLM.md`, or similar files by the docs pipeline. | | **AI from PR comments** | Mined from your team's recurring PR-review comments. Often the highest-yield source — it captures real-world feedback your team has already given. | | **Slack** | Drafted on request from a `@Aviator invariant` command in Slack. | | **GitHub comment** | Drafted on request from a `@aviator invariant` comment on a pull request. | Sources that aren't `manual` produce drafts. Admins promote drafts to active in the UI; that's when they start producing verdicts. ### Creating invariants from Slack Mention the Aviator Slack app in a channel, or DM it, with `@Aviator invariant [repo-name] `; when used inside a thread, the thread's messages become context for the draft, and naming a repository scopes the invariant to it (otherwise it applies account-wide). Aviator replies in the thread with the draft titles and a link to review and approve them.
A Slack thread where a user reports a UI inconsistency, a teammate replies with an @Aviator invariant command, and Aviator responds with a draft invariant and a review link

Turning a Slack thread into a draft invariant

### Creating invariants from a GitHub comment Include `@aviator invariant` in any pull request comment (a regular comment, an inline review comment, or a review body) and Aviator acknowledges with a 👍 reaction, then drafts invariants from the comment and its surrounding discussion. This trigger is limited to repository maintainers; Aviator replies on the pull request with the draft titles and a link to review and approve them.
A PR comment reading @aviator invariant the error message should not be in first person, with Aviator replying with a draft invariant and a review link

Turning a PR comment into a draft invariant

### Conditions Each invariant has zero or more **conditions** that gate when it's eligible to apply: | Condition type | What it matches | | ---------------- | ------------------------------------------------------------- | | `file_path_glob` | Files changed in the runbook match a glob like `src/**/*.py`. | | `language` | The detected language of changed files matches. | An invariant with no conditions is eligible for every runbook. With conditions, it's only eligible when at least one matches the change. Eligibility doesn't mean the invariant *applies* — it just means the next step (selection) will consider it. ### Categories Every invariant belongs to a category. Default categories: | Category | What it covers | | ------------------------- | ----------------------------------------------------------------------------- | | `functional_correctness` | The change behaves correctly for expected and edge-case inputs. | | `test_coverage` | New and changed code is covered by meaningful tests. | | `security` | The change avoids common vulnerabilities and enforces required controls. | | `performance` | The change doesn't introduce avoidable latency, memory, or query regressions. | | `accessibility` | UI changes meet accessibility standards. | | `observability` | Errors emit metrics; logging uses structured fields. | | `backwards_compatibility` | Public surfaces don't break existing consumers. | | `documentation` | Public APIs and behavior changes are documented. | | `code_style` | Code follows team conventions (type usage, naming, layering). | Categories drive grouping in the UI and reporting in the audit trail. ### How invariants apply to a runbook When a runbook is created, a **selector** picks which eligible invariants actually apply. The selector reads the runbook's intent, the user-supplied acceptance criteria, and the change set, and uses an LLM to pick the catalog entries that defensibly fit the change. Selected invariants are materialized as acceptance criteria on the runbook, tagged with `source: baseline_invariant`. From that point on, they flow through the [verification pipeline](/verify/concepts/how-verification-works) like any other criterion — same verdict shape, same evidence, same review-document treatment. Two consequences: * **You don't need to think about invariants when writing the intent.** The selector handles eligibility. Your acceptance criteria stay focused on what's specific to this change. * **Invariant verdicts and user-criterion verdicts look identical in the review document.** The only visible difference is the source tag, and the fact that invariant criteria can't be edited per-runbook (they can be waived). ### Writing a good invariant Three rules of thumb: **Be specific about the assertion, vague about the implementation.** * ✓ "All HTTP handlers must call an authentication middleware before any business logic." * ✗ "Use `AuthMiddleware` from `src/auth/middleware.go`." (Brittle — the check should survive renames and module moves.) **Make the rule verifiable in isolation.** A rule that requires running the whole system is hard to verify. Prefer rules that can be checked from the diff or from a single runtime probe. * ✓ "All migrations must declare a `down` block." * ✗ "All migrations must be reversible." (Can't be checked without running them backwards.) **Use conditions sparingly.** The selector reads the runbook context and picks what fits. Don't over-restrict with conditions — let the selector pass on inapplicable cases. Conditions are useful for hard exclusions (e.g. language-specific rules), not for fine-grained scoping. ### Turning a review comment into an invariant Most invariants worth writing start as a review comment that's been left more than twice. The mining pipeline — `ai_generated_pr_comments` — does this automatically by reading your PR history, but you can also do it by hand: 1. **Find the recurring comment.** Scan PRs over the last quarter. 2. **Write the assertion.** State the rule in one sentence. Don't write the *fix* — the verifier will explain what's wrong. 3. **Pick a category.** Helps with grouping and reporting. 4. **Add conditions if needed.** Most invariants don't need them. 5. **Save as draft, watch a week of verifications.** Draft invariants don't get materialized into runbooks. Use that to confirm the rule reads cleanly before promoting to active. Example, turning a real review comment into an invariant: > Comment on PR #4173: "Please don't write to `users` directly — go through `UserRepository.UpdateProfile`. We had a partial-write bug last quarter from a similar pattern." Invariant body: ``` Writes to the users table must go through UserRepository. Direct INSERT, UPDATE, or DELETE statements against the users table are not allowed outside the repository package. Schema migrations under src/db/migrations are exempt. ``` Conditions: `file_path_glob: src/**/*.go` (skip non-Go files). Category: `functional_correctness`. ### Waivers Invariant verdicts can be waived with a categorized reason, either from the review document or from the [Verify tab on the pull request](/verify/how-to-guides/verify-on-github): | Waiver category | When to use it | | ----------------- | --------------------------------------------------------------------- | | `false_positive` | The invariant fired but the rule misjudged this case. | | `doesnt_apply` | The rule is valid in general but isn't relevant to this PR. | | `accepted_risk` | The failure is real but the PR author accepts the trade-off. | | `fix_in_followup` | The failure is real and will be addressed in a separate follow-up PR. | Every waiver is recorded in the audit trail with the reviewer, the category, and the free-text reason. If you find yourself waiving the same invariant repeatedly, the rule is wrong — tighten its conditions, rephrase the body, or rebuild the rule from real review comments. ### See also * [Setting up org invariants](/verify/setting-up-org-invariants) — step-by-step setup * [Verification layers](/verify/concepts/verification-layers) — how invariants compose with criteria in a run * [How verification works](/verify/concepts/how-verification-works) — the verifier pipeline * [How to: Writing a SKILL.md](/verify/how-to-guides/writing-a-skill-md) — for runtime context, not for rules # Previews A **preview** is an ephemeral environment Verify builds on demand and runs scenarios against. It's how Verify makes behavioral claims real — every "the endpoint returns X" criterion needs the code to actually run somewhere, and the preview is that somewhere. A preview is short-lived: built for the branch's current commit, used by the scenario runner (and optionally a human reviewer), and torn down when the branch moves to a new commit. A re-run on the same commit reuses the running preview — including its state. See [Lifecycle](#lifecycle) for the exact contract. ### Optional, not required Previews are optional. Verify works on day one with code-scan alone — without a preview, every criterion is routed to code-scan (static analysis of the diff) and you get verdicts on structural criteria from the first PR. A preview unlocks **runtime verification**: behavioral criteria like "the endpoint returns 429" or "the modal closes on submit" need the code to actually run, and that requires a preview. Most teams get going with code-scan and add a preview when their criterion list starts asking about behavior the diff alone can't answer. ### Lifecycle
Preview lifecycle: define, build, boot, use, teardown

A preview moves through five phases

| Phase | What happens | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | **Define** | The preview is configured in the Verify settings in the Aviator dashboard (**Verify → Settings → Verify**, with the repo selected). | | **Build** | Aviator boots a container from the cached image and prepares the environment. | | **Boot** | The setup script runs. The declared port becomes reachable. | | **Use** | Scenarios execute against the preview. Reviewers can also open it from the UI. | | **Teardown** | When the branch moves to a new commit (or the preview is stopped), the optional teardown script runs and the container is destroyed. | Whether a run starts fresh depends on whether the branch has moved. When a verification run starts, Aviator compares the branch's current head SHA to the SHA the existing preview was built from: * **New commit since the last run** — the old container is torn down and a fresh one is built and booted. The setup script runs. The run starts from a clean slate. * **Same commit (a re-run)** — Aviator reconnects to the already-running container and skips the setup script entirely. Everything from the previous run persists: database rows, files, logged-in sessions. The practical consequence: if your scenarios mutate data, a re-run on the same commit sees the previous run's leftovers. Push a new commit (or stop the preview) to force a clean boot. ### Composition A preview is composed of inputs from three places: a preview image, your secret store, and your repo.
Preview anatomy: inputs, the preview container, and what consumes it

Where each piece of a preview comes from, and what uses it

* **Image** — a preview image registered with Aviator for your account. Aviator caches the image locally and boots containers from it. Register images through **Settings → Sandbox** in the Aviator UI. * **Secrets** — runtime secrets (DB passwords, API keys) referenced by name from the account secret store and injected as environment variables when the container boots. * **Setup script** — optional. Runs after the container starts and before the port is marked ready. Used for migrations, seeding, warm-up. * **Teardown script** — optional. Runs before destruction to release external resources. * **Port** — the port the runner connects to. The container is considered ready when this port accepts connections. Aviator stitches these together into a single ephemeral container. Your Verify settings are the contract — see [Preview YAML reference](/verify/reference/preview-yaml) for every field. ### Multiple previews per repo A repo can have more than one preview configured. Common patterns: | Pattern | Why | | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **`default`** only | Single API or service. One image, one set of secrets. | | **`default` + `mirror`** | The mirror preview points at a more production-like configuration — bigger seed data, real third-party sandboxes. Used by scenarios that need higher fidelity. | | **`api` + `worker` + `db`** | Multi-service apps. Each scenario picks the preview that matches the code it touches. | | **`light` + `heavy`** | Light boots fast and is used by most scenarios. Heavy boots slow but covers integration tests that need full setup. | Scenarios target a preview by name. If no name is set, scenarios run against `default`. ### When previews are used Previews only spin up when the verification run has at least one **runtime** criterion — a criterion the classifier decides needs to be checked against the running code (see [How verification works](/verify/concepts/how-verification-works)). If every criterion can be verified by code-scan alone, no preview is built and the run finishes faster. This matters for cost: previews are the expensive part of verification. The classifier minimizes their use by routing structural assertions away from the runtime path when possible. ### Reviewer access The review document exposes a per-run "Open preview" link. Clicking it gives the reviewer access to the same ephemeral container the scenarios ran against — same data, same configuration, same code under test. The link expires when the sandbox times out and the preview is torn down. The window is controlled by **Sandbox Timeout (minutes)** under **Verify → Settings → Sandbox** (default 60, up to 240). If reviewers routinely need longer sessions, raise that setting; a preview that has already been torn down can be launched again from the review document. ### Previews vs. CI environments Previews look like CI environments but they're not the same thing: | | Preview | CI environment | | ------------- | ---------------------------------------- | ------------------------------------- | | Triggered by | Verify run | Push, PR open, schedule | | Lifespan | Until the branch moves to a new commit | Per job (minutes to hours) | | Configured in | Verify settings in the Aviator dashboard | CI provider config (GH Actions, etc.) | | Used by | Scenario runner + reviewer | Test runners, build pipeline | | Purpose | Make behavioral verification real | Run the test suite, ship artifacts | ### See also * [Preview YAML reference](/verify/reference/preview-yaml) — the schema * [Creating a preview](/verify/how-to-guides/creating-a-preview) — walkthrough * [Managing previews](/verify/how-to-guides/managing-previews) — bake vs. setup, refresh, cleanup * [Seed data for previews](/verify/how-to-guides/seed-data-for-previews) — fixtures and deterministic state # Audit trails and compliance Every Verify run produces an immutable record of what was submitted, what ran, what was found, and what was decided. This page covers what's recorded, how it composes into a chain you can hand an auditor, and how Verify maps to common compliance frameworks. ### What gets recorded For every runbook: | Record | What it contains | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | Runbook submission | Submitter, timestamp, intent, acceptance criteria, target branch, working branch, repo + commit | | Runbook version | Each iteration of the runbook (steps + acceptance criteria), with version number | | Verification run | Trigger source (manual / ready / approval / queued / linked / criteria-edit), commit SHA, run status, counts (passed / failed / skipped / waived) | | Verification result | One per criterion: verifier path, verdict, evidence reference, reason, location | | Reviewer waiver | Reviewer, timestamp, criterion, category (false-positive / doesn't-apply / accepted-risk / fix-in-followup), free-text reason | Records are immutable. Once written, they can't be modified or deleted by users. ### The audit chain Every merged change has a complete chain: ``` Implementation → Runbook submission → Verification run(s) → Reviewer decisions → Merged ↓ ↓ ↓ ↓ ↓ Commit SHA Submission One result per Waivers + Merge record criterion + evidence approval reference ``` This chain answers the questions an auditor asks: * **What was the change supposed to do?** → The runbook's intent + acceptance criteria. * **Did the running code actually do it?** → Each verification result, with evidence per criterion. * **Who decided this could merge?** → The reviewer's recorded actions on the runbook. * **What was overridden, and why?** → Waivers, each with a category and reason. ### Segregation of duties Compliance frameworks often require segregation — the person making a change shouldn't be the only one who signs off on it. Verify supports this naturally: | Role | Actor | Tracked separately | | --------- | ------------------------------------------------------------------------------------------------ | ------------------ | | Submitter | The user whose MCP token created the runbook | yes | | Reviewer | The person approving (or waiving verdicts), from the review document or the Verify tab on the PR | yes | | Verifier | Automated — the Verify pipeline itself | yes | Account policy can require the reviewer differ from the submitter. The audit chain makes the separation explicit: a single change can't merge without two distinct actors appearing on it. This is stronger than diff review, where the same person can leave a comment and merge their own PR. ### Traceability Every production change links back to: * The runbook submission (the *what was supposed to happen*). * The verification results (the *what actually happened*). * The reviewer's recorded decisions (the *who said yes*, including waivers). * The commit SHA (the *what shipped*). If an auditor asks "why was this change made and how do you know it was safe," you can show all four in one query. ### Compliance framework mapping #### SOC 2 | Trust Service Criteria | How Verify helps | | ------------------------------- | ------------------------------------------------------------------------ | | CC6.1 — Logical access controls | Reviewer decisions track who authorized each change. | | CC6.6 — Authorized changes | Every merged change has a reviewer record. | | CC6.7 — Changes are tested | Verification produces a result for every criterion (user and invariant). | | CC8.1 — Change management | Complete chain: submission → verification → review → merge, immutable. | #### ISO 27001 | Control | How Verify helps | | -------------------------------- | ------------------------------------------------------------- | | A.12.1.2 — Change management | Structured submission + review process for every code change. | | A.12.1.4 — Segregation of duties | Submitter ≠ reviewer, enforceable via account policy. | | A.14.2.2 — System change control | Audit chain links submission to implementation to verdict. | #### HIPAA | Requirement | How Verify helps | | ------------------ | ---------------------------------------------------------------------- | | Access controls | Only authorized reviewers can approve; access is logged. | | Audit controls | Immutable records of every submission, verdict, and reviewer decision. | | Integrity controls | Verification gives evidence that the code matches the declared intent. | ### Exporting reports The audit data is queryable and exportable from the Aviator UI. See [How to export audit logs](/verify/how-to-guides/export-audit-logs) for the current export surface. ### Retention Audit records are retained indefinitely by default. Contact support if you have a regulatory requirement for a different retention or archival policy. ### Immutability Audit records cannot be: * Modified after creation. * Deleted by users. * Backdated. The trail is what happened, not a summary someone wrote afterwards. ### What Verify doesn't do Verify provides an audit trail for code changes. It doesn't: * Replace your ticketing or change-management system. * Provide runtime audit logs (request-level access, API call audits). * Track infrastructure or configuration changes outside source control. * Monitor production access or data handling. Integrate Verify alongside your other compliance tools for full coverage. ### See also * [How to export audit logs](/verify/how-to-guides/export-audit-logs) * [How verification works](/verify/concepts/how-verification-works) * [Why intent-driven verification](/verify/concepts/why-intent-driven-verification) # Reference # Aviator CLI The `aviator` CLI submits intent and acceptance criteria to Verify from your terminal or from inside a coding agent. It is the preferred way to talk to Verify. This is a different tool from `av`, the [Stacked PRs CLI](/aviator-cli). The two are installed separately and don't depend on each other. ### Install ```bash brew install aviator-co/tap/aviator ``` Linux users can install the `.deb` or `.rpm` package from the [releases page](https://github.com/aviator-co/aviator-cli/releases), or download the archive for their platform. Confirm the install: ```bash aviator version ``` ### Authentication The CLI reads an API token from the `AVIATOR_API_TOKEN` environment variable: ```bash export AVIATOR_API_TOKEN= ``` Create a User Access Token at [app.aviator.co/settings/personal/api\_token](https://app.aviator.co/settings/personal/api_token). Submissions are attributed to the user the token belongs to, so each person needs their own — a shared token collapses the audit trail. You can also put the token in a config file. The CLI reads a `config.yaml` (`.json` and `.toml` also work) from the first of these that exists: 1. `$XDG_CONFIG_HOME/aviator/` 2. `~/.config/aviator/` 3. `~/.aviator/` 4. `$AVIATOR_HOME/`, if that variable is set ```yaml aviator: apiToken: ``` A repo-local `.git/aviator/config.yaml` is merged on top of the global one, and environment variables override both. For on-premise installations, point the CLI at your instance with `AVIATOR_API_HOST` (or `aviator.apiHost` in the config file). It defaults to `https://api.aviator.co`. ### Commands | Command | What it does | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `aviator verify` | Submit intent and acceptance criteria for a change you're writing yourself. | | `aviator runbook` | Create a runbook and have Aviator's agent implement the change. | | `aviator show` | Show a runbook or Verify session, e.g. `aviator show r/123`. | | `aviator results` | Show the latest verification results for a session. | | `aviator edit` | Replace the acceptance criteria on an existing session. Takes `--expected-version` to guard against stale edits — read the current version with `aviator show`. | | `aviator init` | Set up your coding agents to capture intent before a PR. See [Set up agent hooks](/verify/how-to-guides/set-up-agent-hooks). | | `aviator hooks` | Manage the hooks `init` installed — `aviator hooks uninstall` removes them. | | `aviator version` | Print the CLI version. | ### `aviator verify` Creates a Verify session seeded with your acceptance criteria. The implementation stays with you — Aviator verifies the PR opened from the working branch against the criteria. ```bash aviator verify \ --repo myorg/myrepo \ --intent "Add rate limiting to the public API so one client can't exhaust capacity" \ --working-branch add-rate-limiting \ --criteria-file criteria.txt ``` | Flag | Required | Description | | ------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--repo` | yes | GitHub repo as `owner/repo`. | | `--intent` | yes | Short, plain-language description of what the change is for. | | `--criteria` | one of | A single acceptance criterion. Repeatable. | | `--criteria-file` | one of | Path to a file with one criterion per line. Preferred for more than two or three criteria — it avoids shell-quoting problems. Mutually exclusive with `--criteria`. | | `--working-branch` | no | The branch the work lives on, so a PR opened from it is verified against these criteria. | | `--target-branch` | no | Base branch to verify against. Defaults to the repo default. | | `--spec` | no | Path to a spec file carrying the key decisions and architecture. | | `--author-email` | no | Attribute the submission to a different user. | The command prints the session URL and the number of criteria it recorded. The first verification run happens when the PR is marked ready for review. To change criteria on a session that already exists, use `aviator edit` — re-running `aviator verify` creates a new session. ### `aviator runbook` Creates a runbook from an intent and hands the implementation to Aviator's agent. Acceptance criteria are optional here. ```bash aviator runbook \ --repo myorg/myrepo \ --intent "Migrate the reporting jobs off the deprecated scheduler" ``` `--repo` and `--intent` are required. `--title`, `--target-branch`, `--spec`, `--criteria`/`--criteria-file`, and `--author-email` are optional, and `--oneshot` (on by default) controls one-shot mode. ### Using the CLI from a coding agent You rarely type `aviator verify` by hand. The `/verify-submit` skill, from the [Aviator agent plugins](https://github.com/aviator-co/agent-plugins), reads the change, drafts the intent and acceptance criteria with you, and calls the CLI for you. Run `aviator init` once per repo to have your agent remind you before a PR is opened. See [Set up agent hooks](/verify/how-to-guides/set-up-agent-hooks). ### See also * [Set up agent hooks](/verify/how-to-guides/set-up-agent-hooks) — the pre-PR reminder * [Your first verification](/verify/your-first-spec) — hands-on tutorial * [Writing effective acceptance criteria](/verify/how-to-guides/writing-effective-acceptance-criteria) * [MCP tools](/verify/reference/mcp-tools) — the legacy submission path # MCP tools {% hint style="info" %} The MCP server is the legacy submission path. The [Aviator CLI](/verify/reference/cli) is the preferred way to submit to Verify — it's what `/verify-submit` uses, and it's where new capability lands. The MCP tools below still work, and existing installs keep functioning. {% endhint %} The Aviator MCP server exposes three tools for creating, reading, and editing the runbook that drives a verification run. This page documents the tool surface. For the conceptual flow, see [How Verify works](/verify/how-it-works). ### Installing the MCP The install snippet — including a scoped token — is available in the Aviator UI under **Settings → Integrations → MCP**. Configure your agent to load the Aviator MCP server using the snippet for your client. Restart the agent after configuring. The Aviator tools should appear in the agent's available-tools list. ### Tools | Tool | Use it for | | ------------- | ----------------------------------------------------------------------- | | `specSubmit` | Submit a new runbook capturing the intent and acceptance criteria. | | `getRunbook` | Read the current state of a runbook — plan, criteria, latest verdict. | | `editRunbook` | Replace acceptance criteria on an existing runbook in a versioned edit. | ### `specSubmit` Submits a runbook to Aviator. It captures two things: the **intent** for the change (what it's for, in plain language), and the **acceptance criteria** the change must satisfy. **When to call it:** when the user explicitly asks to submit to Aviator. Never call it proactively. **Parameters:** | Parameter | Required | Description | | ---------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `repo_name` | yes | GitHub repo name. Accepts `owner/repo` or just `repo` if unambiguous within the account. | | `message` | yes | The intent — a short, plain-language description of what this change is for and why. | | `working_branch` | no | An existing branch where the work already lives. Setting this lets the runbook read what's already pushed and auto-connect a future PR. Omit to have the runbook author the work from scratch. | | `target_branch` | no | The branch the work targets. Omit for the repo default (trunk); pass the parent branch when this work stacks on another in-flight branch. | **Returns:** ```json { "runbook_number": "218", "url": "https://app.aviator.co/r/218", "message": "Runbook creation started. Generation may take a few minutes." } ``` Open the URL to watch the runbook generate, see acceptance criteria, and read verdicts as verification runs. ### `getRunbook` Reads the current state of a runbook. Use it before editing — every edit takes an `expected_version` you read from here. **Parameters:** | Parameter | Required | Description | | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `url` | yes | The runbook URL. Accepts `https://app.aviator.co/r/{number}`, `runbooks/{id}`, or plain `r/{number}`. | | `fields` | no | List of fields to return. Defaults to all. Selectable: `steps_markdown`, `runbook_state`, `acceptance_criteria`. `runbook_version` is always returned. | **Returns (per field):** | Field | Shape | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `runbook_version` | int — always present. Pass back as `expected_version` on edits. | | `steps_markdown` | The runbook plan as markdown. | | `runbook_state` | `target_branch`, `working_branch`, per-step status. | | `acceptance_criteria` | List of `{ordinal, raw_text, source}`. Baseline-invariant criteria are excluded — they're auto-managed. | | `latest_verification` | Returned alongside `acceptance_criteria`. Null if no runs. Includes status, criteria counts, and a list of non-passing results (each tagged with `is_invariant` and `is_waived`). | ### `editRunbook` Edits a runbook artifact. Today only `acceptance_criteria` is editable. Baseline invariants are auto-managed and never included in the payload. **Parameters:** | Parameter | Required | Description | | ------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `runbook_url` | yes | The runbook URL (same formats as `getRunbook`). | | `expected_version` | yes | The `runbook_version` you read with `getRunbook`. If the runbook has changed since, the call fails with a stale-version error — re-read and retry. | | `payload` | yes | JSON object keyed by artifact name. Each value is the COMPLETE new state of that artifact. For `acceptance_criteria`: a JSON array of strings (the new criterion texts in order). One call expresses add/update/remove/reorder in a single atomic edit. | **Example payload:** ```json { "acceptance_criteria": [ "First criterion text", "Second criterion text", "Third criterion text" ] } ``` **Returns:** ```json { "results": {"acceptance_criteria": {...}}, "new_version": 5, "message": "Runbook updated to version 5." } ``` ### Authorization Each MCP token is scoped to the user who generated it: * Submissions are attributed to that user in the audit trail. * Token permissions are bounded by the user's account role — a viewer-only user can't create runbooks for a repo they can't write to. * Tokens can be rotated from the Aviator UI; rotation invalidates the previous token immediately. If a teammate's submission needs to be attributed to them, they need their own install — sharing tokens defeats the audit trail. ### Versioning The MCP tool surface follows backwards-compatibility rules: * Field additions are backwards-compatible. New optional parameters may appear without notice. * Field removals and required-field changes are versioned. The install snippet pins to a major version; breaking changes ship as a new major. * Token format is stable across versions. ### See also * [Aviator CLI](/verify/reference/cli) — the preferred submission path * [How Verify works](/verify/how-it-works) — where submission fits in the loop * [Your first verification](/verify/your-first-spec) — hands-on tutorial * [Running with remote agents](/verify/how-to-guides/running-with-remote-agents) — alternative flow that uses Runbooks directly # Spec format This page documents the shape of what `specSubmit` captures: a **title**, the **intent**, and the **acceptance criteria**. The agent typically generates this from your conversation; documented here so you can read submissions, hand-edit when needed, or build tools against the structure. For the conceptual flow, see [How Verify works](/verify/how-it-works). For the MCP tool itself, see [MCP tools](/verify/reference/mcp-tools). ### Structure A spec is markdown. Two sections are parsed: ```markdown # Title ## Intent [plain-language description] ## Acceptance Criteria [bullet list of verifiable assertions] ``` Section headings must be level-2 (`##`). The parser is case-insensitive on the heading text. ### Title A short description of the change. Appears in the dashboard, runbook UI, and audit trail. ```markdown # Add per-user rate limiting to public API ``` Keep it under \~80 characters. The title is what reviewers see first in the inbox. ### Intent Plain-language description of *what* this change is for and *why*. Captures the constraints that aren't visible from the diff alone. ```markdown ## Intent Cap per-user request rate on /api/v1/public/* endpoints. On overflow, return 429 with a Retry-After header so well-behaved clients can back off. Existing internal endpoints are unaffected. ``` The intent is the contract reviewers approve against. It doesn't describe implementation choices — those live in the code itself. ### Acceptance Criteria A bullet list of verifiable assertions. Each criterion is checked independently during verification. ```markdown ## Acceptance Criteria - /api/v1/public/* endpoints enforce per-user rate limit - Returns 429 when the per-user limit is exceeded - Response includes Retry-After header on 429 - Rate-limit events log through the structured logger - Internal endpoints (/api/v1/internal/*) are unaffected ``` Bullet markers are `-` or `*`. Checkbox syntax (`- [ ]`) is accepted but not required — the checkbox is dropped during parsing. Each criterion should: * **State one claim.** "Returns 429 when exceeded" — not "Returns 429 when exceeded and logs the event." Split compound claims. * **Be verifiable on its own.** A criterion that requires reading the rest of the codebase to evaluate will route to a less-deterministic verifier path. * **Avoid implementation language.** "Uses the new `RateLimiter` struct" is brittle. "Per-user limit is enforced before business logic runs" is durable. See [Writing effective acceptance criteria](/verify/how-to-guides/writing-effective-acceptance-criteria) for more guidance. ### Complete example ```markdown # Add per-user rate limiting to public API ## Intent Cap per-user request rate on /api/v1/public/* endpoints. On overflow, return 429 with a Retry-After header so well-behaved clients can back off. Existing internal endpoints are unaffected. ## Acceptance Criteria - /api/v1/public/* endpoints enforce per-user rate limit - Returns 429 when the per-user limit is exceeded - Response includes Retry-After header on 429 - Rate-limit events log through the structured logger - Internal endpoints (/api/v1/internal/*) are unaffected ``` ### See also * [MCP tools](/verify/reference/mcp-tools) — the `specSubmit` tool that submits this * [Writing effective acceptance criteria](/verify/how-to-guides/writing-effective-acceptance-criteria) * [Concepts: Invariants](/verify/concepts/invariants) — for assertions that should hold across changes # Preview YAML This page documents the `preview` block of your Verify configuration, edited in the Aviator dashboard under **Verify → Settings → Verify** (with the repo selected). For the concept of what a preview is and how it fits in, see [Concepts: Previews](/verify/concepts/previews). ### Shape `preview` is a list nested under the top-level `verify` key. Each entry defines one preview. ```yaml verify: preview: - name: default image: api-preview port: 8000 setup: .aviator/scripts/preview-setup.sh secrets: - DB_PASSWORD - STRIPE_KEY ``` The `verify:` wrapper is required — the schema rejects unknown top-level keys, so a bare `preview:` block fails validation. If a single preview is declared with no `name`, it's treated as `default`. Always set names explicitly when you have more than one. ### Fields | Field | Type | Required | Description | | -------------- | --------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | no | Unique name within the repo. Defaults to `default`. Scenarios target a preview by this name. | | `image` | string | yes | Name of a preview image. Aviator caches the image locally and boots a container from it per run. | | `port` | int | yes | Port the app serves on. Aviator handles exposing this as a public URL. | | `setup` | string | no | Path (in the repo) to a setup script. Defaults to `.aviator/scripts/preview-setup.sh`. Runs after the container starts. | | `teardown` | string | no | Optional path (in the repo) to a teardown script. Runs before the container is destroyed. | | `secrets` | list of strings | no | Account secret keys. Each is injected into the container as an environment variable of the same name. | | `verify_skill` | string | no | Repo-relative path to this preview's [Verify skill](/verify/how-to-guides/writing-a-skill-md) entry point. Overrides the default `.aviator/verify/skills/.md` lookup. | ### Image `image` is the name of a preview image registered with Aviator for your account. Aviator caches the image locally and boots a container from it for each verification run. You can register an image through **Settings → Sandbox** in the Aviator UI. The same image system backs both the preview environment and the coding sandbox — set both to the same image when you want them to share dependencies and tooling. ### Secrets `secrets` is a list of secret keys defined on your account. Each name resolves to a value at boot time and is injected into the container as an environment variable of the same name: ```yaml secrets: - DB_PASSWORD # → env DB_PASSWORD= - STRIPE_KEY # → env STRIPE_KEY= ``` Secrets are managed in the Aviator UI under **Settings → Secrets**. Scoped per account, granted to repos explicitly. The preview container never sees the unresolved name — only the value. ### Verify skill By default, Verify reads this preview's app-driving guidance from `.aviator/verify/skills/.md` (the entry-point file may reference other files in the repo). To point the preview at a file elsewhere, set a single path: ```yaml verify_skill: docs/verify/main.md ``` The path is repo-relative; when set, it replaces the default `.md` lookup. See [Writing a Verify skill](/verify/how-to-guides/writing-a-skill-md). ### Setup script `setup` runs inside the container after start. Use it for things that aren't baked into the image: ```bash #!/usr/bin/env bash set -euo pipefail # Migrations ./bin/api migrate # Light fixtures ./bin/api seed --fixture=tests/fixtures/preview.json ``` Two rules: * **Idempotent.** The script runs every time the preview boots. Don't assume a clean slate elsewhere. * **Fast.** Heavy work (large fixtures, dependency installs) should be baked into the image. Setup-time work is part of every run's latency. See [Managing previews](/verify/how-to-guides/managing-previews) for the bake-vs-setup tradeoff. ### Teardown script `teardown` is optional. When provided, it runs before the container is destroyed. Use it to release external resources the preview acquired (e.g. a per-run tenant in a shared test environment). ### Anatomy
Preview anatomy: inputs, the preview container, and what consumes it

How the YAML fields translate to a running preview

### Examples **Minimal:** ```yaml verify: preview: - name: default image: api-preview port: 8000 setup: .aviator/scripts/preview-setup.sh secrets: - PREVIEW_DB_PASSWORD ``` **Multi-preview repo:** ```yaml verify: preview: - name: default image: api-preview port: 8000 setup: .aviator/scripts/preview-setup.sh secrets: - DB_PASSWORD - STRIPE_KEY - name: worker image: worker-preview port: 9000 secrets: - DB_PASSWORD - QUEUE_URL ``` ### See also * [Concepts: Previews](/verify/concepts/previews) * [Creating a preview](/verify/how-to-guides/creating-a-preview) * [Managing previews](/verify/how-to-guides/managing-previews) # Understanding verification results This page is a reference for the shape of a verification run's output — the run, the per-criterion results, the waivers — and how to read them. For the pipeline that produces these, see [How verification works](/verify/concepts/how-verification-works). For where they appear visually, see the runbook review document in the Aviator UI. ### Where results appear * **The runbook's review document.** Streamed live as the run progresses. The primary surface for reviewers. * **The GitHub PR check.** A single check named `aviator/verify` mirrors the run's overall status. * **The Verify tab on the pull request.** With the [Aviator Chrome extension](/aviator-chrome-extension), the same verdicts render on the PR itself, with the reviewer actions attached. See [Review verification on the pull request](/verify/how-to-guides/verify-on-github). * **A Slack DM to the PR author.** One thread per PR, with controls to re-run, waive, or remove a failing criterion. See [Slack notifications](/verify/reference/slack-notifications). * **`getRunbook` over the MCP.** Returns the latest verification record plus failing results in structured form. The shapes below are what `getRunbook` returns; every other surface is a rendered presentation of the same data. ### Run-level status | Status | Meaning | | ------------- | ----------------------------------------------------------------------- | | `pending` | Queued, hasn't started yet. | | `in_progress` | Currently executing. | | `passed` | Every criterion passed (or was waived). | | `failed` | At least one criterion failed without a waiver. | | `error` | The run itself errored — pipeline issue, not a code verdict. | | `deferred` | Waiting for baseline-invariant selection to finish before it can start. | ### Run record shape A run's record carries trigger context plus aggregate counts: | Field | Description | | ------------------ | ------------------------------------------------------------------------------------------------- | | `status` | One of the values above. | | `trigger_source` | What kicked the run off: `manual`, `ready`, `approval`, `queued`, `linked`, `criteria_edit`. | | `runbook_version` | Version of the runbook that was verified (matches what `getRunbook` returned at submission time). | | `commit_sha` | The commit verified. | | `criteria_total` | Total criteria evaluated in this run. | | `criteria_passed` | Passed. | | `criteria_failed` | Failed (excluding waived). | | `criteria_skipped` | Could not be evaluated (e.g. preview boot failed). | | `criteria_waived` | Failed but explicitly waived by a reviewer. | | `error_message` | Only set when `status = error`. | ### Per-criterion result shape Each criterion produces one result: | Field | Description | | -------------- | ----------------------------------------------------------------------------------------------------------------------- | | `criterion` | The criterion text. | | `is_invariant` | True if the criterion was materialized from an [invariant](/verify/concepts/invariants) (source: `baseline_invariant`). | | `is_waived` | True if a reviewer has waived this verdict. | | `status` | `pass`, `fail`, `warn`, or `error` (see below). | | `evidence` | Structured reference to the captured artifact backing the verdict. | | `reason` | Verifier-produced explanation when present. | | `location` | File + line range when the verifier could attribute one. | ### Criterion-level status | Status | Meaning | | ------- | ------------------------------------------------------------------------------------------------------------------------ | | `pass` | Implementation satisfies the criterion. | | `fail` | Implementation violates the criterion (or the verifier couldn't confirm it should pass). | | `warn` | Concerning but not failing — the verifier flagged something the reviewer should look at without blocking the merge gate. | | `error` | The verifier itself threw an exception or returned an inconclusive result. Treat as needing human review. | ### Evidence by verifier path The shape of `evidence` depends on which verifier produced the verdict: | Verifier path | Evidence shape | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | Code-scan | File + line range plus the relevant code snippet (diff or AST excerpt). | | Runtime | One or more of: `screenshot`, `console_log`, `dom_snapshot`, `api_response`, plus a `trace` (full agent transcript for the scenario run). | The trace is captured for every runtime scenario regardless of success — it's how you debug a verdict you disagree with. ### Reading invariant verdicts Invariant-sourced criteria look identical to user criteria on the result record. The only signal is `is_invariant: true`. They run through the same pipeline and produce the same verdict + evidence shapes. When an invariant verdict is wrong (or doesn't apply to this PR), the reviewer waives it — from the review document, or from the [Verify tab on the PR](/verify/how-to-guides/verify-on-github) — with a category: | Waiver category | When to use it | | ----------------- | --------------------------------------------------------------------- | | `false_positive` | The invariant fired but the rule misjudged this case. | | `doesnt_apply` | The rule is valid in general but isn't relevant to this PR. | | `accepted_risk` | The failure is real but the PR author accepts the trade-off. | | `fix_in_followup` | The failure is real and will be addressed in a separate follow-up PR. | Every waiver is recorded with the reviewer, the category, and a free-text reason. See [Audit trails and compliance](/verify/concepts/audit-trails-and-compliance). ### Reading the PR check The PR check named `aviator/verify` mirrors the run's overall status: | Run status | GitHub check state | | ------------- | -------------------------------------------- | | `pending` | `queued` | | `in_progress` | `in_progress` | | `passed` | `success` | | `failed` | `failure` | | `error` | `failure` | | `deferred` | `in_progress` until the run actually starts. | The check summary surfaces the aggregate counts (`X/Y criteria passed`) and links back to the runbook for the full evidence. The check doesn't only move when a run finishes. Waiving a verdict, or removing an acceptance criterion, recomputes the run's counts against the criteria that are still active and re-posts the check right away — so the gate can flip to `success` without a new run. ### See also * [How verification works](/verify/concepts/how-verification-works) * [Fixing verification failures](/verify/how-to-guides/fixing-verification-failures) * [MCP tools — `getRunbook`](/verify/reference/mcp-tools#getrunbook) * [Audit trails and compliance](/verify/concepts/audit-trails-and-compliance) # GitHub integration Reference for how Verify integrates with GitHub. ### GitHub App Verify uses the Aviator GitHub App for repository access. Install it from the Aviator UI under **Settings → Integrations → GitHub**, or directly from the GitHub Marketplace. #### Permissions The app requests the permissions needed to read code, post checks, and surface verification results on PRs: | Permission | Access | Why | | ------------------- | ---------- | ----------------------------------------------- | | Repository contents | Read | Read the diff and source for verification | | Pull requests | Read/Write | Surface verification context on PRs | | Checks | Read/Write | Create and update the `aviator/verify` PR check | | Metadata | Read | Basic repository info | If you don't see Verify behaving as expected, the most common cause is that the GitHub App doesn't have access to the repo. Change access in GitHub under **Organization Settings → Installed GitHub Apps → Aviator → Configure**. ### The PR check Every verification run is mirrored to GitHub as a single PR check. * **Check name:** `aviator/verify` * **Where it shows up:** the PR's "Checks" tab and the merge-readiness summary. With the [Aviator Chrome extension](/aviator-chrome-extension), a [**Verify** tab](/verify/how-to-guides/verify-on-github) also appears in the PR tab row for acting on the run from the PR. #### Check states The check state tracks the verification run's status: | Verification run status | GitHub check state | Notes | | ----------------------- | ------------------ | ------------------------------------------------------ | | `pending` | `queued` | Run is enqueued but hasn't started. | | `in_progress` | `in_progress` | Run is executing. | | `passed` | `success` | All criteria passed or were waived. | | `failed` | `failure` | At least one criterion failed without a waiver. | | `error` | `failure` | The run itself errored — surfaced as a check failure. | | `deferred` | `in_progress` | Waiting for invariant selection before the run starts. | The check summary links back to the runbook in the Aviator UI for the full review document. ### Branch protection To require verification before merge, add `aviator/verify` to your repo's branch protection: 1. Repository **Settings → Branches → Add rule** (or edit an existing rule for your protected branch). 2. Enable **Require status checks to pass before merging**. 3. Search for and select **aviator/verify**. Recommended settings: ``` ☑ Require status checks to pass before merging ☑ aviator/verify ☑ Require branches to be up to date before merging (optional) ``` See [Configuring branch protection](/verify/how-to-guides/configuring-branch-protection) for the step-by-step. ### See also * [Configuring branch protection](/verify/how-to-guides/configuring-branch-protection) * [Connect a repository](/verify/how-to-guides/connect-a-repository) * [Understanding verification results](/verify/reference/understanding-verification-results) * [Slack notifications](/verify/reference/slack-notifications) # Slack notifications When a verification run finishes, Verify DMs the PR author the result — and lets them re-run verification, waive a failing invariant, or remove a failing criterion without leaving Slack. For the underlying result shapes, see [Understanding verification results](/verify/reference/understanding-verification-results). ### Prerequisites * Your workspace has the Slack integration connected. See [Slack Integration Guide](/api/personal-integrations#initial-slack-setup). * The recipient has run `/aviator connect` and associated their GitHub handle with their Aviator user. * **On-premise only:** the Slack app is created by hand, and needs **Interactivity** enabled for the action buttons to work — the DMs arrive without it, but clicking a button does nothing. See [Create your Slack App](/manage/on-premise-installation/slack-integration#create-your-slack-app), in particular [Enable interactivity](/manage/on-premise-installation/slack-integration#enable-interactivity). ### The thread model Verify keeps **one thread per pull request**, in the PR author's DM with the Aviator app. The first notification posts the thread root; every later result posts as a reply, so the whole PR reads as one conversation. The root doubles as a **live status summary** — it re-renders after each result, and after any waive or removal, rather than waiting for the next run. {% hint style="info" %} Root edits are best-effort. If your workspace forbids message edits the summary can go stale; the thread replies stay complete, and the `aviator/verify` check is always authoritative. {% endhint %} ### When a notification is sent Verify sends **one notification per run**, and only on **terminal** run statuses: | Run status | Behavior | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `failed` | Always sent. | | `error` | Always sent — "Verify couldn't complete". | | `passed` | Sent **only** if the previous completed run failed, as a recovery message. First-try passes are silent, and you never get two success messages in a row. | | `pending`, `in_progress`, `deferred` | Not sent — no outcome yet. A deferred run posts normally once it actually runs. | {% hint style="info" %} Superseded runs — cancelled because a newer run replaced them — don't send a notification. {% endhint %} ### Anatomy of the message A failure summary looks like this: ``` ❌ Verify failed on acme/api#482 3 of 7 criteria failed · ⚪ 1 waived · commit a1b2c3d · triggered by push Acceptance criteria 2. Returns 429 when the rate limit is exceeded ⋮ src/api/limiter.go:88 — Handler responded 200 on the 11th request… Baseline invariants 5. auth-required-on-handlers ⋮ src/handlers/admin.go:23 — Handler does not call auth middleware… …and 1 more — view all in Aviator [ 🔁 Re-run ] ```
A Verify failure DM in Slack, showing the headline, context line, failing acceptance criteria with file:line and reason, a link to the run in Aviator, and the Re-run button

A failure notification in Slack, with the per-criterion ⋮ menus and the Re-run button.

#### Headline | Condition | Headline | | ------------------------------------------------ | ------------------------------- | | Run passed, or every remaining failure is waived | ✅ **Verify passing** | | Run failed with unwaived failures | ❌ **Verify failed** | | Run errored | ⚠️ **Verify couldn't complete** | {% hint style="info" %} **Passing with exceptions.** When every remaining failure is waived, the run's status stays `failed` — not every criterion literally passed — but the merge gate is green. Slack shows it as passing so it never contradicts the `aviator/verify` check for the same run. {% endhint %} #### Context line A `·`-joined summary: the **counts** (`X of Y criteria passing`, `X of Y criteria failed`, or `last run couldn't complete — not caused by your code`), the **waived count** when any, the **short commit SHA**, and a **trigger label** such as `triggered manually` — see [When a run is triggered](/verify/concepts/how-verification-works#when-a-run-is-triggered). Unmapped triggers read `triggered automatically`. #### Failure listing * Only **active, unwaived** failures appear. Anything waived or deleted since the run drops out, matching the merge gate. * Grouped into **Acceptance criteria** then **Baseline invariants**; an empty group is omitted. * Entries are numbered by position in the criteria list. **These numbers match the GitHub check run**, so you can carry a number between surfaces. * **At most 3** are shown; the rest collapse into `…and N more`. * Each shows `file:line` when the verdict carried a location, plus the reason **truncated to 120 characters**. Full evidence is never inlined in Slack — follow the link to the review document. ### Actions | Control | Where it appears | What it does | | ----------------------- | -------------------------------------------------- | ----------------------------------------------- | | 🔁 **Re-run** | Failed and errored summaries only | Re-runs verification on the PR | | ⋮ **Waive failure…** | Next to a failing **baseline invariant** criterion | Opens a modal to waive with a category + reason | | ⋮ **Remove criterion…** | Next to a failing **acceptance criterion** | Opens a confirmation modal to delete it | The ⋮ menu offers only the action that's legal for that criterion's type. Every action — success, refusal, or error — posts a visible reply in the thread; nothing fails silently. #### 🔁 Re-run Enqueues a fresh run and replies in the thread: ``` 🔄 Re-running verification — results will post in this thread. ``` Rapid double-clicks collapse into a single run. If the enqueue fails, or Verify is no longer enabled for the account, the reply says so instead. #### Waive failure Opens a modal with two required fields: **Category** (see [Invariants](/verify/concepts/invariants#waivers)) and **Justification** — free text, where a whitespace-only value is rejected inline and the modal stays open. On submit, Verify records the waiver, refreshes the run's counts, re-posts the `aviator/verify` check so the merge gate can unblock, confirms in the thread, and re-renders the root: ``` ✅ @user waived Handler does not call auth middleware (Accepted risk) — shipping behind a feature flag, follow-up in ENG-2214. ``` * A waiver is an **overlay** — the underlying verdict keeps its engine result for [audit](/verify/concepts/audit-trails-and-compliance). * Waiving is **idempotent**; waiving an already-waived criterion returns the existing waiver. * Only **invariant-derived** criteria can be waived. Task criteria are rejected with a pointer to edit or delete them instead. #### Remove criterion A confirmation modal, no inputs. On submit the criterion leaves the runbook's acceptance criteria and future runs won't check it: ``` @user removed Returns 429 when the rate limit is exceeded — future verify runs won't check it. ``` ### Old messages still work Controls act on the PR's **current** state, not the state captured when the message was posted. Re-run on a months-old message verifies the PR as it is today, and criterion controls resolve through the criterion's history — so they still target the right criterion even though editing criteria re-mints them. ### Notification settings Verify notifications are sent to the PR author as a Slack DM and are enabled by default — both the failure DM and the recovery DM. Per-event opt-out is not currently available in Aviator settings. To stop all Aviator DMs, disconnect your Slack account under `Settings > Personal > Integrations`. ### Troubleshooting | Symptom | Likely cause | | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | No DM at all | Slack isn't connected for the account, the user hasn't run `/aviator connect`, or their GitHub handle isn't associated with their Aviator user. | | The run passed but no DM arrived | Expected. Passes are only announced when they follow a failure. | | The thread root looks stale | Your workspace forbids message edits. Read the thread replies, or open the run in Aviator. | | Buttons do nothing or refuse | Verify is no longer enabled for the account, or your Slack account isn't linked to an Aviator user on this account. On-premise: check that **Interactivity** is enabled on the Slack app. | | Slack shows passing, but a criterion still shows failed in the UI | Every remaining failure is waived. The merge gate is green; see [Passing with exceptions](#headline). | ### See also * [Understanding verification results](/verify/reference/understanding-verification-results) * [Fixing verification failures](/verify/how-to-guides/fixing-verification-failures) * [GitHub integration](/verify/reference/github-integration) * [Concepts: Invariants](/verify/concepts/invariants) * [Slack Integration Guide](/api/personal-integrations) # Configuration reference Configuration surface for Verify — per-repo verify.yaml and account-level settings. This page is the umbrella reference for Verify configuration. Per-concept references live on their own pages and are linked from here. ### Per-repo: `verify.yaml` The root of all per-repo configuration. Edit it in the Aviator dashboard under **Verify → Settings → Verify** with the repository selected — it is not a file you commit to the repo. Each save is versioned, and earlier versions can be restored from the editor's history. #### Top-level keys | Key | Type | Default | Description | | ---------------- | ---- | ------- | ------------------------------------------------------------------------------------ | | `verify` | map | none | Root key. All Verify configuration nests under it. | | `verify.preview` | list | none | One or more preview definitions. See [Preview YAML](/verify/reference/preview-yaml). | Example: ```yaml verify: preview: - name: default image: api-preview port: 8000 setup: .aviator/scripts/preview-setup.sh secrets: - DB_PASSWORD - STRIPE_KEY ``` Unknown keys are rejected: saving fails with a validation error rather than silently ignoring them. A common mistake is starting the document at `preview:` — it must nest under `verify:`. ### Per-repo: preview block Detailed in its own reference page: → [Preview YAML reference](/verify/reference/preview-yaml) ### Account-level settings Configure in **Verify → Settings**. These apply across the account rather than per repository. #### Sandbox Set under **Verify → Settings → Sandbox**. | Setting | Type | Default | Description | | ----------------------------- | -------- | --------------- | ----------------------------------------------------------------------------- | | **Sandbox Timeout (minutes)** | integer | `60` | How long a sandbox stays active. Accepts 1–240. | | **Sandbox Image** | template | Aviator default | The image sandboxes boot from. Custom images are registered on the same page. | When the timeout expires, the sandbox is torn down and the preview link stops resolving. See [Managing previews](/verify/how-to-guides/managing-previews). ### Invariants Invariants have their own configuration surface in **Settings → Invariants**. Field-level reference and writing guidance are on the concept and tutorial pages: * [Concepts: Invariants](/verify/concepts/invariants) — sources, categories, selection * [Setting up org invariants](/verify/setting-up-org-invariants) — step-by-step setup ### MCP The MCP install and tool surface are on a dedicated reference page: → [MCP tools](/verify/reference/mcp-tools) ### See also * [Preview YAML reference](/verify/reference/preview-yaml) * [MCP tools](/verify/reference/mcp-tools) * [Spec format](/verify/reference/spec-format) * [Concepts: Invariants](/verify/concepts/invariants) # How to guides # Connect a repository This guide explains how to connect a GitHub repository to Aviator Verify. ### Prerequisites * An Aviator account * Admin or owner access to the GitHub repository * The Aviator GitHub App installed on your organization ### Steps #### 1. Install the GitHub App (if not already installed) Go to **Settings → Integrations → GitHub** in your Aviator dashboard. If you see “Not connected,” click **Connect GitHub** and follow the OAuth flow to install the Aviator GitHub App on your organization. Select which repositories the app can access: * **All repositories** — Aviator can access any repo in your org * **Select repositories** — Choose specific repos #### 2. Enable the repository for Verify Go to **Verify → Repositories**. Click **Add Repository**. Select the repository from the dropdown. Only repositories the GitHub App can access will appear. Click **Enable**. #### 3. Configure repository settings After enabling, you can configure: | Setting | Description | Default | | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------- | | **Auto-create runbook on PR open** | Opening a PR in this repository automatically creates a runbook | Off | | **Generate invariants from PR comments** | Aviator reviews merged-PR comments weekly to propose new baseline invariants | On after onboarding | | **`verify.yaml`** | Per-repo Verify configuration, including preview environments, edited under **Verify → Settings → Verify** | Empty | A preview is **not required** to get started — Verify runs code-scan only without one and produces verdicts on structural criteria from the first PR. Add a preview later when behavioral verification matters. See [Configuration reference](/verify/reference/configuration-reference) for the full surface, and [Preview YAML](/verify/reference/preview-yaml) for the preview block. #### 4. Set up the repository's coding agents Enabling the repository tells Aviator to verify it. `aviator init` sets up the other half — the coding agents that capture the intent Verify checks against. Install the [Aviator CLI](/verify/reference/cli), then run this once from a clone of the repository: ```bash aviator init ``` Choose the repo-wide setup when it asks who it's for, then pick the agents your team uses. It writes the hook configuration into the repository — `.claude/settings.json` for Claude Code, `.codex/hooks.json` for Codex. Commit those files and the setup travels with the repo. From then on, every agent session in this repository starts with a standing instruction to capture intent and acceptance criteria before opening a pull request, and gets reminded again at the PR itself. Everyone on the team needs three things: * The Aviator CLI — `brew install aviator-co/tap/aviator`, or the deb/rpm package on Linux. * Their own API token, set as `AVIATOR_API_TOKEN` or in `~/.config/aviator/config.yaml` — see [Authentication](/verify/reference/cli#authentication). Submissions are attributed to the token's owner, so a shared token collapses the audit trail. * The `/verify-submit` skill from the [Aviator agent plugins](https://github.com/aviator-co/agent-plugins). Codex users also need to run `/hooks` inside Codex and trust the hook. See [Set up agent hooks](/verify/how-to-guides/set-up-agent-hooks) for personal setup, provisioning, and removal. #### 5. Verify the connection To confirm everything works: 1. Make a small change in this repo with your coding agent. 2. Let it open a pull request. The standing instruction should prompt it to run `/verify-submit` first — if it doesn't, run `/verify-submit` yourself. 3. Open the session URL the CLI prints — you should land on the review document for this repo. 4. Check that the Aviator Verify check appears on the pull request. If the hook never fires, start a fresh agent session — the standing instruction is delivered at session start, so a session already running when you ran `init` won't have it. If the check doesn't appear or the review document fails to load, see [Fixing verification failures](/verify/how-to-guides/fixing-verification-failures). ### Adding more repositories Repeat steps 2-4 for each repository you want to enable. Step 4 is per-repo when you set it up for the whole team, since the hook config is committed to each repository. If a repository doesn’t appear in the dropdown, check that: * The Aviator GitHub App has access to it * You have admin access to the repository ### Removing a repository Go to **Verify → Repositories**, find the repository, and click **Remove**. This disables Verify for that repository. Existing submissions and verification history are retained. ### See also * [Set up agent hooks](/verify/how-to-guides/set-up-agent-hooks) * [Aviator CLI](/verify/reference/cli) * [Configuring branch protection](/verify/how-to-guides/configuring-branch-protection) * [GitHub integration](/verify/reference/github-integration) * [Configuration reference](/verify/reference/configuration-reference) # Set up agent hooks Intent is easiest to capture while the reasoning behind a change is still in the agent's context. Once the PR is open and the session has moved on, it has to be reconstructed from the diff — which is exactly the information Verify needs and the diff doesn't carry. `aviator init` sets up your coding agents to capture that intent as you open a pull request. It installs two hooks: a standing instruction at the start of every session, and a reminder that fires when the agent is about to open a PR. ### Prerequisites * The [Aviator CLI](/verify/reference/cli) * An API token, as `AVIATOR_API_TOKEN` or in `~/.config/aviator/config.yaml` — see [Authentication](/verify/reference/cli#authentication) * The `/verify-submit` skill from the [Aviator agent plugins](https://github.com/aviator-co/agent-plugins) * A supported agent — Claude Code or Codex ### Run init From inside the repository: ```bash aviator init ``` It asks who the setup is for and which agents to cover, then writes the hook configuration. Re-run it any time to add an agent or reconcile existing config — it's idempotent, and reports whether each hook was added, updated, or already in place. #### Who the setup covers This decides where the configuration lands. | Set up for | Where the hook is written | Who it covers | | -------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------- | | The whole repo | `.claude/settings.json` or `.codex/hooks.json` in the repository | Everyone who works on this repo, once you commit the files | | Just you | Your own agent config — `~/.claude/settings.json`, `~/.codex/hooks.json` | You, in every repository on this machine | Set it up for the **whole repo** to roll Verify out to a team — committing the files is what shares it. Set it up for **just yourself** to get the reminder without adding anything to the repository; it covers every repo you touch on this machine. If you've set `CLAUDE_CONFIG_DIR` or `CODEX_HOME`, the CLI honors them. #### Which agents to cover Pick the agents your team actually uses. A repo-wide setup preselects all supported agents, since it can't detect what your teammates run; a personal setup preselects the ones it finds on this machine. Everyone the setup covers needs the CLI installed, their own `AVIATOR_API_TOKEN`, and the `/verify-submit` skill. ### What the hooks do `init` adds two hook entries per agent, each calling back into the CLI. **`SessionStart`** runs `aviator hooks session-start`. It gives the agent a standing instruction: this repository uses Aviator Verify, capture the change's intent and acceptance criteria with `/verify-submit` before opening a pull request. This is the one doing the real work — session start is the only point that reaches the agent reliably ahead of a PR. If the skill isn't installed, the instruction also says how to get it for that agent. **`PreToolUse`** runs `aviator hooks pre-tool-use`. It watches for PR-opening calls — `gh pr create`, `av pr`, and the GitHub MCP server's `create_pull_request` tool — and injects a reminder when it sees one. Treat it as a backstop rather than a gate: Claude and Codex deliver this context alongside the tool result, so the agent usually reads it just after the PR command has run. Every other tool call passes through untouched. Both hooks only inject text. Neither submits anything and neither blocks a tool call. `/verify-submit` is what reads the change, drafts the intent and acceptance criteria with you, and calls [`aviator verify`](/verify/reference/cli#aviator-verify). ### Codex needs one extra step Codex won't fire a hook it hasn't been told to trust. After running `aviator init`, run `/hooks` inside Codex and trust the Aviator hook. Until you do, the configuration is in place but nothing fires. ### Non-interactive use `aviator init` is meant to be run interactively — the prompts are how you make the scope decision. For provisioning scripts, machine images, and other places where nobody is there to answer, pass the answers as flags instead: ```bash aviator init --scope team --agents claude,codex --yes ``` | Flag | Description | | ---------- | ------------------------------------------------------------------------------------------------------------------------- | | `--scope` | `team` or `self`. Prompts if omitted. | | `--agents` | Comma-separated agent ids (`claude`, `codex`). Defaults to all supported agents for team scope, detected agents for self. | | `--yes` | Skip the prompts. Requires `--scope`. | Outside a terminal the command can't prompt, so `--scope` is required there. ### Removing the hooks ```bash aviator hooks uninstall ``` This clears both scopes by default, since self-scope config lives outside the repo and is easy to forget. Narrow it with `--scope team` or `--scope self`. Uninstall removes only Aviator's own hook entries. Other hooks, and every other key in the settings file, are left alone. ### See also * [Aviator CLI](/verify/reference/cli) — install, authentication, and the full command surface * [Writing effective acceptance criteria](/verify/how-to-guides/writing-effective-acceptance-criteria) * [Your first verification](/verify/your-first-spec) # Writing effective acceptance criteria Good acceptance criteria are specific, verifiable, and complete. This guide shows how to write criteria that lead to accurate verification. ### Be specific Vague criteria can’t be reliably verified. **Too vague:** ```markdown -[ ] Endpoint is secure ``` **Specific:** ```markdown -[ ] Requires authentication via Bearer token -[ ] Returns 401 for missing or invalid token -[ ] Returns 403 if user lacks permission ``` **Too vague:** ```markdown -[ ] Handles errors properly ``` **Specific:** ```markdown -[ ] Returns 404 if resource not found -[ ] Returns 503 if downstream service unavailable -[ ] Error responses include correlation ID ``` ### Include both positive and negative requirements Say what should happen *and* what shouldn’t. ```markdown -[ ] Response includes: status, renewal_date, plan_name -[ ] Response excludes: internal_id, billing_provider_id ``` ```markdown -[ ] Creates new user record -[ ] Does not modify existing user records ``` Negative requirements often catch bugs that positive requirements miss. ### Cover error cases Don’t just test the happy path. ```markdown -[ ] Returns 400 for malformed request body -[ ] Returns 404 if subscription not found -[ ] Returns 409 if resource already exists -[ ] Returns 503 if payment service unavailable ``` Think about what could go wrong and how your code should respond. ### Add performance requirements only when meaningful ```markdown -[ ] P99 latency under 200ms -[ ] No N+1 queries ``` Only include these if they’re actual requirements. Don’t add “P99 under 100ms” to everything just because it sounds good. ### Express constraints as criteria Things that must *not* happen: ```markdown -[ ] No new external dependencies -[ ] Does not modify database schemas -[ ] Does not call billing-service directly -[ ] No breaking changes to existing API ``` ### Use concrete values **Vague:** ```markdown -[ ] Returns appropriate status code ``` **Concrete:** ```markdown -[ ] Returns HTTP 201 on successful creation ``` **Vague:** ```markdown -[ ] Response includes required fields ``` **Concrete:** ```markdown -[ ] Response includes: id, name, created_at ``` ### Group related criteria For readability, group criteria by concern: ```markdown ## Acceptance Criteria ### Endpoint behavior -[ ] Endpoint: `POST /api/v1/users` -[ ] Accepts JSON request body -[ ] Returns HTTP 201 on success ### Validation -[ ] Returns 400 if email is missing -[ ] Returns 400 if email format is invalid -[ ] Returns 409 if email already exists ### Response -[ ] Response includes: id, email, created_at -[ ] Response excludes: password_hash ### Security -[ ] Requires admin authentication -[ ] Password is hashed before storage ``` ### Common patterns #### REST endpoint ```markdown -[ ] Endpoint: `GET /api/v1/resource/{id}` -[ ] Requires authentication -[ ] Returns 200 with resource data -[ ] Returns 404 if not found -[ ] Returns 403 if user lacks access -[ ] Response format matches ResourceSchema ``` #### Data mutation ```markdown -[ ] Creates/updates record in database -[ ] Validates input before persisting -[ ] Returns created/updated resource -[ ] Emits event to message queue -[ ] Operation is idempotent ``` #### Integration ```markdown -[ ] Calls external-service API -[ ] Handles timeout with retry -[ ] Handles 5xx with graceful degradation -[ ] Logs external call with correlation ID ``` ### Criteria to avoid **Implementation details:** ```markdown # Bad - specifies how, not what -[ ] Uses Redis for caching -[ ] Implements retry with exponential backoff ``` Better to focus on behavior: ```markdown # Good - specifies observable behavior -[ ] Caches response for 60 seconds -[ ] Retries failed requests up to 3 times ``` **Subjective judgments:** ```markdown # Bad - can't be objectively verified -[ ] Code is clean and readable -[ ] Performance is acceptable ``` **Compound criteria:** ```markdown # Bad - multiple things in one criterion -[ ] Validates input, creates record, and sends email ``` Split into separate criteria: ```markdown # Good - one thing per criterion -[ ] Validates input against schema -[ ] Creates record in database -[ ] Sends confirmation email ``` ### See also * [Your first verification](/verify/your-first-spec) * [Spec format](/verify/reference/spec-format) * [How verification works](/verify/concepts/how-verification-works) # Writing a Verify skill A **Verify skill** is a short markdown file that tells Verify how to drive your running app — how to sign in, what's worth checking, and where things live. It's the app-specific knowledge the agent can't infer from your code alone. Verify reads your skill at two points: * **Planning** — when Verify turns an acceptance criterion into test scenarios, it reads your skill to ground the plan in your real login flow, routes, and what the preview can actually show. * **Running** — when it drives a headless browser against your preview to capture evidence, it follows your skill's sign-in steps and navigation. The test for what belongs here: the things a new engineer would have to ask before they could test your app sensibly are exactly what goes in a skill. ### Where the skill lives Each preview has one **entry-point** skill at `.aviator/verify/skills/.md`, named after the preview it describes. So the `default` preview reads `.aviator/verify/skills/default.md`. The entry point can **reference other files** in your repo. If your guidance grows, split it by concern and point at the pieces from the entry point — Verify reads those too, in place: ``` .aviator/verify/skills/ ├── default.md # entry point — points at the files below ├── auth.md # how to sign in └── app.md # navigation + what's observable ```
Verify reads your skill before planning and running scenarios

Verify reads your skill — and any files it references — before planning and running scenarios.

The skill is read from the **commit under verification**, so it versions with your code — update it in the same change that alters the behavior. To point a preview at a file somewhere other than the default location, set `verify_skill`: ```yaml verify: preview: - name: default image: api-preview port: 8000 verify_skill: docs/verify/main.md ``` `verify_skill` is a single repo-relative path; when set, it replaces the default `.md` lookup for that preview. See [Preview YAML](/verify/reference/preview-yaml). ### Credentials: never hard-code them When a flow needs to log in, **don't put real credentials in the skill.** Store them as account secrets (**Settings → Secrets** in the Aviator UI) and reference them by name with a `{{ secrets. }}` placeholder: ```markdown # Signing in The app redirects to `/login` on first load. To sign in: 1. Navigate to the preview URL. 2. Fill the email field with `{{ secrets.app_admin_email }}`. 3. Fill the password field with `{{ secrets.app_admin_password }}`. 4. Click "Log in". ``` When Verify drives the browser it substitutes the real value at the moment it fills the field. The value never appears in the skill, the plan, the prompt, or the run transcript — only the placeholder does. You can reference any account secret this way, and placeholders also work embedded in a string (e.g. `Authorization: Bearer {{ secrets.api_token }}`). > This is separate from a preview's `secrets:` list, which injects secrets as **environment variables into the preview container** so your app can boot. The same secret store backs both — `{{ secrets.* }}` is specifically for credentials Verify types into your UI. See [Preview YAML → Secrets](/verify/reference/preview-yaml). ### What to include Aim for the smallest set of facts that lets Verify run a scenario without trial and error: | Category | What to write | | ------------------------ | ---------------------------------------------------------------------------------------------- | | **Sign-in** | The login flow, step by step, with `{{ secrets.* }}` placeholders for credentials. | | **What's observable** | What the running preview can and can't show (see below). | | **Navigation** | Key routes and how to reach important screens. "The article list is at `/unread/list`." | | **Test data / fixtures** | Named records in the seeded data, referenced by stable name, not ID. | | **Side effects** | What's real vs. mocked. "Stripe runs in test mode — no real charges. Email sends are dropped." | | **Gotchas** | Things that bit a previous run. "First request after boot takes \~3s due to JIT warmup." | ### Tell Verify what's observable This is the most valuable thing a skill adds. Verify confirms a criterion by **driving the running app and watching what it does** — rendered UI, DOM, computed styles, console output, API responses. It can see that the app *initiated* something, but not a result that a background job, queue, or external system has to produce. So call out what your preview can and can't exercise: ```markdown ## What's observable here This is the full web UI driven through a browser, so rendered state, DOM, computed styles, and console output are all fair game. Background workers and outbound email are NOT exercised in the preview — don't try to verify anything that depends on them. ``` This keeps Verify from burning a run trying to confirm something the preview structurally can't show — it verifies the responsible code path instead. ### What to leave out Verify can already read your code. Don't repeat what it can see: * **Architecture descriptions.** "We use Express with Postgres" — the agent can see this. Don't restate it. * **Endpoint catalogs.** It will discover endpoints from the router. You don't need to list them. * **Code conventions.** That's invariants, not skills. ([Invariants](/verify/concepts/invariants)) * **Implementation history.** "We used to use library X, switched to Y in 2024." Irrelevant for running scenarios. * **The change under test.** That's the intent, submitted via MCP per change. A bloated skill hurts more than a thin one. Every irrelevant line dilutes the context and slows the agent down. ### Example A `default.md` entry point that references two more files in the same directory: `default.md`: ```markdown # App verify guidance Read both of these files (same directory) before planning or driving: - `auth.md` — how to sign in. - `app.md` — getting around, and what the preview can exercise. ``` `auth.md`: ```markdown # Signing in The app gates everything behind a login form at `/login`. 1. Navigate to the preview URL. 2. Fill the email field with `{{ secrets.app_admin_email }}`. 3. Fill the password field with `{{ secrets.app_admin_password }}`. 4. Click "Log in" — you land on the dashboard. ``` `app.md`: ```markdown # Driving the app - The main view is a list of saved items, rendered as cards, at `/unread/list`. - Settings live at `/config`; per-item actions (archive, star, delete) are on each card. ## What's observable here Rendered UI, DOM, computed styles, and console output are all available as evidence. Background jobs and outbound email are not exercised in the preview. ``` A single self-contained `default.md` works just as well — split into referenced files only when one file gets unwieldy. ### Tips * **Keep it short.** If a file passes \~150 lines, split it into referenced files. * **Lead with the facts that change behavior.** Sign-in and observability first; gotchas last. * **Use stable identifiers.** Reference fixtures by name (`org "acme"`), not by ID. IDs change when seed data is regenerated. * **Update the skill when you change the seed or the login flow.** A stale skill produces confidently wrong scenarios. * **Never put a literal secret in a skill.** Use `{{ secrets. }}` placeholders. ### See also * [Concepts: Invariants](/verify/concepts/invariants) — for rules that apply across changes * [How Verify works](/verify/how-it-works) — where skills fit in the verification pipeline * [Preview YAML](/verify/reference/preview-yaml) — `verify_skill` and `secrets` # Creating a preview This guide walks through adding a preview to a repo end-to-end: registering an image with Aviator, declaring runtime secrets, configuring the preview in your Aviator Verify settings, and confirming it boots. By the end, you'll have a working `default` preview that scenarios can run against. > Previews are optional. Verify works without one — every criterion routes to code-scan in that case. Add a preview when behavioral verdicts (endpoint contracts, UI flows, error shapes) start mattering for your team. See [Concepts: Previews](/verify/concepts/previews). **Time:** \~15 minutes **Prerequisites:** * Admin access to your Aviator account (to manage images and secrets) * The repo connected to Aviator — see [Connect a repository](/verify/how-to-guides/connect-a-repository) For the concept overview, see [Concepts: Previews](/verify/concepts/previews). For the full field list, see [Preview YAML reference](/verify/reference/preview-yaml). ### Step 1: Register a preview image Aviator boots a preview container from an image you register under **Settings → Sandbox**. Give it a short, stable name — e.g. `api-preview` — which you'll reference from your preview config. There are two ways to register one. #### Option A: From a Dockerfile Provide a Dockerfile and Aviator builds the image for you. Choose this when your service's environment can be assembled from packages and you want Aviator to own the build. Bake in everything your service needs to run — the language runtime, system packages, and dependencies (`apt` / `npm` / `pip` installs). `git`, `git-lfs`, and the coding-agent CLI are added automatically. `COPY` and `ADD` **aren't supported**, so you can't bake your own source into the image this way — Aviator checks out the branch under test at boot, and your setup script (Step 4) builds and starts the service from that checkout. #### Option B: From a container registry image Point Aviator at an image you already publish to your own registry — AWS ECR, Google Artifact/Container Registry, or any generic registry (Docker Hub, GHCR, Artifactory, self-hosted). Choose this when you want the preview to run your *real* environment rather than one reconstructed from a Dockerfile: * **Bake in anything, not just packages.** Since an Aviator-built Dockerfile can't `COPY`/`ADD`, it can only install public packages. A registry image can contain whatever your build produces — compiled binaries, private artifacts, a fully prepared environment. * **Reuse the image your CI already builds.** One build definition instead of a second Dockerfile to keep in sync, so previews mirror what you actually ship — and it covers private base images and build-time secrets Aviator's builder can't use. * **Faster, prod-like boots.** Bake the repository into the image (see **Repo root** below) and Aviator fetches the branch in place instead of cloning it fresh each run. 1. Open **Settings → Sandbox → Add → From Registry Image**. 2. Select the registry type and enter the fully-qualified image reference, e.g. `123456789.dkr.ecr.us-west-2.amazonaws.com/api:latest`. 3. Provide **read-only** pull credentials. Scope them as tightly as possible — Aviator only needs to pull and inspect the image: * **AWS ECR** — an access key ID, secret access key, and region for an IAM principal with `AmazonEC2ContainerRegistryReadOnly` on the repository. * **GCP** — a service-account key (JSON) with `roles/artifactregistry.reader` on the repository. * **Generic** — a username and password/token (omit both for a public image). Credentials are stored encrypted and are never displayed again. 4. *(Optional)* If your image already has the repository checked out — for example, baked in during your own CI build — set **Repo root** to that path (e.g. `/code`). The preview runs from there instead of cloning the branch fresh. Leave it blank to have Aviator clone the branch at boot. The image **must include `git`** — Aviator checks out (or updates) the branch under test inside the container at boot. After you register the image, Aviator pulls and prepares it. The image card shows **Building**, then **Success** once it's ready — reference it from your preview config at that point. **Keeping a registry image current.** When you push a new image to the same tag, Aviator re-pulls and rebuilds it automatically within the hour — and only when the tag's digest has actually changed, so an unchanged tag costs nothing. To pull a new build immediately, open the image under **Settings → Sandbox** and click **Check for Update**. To pin an exact build that never moves, reference an image digest (`…@sha256:…`) instead of a tag. If you don't have a service image yet, the simplest path is to start with a base image that has your runtime (e.g. `node:20`) and let the setup script install dependencies. As your previews mature, bake more into the image. ### Step 2: Declare runtime secrets Anything your service reads from the environment at runtime — DB password, third-party API keys — needs to be a secret. For each runtime secret: 1. Open **Settings → Secrets**. 2. Add the secret using the *exact name* your service expects (e.g. `DB_PASSWORD`, not `PG_PASSWORD`). 3. Grant it to the repo. Naming matters: each secret is injected into the preview as an environment variable of the same name. ### Step 3: Configure the preview Previews are configured in your **Aviator Verify settings**. Add a `preview` entry under `verify`: ```yaml verify: preview: - name: default image: api-preview port: 8000 setup: .aviator/scripts/preview-setup.sh secrets: - DB_PASSWORD - STRIPE_KEY ``` The fields to get right: * **`image`** — the image name you registered in Step 1. * **`port`** — the port your service listens on inside the container. This is what the scenario runner connects to. * **`setup`** — path in your repo to the script that starts your service (see Step 4). * **`secrets`** — every name listed here must exist in your secret store and be granted to this repo, or the boot fails with a clear error. ### Step 4: Write the setup script The setup script is what **starts your service** — it's the launch point of every preview, so this step isn't optional. Aviator runs it once inside the container at boot, after checking out the branch under test, and it's also where migrations, fixtures, and warm-up belong. Your declared secrets and a `PREVIEW_URL` variable are injected into the script's environment. Create `.aviator/scripts/preview-setup.sh` in the repo (at the path your `setup` field points to). The script below is a sample — the exact commands are specific to your repo and service: ```bash #!/usr/bin/env bash set -euo pipefail # Build/install from the checked-out source, run migrations, seed fixtures. ./bin/api migrate ./bin/api seed --fixture=tests/fixtures/preview.json # Start the service in the BACKGROUND so this script can exit — Aviator marks # the preview ready once the script returns. Listen on your configured port. nohup ./bin/api serve --port 8000 > /tmp/preview.log 2>&1 & # There's no health check, so give the port a moment to come up before exiting. sleep 2 ``` Two rules of thumb: * **Start the service in the background and let the script return.** Aviator marks the preview ready when the script exits `0` — there's no separate health check — so background your server and confirm the port is listening before the script finishes. If the script runs the server in the foreground, it never returns and the boot times out. * **Keep it fast.** Heavy, rarely-changing work (dependency installs, compilation) belongs in the image, not here — setup time is part of every run's latency. See [Managing previews](/verify/how-to-guides/managing-previews) for the bake-vs-setup tradeoff. ### Step 5: Commit and trigger a verification Save your preview config in Verify settings and commit the setup script to the repo. Submit through the MCP (or open a PR if you've already submitted). When the next verification run starts, watch the run timeline in the review document: * You'll see a "Building preview" line while the container is prepared. * Then "Booting preview" while the setup script runs. * Then "Preview ready" once the setup script finishes. * Scenarios start executing. If the preview fails to boot, the failure shows up here with the exit code and the last lines of container output. ### Step 6: Verify the open-preview link Once a run produces a verdict, open the review document. The run panel includes an **Open preview** link. Click it — it should land you at the preview's port, accessible from your browser. If the link works, the preview is fully wired. Reviewers can now open it for ad-hoc exploration during review. ### Common boot failures | Symptom | Likely cause | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `image not found` | The image name in your preview config doesn't match what's registered in **Sandboxes**, or the image isn't granted to this repo. | | Registry pull / auth error | For a registry-sourced image, the stored pull credentials are wrong, expired, or lack read access to the repository. Re-enter them in **Sandboxes**. | | Missing secret error | A name in `secrets` isn't defined or isn't granted to this repo. | | Setup script never finishes / times out | The script runs the server in the foreground and blocks. Start it in the background so the script can exit. | | Scenarios can't reach the service | It bound to `localhost` only, or exited before listening. Bind to `0.0.0.0` on the configured port and confirm it's up before the setup script returns. | | Setup script exits non-zero | Migration failure or fixture path wrong. The container output shows the exact line. | ### Next steps * [Managing previews](/verify/how-to-guides/managing-previews) — bake-vs-setup, refresh, multi-preview repos * [Seed data for previews](/verify/how-to-guides/seed-data-for-previews) — fixture strategies * [Writing a SKILL.md](/verify/how-to-guides/writing-a-skill-md) — give the scenario runner context for your preview # Managing previews Once a preview is up, the ongoing work is keeping it fast, fresh, and matched to what scenarios need. This guide covers the recurring decisions. For setup, see [Creating a preview](/verify/how-to-guides/creating-a-preview). For the schema, see [Preview YAML reference](/verify/reference/preview-yaml). ### Bake into the image or run at setup time? Every preview has work that has to happen before scenarios run: dependencies, migrations, fixtures, warm-up. You can put that work in the image build, or in the `setup` script that runs at boot. | Put it in the image when… | Put it in `setup` when… | | ------------------------------------------------------------------------ | ------------------------------------------------------------------ | | It's slow (`npm install`, downloading large blobs, compiling assets) | It depends on per-run state (current DB schema, fresh fixtures) | | It rarely changes (system packages, language runtime, compiled binaries) | It changes every run (re-seeding, applying migrations from `HEAD`) | | It doesn't depend on secrets that are only available at runtime | It needs runtime secrets to complete | Default: bake heavy work into the image. Use `setup` for the things that genuinely have to be fresh. A preview that takes 90 seconds to boot is a preview people stop trusting. If you're consistently over 60 seconds, look at what's in `setup` and move what you can into the image. ### Keeping the image fresh `image` references a preview image registered with Aviator. Aviator caches the built image and boots from the cached copy per run, so a preview is only as fresh as the last build it picked up. How it refreshes depends on how the image is registered: * **Registry-sourced images.** Push a new image to the same tag. Aviator re-pulls and rebuilds it automatically within the hour — and only when the tag's digest actually changed. To pull a new build immediately, open the image under **Settings → Sandbox** and click **Check for Update**. * **Dockerfile-built images.** Aviator builds these from the Dockerfile you provided at registration. When the contents need to change, register the updated image under **Settings → Sandbox**. Two patterns work well for registry-sourced images. Both are set on the image itself under **Settings → Sandbox**, not in `verify.yaml` — the `image` field there only names the registered image: * **Track a moving tag.** Register the image against a stable tag (e.g. `registry.example.com/api-preview:latest`) and let each new push flow through. Simple, recommended for most teams. * **Pin an exact build per release.** For changes that must be reproducible months later for compliance, register the image by digest (`…@sha256:…`) or by a versioned tag (e.g. `registry.example.com/api-preview:v1.42.0`). The exact image used at verification time stays fixed. ### Multi-preview repos Most repos start with a single `default` preview. Add a second when: * **You have multiple services in one repo** that scenarios need to hit independently. Declare each as its own preview with its own port. * **Some scenarios need much higher fidelity** (real third-party sandboxes, big seed data) and others don't. Split into a fast `light` preview and a slow `heavy` one — most scenarios target `light`; the few that need fidelity opt into `heavy`. * **You have a prod-mirror configuration** worth verifying against. Declare it as `mirror` with production-like secrets and seed. Don't proliferate previews for small differences. Two previews that share most of their setup are easier to maintain as one preview with a setup-script switch. ### TTL and cleanup Previews are torn down when their sandbox times out. The "Open preview" link in the review document stays live until then, so reviewers can poke at the container; once it expires, it's gone. The window is the account-wide **Sandbox Timeout (minutes)** setting under **Verify → Settings → Sandbox** — 60 minutes by default, adjustable from 1 to 240. If reviewers need longer access for a security review, a customer demo, or a deep debugging session, raise that setting, or launch the preview again from the review document when they're ready to look. ### Updating an existing preview Most preview changes are low-risk — your setup script is read fresh from the branch every run, and config changes take effect as soon as you save them in Verify settings. Two exceptions: * **Removing a secret your service still reads.** The preview will boot, but the service will fail at first use. Update the service first, then trim the secret. * **Changing `port`.** Update the service's port binding in the same commit. A port mismatch makes the preview never become ready, which looks like a generic boot failure. ### See also * [Creating a preview](/verify/how-to-guides/creating-a-preview) * [Seed data for previews](/verify/how-to-guides/seed-data-for-previews) * [Preview YAML reference](/verify/reference/preview-yaml) * [Concepts: Previews](/verify/concepts/previews) # Seed data for previews Scenarios are only as good as the state they run against. A preview that boots with no data, or with different data each run, produces verdicts you can't trust. This guide covers the seeding strategies that work. For where seed work fits in a preview, see [Creating a preview](/verify/how-to-guides/creating-a-preview) and [Managing previews](/verify/how-to-guides/managing-previews). ### Why determinism matters Verify treats verification as reproducible. If a scenario passes today, it should pass tomorrow on the same code. Non-deterministic seeds break that contract — a scenario might rely on "the first user" or "any active subscription," and which one shows up depends on insertion order. Three properties to preserve: * **Known records.** Scenarios should reference data by stable identifiers, not by "find any." * **Reset between runs.** Every Verify run starts from the same state. State from the previous run never leaks in. * **Reset between scenarios within a run.** If two scenarios mutate the same record, one shouldn't see the other's changes. If any of these break, the noise floor of verification rises. Verdicts get less reliable. People start ignoring them. ### Strategy 1: Fixture files in the repo Best for small, stable datasets. Commit a JSON or SQL file to the repo and load it in `setup`. ```yaml verify: preview: - name: default image: ghcr.io/acme/api:edge port: 8000 setup: .aviator/scripts/preview-setup.sh secrets: - DB_PASSWORD ``` ```bash # .aviator/scripts/preview-setup.sh #!/usr/bin/env bash set -euo pipefail ./bin/api migrate ./bin/api seed --fixture=tests/fixtures/preview.json ``` ```json // tests/fixtures/preview.json { "orgs": [ { "slug": "acme", "plan": "pro" }, { "slug": "starter-co", "plan": "starter" } ], "users": [ { "email": "admin@acme.test", "org": "acme", "role": "admin" }, { "email": "member@acme.test", "org": "acme", "role": "member" } ] } ``` Reference these by name in your SKILL.md (`org "acme"`, `user "admin@acme.test"`). Scenarios read the skill, use the named records, and produce reproducible verdicts. **Pros:** Diff-able, reviewable, no external dependencies. **Cons:** Doesn't scale past a few hundred records; tedious to maintain for evolving schemas. ### Strategy 2: Factories at setup time Best for medium-sized seeds where you want code, not data, in the repo. Run a factory script in `setup`. ```bash # .aviator/scripts/preview-setup.sh #!/usr/bin/env bash set -euo pipefail ./bin/api migrate ./bin/api factory create-org --slug=acme --plan=pro ./bin/api factory create-user --org=acme --email=admin@acme.test --role=admin ./bin/api factory create-subscription --org=acme --plan=pro --status=active ``` Factories work especially well when your test suite already uses them — share the code. Build factories with explicit arguments (`--slug=acme`) rather than randomness, so the seed is the same every run. **Pros:** Scales further than fixtures, evolves with your schema. **Cons:** Setup time grows linearly with seed size. ### Strategy 3: Anonymized prod snapshot Best for high-fidelity scenarios — pricing logic, complex permission graphs, performance under realistic data shape. Snapshot prod, scrub PII, store the snapshot, restore it at boot. ```bash # .aviator/scripts/preview-setup.sh #!/usr/bin/env bash set -euo pipefail ./bin/api restore --snapshot=$ANONYMIZED_SNAPSHOT_URL ``` The snapshot lives somewhere persistent (S3, GCS) and is refreshed on a schedule by a separate job that pulls from prod, scrubs, and uploads. The preview just restores. **Pros:** Realistic shape, finds bugs synthetic data misses. **Cons:** Operational overhead — snapshot pipeline, PII scrubbing, access control on the snapshot. Most teams don't need this until they're verifying behavior that depends on real data shape (large-tenant queries, pricing on actual subscription distributions). Start simpler. ### Strategy 4: Baked baseline + setup additions Best for repos that boot fast but need a couple of per-run mutations. Bake the baseline into the image, mutate at setup time. * **Image build (CI):** runs migrations and seeds the baseline (10k orgs, 100k users from anonymized prod, or whatever your baseline is). This is slow, runs once per image push. * **Setup script (per run):** applies any migrations new in this PR, adds the named records the scenarios reference. This combines the speed of pre-baked state with the freshness of per-run additions. It's the right answer once your baseline is too big for setup-time loading but you still need pull-time data. ### Reset between scenarios within a run If your scenarios mutate state, give them a reset hook so they don't interact. ```bash # Expose an internal reset endpoint, gated by an env flag the preview sets. POST /__test__/reset ``` The scenario runner calls `/__test__/reset` between scenarios. The reset truncates user-created tables and re-seeds from the baseline. Two rules: * **Gate the reset endpoint.** Only enable when `AVIATOR_TEST_MODE=1` (or equivalent). It must be unreachable in production. * **Reset to a known state, not an empty one.** Resetting to empty makes every scenario re-create fixtures, which is slow and error-prone. ### Anti-patterns to avoid * **Random IDs in fixtures.** `user-{{uuid}}` makes scenarios reference moving targets. Use stable slugs. * **Time-relative data.** "User created 7 days ago" produces different data depending on when verification runs. Use fixed `created_at` values. * **Implicit dependencies between fixtures.** "Add this user *after* that org" baked into ordering is brittle. Make the dependency explicit (foreign key references). * **No reset between runs.** Trusting that the previous preview was torn down isn't a strategy. Re-seed every run. ### See also * [Creating a preview](/verify/how-to-guides/creating-a-preview) * [Managing previews](/verify/how-to-guides/managing-previews) * [Writing a SKILL.md](/verify/how-to-guides/writing-a-skill-md) — how scenarios learn what your seed contains # Running with remote agents The default Verify flow is local: you work with your coding agent in your editor, and the [Aviator MCP](/verify/reference/mcp-tools) submits the intent when you're done. This guide covers the alternative — running the agent remotely through Aviator Runbooks. Use this when: * You want batch or off-hours work — kick off ten tasks at once and come back to a stack of verified PRs. * The change is driven by someone who isn't a developer — a PM filing a ticket-style task, a security team asking for a small fix across many repos. * You want full attribution and reproducibility — the agent runs in a sandbox Aviator manages, with the same configuration every time. This is a secondary flow. Most changes go through the local-MCP path. If you're picking your first integration, start there. ### How the remote flow differs In the local flow: ``` You + agent (local) → Aviator MCP → Verify → Review document ``` In the remote flow: ``` Runbook (with task description) → Aviator-hosted agent (sandbox) → MCP → Verify → Review document ``` The implementation step moves into a sandbox. The MCP, verification, and review document are identical. The only thing that changes is *where* the agent runs and *who* drives it. ### Setting it up You need: * A Verify-connected repo, with at least one preview defined. * A Runbook for the task — see [Runbooks: Getting started](/runbooks/getting-started) for the basics. * The same MCP install as the local flow, scoped to the Aviator-hosted agent. The MCP is preconfigured for Aviator-hosted agents — you don't install it separately. Submissions are attributed to the user who triggered the runbook, not to the agent. ### Writing a runbook for Verify For runbook authoring conventions — task description, scope, personas, sandboxes — see [Runbooks: Getting started](/runbooks/getting-started) and [Runbooks: Runbook Format](/runbooks/concepts/runbook-format). The Verify-specific addition: write a task description specific enough that the remote agent can implement against it without follow-up clarification. Vague descriptions produce runbooks that drift from the intent — and intent drift is exactly what verification is supposed to catch, so this gets noisy fast. When the runbook completes, verification runs against the resulting branch the same way it does for any other submission. The review document is identical. ### What you trade off Compared to the local flow: | Local flow | Remote (Runbook) flow | | ------------------------------------------- | --------------------------------------------------- | | Fast iteration, immediate context | Higher latency per task | | You see and steer the agent's choices live | The runbook description must be precise upfront | | Editor + branch state is your environment | A clean sandbox per run | | Submits attributed to the implementing user | Submits attributed to whoever triggered the runbook | | One task at a time | Batch many tasks | The remote flow is better for *replication*. The local flow is better for *exploration*. ### Reviewing the result The review document is identical to the local flow — same intent + criteria + verdicts + evidence surface. Reviewers can tell a remote-flow change apart by the submission attribution (it shows the Runbook ID alongside the user), but they don't have to treat it differently. If a remote run fails verification, two recovery paths: * **Re-run the runbook.** If the failure was transient or the description needed sharpening, re-running is the cheapest fix. * **Take it local.** Check out the branch, open it in your local agent, iterate, and submit through the local MCP. The audit trail records both submissions. ### See also * [How Verify works](/verify/how-it-works) * [MCP tools](/verify/reference/mcp-tools) * [Runbooks: Getting started](/runbooks/getting-started) — for the runbook side of the flow # Configuring branch protection ## How to Require Aviator Verify to pass before PRs can merge by adding it to your GitHub branch protection rules. ### Prerequisites * Admin access to the GitHub repository * Aviator Verify enabled for the repository ### Steps #### 1. Open branch protection settings In GitHub, go to your repository’s **Settings → Branches**. Click **Add rule** or edit an existing rule for your protected branch (usually `main`). #### 2. Require status checks Under “Protect matching branches,” enable: ``` ☑ Require status checks to pass before merging ``` #### 3. Add the Verify check as required In the search box, type "aviator" and select: ``` ☑ aviator/verify ``` This makes verification a required check. PRs cannot merge until verification passes. #### 4. Optional: Require branches to be up to date Enable this option if you want PRs to be based on the latest target branch: ``` ☑ Require branches to be up to date before merging ``` This helps avoid merge conflicts but can create more CI load. #### 5. Save changes Click **Save changes** at the bottom of the page. ### Testing the configuration Open a PR for a branch where you've submitted an intent through the MCP. You should see: 1. The `aviator/verify` check appears. 2. Merge button is disabled until verification passes. 3. If verification fails, the PR cannot be merged (without admin override). ### Bypassing verification Admins can bypass branch protection for emergencies. This is logged in both GitHub and Aviator audit trails. To allow admin bypass: ``` ☑ Allow specified actors to bypass required pull requests ``` Add users or teams who should have bypass ability. ### Multiple required checks You likely have other required checks (CI, tests, linting). The `aviator/verify` check works alongside them: ``` Required checks: ☑ ci/build ☑ ci/test ☑ aviator/verify ``` All checks must pass for the PR to merge. ### See also * [Connect a repository](/verify/how-to-guides/connect-a-repository) * [GitHub integration](/verify/reference/github-integration) # Fixing verification failures A failed verification means either your code, your criteria, an invariant, or your preview needs attention. This guide covers the common failure shapes and how to resolve each. ### Reading the failure Open the review document for the run. Every failed verdict shows: * **The criterion text** (or invariant title). * **Which verifier path handled it** (Code-scan or Runtime). * **Evidence** — the snippet, request/response, screenshot, or other captured artifact that produced the verdict. * **A reason** describing why the verdict went the way it did. * **A location** — file + line range, when the verifier could attribute one. The failing rows and their reasons also render on the pull request itself in the [Verify tab](/verify/how-to-guides/verify-on-github), if you have the Aviator Chrome extension. Most failures fall into one of the patterns below. ### Criterion failed on Runtime The most common failure. The runtime runner drove your preview and the assertion didn't hold. ``` ✗ Returns 429 when rate exceeded Verifier: Runtime Evidence: POST /api/v1/public/users → 200 (expected 429) ``` **Cause:** the implementation doesn't behave as the criterion asserts. **Fix:** make the code do what the criterion says. Push the fix; verification re-runs. If you're confident the code is right and the runtime check is wrong, see *When the failure is the verifier* below. ### Criterion failed on Code-scan The verifier inspected the diff or AST and the assertion didn't hold. ``` ✗ No new direct dependencies Verifier: Code-scan Evidence: package.json:42 — added "got@^12.0.0" ``` **Cause:** the assertion is structural and the diff violates it. **Fix:** either remove the offending change or update the criterion if the change is intentional. Edit the criteria via [`editRunbook`](/verify/reference/mcp-tools#editrunbook) (for user criteria) or work with the reviewer to waive (for invariant criteria). ### Invariant violation A team-defined rule flagged the change. ``` ✗ auth-required-on-handlers (invariant) Verifier: Code-scan Evidence: src/handlers/admin.go:23 — Handler AdminUsers does not call an authentication middleware before responding. ``` **Cause:** the code breaks a rule that applies across changes. **Fix paths, in order of preference:** 1. **Fix the code.** Most of the time the rule is right and the change just missed it. 2. **Waive the verdict with a category.** For legitimate cases the invariant didn't anticipate, the reviewer waives from the review document or from the [Verify tab on the PR](/verify/how-to-guides/verify-on-github). Pick the right category: * `false_positive` — the rule fired but misjudged this case. * `doesnt_apply` — the rule is valid in general but isn't relevant to this PR. * `accepted_risk` — the failure is real but the author accepts it. * `fix_in_followup` — will be addressed in a separate PR. 3. **Fix the invariant.** If you're waiving the same invariant repeatedly across changes, the rule is wrong. Tighten its conditions or rewrite the body. See [Concepts: Invariants](/verify/concepts/invariants). ### Preview boot failure Verification couldn't bring up the preview, so no runtime checks ran. ``` ✗ Preview did not become ready Phase: setup script Exit code: 1 Container output (last lines): [...] ``` **Cause:** something is wrong with the preview itself — secret missing, setup script broken, image incompatible. **Fix:** start with the container output. The common shapes are listed under [Creating a preview — Common boot failures](/verify/how-to-guides/creating-a-preview#common-boot-failures). For ongoing flakiness, see [Managing previews — When the preview is wrong](/verify/how-to-guides/managing-previews#when-the-preview-is-wrong). ### Runtime run terminated The preview booted, but the runtime runner stopped without producing a clean verdict. Common termination reasons: | Reason | What it means | | ----------------- | --------------------------------------------------------------- | | `caps_exceeded` | Hit a hard cap on tool calls, wall time, or per-scenario cost. | | `loop_detected` | The runner got stuck repeating the same action or page state. | | `stuck` | A periodic check determined the runner wasn't making progress. | | `give_up` | The runner explicitly decided it couldn't verify the criterion. | | `unhandled_error` | The runner raised an unclassified exception. | **Fix:** * `caps_exceeded` or `give_up`: often the criterion is too expensive to verify or the preview is too slow. Move setup work into the image, or rewrite the criterion to be tighter. * `loop_detected` or `stuck`: usually the preview's UI/state is non-deterministic between runs. See [Seed data for previews](/verify/how-to-guides/seed-data-for-previews). * `unhandled_error`: file as a bug from the review document. ### When the failure is the verifier, not the code Occasionally a verdict is wrong. Before assuming it's a bug: * **Re-read the evidence.** Does it actually contradict the criterion, or are you and the verifier interpreting the criterion differently? * **Check the criterion phrasing.** "Requires authentication" might be read as "calls AuthMiddleware" or "rejects unauthenticated callers." Tighten the phrasing — for user criteria, via `editRunbook`; for invariants, by editing the catalog entry. * **Check the preview.** If a runtime verdict is wrong, the preview's state at run time might be wrong — wrong fixtures, stale seed. See [Seed data for previews](/verify/how-to-guides/seed-data-for-previews). If you've ruled all that out and the verdict still seems wrong, click **Report verdict** in the review document. Include the run ID; the team uses these to improve the classifier and verifier accuracy. ### Re-running verification After fixing issues, verification re-runs automatically on the next push. You can also trigger it manually from the runbook UI, with **Rerun verification** in the [Verify tab on the PR](/verify/how-to-guides/verify-on-github), or with the **🔁 Re-run** button on the Slack notification for the run. Re-running is safe — verifier results are stable on identical input, and the system caches runtime evidence per criterion + change set. If you got the failure as a Slack DM, you can also waive a failing invariant or remove a failing criterion directly from that message. See [Slack notifications](/verify/reference/slack-notifications#actions). ### Getting help If you can't resolve a failure: * Check the run timeline and container output from the review document. * Ask on Discord: [discord.gg/aviator](https://discord.gg/MmQWrY9xrA). * Email support: . Include the runbook number when asking. It's in the URL of the review document (`r/{number}`). ### See also * [How verification works](/verify/concepts/how-verification-works) — the verifier pipeline * [Concepts: Invariants](/verify/concepts/invariants) — waivers and the catalog * [Managing previews](/verify/how-to-guides/managing-previews) — preview reliability * [Writing effective acceptance criteria](/verify/how-to-guides/writing-effective-acceptance-criteria) * [Review verification on the pull request](/verify/how-to-guides/verify-on-github) — resolving failures from GitHub * [Slack notifications](/verify/reference/slack-notifications) — triaging failures from Slack # Review verification on the pull request With the [Aviator Chrome extension](/aviator-chrome-extension), a **Verify** tab appears in the pull request's tab row, next to Files changed. It's a trimmed-down, live view of the review document, so you can see why verification failed and unblock the PR without leaving GitHub.
The Verify tab selected on a GitHub pull request, showing a failed run header with rerun and open-in-Aviator buttons, four passing acceptance criteria, a failing invariant with its reason, and a waived invariant struck through with its category and reason

The Verify tab on a pull request with a failing run

### Prerequisites * The [Aviator Chrome extension](/aviator-chrome-extension) installed and signed in to Aviator. * The pull request has a verification run. The tab only appears once a run exists; PRs without one show nothing. * An Aviator backend on version `2026.07.16-2` or newer. Aviator Cloud is always current. A self-hosted deployment older than that doesn't serve the data the tab reads, so the tab stays hidden. ### What it shows The tab opens in place of GitHub's content and mirrors the review document: * **The run outcome** — passed, passed with N accepted failures, failed, errored, in progress, or deferred (waiting on invariant selection). * **The verified commit** and when and why the run happened (for example, "9 days ago on queue"). * **Every criterion and invariant** with its verdict icon and, for failing rows, the reason it failed. Invariants appear in their own section and link to their invariant page. * **Waived rows** are struck through and show the waiver's category and reason. The outcome tracks the `aviator/verify` check, so the tab never disagrees with it — a run that failed but has every failure waived reads as passed with accepted failures. Live updates arrive on their own; you don't need to refresh. Use **Open in Aviator** for the full review document (evidence, scenarios, run history). ### Rerun verification Select **Rerun verification** to start a new run against the PR's latest commit. It's unavailable while a run is in progress, and when the runbook has no open PR to verify against it explains why. ### Waive an invariant failure Failing invariant rows offer **Waive…**. Invariants are account-wide rules you can't edit here, so failures clear by waiving them: 1. Pick a category — **False positive**, **Doesn't apply**, **Accepted risk**, or **Fix in follow-up**. 2. Give a reason (required). 3. Confirm. The failure clears, the counter and header update, and the `aviator/verify` check is re-posted.
The waive dialog showing the invariant text, a category dropdown set to False positive, and a required reason field

Waiving an invariant failure with a category and a reason

### Remove an acceptance criterion Failing acceptance-criteria rows offer **Remove…**, which drops a criterion you don't want verified. The last remaining criterion and account invariants can't be removed; when the backend refuses, the tab tells you why. ### Related * [Fixing verification failures](/verify/how-to-guides/fixing-verification-failures) — the failure shapes and how to resolve each. * [GitHub integration](/verify/reference/github-integration) — the `aviator/verify` check and its states. # Export audit logs Export verification and reviewer-decision records from the Aviator UI for compliance reporting. For the underlying concept and what each record contains, see [Concepts: Audit trails and compliance](/verify/concepts/audit-trails-and-compliance). ### Prerequisites * Admin access to your Aviator organization. * At least one verification run in the period you want to export. ### Exporting from the dashboard #### 1. Open the audit view Go to **Verify → Audit**. You'll see a chronological list of records: runbook submissions, verification runs, individual verdicts, and reviewer actions (approvals + waivers). #### 2. Filter the data Use the filters at the top to narrow: | Filter | Options | | --------------- | --------------------------------------------------------------------- | | Date range | Last 7 days, 30 days, 90 days, or custom range. | | Repository | All repos, or a specific repo. | | Record type | Runbook submissions, verification runs, verdicts, waivers. | | Actor | All users, or a specific submitter or reviewer. | | Verdict status | `pass`, `fail`, `warn`, `error`. | | Waiver category | `false_positive`, `doesnt_apply`, `accepted_risk`, `fix_in_followup`. | #### 3. Export Click **Export** and choose a format: * **JSON** — machine-readable, good for downstream processing. * **CSV** — spreadsheet-compatible. * **PDF** — formatted report, good for handing to auditors directly. The export contains every record matching your filters. ### Record shapes Each record carries fields specific to its type. Common shapes: **Runbook submission:** | Field | Description | | --------------------- | --------------------------------------------------------------- | | `runbook_number` | User-facing runbook ID. | | `submitted_by` | Submitter (the user the MCP token belonged to, or the UI user). | | `submitted_at` | ISO 8601 timestamp. | | `repo`, `branch` | Git context — target branch and working branch. | | `commit_sha` | The HEAD commit at submission time. | | `intent` | The submitted intent (plain-language description). | | `acceptance_criteria` | The acceptance criteria submitted with the runbook. | **Verification run:** | Field | Description | | ----------------------------------------------------- | ---------------------------------------------------------------------- | | `run_id` | Unique ID. | | `runbook_number` | The runbook this run belongs to. | | `runbook_version` | The version of the runbook that was verified. | | `trigger_source` | `manual`, `ready`, `approval`, `queued`, `linked`, or `criteria_edit`. | | `commit_sha` | The commit verified. | | `status` | `pending`, `in_progress`, `passed`, `failed`, `error`, `deferred`. | | `criteria_total / passed / failed / skipped / waived` | Aggregate counts. | | `started_at`, `completed_at` | ISO 8601 timestamps. | **Verification result (one per criterion):** | Field | Description | | -------------- | ----------------------------------------------------------------------- | | `run_id` | The run this result belongs to. | | `criterion` | The criterion text. | | `is_invariant` | True if the criterion was materialized from an invariant. | | `is_waived` | True if the verdict has been waived. | | `status` | `pass`, `fail`, `warn`, `error`. | | `evidence` | Reference to the captured evidence (snippet, screenshot, API response). | | `reason` | Verifier-produced explanation. | | `location` | File + line range when applicable. | **Waiver:** | Field | Description | | ----------- | --------------------------------------------------------------------- | | `result_id` | The verification result being waived. | | `waived_by` | Reviewer. | | `waived_at` | ISO 8601 timestamp. | | `category` | `false_positive`, `doesnt_apply`, `accepted_risk`, `fix_in_followup`. | | `reason` | Free-text justification. | ### Generating compliance evidence For an audit period, the most useful exports: * **Runbook submissions + reviewer decisions.** Demonstrates that every change had an explicit submission and a recorded sign-off. * **Verification results.** Demonstrates that every change was systematically verified against its criteria and against the team's invariants. * **Waivers.** Demonstrates that exceptions were categorized, attributed, and reasoned. Filter to your audit period and export as PDF for sharing. See [Concepts: Audit trails and compliance — Compliance framework mapping](/verify/concepts/audit-trails-and-compliance#compliance-framework-mapping) for SOC 2 / ISO 27001 / HIPAA references. ### Retention Audit records are retained indefinitely by default. If you need a specific retention or archival policy, contact support. ### See also * [Audit trails and compliance](/verify/concepts/audit-trails-and-compliance) * [Configuration reference](/verify/reference/configuration-reference) # Releases Documentation for Aviator Releases, a dashboard that helps engineering teams manage deployments across environments in a central dashboard. ## Introduction Aviator Releases enables product engineers to manage their own releases. It is a unified dashboard that helps teams confidently manage releases, deployments, cherrypicks and rollbacks across multiple services and environments. It integrates with existing CI/CD systems (Jenkins, Buildkite, CircleCI, GitHub Actions, ArgoCD, Spinnaker, FluxCD) and GitHub. {% embed url="" %} Aviator Release Demo {% endembed %} ## Why Releases Deploying software in most organizations is still a brittle process, often managed by a small team of engineers. There is rarely a defined process for handling deployment failures. Rollbacks and cherry-pick processes can be challenging to execute, and are prone to human errors. Product engineers often struggle to track changes—whether they have been deployed to production, rolled back, or pending verification in staging. Modern deployment tools like Spinnaker and ArgoCD mitigate some concerns around automatic rollbacks and early failure detection. However, those tools can be hard for product engineers to use confidently, and lack the understanding of the code changes that are being deployed. Releases helps democratize the process of deployments and release management for all engineers, while providing the right set of access control and guardrails to avoid human errors. ## Who is Releases for Aviator Releases can be relevant at any stage of the company. It’s most useful when you have a frequent but manual deploy process, where you are looking to incorporate better standardization. it can help provide more visibility and control to the product engineering tams, and to improve the engineering practices internally. # Getting Started with Releases How to get started with Aviator Releases Releases. This page guides you to setting up Aviator Releases using GitHub actions step by step. This document guides you to the set up for setting up [Aviator Releases](https://www.aviator.co/releases) using GitHub actions. ## Prerequisite * This guide requires using GitHub Action for your build and deploy workflows. [Look up your CI / CD guide](/releases-beta/how-to-guides/working-with-your-ci-cd) If you are using other tool. * \[For existing MergeQueue users only] If you have set up Aviator for MergeQueue before, make sure you have approved the new permission request for `Actions`, so that Aviator is able to fetch and trigger the GitHub actions for you. ### Separate Build and Deploy Steps To use Aviator Releases, we recommend [separating out the build and deploy process](/releases-beta/concepts/two-step-delivery) if you run them together. This ensures a simpler and meaningful process for rollbacks. You may still use it with a single step, in which case the creation of [Release](/releases-beta/concepts/terminology#release) will be a no-op. ## Set up instructions 1. Set up an Aviator account, and walk through the initial onboarding to connect your GitHub repository. Select Releases as the capability. {% hint style="info" %} If you have trouble connecting the GitHub app, please read the [troubleshooting doc](/manage/faqs/troubleshooting-github-app-connection). {% endhint %} 2. Now, click on [Releases](https://app.aviator.co/releases) tab in the menu and navigate to creating your first project. 3. In this release project, create new deploy environments and specify the workflows to be triggered 1. The dropdown for workflows should auto-populate with your GitHub actions workflows. ### GitHub Actions set up 1. Generate an Aviator API token at 2. Set up environment variables and secrets for your GitHub repo at `Settings > Secrets and Variables > Actions` 1. Add a repository secret `AVIATOR_API_TOKEN` as the API token generated above #### Build workflow 1. While setting up Aviator Releases, we recommend duplicating your existing workflow files in GitHub for a smoother transition. 1. NOTE: The new workflows need to be merged to your default branch in order to take effect 2. Aviator triggers GitHub using [workflow dispatch REST API](https://docs.github.com/en/actions/using-workflows/manually-running-a-workflow#running-a-workflow-using-the-rest-api). This workflow requires specifying the following params in your workflow file: ```yaml on: workflow_dispatch: inputs: aviator_release_cut_id: description: "Database ID of release cut" required: false type: string aviator_release_cut_commit_hash: description: "Commit SHA, branch name, or tag of the HEAD to be built" required: false type: string aviator_release_candidate_version: description: "Name of the version" required: true type: string ``` 1. Add a workflow job at the beginning of the build workflow to sync the workflow run ID with Aviator. This helps Aviator track the CI action that’s running your build. ```yaml steps: - if: inputs.aviator_release_cut_id != '' name: Sync workflow run ID via Aviator API uses: fjogeleit/http-request-action@v1 with: url: 'https://api.aviator.co/api/v1/sync-build-github-action' method: 'POST' bearerToken: ${{ secrets.AVIATOR_API_TOKEN }} data: '{"release_cut_id": "${{ inputs.aviator_release_cut_id }}", "workflow_run_id": "${{ github.run_id }}"}' - name: Checkout the repository uses: actions/checkout@v4 with: # if custom commit_sha is not defined, this should fall back to the head branch ref: "${{ inputs.aviator_release_cut_commit_hash }}" lfs: true submodules: 'recursive' ``` 2. After this you can keep your regular builds steps as is. Make sure to tag the build artifacts with the release version using `${{inputs.version}}` so that you can refer to them in the deployment step. #### Deploy workflow 1. Similar to the Build step, duplicate the Deploy step as well and apply the following parameters in your workflow file. Note that these are different than the build params: ```yaml on: workflow_dispatch: inputs: deployment_id: description: "Database ID of deployment" required: false type: string commit_sha: description: "Commit SHA, branch name, or tag of the HEAD" required: false type: string version: description: "Version to deploy" required: true type: string ``` 1. Add a workflow job at the beginning of the deploy workflow to sync the workflow run ID with Aviator. Note that the API URL is different from the build workflow. ```yaml steps: - if: inputs.deployment_id != '' name: Sync workflow run ID via Aviator API uses: fjogeleit/http-request-action@v1 with: url: '' method: 'POST' bearerToken: ${{ secrets.AVIATOR_API_TOKEN }} data: '{"deployment_id": "${{ inputs.deployment_id }}", "workflow_run_id": "${{ github.run_id }}"}' - name: Checkout the repository uses: actions/checkout@v4 with: # if custom commit_sha is not defined, this should fall back to the head branch ref: "${{ inputs.commit_sha }}" lfs: true submodules: 'recursive' ``` 2. After this you can keep your regular deploys steps as is. Make sure to use the same build artifacts with the release version using `${{inputs.version}}` so that you can refer to them in the deployment step. Now you should be ready to use Aviator Release Management! # Concepts for Releases Learn about core concepts for the Releases dashboard, including terminology, two-step delivery, rollbacks, cherry-picks, dogfood, canary and rollout, and other. # Terminology for Releases Read the terminology for the Releases feature. The common terms are important to understand the Release process with Aviator and use the tool more effectively. ## Release Project Release Project is the top-level entity in Aviator Releases. It refers to software that need to be built and deployed at the same time. Typically, this could be a single service, application, or library, however it could also be defined as multiple services that have to be deployed altogether. Aviator recommend designing the release projects as granular as possible for better manageability. A Release Project can have one more [Environments](#environment), and is tied to a unique CD workflows for each of those environments. When cutting a Release, it would trigger a separate CI workflow to build the artifacts that are then deployed to specific environments using the CD workflow. Each Release Project is tied to 1 GitHub repository, but each GitHub repository can have multiple Release Projects, each of which represent separate [Changelog](#changelog). ## Environment Each [Release Project](#release-project) can have one or more Environments that are deployed and managed. Most common examples of Environments are `staging` and `production`, but depending on different use cases, there may be additional environments you can configure, e.g. `canary`, `dogfood`, `pre-production`, `sandbox`, `development`. There is no limit to the number of Environments for a Release Project. Each Release Project defines its own Environments, and there is no correlation of same Environments across different release projects. ## Changelog Changelog is one of the core utilities of Aviator Releases. The Release dashboard records of all the changes as pull requests in a linear history. This way it’s easy to understand which code changes (PRs) are deployed with which [Release](#release) and on which [Environments](#environment). Each Release project has a it’s own Changelog makes it easier for any engineer to track and understand when their changes were deployed. The Changelog also represents the [Cherry-picks](#cherry-pick), and tie them to the corresponding Release and [Deployments](#deployment). Changelogs help users and developers track the progression of the software, understand what has changed from one version to another, and ensure compatibility with other systems or components. ## Release A Release is a snapshot of a commit SHA from the primary branch that is tested and eventually deployed to various environments. A release contains one or more [Release Candidates](#release-candidate). Each Release is also associated with a unique [Release Version](#release-version). A Release is also associated with a git commit SHA, but this association is mutable. That means, as new Release Candidates are created the the git commit SHA for a Release can change. ## Release Candidate Each Release has one or more Release Candidates represented by `rc1`, `rc2` and so on. A Release Candidate is always tied a unique git commit SHA and is immutable. There is no limit in the number of Release candidates as Release can have. Typically there is only one valid Release Candidate for any Release at a given time. The default Release Candidate for a Release is represented by `rc1`. ## Release version Each [Release](#release) is represented by a unique version that typically follows a pattern. Aviator Releases supports some common release version patterns such as [CalVer](https://calver.org/) and [SemVer](https://semver.org/). But you can also define and maintain your own release version formats. Aviator does not automatically create Git-tags for the release. ## Release Candidate version Release Candidate version is a full string that uniquely identifies both a Release and a corresponding Release Candidate version. A Release Candidate version is created by appending `.rc` to a [Release version](#release-version). For instance, if the Release version is: `2024.06.20` then the first Release Candidate version will be `2024.06.20.rc1`, and second one will be `2024.06.20.rc2` and so on. ## Build To effectively use Aviator Releases we recommend [separating out the build and deployment process](/releases-beta/concepts/two-step-delivery). A build process is where the artifacts are built. A common example of this would be building one or more docker images. To manage the Build process via Aviator Releases, you will have to configure Aviator to trigger your build CI workflow. When triggering this pipeline, Aviator passes the [Release Candidate version](#release-candidate-version) as a parameter along with the git commit SHA to your build CI workflow. This helps uniquely tie the build to a given Release Candidate to generate the artifacts. ## Deployment Deployment is an action of taking existing build artifacts and pushing those to a specific Environment. To manage the Deployments via Aviator Releases, you will have to configure Aviator to trigger your a deployment CI or CD workflow. Similar to the Build step, the Deployment step also passes the Release Candidate version and the git commit SHA to your configured workflow. ## Cherry-pick Cherry-pick allows you to apply any arbitrary commit on top of an existing Release Candidate to create a new Release Candidate. This is also sometimes called hotfixing. This can be useful in scenarios where you don’t want to cut a new release because there are too many commits being introduced since the last release was cut and you want to surgically hotfix a bug. Find more details on the [Cherry-picks](/releases-beta/concepts/cherry-picks) page. # Two-step delivery Find out how two-step delivery works with Releases. We recommend splitting the delivery in two steps when you use trunk based development and Aviator Releases. In many software companies with cloud based infrastructure, releases and deployments are used interchangeably. When using [trunk based development](https://trunkbaseddevelopment.com/) and Aviator Releases, we recommend splitting the delivery in two steps. {% embed url="" %} In a two-step delivery workflow we create a build, and then deploy that build to an environment, or more appropriately: * **Cutting a release**: Cutting a release is an act of snapshotting your mainline (trunk) by picking a commit SHA, thereby creating a release candidate. In this process, you build the artifacts that will be then used to deploy. The advantage is to build this once and deploy the same artifacts to all environments ensuring that you are testing the same images in all environments. After cutting a release, it may be deployed to various pre-production environments (sandbox, staging, etc) for validation. During this process, any critical bugs identified may also be fixed and cherry-picked to create additional release candidates. * **Deployment**: Once a release candidate is validated in pre-production environments, it is then deployed to production using the same build artifacts. That means, we build once during the process of cutting a release, and then those build artifacts are then deployed to all environments. ## Advantages There are a few advantages of separating builds and deploys: * **Isolation of Concerns**: Build step focuses on code compilation and testing; deploy focuses on moving artifacts to environments. * **Consistency**: Ensures the same artifact is used across environments reducing environmental discrepancies. * **Faster Feedback Loops**: Quick feedback on build issues without waiting for deployment. * **Simplified Rollbacks**: Easier to revert by redeploying previous artifacts. * **Scalability**: Build and deploy processes can be scaled and optimized independently. * **Better Resource Utilization**: Optimize resources separately for build and deployment. ### Aviator Releases Using separate steps helps Aviator streamline the process of creating one-click rollbacks, and ensure that the same builds can verified in an environment before those are released to production. ## Continuous Delivery (CD) The modern CD tools such as [Argo, Flux and Spinnaker](https://www.aviator.co/blog/comparing-flux-cd-argo-cd-and-spinnaker/) enable teams to manage continuous delivery, where each commit is deployed to production as soon as it’s pushed the mainline. This is possible with a comprehensive and reliable test infrastructure when manual verification or intervention is not needed. In the system with Continuous Delivery, splitting release and deployments steps are less important, since manual rollbacks and cherrypicks are rare. For such cases, you can configure Aviator release and deployment in a single step. # Rollbacks Learn about the rollback feature in Releases. Rollbacks enable you to revert to a previous stable version when a newly deployed version has issues. Rollback is a mechanism to revert to a previous stable version when issues are detected in a newly deployed version. Rollbacks can be categorized in 3 different buckets: * **Automatic Rollbacks**: This is system's ability to detect failures or issues in the newly deployed version and revert to the previous stable version without manual intervention. * **Automated Rollbacks**: Some predefined scripts or processes facilitate a rollback, but they may require some level of manual initiation. * **Manual Rollbacks**: The rollback has be performed in an adhoc fashion with making a few configuration changes directly into the system. Aviator enable teams to introduce automatic and automated rollbacks in their workflow by streamlining the release and deploy processes. ## Automated Rollbacks By splitting up the release process in build and deploy steps, automated rollbacks become as simple as rolling forward. Effectively, using Aviator Releases one deploys all versions of the build in the same fashion This makes the process of rollback very transparent eliminating human errors. # Cherry-picks Read about the cherry-picks feature in Releases. Aviator enables teams to cherry-pick the same commit or pull request to multiple releases in one action. Cherry-picks are useful for hot fixes where a bug has been identified in a released version, but the mainline has moved much further along so cutting a new release may require a longer validation cycle. In such cases, a small localized fix can be “cherry-picked” on top of an existing release without cutting a new release from mainline. Although the git CLI supports a simple cherry-pick action, doing manual cherry-picks can be prone to errors. For instance: * accidentally cherrypick on top of a wrong branch * accidentally deploy a wrong branch after cherrypicking * forget to cherrypick to all environments / release candidates * accidentally deploy the next release that does not contain that cherrypick after pushing the cherrypick (so rollback the fix) Aviator Releases standardizes the process of cherry-picking while also enabling teams to cherry-pick the same commit or pull request to multiple releases in a single action. Aviator Releases also ensures that once a cherry-pick is created, that any subsequent releases and deployments cannot be created without that cherry-picked commit avoiding any human errors. # Dogfood, Canary and Rollout Check out deployment strategies that limit the impact in case of a failure during deployment. Dogfood, Canary, and Rollout explained with usage guidelines. These deployment strategies that limit the impact in case of a failure during deployment. ## Dogfood Dogfooding is a strategy to deploy the service to a cluster that is only serving internal traffic. This is a common practice for the products and services that can be tested easily within the company. For instance, if you have a business messaging app, it’s easy to test such an app by dogfooding it internally for communication. If the software serves a very specific type of user base such as a brick and mortar store, or a dentist, dogfooding may not be applicable. ## Canary Canary is a process of deploying the application or service to a small segment of the users to ensure any failures can be quickly diagnosed and resolved without impacting the entire user base. If an issue is identified after Canary deployment, the deployment may be flagged as bad and rolled back. This can be performed via automated or automatic rollbacks. Once the Canary deployment is successful , a full production deployment may be performed automatically after some time or triggered manually. ## Rollouts Rollouts process is somewhat similar to Canary deployment, where the application is first deployed to a small segment of the users. Unlike Canary, rollouts are performed as a percentage of the traffic that is gradually rolled out to all user base if no issue are identified. If an issue is identified, the rollouts are stopped and an automatic or automated rollback can be performed back to the last successful release. ## Applying these strategies with Aviator Releases Each of the above deployment strategies can be configured using separate Environments within a single Release project. These environments are then deployed to those specific clusters depending on the configuration and can then be monitored for any defects before a production environment is deployed. This provides a clear picture of all the environments and their current deployed versions for a simpler release orchestration. {% hint style="info" %} Note: Currently Aviator does not support gradual rollouts as a single environment with parameters. These can be configured as separate environments as a step function rollout. {% endhint %} # Release notes Read the concepts behind release notes from Aviator Releases. Helpful for internal teams, QA, support, for a shared understanding of recent code changes. Defining Release notes or Changelog is a common practice for open-source software or a software that is typically shared as pre-built binaries to external users. But release notes can also be very helpful for internal teams: * Facilitates collaboration between development, QA, support, and other teams by providing a shared understanding of what has been modified. * Helps onboarding and training by providing a clear history of enhancements to the product. * Aids in troubleshooting issues by providing context about recent updates that may have introduced new bugs or resolved existing ones. * Serves as a valuable reference for maintaining internal documentation and user manuals. * Enables internal teams to provide informed feedback on new features and changes, contributing to continuous improvement. * Helps ensure compliance requirements are met by documenting changes systematically. * Assists in future planning and development by reviewing past changes and understanding their impact. * Equips support teams with the necessary information to address customer inquiries about new features or fixes. * Tracks the performance of development teams by documenting deliverables and milestones. ## Compared to GitHub GitHub also natively offers Releases that can generate Release notes. But these release notes can get messy when using a monorepo where multiple micro-services released from a single repository. Since GitHub’s Releases are linear with no separation of micro-services, the release notes are generated as a diff from the last release that may or may not be tied to the same service. Additionally, the default release notes generated from GitHub do not account for different code paths or dependencies to understand what changes are associated with which micro services. Using Aviator Releases, you can manage separate release notes and changelog history for each service even within the same repository. Moreover, using separate release fag formats and declarative code paths, the release notes can be uniquely generated for each service. # How-to Guides for Releases Access all how-to guides for Releases from this page. Get instructions on creating a release project, configuring environments, and other key operations. # How to Create a Release Project Follow our guide for instructions on creating a release project with examples. A Release Project is a top-level entity you must build and deploy simultaneously. A [Release Project](/releases-beta/concepts/terminology#release-project) is a top-level entity that needs to be built an deployed simultaneously.
## Choosing the granularity Typically, even in a large micro-services architecture many services may be deployed together. In such cases, all of those will be tied to a single Release Project. If each service is deployed separately, then each of them should be it’s own Release project. Each Release Project can have multiple environments that it deploys to, in which case all of those environments should share the same Release Project. Aviator recommends making release projects as granular as possible for better manageability. ## Release Configuration Other than the GitHub Repository name, all the Release Project properties are mutable. ### Project Name A readable string representing the name of the project. Do not use any special characters other than `-`, `_` or a space. The project name is only used for reference purposes and is mutable. Each project name within the Aviator workspace should have a unique name. ### Repository Represents the primary GitHub repository containing the project’s source code. This is the repository that Aviator uses to generate the Changelog. When using the GitOps based tool such as ArgoCD or FluxCD, we recommend choosing your application repository as the primary repository. Each Release Project is tied to 1 GitHub repository, but each GitHub repository can have multiple Release Projects, each of which represent separate Changelog. ### Git tag patterns \[Coming soon] Aviator autogenerates and publishes [Git-tags](https://git-scm.com/book/en/v2/Git-Basics-Tagging) to track the Release Candidates. Aviator supports 2 Git tag patterns: [Semver](https://semver.org/) and [Calver](https://calver.org/) to automatically generate versions for new releases. These versions are not enforced and can be overridden. This is an optional field, can be left blank. ### File code path A list of glob patterns that presents the file paths associated with the Release Project. This file follows similar patterns to [gitignore](https://git-scm.com/docs/gitignore#_pattern_format). This is a great way to filter out the changes associated with This is an optional field, can be left blank. When left blank, all code changes in the repository are associated with the Release Project. The field is mutable, but it does not update the Changelog associated with the past release versions. ## Build Configuration If you are using a [two-step delivery](/releases-beta/concepts/two-step-delivery) workflow, the build step should be configured here. A more detailed explanation of each supported CI is explained in the [CI specific pages](/releases-beta/how-to-guides/working-with-your-ci-cd). ### Not configured If you want to skip the build step entirely (one-step delivery workflow), you can choose the build step to be “Not configured”. You can also choose this step if you decide to build an entirely API-driven workflow. ### GitHub Actions

Build step with GitHub

Since GitHub authorization already provides Aviator access to the GitHub workflows, integrating GitHub actions is fairly straight-forward. Simply select the GitHub workflow that should be triggered for creating a build. If you do not see your workflow in the dropdown, click on “Fetch workflows” to async fetch the workflow names. Note: Please read the full [GitHub Actions guide](/releases-beta/how-to-guides/working-with-your-ci-cd/github-actions-workflow) to configure the GitHub workflow parameters and callbacks to ensure your build and deploy workflows can be tracked in Aviator. ### Buildkite

Build step with Buildkite

Aviator uses Buildkite [API access tokens](https://buildkite.com/docs/apis/managing-api-tokens) to trigger the Buildkite workflows. Please read the [Buildkite setup guide](/releases-beta/how-to-guides/working-with-your-ci-cd/buildkite-workflow) to understand how to configure the access tokens with the right permissions. On the Release Project page, configure the Organization slug and Pipeline slug to be able to trigger and track the Buildkite pipelines. Please note that these slugs are case sensitive. No callbacks are necessary to track the workflows in Aviator with Buildkite. ## Editing a Release Project Once a Release Project is created, you can edit all the project configuration other than the GitHub Repository name. None of the changes should impact your existing released versions and changelog. To edit the Release Project, go to the Releases main dashboard, and click on the Gear icon next to the Release Project name.
Once the Release Project is created, next step is to [set up the environments](/releases-beta/how-to-guides/configuring-environments). # How to Configure Environments Read how to configure requirements in Aviator Releases in our how-to guide. Set up a separate environment for anything that can be uniquely deployed. [Environments](/releases-beta/concepts/terminology#environment) in Aviator Releases is tied to a Release Project, and there is no correlation of same Environments across different release projects. Set up a separate environment for anything that can be uniquely deployed. Common examples of envinroments are `production`, `staging`, or `development` but it can vary depending on how you manage your environments. For instance, even [dogfood, canary or rollouts](/releases-beta/concepts/dogfood-canary-and-rollout) should be configured separately. To create an environment, go to the configuration page for the Release Project, and click on “Add Environment” button at the bottom right of the page.

Create Release Environment config

### Environment name The name of the environment as string. Do not use spaces or special characters other than `-` or `_`. Changing this environment name will also update the all the past deployments associated with this environment. ### Tier The tier can be `production`, `staging` or `development`. This is only used for internally grouping purposes. Tiered environments are grouped and can be reviewed on the main dashboard for Releases. This property is also mutable. ### Require PR verification before deploy When enabled, the authors of the PRs are requested to verify their changes (PRs) before those changes can be deployed. If you use a deployment spreadsheet to verify the changes before a deployment, this configuration could easily replace the same with clear audit logs. Learn more about the PR verification workflow. These PRs can be marked as verified within the Aviator dashboard, and the user can bypass the requirement for any time-sensitive deploys. ### Notify owners after deployment success If enabled, the authors of the PRs whose changes are in the current release will get notified on Slack that their changes are deployed to the specified environment. This could be a good way to remind authors to verify their changes. For instance, if you would like the PR verification workflow in `production`, you can notify owners after deployment in `staging` as a reminder to verify the changes. ## Deployment configuration The deployment configuration is similar to the [build configuration](/releases-beta/how-to-guides/creating-a-release-project#build-configuration) in the Release Project. ### Not configured If your CD service is not supported you can use an API driven workflow, you can choose the deployment step to be “Not configured”. In this configuration, Aviator listens to API calls to manage the deployment status. ### GitHub Actions

Deployment with GitHub Actions

Since GitHub authorization already provides Aviator access to the GitHub workflows, no additional authorization is needed. Simply select the GitHub workflow that should be triggered for creating a build. If you do not see your workflow in the dropdown, click on “Fetch workflows” to async fetch the workflow names. Additional workflow parameters can be provided as key-value pairs. These parameters are then passed as is to the GitHub workflow. {% hint style="info" %} Note: Please read the full [GitHub Actions guide](/releases-beta/how-to-guides/working-with-your-ci-cd/github-actions-workflow) to configure the GitHub workflow parameters and callbacks to ensure your build and deploy workflows can be tracked in Aviator. {% endhint %} ### Buildkite

Deployment with Buildkite

Aviator uses Buildkite [API access tokens](https://buildkite.com/docs/apis/managing-api-tokens) to trigger the Buildkite workflows. Please read the [Buildkite setup guide](/releases-beta/how-to-guides/working-with-your-ci-cd/buildkite-workflow) to understand how to configure the access tokens with the right permissions. On the Deployment configuration, set the Organization slug and Pipeline slug that will be used to trigger and track the Buildkite pipelines. Please note that these slugs are case sensitive. No callbacks are necessary to track the workflows in Aviator with Buildkite. # How to Create a Scheduled Release Use our guide for instrutions on how to schedule a release. Set up automated creation of a new version of your software at specific intervals or times. Scheduling a release means setting up an automated process that triggers the creation of a new version of your software at specific times or intervals. Instead of manually deciding when to create a new version (release), you just define a cron expression of the schedule, and Aviator handles it automatically. ## Enable Scheduled Release Cut and Define the Schedule 1. Go to the **Project Config** to enable scheduling for the release project. ![](/files/OxvO4M6n2HHMm60ZJewK) 2. Scroll down to find the **Enable scheduled release cut** option. 3. Toggle the button to enable scheduling if it’s not already enabled. 4. Add a cron expression to specify when the release cuts should occur.\ **Note:** This is a standard Unix cron expression that operates in the timezone specified in the account setting. You can refer to the cron expression format [here](https://crontab.guru/). 5. Save the configuration after adding the cron expression. ![](/files/gojOsVocPIZD6dMPpPdn) ## Verify the Release Cut 1. Ensure that the release cut occurs successfully at the scheduled time. 2. Check the release details to confirm that the release was created as expected. 3. If configured, an automated deployment of an environment will also be triggered after the successful build. ![](/files/LPDyn3nHAQ2h51JZyp4o) ## FAQ ### What happens if there are no new commits since the last release cut version? If there are no new commits since the last release cut version, the release cut will be skipped. This ensures that only meaningful changes trigger a new release, avoiding unnecessary builds and deployments. # Create Custom Workflow Parameters Use our guide to create custom workflow parameters. This setting allows you to manage key-value pairs that can be accessed during software build and deployment. Custom workflow parameters allow you to define and manage key-value pairs that can be accessed during your software build and deployment process. This ensures that your release pipeline can use specific configurations for different environments, such as staging, production, or custom setups. ## Setting Workflow Parameters for Your Build Workflow 1. Go to the Project Config in your Aviator release project. 2. In the build section, fetch and select your workflow. ![](/files/2ptMMUCh5iqgiJmnnNQ8) 3. In the "Additional workflow parameters", click "ADD" button to define a new workflow parameter. 4. Enter the key (name of the variable) and the value. Instead of a custom value, you can also select a pre-defined value from the dropdown. * **Key**: This should be a unique identifier for the variable (e.g., user). * **Value**: This is the specific configuration or secret (e.g., Aviator). 5. After adding all necessary variables, click Save to store the configurations. ![](/files/EYluhJ8n6E16DCk31xXM) ## Setting Workflow Parameters for Your Deploy Workflow 1. Navigate to the **Environment Config** for the environment where you'd like to apply custom workflow parameters (e.g., staging or production). * **Note:** If you haven't created any environment yet, go to your **Project Config** and click on **"Add Environment"** to create one. 2. In the **"Additional workflow parameters"** section, click the **"ADD"** button to define the key and value for this environment. 3. Save the changes to apply the parameters specific to that environment. ![](/files/X9Os6dyZMCmWNt82cSDY) ## Add the Custom Workflow Parameters to your Pipeline 1. Go to your CI/CD workflow (GitHub Actions in this case). 2. Add the variables as inputs on workflow dispatch. ```yaml name: build on: workflow_dispatch: inputs: user: description: "Username" required: false type: string commit_hash: description: "Commit hash" required: false type: string env_name: description: "Environment name" required: false type: string ``` 3. Create a job which prints these inputs. ```yaml jobs: build: runs-on: ubuntu-latest steps: - name: Print the inputs run: | echo "Inputs:" echo "user=${{ inputs.user }}" echo "commit_hash=${{ inputs.commit_hash }}" echo "env_name=${{ inputs.env_name }}" ``` 4. On the release dashboard, cut a release to trigger the workflow. ![](/files/W63CBhXQhS61i3QTtjH8) 5. Verify the logs of the triggered pipeline to get the value of the custom workflow parameters in the workflow. ![](/files/SJt3zCXHTseQfKTpIGPF) # How to Manage Cherry-Picks Visit our guide to get instructions on managing cherry-picks for smooth release. In Aviator, cherry-picks selectively integrate changes into release candidates. Cherry-picking enables you to apply specific commits from one branch to another without merging the entire branch. In Aviator’s release management system, cherry-picks are used to selectively integrate changes into release candidates. This guide will walk you through the process of managing cherry-picks in Aviator, ensuring smooth handling of your release process. ## Prerequisites Before you begin cherry-picking, ensure that your repository is properly connected to Aviator and that a release project has been created. Refer the guide [here](/releases-beta/getting-started) to know more about release projects. 1. Set up your repository and connect it to Aviator. 2. [Create a release project](/releases-beta/how-to-guides/creating-a-release-project) if it does not already exist. ## Cherry-Picking a Pull Request (PR) ### Step 1: Cut a New Release 1. Open Aviator’s Release Dashboard. 2. Cut a release from the listed PRs on the dashboard. ![](/files/8g6W5BhCd8x6m25AkV2f) Note: You can skip this step if you already have an existing release to base the cherry-pick on, as this is the release where you will apply the cherry-pick. ### Step 2: Cherry-Pick the PR 1. Find the specific PR you wish to cherry-pick into your release. 2. Click on the PR to open the detailed view. 3. Click the "Cherry-pick" button in the PR details view. ![](/files/0fzVBfusREN4EzIpRyMU) 4. Click the button to initiate the cherry-pick process. Aviator will apply the changes from the PR into the selected release candidate in the dropdown. It may take a few seconds to apply the cherry-pick. 5. Aviator enables users to apply the same PR to multiple release candidates. You can select the oldest release version, and Aviator will apply the cherry-pick to all versions newer than that. 6. Users can also deselect the versions they don't want to apply cherry-picks to. Aviator will automatically detect which versions already include this PR and skip the cherry-pick for those versions. ![](/files/WbT2lj786dBx6sWO7SBO) During the cherry-pick process, Aviator will display the current status of the operation. It may take a few seconds for the cherry-pick to be applied, and the status will indicate the progress: * **In Progress**: The cherry-pick is still in progress. ![](/files/A9HpnllHYeUZ4uQwp79h) * **Success**: The cherry-pick has been successfully applied, and a new release candidate has been created. ![](/files/pYPCy6nbw7PZTnIYlsVJ) * **Failure**: The cherry-pick failed due to confliects, and Aviator will create a draft PR to resolve the issue. ![](/files/vHzLdsiriA5wXcA1SwZk) If the cherry-pick is successful, a second release candidate of the release will be created. ## Cherry-Pick Failures In some cases, a cherry-pick may fail due to conflicts between the changes in the PR and the current state of the release branch. Aviator will flag such conflicts and provide tools to help resolve them. To learn how to resolve cherry-pick failures, refer to the detailed guide on [Resolving Cherry-Pick Failures](/releases-beta/how-to-guides/resolving-a-cherry-pick-failure). ![](/files/Hy6jET5lV3a3vLqnISrj) # How to Resolve a Cherry-Pick Failure A cherry-picking workflow may fail due to a merge conflict. This document explains how to manage such an issue during a release process in Aviator. Cherry-picking is a common workflow in version control that allows you to selectively integrate specific changes (commits) from one branch into another. However, sometimes a cherry-pick can fail due to a merge conflict. This document will guide you through managing such a scenario during a release process in Aviator. Setup a repository, connect it to Aviator and [create a release project](/releases-beta/how-to-guides/creating-a-release-project) for this guide. ## Create a Base Release 1. Add a new file to your repository. For example, `echo "base" > test.txt` 2. Commit and push the changes to a different branch. 3. Create a pull request to the main branch. 4. Merge the PR into the main branch. ![](/files/z0DHsloS4m9SJDkdlgdi) 5. Cut a release: Create a new release based on this PR. ![](/files/8g6W5BhCd8x6m25AkV2f) ## Raise Two more Pull Requests 1. First PR: `Ensure test.txt` contains "base". 2. Modify the file: `echo "modify1" > conflict-test.txt`. 3. Commit and merge the PR. ![](/files/OOyn0jrcLj017BQaC8cL) 4. Second PR: Confirm that conflict-test.txt shows modify1. 5. Modify the file again: `echo "modify2" > conflict-test.txt`. 6. Commit and merge the PR. ![](/files/c4Y8RKJxIIo8kYRSqEGQ) ### Merge Conflict To create a merge conflict for this tutorial, go to the Aviator Release Dashboard, select the last PR (the one with modify2), and follow these steps: 1. Select the PR: Click on the PR you want to cherry-pick(last PR in this case). 2. Cherry-Pick: Click the "Cherry-Pick" button to integrate the changes into the base release. ![](/files/Hy6jET5lV3a3vLqnISrj) You will see a red tag indicating "Failed to cherry-pick" due to a merge conflict. If you encounter this issue, follow the steps below to resolve it. ## Resolve Merge Conflicts 1. On this failed cherry-pick commit, you should see the conflict resolution PR link. Open the link and follow the instructions to resolve the conflict. ![](/files/NgggCrBj98ifNZFBrKKJ) 2. Follow the steps provided by the Aviator-app bot to fetch the code in your dev environment and resolve the conflict. Alternatively, you can use GitHub UI to resolve the conflicts. The conflicting files will be flagged with diffs showing the necessary resolutions. You can click on each conflict and choose which changes to keep or edit the file directly in the GitHub UI. Ensure you select the changes from the second PR as the final result. ![](/files/pguhpj2F0OyIM1ww809E) 3. After pushing the changes, you should be able to merge the PR. You may optionally also get this reviewed by a coworker for correctness. ![](/files/fy7pFVJnTQL8CiLOozAI) 4. Once PR is merged successfully, go back to the UI and click "Accept Resolution". This step is crucial for the resolution to be recognized and accepted as a new Release Candidate (RC). 5. After accepting the resolution, the Release Candidate (RC) should now reflect the correct version of the file following the conflict resolution. ![](/files/C0IvYEF2GyKEXHwO154y) # Working with your CI / CD Read about the specifics of using Aviator with CI / CD and supported integrations. Aviator Releases works with your existing CI or CD system, not replace one. Aviator Releases works with your existing CI or CD system, not replace one. While in beta, Aviator supports two integrations to manage your build and deploy workflows: GitHub Actions and Buildkite. Alongside, you can also use Aviator Releases with any other CI system using the API based workflow. Still looking for additional CI support? Please email us at . --- [Next Page](/llms-full.txt/1)