Providing feedback

This guide helps you provide feedback on code generated by Runbooks to ensure high-quality results and iterative improvements.

Slash command /aviator revise

You can review the generated PR and use the /aviator revise slash command to trigger automatic code revisions based on your feedback. There are a few ways this command can be submitted:

As a PR Comment

The command can be posted as a top level PR comment to address all unresolved review comments on the PR. This can be helpful for scenarios where multiple reviewers have left comments.

/aviator revise as a PR comment

If you want to exclude certain comments to be addressed, please mark them as resolved manually. Runbooks agents will ignore any resolved comments in the PR.

Runbooks agents reacts with a 👍 emoji to acknowledge the command.

This comment can also be posted as part of the review summary:

/aviator revise as a review summary

If posted as the review summary, Runbooks agents will accept all the unresolved comments (not just the ones part of this review).

Posting with inline instructions

You can optionally also provide some inline instructions when posting the /aviator revise comment. In such case, the Runbooks agents will also take those instructions into account when revising the PR.

In case there are no unresolved comments, /aviator revise <instructions> can also be used to be provide a single inline instruction to revise the PR.

If there are no unresolved comments and no inline instructions are provided, Runbooks agents will return an error, posted as a comment.

As a review comment

If you want Runbooks agents to only address a single review comment, you can post /aviator revise as a review comment itself. In such case, the Runbooks agents will ignore all the other open review comments in the PR.

/aviator revise as a review comment

Similarly, you can also revise an existing comment using the same strategy by posting /aviator revise as a reply thread on a existing review comment.

/aviator revise as a review thread

If there are multiple replies in a review thread, Runbooks agents will pick all the replies together to address the change requested.

Troubleshooting

I posted /aviator revise but it didn't do anything.

Ensure that the comment was not posted as pending comment. And make sure that it's a Runbooks generated PR. Currently /aviator revise only works on PRs generated by Runbooks.

If there is an emoji reaction, the Runbooks agents might be still processing the request. Check the Runbooks URL posted in the PR description for real-time status.

Make sure no other tasks is being run for that particular Runbook. To avoid conflicts, Runbooks agents only perform one action at a time for a given Runbook.

If the problem persists, please contact us [email protected].

Best Practices

Be Specific and Concrete

❌ Vague: "This doesn't look right."

✅ Specific: "Preserve the existing error handling logic in handleAuth(). Currently, it removes the try-catch block which we need for logging."

Provide Context

❌ Missing context: "Add tests."

✅ With context: "Add unit tests using Jest, following the pattern in tests/auth/oauth.test.ts. Focus on testing the error paths since this code handles sensitive authentication logic."

Reference Existing Code

✅ Good feedback:

"The new UserService class should follow the same pattern as
OrderService in src/services/OrderService.ts, including:
- Constructor dependency injection
- Private helper methods for validation
- Consistent error handling with ServiceError class"

Address One Issue at a Time

When providing feedback on multiple issues, break them into separate, numbered points:

Feedback on Step 4:

1. The database query needs pagination - we have 10M+ users
2. Add an index on the email column for performance
3. Use prepared statements to prevent SQL injection
4. Return a consistent error format matching our API spec

Include Error Messages

When something fails, include the actual error:

"Step 3.2 execution failed with this error:

TypeError: Cannot read property 'map' of undefined at line 45

The issue is that the code assumes `users` is always an array,
but it can be undefined when the API returns an error."

Feedback Examples

Example 1: Using /aviator revise for PR Feedback

Step 1: Review the Generated PR

Runbook executes Step 3.1 and creates PR #456. You review and find several issues.

Step 2: Add Specific Line Comments

[Comment on line 23 in UserService.ts]
This should handle the case where email is null or undefined.
The current code will throw an error.

[Comment on line 45 in UserService.ts]
We need to log this error to our monitoring system using
logger.error() before throwing.

[Comment on line 78 in UserService.test.ts]
Missing test case for when the API returns a 429 rate limit error.

Step 3: Choose Your Revision Strategy

Option A - Process All Comments Together:

[Add as top-level PR comment]
/aviator revise

This will:

• Pick up all three unresolved comments

• Generate code changes addressing each issue

• Push a new commit with all fixes

Option B - Fix Issues One at a Time:

[Reply in the thread for line 23]
/aviator revise

[After reviewing that fix, reply in the thread for line 45]
/aviator revise

[After reviewing that fix, reply in the thread for line 78]
/aviator revise

Option C - Provide Inline Feedback:

/aviator revise Add null check for email field at line 23: if (!email) throw new ValidationError('Email is required')

Step 4: Review and Iterate

After Aviator pushes the changes:

1. Review the new commit

2. If satisfied, resolve the conversation

3. If more changes needed, add another comment and use /aviator revise again

Example 2: Refining Requirements

Initial request: "Migrate from Redux to React Context"

Runbook generates: Basic context migration

Your feedback:

"Good start, but we need to:
1. Keep Redux for server state (API calls)
2. Use Context only for UI state (theme, sidebar)
3. Maintain the same action creator pattern for consistency
4. Add TypeScript types for all context values"

Example 3: Correcting Technical Details

Step 3.1: "Update imports to use ES modules"

Your feedback:

"We're using CommonJS (require/module.exports) because we need
to support Node 14. Please keep the CommonJS syntax and instead
focus on organizing the imports alphabetically and grouping
internal vs external dependencies."

Example 4: Adding Implementation Details

Step 2: "Add error handling to API calls"

Your feedback:

"Please implement error handling following this pattern:

1. Use our custom ApiError class from src/errors/ApiError.ts
2. Catch network errors separately from API errors
3. Log errors using our structured logger (src/utils/logger.ts)
4. Return user-friendly messages from src/constants/errorMessages.ts
5. Preserve the original error in the log for debugging

Example from src/services/UserService.ts lines 45-60."

Example 5: Providing Code Examples

Step 4.2: "Update tests to match new API"

Your feedback:

"Here's the pattern for testing async API calls in our codebase:

```typescript
describe('fetchUserData', () => {
  it('should handle success response', async () => {
    mockApiClient.get.mockResolvedValue({ data: mockUser });
    const result = await fetchUserData(123);
    expect(result).toEqual(mockUser);
  });

  it('should throw ApiError on failure', async () => {
    mockApiClient.get.mockRejectedValue(new Error('Network error'));
    await expect(fetchUserData(123)).rejects.toThrow(ApiError);
  });
});
```

Please follow this pattern for all the new tests."

Quick Reference: /aviator revise Commands

Command	Where to Use	What It Does
/aviator revise	Top-level PR comment	Processes all unresolved comments in the PR
/aviator revise	Comment thread reply	Processes only that specific thread's feedback
/aviator revise <feedback>	Anywhere in PR	Applies the inline feedback immediately

Key Takeaways

✅ Do:

• Use /aviator revise for code-level feedback on PRs

• Add specific line comments before triggering revisions

• Provide context and examples in your review comments

• Iterate: review changes, add more feedback, use /aviator revise again

• Edit runbook steps directly for plan-level changes

❌ Don't:

• Don't wait until everything is wrong to provide feedback

• Don't provide vague feedback like "this doesn't work"

• Don't forget to test changes after they're applied

• Don't skip adding comments before using top-level /aviator revise

Remember: Runbooks learns from your feedback. The more detailed and constructive your input, the better it becomes at understanding your codebase and requirements.