feat(evals): add behavioral eval for error recovery and self-correction#23361
feat(evals): add behavioral eval for error recovery and self-correction#23361ayazhankadessova wants to merge 1 commit intogoogle-gemini:mainfrom
Conversation
Add error_recovery.eval.ts with 2 test cases that validate the agent ability to detect errors, diagnose issues, apply fixes, and verify corrections: 1. Type error recovery: agent fixes a TypeScript type mismatch and runs tsc to verify the project compiles 2. Test failure recovery: agent identifies an off-by-one bug in source code (not the test), fixes it, and re-runs the test suite Both use controlled error injection and assert on behavioral signals (tool call patterns) rather than content matching. Fixes google-gemini#21990
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the evaluation suite by adding a new behavioral evaluation focused on an agent's error recovery capabilities. It provides structured scenarios where an agent must identify and resolve common programming errors, such as type mismatches and logical bugs, demonstrating its ability to observe, diagnose, fix, and verify solutions. This improves the robustness testing of agents by simulating real-world debugging challenges. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new behavioral evaluation for error recovery, which is a great addition for testing the agent's robustness. The implementation is solid, with two well-defined test cases for type errors and test failures. I've identified two high-severity issues in the test setup: one is a brittle assertion for checking which file was edited, and the other is an incorrect version for the vitest dependency which could cause the test to fail. My review includes suggestions to fix both.
Note: Security Review is unavailable for this PR.
| test: 'vitest run', | ||
| }, | ||
| devDependencies: { | ||
| vitest: '^3.0.0', |
There was a problem hiding this comment.
The specified vitest version ^3.0.0 does not exist. This seems to be a typo and will likely cause the test setup to fail during dependency installation. The latest stable version of vitest is 1.x. I suggest changing this to ^1.0.0.
| vitest: '^3.0.0', | |
| vitest: '^1.0.0', |
| const editedSource = editCalls.some((log) => | ||
| log.toolRequest.args.includes('src/utils.ts'), | ||
| ); |
There was a problem hiding this comment.
This check is a bit brittle as it uses String.prototype.includes() on a JSON string. A more robust approach would be to parse the JSON and check the file_path property directly. This avoids potential false positives if the file path string appears in other arguments (like old_string or new_string) and ensures correctness if the path contains characters that get escaped in JSON.
| const editedSource = editCalls.some((log) => | |
| log.toolRequest.args.includes('src/utils.ts'), | |
| ); | |
| const editedSource = editCalls.some((log) => { | |
| try { | |
| const args = JSON.parse(log.toolRequest.args) as { file_path?: string }; | |
| return args.file_path === 'src/utils.ts'; | |
| } catch { | |
| return false; | |
| } | |
| }); |
References
- The
toolRequest.argsproperty is a JSON string, not an object. It must be parsed usingJSON.parse()before its properties can be accessed.
gemini-cli
bot
added
the
area/platform
|
Hi there! Thank you for your interest in contributing to Gemini CLI. To ensure we maintain high code quality and focus on our prioritized roadmap, we have updated our contribution policy (see Discussion #17383). We only guarantee review and consideration of pull requests for issues that are explicitly labeled as 'help wanted'. All other community pull requests are subject to closure after 14 days if they do not align with our current focus areas. For this reason, we strongly recommend that contributors only submit pull requests against issues explicitly labeled as 'help-wanted'. This pull request is being closed as it has been open for 14 days without a 'help wanted' designation. We encourage you to find and contribute to existing 'help wanted' issues in our backlog! Thank you for your understanding and for being part of our community! |
1 participant
Summary
Add a new behavioral eval (
error_recovery.eval.ts) that tests the agent's ability to recover from errors through the observe-diagnose-fix-verify loop.Test cases
1. Type error recovery (
USUALLY_PASSES)tsc/build to verify, and the type error is resolved2. Test failure recovery (
USUALLY_PASSES)>=instead of>) that causes test failureDesign decisions
validation_fidelity.eval.tsfor similar multi-step tasksUSUALLY_PASSESpolicy: Appropriate for behavioral evals with inherent LLM varianceFixes #21990
Test plan
validation_fidelity.eval.tsevalTest,EDIT_TOOL_NAMES, and tool log assertions consistent with codebase conventionsUSUALLY_PASSESper eval guidelines