Skip to content

feat: add configurable timeout for evals (5-10 min) #5865

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 20, 2025

Conversation

roomote[bot]
Copy link

@roomote roomote bot commented Jul 18, 2025

This PR adds a configurable timeout setting to the evals setup (/runs/new) that allows users to change the current 5-minute timeout up to 10 minutes.

Changes Made

Frontend Changes

  • Added timeout field to CreateRun schema with min 5, max 10, default 5 minutes
  • Added timeout slider UI component to /runs/new page with proper labeling
  • Updated form to include timeout in default values and submission

Backend Changes

  • Updated database schema to include timeout column in runs table
  • Created migration to add timeout column with default value of 5 minutes
  • Updated runTask.ts to use configurable timeout from run settings instead of hardcoded EVALS_TIMEOUT
  • Modified createRun action to pass timeout parameter through to backend

UI/UX

  • Slider positioned between concurrency and description fields
  • Shows current value with "min" suffix (e.g., "5 min", "10 min")
  • Default value is 5 minutes (bottom of slider)
  • Maximum value is 10 minutes (top of slider)
  • Follows same design pattern as existing concurrency slider

Testing

  • Form validation ensures timeout is between 5-10 minutes
  • Database migration adds column with proper default value
  • Backend correctly uses the configured timeout value for task execution

Addresses the Slack request to add configurable timeout with slider interface.


Important

Adds configurable timeout (5-10 min) for evals with frontend slider and backend support, updating database schema and tests.

  • Behavior:
    • Adds configurable timeout (5-10 min) to evals setup in createRun function in runs.ts.
    • Timeout slider added to /runs/new page in new-run.tsx.
    • Default timeout is 5 minutes, max is 10 minutes.
  • Backend:
    • Updates createRun function to accept timeout parameter.
    • Modifies runTask.ts to use the configurable timeout instead of hardcoded value.
    • Updates redis.ts to use timeoutSeconds for runner registration.
  • Database:
    • Adds timeout column to runs table in schema.ts and migration file 0001_add_timeout_to_runs.sql.
    • Updates copyRun.spec.ts to include timeout in test cases.
  • UI/UX:
    • Slider positioned between concurrency and description fields, shows current value with "min" suffix.
    • Follows design pattern of existing concurrency slider.
  • Testing:
    • Ensures form validation for timeout range.
    • Verifies database migration and backend usage of timeout value.

This description was created by Ellipsis for 19d0f28. You can customize this summary. It will automatically update as commits are pushed.

- Add timeout field to CreateRun schema with min 5, max 10, default 5
- Add timeout slider UI component to /runs/new page
- Update database schema to include timeout column in runs table
- Create migration to add timeout column with default value of 5
- Update runTask.ts to use configurable timeout from run settings
- Pass timeout parameter through the createRun action
@roomote roomote bot requested review from mrubens, cte and jr as code owners July 18, 2025 01:57
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. enhancement New feature or request UI/UX UI/UX related or focused labels Jul 18, 2025
@hannesrudolph hannesrudolph added the Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. label Jul 18, 2025
@hannesrudolph
Copy link
Collaborator

@roomote-agent can you fix that failing test?

Copy link
Author

roomote bot commented Jul 18, 2025

I can see the failing test - there appears to be a compilation failure. Let me check out the branch and investigate the issue.

@hannesrudolph hannesrudolph moved this from Triage to PR [Needs Prelim Review] in Roo Code Roadmap Jul 18, 2025
Copy link
Author

roomote bot commented Jul 18, 2025

I can see the request to fix the failing test. Let me check out the branch and investigate the current status of the tests.

@hannesrudolph hannesrudolph added PR - Needs Preliminary Review and removed Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. labels Jul 18, 2025
- Added timeout: 5 to both createRun calls in copyRun.spec.ts
- This fixes the test failure caused by the new required timeout field in the runs schema
- The timeout field was added in the configurable timeout feature but the test was not updated
Copy link
Author

roomote bot commented Jul 18, 2025

Fixed the failing test!

Issue identified: The test in packages/evals/src/db/queries/__tests__/copyRun.spec.ts was failing because it was missing the new required timeout field when creating runs.

Solution applied:

  • Added timeout: 5 to both createRun calls in the test file
  • This aligns with the new schema that includes a timeout field with a default value of 5 minutes

Results:

  • All CI checks are now passing ✅
  • The configurable timeout feature is working correctly
  • Test coverage is maintained

The PR is ready for review and merge!

Copy link
Collaborator

@daniel-lxs daniel-lxs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Jul 18, 2025
@daniel-lxs
Copy link
Collaborator

@roomote-agent - There's a critical issue with the Redis key expiration that needs to be fixed.

Problem: The registerRunner function in packages/evals/src/cli/redis.ts:25 uses the hardcoded EVALS_TIMEOUT (5 minutes) for Redis key expiration, but tasks can now run up to 10 minutes with the configurable timeout.

Impact: When users set a timeout longer than 5 minutes, the Redis runner keys will expire before the task completes, potentially causing issues with task tracking.

Solution: Update the registerRunner function to accept and use the configurable timeout:

// In redis.ts
export const registerRunner = async ({ runId, taskId, timeoutSeconds }: { runId: number; taskId: number; timeoutSeconds: number }) => {
  const redis = await redisClient()
  const runnersKey = getRunnersKey(runId)
  await redis.sAdd(runnersKey, `task-${taskId}:${process.env.HOSTNAME ?? process.pid}`)
  await redis.expire(runnersKey, timeoutSeconds)
}
// In runTask.ts line 38
await registerRunner({ runId: run.id, taskId, timeoutSeconds: (run.timeout || 5) * 60 })

This ensures the Redis keys remain valid for the entire duration of the task execution.

Copy link
Author

roomote bot commented Jul 18, 2025

👋 I've seen your comment about the Redis key expiration issue. You're absolutely right - this is a critical problem that needs to be fixed. I'll check out the branch and implement the solution to make the registerRunner function use the configurable timeout instead of the hardcoded EVALS_TIMEOUT. Working on it now!

- Updated registerRunner function to accept timeoutSeconds parameter
- Modified call in runTask.ts to pass configurable timeout instead of hardcoded EVALS_TIMEOUT
- Removed unused EVALS_TIMEOUT import from redis.ts
- Ensures Redis keys remain valid for the entire duration of task execution (up to 10 minutes)
Copy link
Author

roomote bot commented Jul 18, 2025

Redis timeout issue fixed!

Problem resolved: The Redis key expiration issue has been successfully addressed. The function was using the hardcoded (5 minutes) for Redis key expiration, but tasks can now run up to 10 minutes with the configurable timeout.

Changes implemented:

  • Updated function in to accept a parameter
  • Modified the call in to pass the configurable timeout: (run.timeout || 5) * 60
  • Removed unused EVALS_TIMEOUT import to clean up the code

Result: Redis keys now remain valid for the entire duration of task execution, whether it's 5 minutes or the full 10 minutes. All CI checks are passing ✅

The fix ensures that when users set a timeout longer than 5 minutes, the Redis runner keys will not expire prematurely, preventing any issues with task tracking.

Copy link
Collaborator

@daniel-lxs daniel-lxs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@daniel-lxs daniel-lxs moved this from PR [Needs Prelim Review] to PR [Needs Review] in Roo Code Roadmap Jul 19, 2025
@mrubens mrubens merged commit c96b399 into main Jul 20, 2025
14 checks passed
@mrubens mrubens deleted the feature/configurable-evals-timeout branch July 20, 2025 02:20
@github-project-automation github-project-automation bot moved this from PR [Needs Review] to Done in Roo Code Roadmap Jul 20, 2025
@github-project-automation github-project-automation bot moved this from New to Done in Roo Code Roadmap Jul 20, 2025
cte added a commit that referenced this pull request Jul 22, 2025
cte added a commit that referenced this pull request Jul 22, 2025
pwilkin pushed a commit to pwilkin/Roo-Code that referenced this pull request Jul 22, 2025
* feat: add configurable timeout for evals (5-10 min)

- Add timeout field to CreateRun schema with min 5, max 10, default 5
- Add timeout slider UI component to /runs/new page
- Update database schema to include timeout column in runs table
- Create migration to add timeout column with default value of 5
- Update runTask.ts to use configurable timeout from run settings
- Pass timeout parameter through the createRun action

* fix: remove unused EVALS_TIMEOUT import

* fix: add timeout field to createRun calls in copyRun test

- Added timeout: 5 to both createRun calls in copyRun.spec.ts
- This fixes the test failure caused by the new required timeout field in the runs schema
- The timeout field was added in the configurable timeout feature but the test was not updated

* fix: use configurable timeout for Redis key expiration in registerRunner

- Updated registerRunner function to accept timeoutSeconds parameter
- Modified call in runTask.ts to pass configurable timeout instead of hardcoded EVALS_TIMEOUT
- Removed unused EVALS_TIMEOUT import from redis.ts
- Ensures Redis keys remain valid for the entire duration of task execution (up to 10 minutes)

---------

Co-authored-by: Roo Code <[email protected]>
Co-authored-by: hannesrudolph <[email protected]>
pwilkin pushed a commit to pwilkin/Roo-Code that referenced this pull request Jul 22, 2025
@ellipsis-dev ellipsis-dev bot mentioned this pull request Jul 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request lgtm This PR has been approved by a maintainer PR - Needs Review size:M This PR changes 30-99 lines, ignoring generated files. UI/UX UI/UX related or focused
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants