Skip to content

Conversation

@cretz
Copy link
Member

@cretz cretz commented Mar 21, 2025

What was changed

Create simple/naive nightly benchmark that uses fibers. This is mostly a sanity test that Ruby performs well with some load, we just happen to run it nightly. No big surprises here. We use the same methodology as Python and .NET nightlies. For example, here is the 10000 workflow bench on the latest Ruby run compared to the last night's Python run and .NET run:

Ruby:

{workflow_count: 10000, max_cached_workflows: 10000, max_concurrent: 10000, max_mem_mib: 1464, start_seconds: 23.889, result_seconds: 117.192, workflows_per_second: 85.33}
{workflow_count: 10000, max_cached_workflows: 10000, max_concurrent: 10000, max_mem_mib: 1465, start_seconds: 23.253, result_seconds: 142.239, workflows_per_second: 70.304}
{workflow_count: 10000, max_cached_workflows: 1000, max_concurrent: 1000, max_mem_mib: 1466, start_seconds: 23.022, result_seconds: 116.619, workflows_per_second: 85.749}
{workflow_count: 10000, max_cached_workflows: 1000, max_concurrent: 1000, max_mem_mib: 1466, start_seconds: 23.059, result_seconds: 122.289, workflows_per_second: 81.774}

Python:

{"workflow_count": 10000, "sandbox": true, "max_cached_workflows": 10000, "max_concurrent": 10000, "max_mem_mib": 79.6, "start_seconds": 29.7, "result_seconds": 123.8, "workflows_per_second": 80.8}
{"workflow_count": 10000, "sandbox": true, "max_cached_workflows": 10000, "max_concurrent": 10000, "max_mem_mib": 79.4, "start_seconds": 29.9, "result_seconds": 136.9, "workflows_per_second": 73.0}
{"workflow_count": 10000, "sandbox": true, "max_cached_workflows": 1000, "max_concurrent": 1000, "max_mem_mib": 80.3, "start_seconds": 29.7, "result_seconds": 133.8, "workflows_per_second": 74.7}
{"workflow_count": 10000, "sandbox": true, "max_cached_workflows": 1000, "max_concurrent": 1000, "max_mem_mib": 79.4, "start_seconds": 30.5, "result_seconds": 128.5, "workflows_per_second": 77.8}

.NET:

{ WorkflowCount = 10000, MaxCachedWorkflows = 10000, MaxConcurrent = 10000, MaxMemoryMib = 515, StartDuration = 00:00:18.4229059, ResultDuration = 00:01:53.2857397, WorkflowsPerSecond = 88.27 }
{ WorkflowCount = 10000, MaxCachedWorkflows = 10000, MaxConcurrent = 10000, MaxMemoryMib = 507, StartDuration = 00:00:16.4261393, ResultDuration = 00:01:37.9173815, WorkflowsPerSecond = 102.13 }
{ WorkflowCount = 10000, MaxCachedWorkflows = 1000, MaxConcurrent = 1000, MaxMemoryMib = 510, StartDuration = 00:00:15.1196929, ResultDuration = 00:01:27.6652203, WorkflowsPerSecond = 114.07 }
{ WorkflowCount = 10000, MaxCachedWorkflows = 1000, MaxConcurrent = 1000, MaxMemoryMib = 518, StartDuration = 00:00:15.2925597, ResultDuration = 00:01:32.4417527, WorkflowsPerSecond = 108.18 }

Can see Ruby is probably a tad-yet-negligibly faster than Python, which is to be expected given the Python sandbox, and both are slower than .NET which is obviously expected. Python has a lot less higher max mem because its GC runs way more frequently than Ruby and .NET (not even sure Ruby's runs at all in this test, but I will confirm there are no mem leaks). Lots of these numbers are variable because they are probably pushing the CLI dev server too hard (lots of RPS warn limit logs). And obviously there is no tuning done on the weak 4 core GH worker or the worker options to get the most, so don't read these as "proper workflow/sec numbers" by any means.

Checklist

  1. Closes [Feature Request] Simple nightly smoke benchmark #233

@cretz cretz force-pushed the nightly-benchmark branch 4 times, most recently from 0f52289 to 5d52b71 Compare March 21, 2025 14:41
@cretz cretz force-pushed the nightly-benchmark branch from 5d52b71 to ad53333 Compare March 21, 2025 15:07
@cretz cretz requested a review from a team March 21, 2025 15:09
@cretz cretz marked this pull request as ready for review March 21, 2025 15:09
@cretz cretz merged commit 6218c1f into main Mar 21, 2025
7 checks passed
@cretz cretz deleted the nightly-benchmark branch March 21, 2025 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Simple nightly smoke benchmark

3 participants