Self-hosted runners is idle but not picking up jobs

[ethanppl]

Select Topic Area

Bug

Body

Problem

I have 8 self-hosted runners that are configured the same. 4 of them started to stop working yesterday (22 Apr) around 08:44 UTC.

When a workflow run, 4 of the workers are active picking up jobs, the other 4 stays idle as if there is no job. The 4 runners are not offline, they are marked as online, connected, idle, yet not picking up jobs. The expected behaviour is all 8 of them can pick up jobs.

It is not related to any concurrency limit, it's consistently that specific 4 runners not picking up the jobs across different repos in the organization.

What it looks like

The runners page in the organization, 4 active, 4 idle:

But at the same time, there are plenty of checks waiting for runner to pick up:

About the runners

• All the 8 runners are all Linux runners self-hosted in a Hetzner box in x64 architecture.
• All the 8 runners are running the latest actions runner version 2.315.0
• All the 8 runners have the same label self-hosted and in the same runner group Default
• All the 8 runners are added to the same organization as organization runners
• All the 8 runners are set up in the same script.

I couldn't figure out why 4 of them works, why 4 of them suddenly stopped working.

What I have tried to fix it

• Stopping the service and then starting them again: sudo ./svc.sh stop and then sudo ./svc.sh start. This does mark the service as offline and then idle again, but still not picking up jobs
• Restarting the machine: sudo reboot, similarly the service is offline then online but not picking up jobs
• Troubleshooting network connectivity with ./run.sh --check --url <org_url> --pat <my_pat>, all passed

• Checking the runner logs in _diag directory, only this error since it stopped working

  [2024-04-22 14:31:57Z ERR  MessageListener] Catch exception during get next message.
  [2024-04-22 14:31:57Z ERR  MessageListener] System.TimeoutException: The HTTP request timed out after 00:01:00.
   ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
   ---> System.Net.Sockets.SocketException (125): Operation canceled
     --- End of inner exception stack trace ---
     at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
     at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
     at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
     at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
     at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
     at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
     --- End of inner exception stack trace ---
     at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
     at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
     at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
     at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
     at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
     at GitHub.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
     --- End of inner exception stack trace ---
     at GitHub.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
     at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
     at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
     at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
     at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
     at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
     at GitHub.Runner.Listener.MessageListener.GetNextMessageAsync(CancellationToken token)
  [2024-04-22 14:31:57Z ERR  MessageListener] #####################################################
  

  • I found that others facing this error in this discussion
  • But following what others did to disabling ipv6 in the service doesn't work, nothing changed

• Checking the last worker logs in _diag directory, nothing special

• Checking journalctl, no errors logged, as if there is no new job ever requested

  Apr 23 04:37:54 github-runner-1-ubuntu-4gb-ash runsvc.sh[2804]: Starting Runner listener with startup type: service
  Apr 23 04:37:54 github-runner-1-ubuntu-4gb-ash runsvc.sh[2804]: Started listener process, pid: 2811
  Apr 23 04:37:54 github-runner-1-ubuntu-4gb-ash runsvc.sh[2804]: Started running service
  Apr 23 04:37:55 github-runner-1-ubuntu-4gb-ash runsvc.sh[2804]: √ Connected to GitHub
  Apr 23 04:37:55 github-runner-1-ubuntu-4gb-ash runsvc.sh[2804]: Current runner version: '2.315.0'
  Apr 23 04:37:55 github-runner-1-ubuntu-4gb-ash runsvc.sh[2804]: 2024-04-23 04:37:55Z: Listening for Jobs
  

• Checking their versions, 2.315.0, all at the latest version

• Docker is installed and active (sudo systemctl is-active docker.service)

Anyone faced this before? Any clue what else can I check? Would this be a bug in GitHub side? Thanks in advance!

[BbolroC]

I am facing with the same situation (1 dropped out of 5) for 2.315.0. I have not experienced this in the previous versions.

[BbolroC]

I have upgraded a runner to 2.316.0 and downgraded it to 2.314.1 (which had been working quite some time without the issue), but no success. The runner got into an idle state and not picking up jobs some time after it was registered.

Interestingly, the runners for 2.315.0 which have not been affected by the issue are functioning so far.

[ethanppl]

It doesn't look like it is version related. I have runners that work in 2.315.0 and runners that doesn't work in 2.315.0. They have the same setup, but some of them doesn't work anymore.

[BbolroC]

Yeah, I was tryting to deliver your point. Thanks for the confirmation. 😉

[ohookins]

Also having the same problem here, with the Actions Runner Controller. I suspect the API is having issues (the managed runners are behaving similarly poorly) but the status page is not really reflecting reality.

[asutosh23]

I am also facing the same issues with the ARC runners
tried to delete the runner deployment and deploy again
worked in few cases, but not for all

[ohookins]

The issue I was having earlier was due to actions/actions-runner-controller#3450 - reverting to 0.9.0 of ARC fixed the problem and I'm awaiting the next release which incorporates the fix. Meanwhile I have noticed we also have the issue of a runner starting up properly, connecting to the Github API and then not receiving the queued job. So that seems to align more with the topic of this thread.

[shashank1992jain]

I am also having exactly same problem.Please can someone advise?
@ethanppl any workaround?

[ethanppl]

I haven't figured it out yet

[shashank1992jain]

any updates, this issue is blocking a lot of things for us

[Martiix]

Experiencing the same thing, restarting the runner service doesn't change anything, recreating the runner fixes the issue, but it's a hassle if the runners keep going into idle state and have to be recreated often. Still don't know the cause.

[Martiix]

It might be related to the process of bumping the version though, logs from the runner services indicated that the runner version was upgraded to 2.316.0 and then got stuck in "listening for jobs"

[Martiix]

This is the sudo systemctl status output for the runner service of a runner stuck in idle mode:

[BbolroC]

Do you think that the update is triggered even if --disableupdate is passed for registration? I am facing with the same issue with --disableupdate.

[Martiix]

Then my assumption is likely wrong

[shashank1992jain]

I think it was working fine until version 2.304 because there were no issues with that version.Github forces updating the version even if it is not tested properly

[shashank1992jain]

Recreating a runner makes other runners pick up the jobs automatically. This is very difficult to understand

[ethanppl]

2 runners started to pick up jobs now. So instead of 4 out of 8 not working, it improved to 2 out of 8 not working, but I didn't change anything. Probably GitHub is working on something to fix it.

The runners are updated to 2.316.0 but I don't think that matters as 2 of them are still not working.

[BbolroC]

Oh, interesting. some of the idle ones finally woke up?

[ethanppl]

yes, two of them woke up

[Rameeskc]

I think it was working fine until version 2.304 because there were no issues with that version.Github forces updating the version even if it is not tested properly

[jcahigal]

Same error here, after (auto) updating from 2.315 to 2.316. Reinstalling runners (Current runner version: '2.316.0') works and now they are processing jobs again.

[shashank1992jain]

@jcahigal but it doesn't last long, it would again fail to pick up jobs if you run it again

[shashank1992jain]

I mean perform another execution

[gabrielmocanu]

But why don't you try to reconfigure all runners?

[shashank1992jain]

All runners have been reconfigured but the issue still persists

[gabrielmocanu]

Hmm you are right, it's seems the issue still persists for me also

[jcahigal]

weird, the reinstalled runners, in case, are still working properly. I have workflows triggered with a cron (nightly) and others just for events, all are working ok since I've reinstalled the runners

[gabrielmocanu]

Do you guys use these self-hosted runners most of the time?
I have more self-hosted runners and this issue persists only in one that is running every 10 minutes, I am wondering if this can be the cause.

[ohookins]

I have self-hosted runners, deployed with ARC (so they are not running except on demand). It seems random chance whether they connect and pick up a job or sit there idle for a long time.

[Martiix]

I am using over a hundred self-hosted runners, that all run continuously every day more or less, and this has happened to a few of them 2-3 times.

[Martiix]

I have 105 self-hosted runners most running continuously every day. I had two waves of runners going idle, probably related to when they updated. All seems to be in order after the second round of recreating the runners. All runners are currently working as expected.

[gabrielmocanu]

Same for me, now it seems to work.

[ethanppl]

Same, around a few days ago, all runners are working as expected again.

[oprokhorov]

Have this on 2.317.0, sometime around late June our runner have just stopped picking up jobs. The repo is private, all config.sh checks pass, removing and adding the runner again did not help.
upd: re-installed older 2.316.1 but it didn't work, removed the label and referred to it as 'self-hosted' and it picked up the job, re-added the label again and it works fine on. Not sure which exact one of these actions helped, but okay.
upd Aug 14: re-installed the latest version of the runner and it's working fine since. I think my confusion was related to the fact that i tried to refer to runner by it's name, rather than by it's custom tag. But i still have no idea on what exactly broken runner initially.

[garrynewman]

Same issue this morning. Changed the labels and it started working - thanks!

[ethanppl]

The problem occurs again. I tried adding and removing labels, but it didn't fix the issue.

I had been referring to self-hosted runners as self-hosted, so I added a new label test to one of the idle runners. Tried running workflows with the new label test and the job is being picked up. The runner that is labelled as test is still idled.

[kadosh1000]

Removing and re-adding a label did the trick for me. Thanks !!

[kondratovbr]

Experiencing this in March 2025.

[miguel-fernandez-corral]

Me too, using version 2.322.0 on Windows Server 2022. But at least, re-running the runner starts to pick enqueued jobs.

[MartinPavlik]

Same here on multiple repositories

[HnadaMohamed]

same here with 2.322.0

[klonwar]

Still hope GitHub will do something with that...

[ChristopherHX]

This is related to a feature toggle (use run service that replaces the pipelines service), my custom runner regression repository collected these timestamps of it being enabled for me. My custom actions runner has been broken inside these timeframes and I just had to look at the cancelled runs.

• Feb 27. 2025, 21:27 UTC+1 - Feb 28. 2025, 15:39 UTC+1
• Mar 4. 2025, 19:35 UTC+1 - Mar 5. 2025, 14:20 UTC+1
• Mar 6. 2025, 20:20 UTC+1 - Mar 8. 2025 19:40 UTC+1 (shortly after incident has been closed, the feature toogle has been turned off)
• Mar 11. 2025, 20:23 UTC+1 - Now

My experience is the following with the new run service enabled in these time slots

• Ephemeral runners are the only runners that pick all jobs reliable
  • See GitHub Hosted Runner
  • See Actions Runner Controller ARC
  • ./config --ephemeral
  • jit config runners
• Reregistering the runner helps once
  • also a requirement of Ephemeral runners
• Cancelling a job helps
  • I see then directly a in progress indicator, but I need to rerun the job for it to be picked up
  • possible makes the stuck runner get a job
  • Rerunning jobs helps only sometimes
• Queuing a new workflow helps (possible makes the stuck runner get a job)
• Registering/Deleting another runner helps (possible makes the stuck runner get a job)

This discussion happens here as well actions/runner#3609 (comment) onwards.

EDIT This feature is back again, queue behavior is probably not very much different now as during the comments

[bengabp]

What the heck, I was like surely this issue has been resolved since Apr 2024. I have this same issue. Why isnt github replying ?? Dont they have an active community manager or discussion manager ?

[zzzhr1990]

It seems that the issue has occurred again, and even restarting the service doesn't work now.

[ChristopherHX]

FYI Incident confirmed by GitHub https://www.githubstatus.com/incidents/m7vl0x8k3j9c

GitHub Hosted Runners now broken for me as well, GitHub Actions is currently dead for me.

EDIT Resolved and incident closed

[18visions]

My self-hosted runners still aren't working, sadly. They were working prior to the incident, but not after it's resolved. The idea of uninstalling, removing, and re-registering them is enough to make me use a different pipeline for my deployments.

[berkaycit-rotatelab]

is there any solution?

[Cameron-McBroom]

I'm also experiencing the same issue when running a single self hosted runner on ubuntu 24.04. I am running it as systemd service.

I notice that if I have a number of jobs queued they all get picked up and run. As soon as there some idle time then it stops picking up jobs.

I have to restart the systemctl service everytime I want it to start picking up the jobs again

[5priyanka]

I have the same issue. I uninstalled the services, reinstalled them, registered the runner again, and also changed the labels, but no luck. Every time, I need to restart for it to pick up the job. This has been happening for the last two weeks; otherwise, it has worked well for three years. Do we have a solution? I’m not keeping track of who made changes, and I have to restart the service. Thank you.

[giellamo]

It is also happening to me.

We have just one self-hosted repo who is not picking up jobs if I don't restart the service.

[ronistrom]

Having same problem here also

[moritzsur]

Same here... Triggering new workflows sometimes seems to get the old ones out of the queue stage and starts them.

[zacharytyhacz]

Same here, I have to restart the runner action. Github status page claims "all systems operational" but besides here I dont see a submit this as a formal issue. @queenofcorgis can you help us please

@

[naueramant]

Same here on v2.322.0, it is really disrupting. Hope that this will be fixed soon @github

[klonwar]

Abandon hope, all who enter here
Suffer from this since Aug 2024

[neziw]

Same here on 2.322.0, newly created runner, any workaround?

[naueramant]

Same here on 2.322.0, newly created runner, any workaround?

Haven't tried it yet, but maybe one could just restart the runner after each job with the post job hook:

ACTIONS_RUNNER_HOOK_JOB_COMPLETED=pkill -2 Runner.Listener

I know it's not ideal but maybe as a workaround for now..

sources:

• https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/running-scripts-before-or-after-a-job#triggering-the-scripts
• How to gracefully shutdown GitHub action runner? actions/runner#2190

[aorith]

Hey @naueramant

I just tried your workaround and it seems to work fine.
I had to tweak it a little bit:

1. Edit the systemd unit /etc/systemd/system/actions.runner.* to add Restart=always + systemctl daemon-reload
2. Put the kill command in a script and give exec permissions:

#!/bin/sh
/usr/bin/pkill -2 Runner.Listener

3. Add an entry to the .env file:

ACTIONS_RUNNER_HOOK_JOB_COMPLETED=/path/to/restart-runner.sh

I still can't believe that we're forced to use a workaround like this

[naueramant]

Ah nice that it worked! But yeah really horrible 😂

[zacharytyhacz]

I don't force kill the process, I have the service enabled on my systemctl and I just simply restart it. It's annoying to do it for every job though

[kondratovbr]

Mine is running in a Docker container and restarts don't do anything. It picked a few jobs two days ago on its own and now it isn't picking anything up again. Just says Listening for Jobs.

[klonwar]

Hasn't reproduced for me for at least a week now

[neziw]

Same, I assume it has been fixed

[mklauber]

I'm still having issues with this. Using the k8s/helm chart deployment, workers not scaling despite tasks in the queue. Manually assigning a minWorker count results in tasks being picked up and processed.

[iamsrikanthnani]

[FIXED]

It took me about 3 hours to realize the issue. After adding the label as the runner name, the job was picked up automatically, and everything worked as expected.

Issue: Identical Runner Names ❌

When all runner names were the same, jobs remained queued and did not execute.

Example (without unique labels):

- dev: runs-on: [self-hosted, Linux, X64, dev]  
- staging: runs-on: [self-hosted, Linux, X64, staging]  
- production: runs-on: [self-hosted, Linux, X64, production]  

Solution: Unique Labels for Each Runner ✅

After adding unique labels to each runner, jobs were picked up correctly, and everything worked as expected.

Example (with unique labels, same as above):

- dev: runs-on: [self-hosted, Linux, X64, dev]  
- staging: runs-on: [self-hosted, Linux, X64, staging]  
- production: runs-on: [self-hosted, Linux, X64, production]  

Conclusion

If using self-hosted runners, ensure each runner has a distinct label to avoid jobs being stuck in the queue.

[melyux]

I only have one runner.

[zacharytyhacz]

thats the fix? Github with all their money, resources, talent, and teams of developers?

bruh

[iamsrikanthnani]

My runner version is: v2.323.0

[DamienSAVNTEC]

Thanks it worked for me ! I just tagged my runners with their specific names and it worked instantly.

[AbbasAlibhaiER]

This worked for me too! Worked ASAP. Thank you!

[savager]

Hi, I'm still experiencing this. I have only one self hosted runner. Not a lot of helpful information here.

[Sunethan]

I just found that if I change the organization-based runner to the repo-based runner, than it will work. Maybe it's because I am using the enterprise GitHub and somehow I don't really have the access to my self-hosted runner!?

[deribaucourt]

Same here. Configuring the runner as repo-based works, but my organization runner is never being picked-uped and stays 'idle'

[deribaucourt]

I fixed it in my case by allowing to run on public repositories of my organization under the runner's group.

[aestheticdyn]

don't run self-hosted on public repositories, it's a security risk, people can just inject code and run it through pull requests

[leventov]

For whoever scrolled to the bottom: for me, the reason was that I was using runs-on: <my-runner-name>. Changing to runs-on: [self-hosted] has fixed the issue. As per https://github.com/orgs/community/discussions/120813#discussioncomment-9938350.