Significantly increased runner queue wait times and intermittent listener restarts after upgrading to ARC v0.14.1

[wesco-prathapmotupalli]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.14.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1.  Deploy `actions-runner-controller` and `gha-runner-scale-set` version v0.14.1 to an AKS cluster.
2.  Configure a runner scale set with:
       `minRunners` set (e.g., 50)
       Moderate to high concurrency workflows targeting the scale set.
3.  Trigger a large number of workflows simultaneously (bursty workload).
4.  Observe the GitHub Actions UI:
       Jobs remain in queued state for 10–15 minutes.
       Scaling beyond the minimum runner count is significantly delayed.
5.  Observe Kubernetes:
       Listener pod restarts or becomes intermittently unavailable during this period.
6.  After the listener stabilizes, runners eventually spin up and jobs start executing.

Describe the bug

A significant delay in runner allocation and scaling was observed after upgrading actions-runner-controller (ARC) and gha-runner-scale-set to v0.14.1.
Workflow jobs remain queued for extended periods (approximately 10–15 minutes) before runners are allocated. This behavior becomes more pronounced when a large number of jobs start concurrently and the scale set reaches the configured minimum runner count (e.g., 50).
Additionally, during these periods, the listener component was intermittently restarting or unavailable, which correlated with delayed scaling activity and difficulty maintaining the minimum number of runners.
This issue appears to be a regression, as the same workload behaved normally prior to the upgrade.

Describe the expected behavior

Runners should be provisioned within seconds to a few minutes after jobs are queued.
Scaling beyond the configured minimum runners should start promptly when demand exceeds capacity.
The minimum runner count should be maintained consistently.
The listener component should remain stable and continuously connected under load.
Overall scaling behavior should be comparable to pre‑v0.14.1 releases under the same workload.

Additional Context

ARC version: `v0.14.1`
   gha-runner-scale-set version: `v0.14.1`
   Kubernetes: AKS
   Runner type: Self‑hosted GitHub Actions scale sets
   Regression observed immediately after upgrade
   Rolling back to a previous ARC version restores normal scaling behavior
   This issue closely mirrors the behavior discussed in:
       <https://github.com/actions/actions-runner-controller/issues/4460>

Controller Logs

NAME                                     READY   STATUS    RESTARTS   AGE
arc-gha-rs-controller-67ccddfd87-6hfxv   1/1     Running   0          2d2h
prod-67d8f47f-listener                   1/1     Running   0          3h58m
prod-large-5677b8c9-listener             1/1     Running   0          2d2h


prod-67d8f47f-listener  restarted

Runner Pod Logs

At the time when workflows are queued for extended periods (10–15 minutes), there are no runner pods available or being created to service the queued jobs.

Observed behavior:
- Jobs remain in `queued` state in GitHub Actions while no new runner pods are scheduled
- Runner pods only start appearing several minutes later, after which jobs begin executing
- This delay occurs especially when many jobs start simultaneously and the scale set has already reached the configured minimum runners

The delayed job execution is directly correlated with the absence or delayed creation of runner pods rather than runner pod failures.


kubectl get pods -n arc-system
NAME                                     READY   STATUS    RESTARTS   AGE
arc-gha-rs-controller-67ccddfd87-rgjg5   1/1     Running   0          2d1h
prod-6cb9f565-listener                   1/1     Running   0          69s
prod-large-7b7c869c-listener             1/1     Running   0          2d1h

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[wesco-prathapmotupalli]

This issue is reproducible in both ARC v0.14.0 and v0.14.1. We are observing frequent listener restarts in the ARC system, as shown below:

kubectl get pods -n arc-system NAME READY STATUS RESTARTS AGE arc-gha-rs-controller-7dc57fd99d-tsjrp 1/1 Running 0 19h prod-67d8f47f-listener 1/1 Running 0 12m prod-large-5677b8c9-listener 1/1 Running 0 19h

[nikola-jokic]

Hey, can you please post the listener log?

[wesco-prathapmotupalli]

@nikola-jokic please check the attached logs

prathap@Prathap github-linux-runners % kubectl get po -n arc-system
NAME READY STATUS RESTARTS AGE
arc-gha-rs-controller-7dc57fd99d-tsjrp 1/1 Running 0 20h
prod-67d8f47f-listener 1/1 Running 0 48m
prod-large-5677b8c9-listener 1/1 Running 0 20h

prathap@Prathap github-linux-runners % kubectl get po -n arc-system
NAME READY STATUS RESTARTS AGE
arc-gha-rs-controller-67ccddfd87-rgjg5 1/1 Running 0 2d2h
prod-6cb9f565-listener 1/1 Running 0 44m
prod-large-7b7c869c-listener 1/1 Running 0 2d2h
prathap@Prathap github-linux-runners %

ncus-controller-logs.txt
eastus-controller-logs.txt

[nikola-jokic]

Hey @wesco-prathapmotupalli,

In order to properly inspect the failures, we need a log of the listener that actually restarted, so we can inspect what caused the restart.

[wescoPete]

@nikola-jokic the listener logs aren't showing us much, but will attach here. See also a screenshot showing several instances of an alert we configured to catch when the listener pod is restarting

prod-67d8f47f-listener.log

[nikola-jokic]

Hey @wesco-prathapmotupalli,

This log is not enough, we do need a log of the listener that exited, which should display something like an error and exiting. Maybe inspect the kubelet log? Without a way to reproduce the issue, or a least have a log that clearly shows why listener is exiting, I don't know how to tackle this issue.

[USBAkimbo]

Hey - I'm also getting a similar issue on AKS on 0.14.1

Environment:

• ARC v0.14.1 (also reproduced on v0.13.1),
• AKS
• Org-level scale set
• updateStrategy: immediate
• minRunners: 0 (same issue with minRunners: 1)

Symptom:

• Runners spin up, register with GitHub (get a RunnerID), then exit with code 0 in 1 second with no output
• Job stays queued indefinitely
• EphemeralRunners cycle through Pending to Completed every 3-4 seconds
• Controller logs show "finished successfully, exitCode: 0" but ARC says it's healthy

Listener state during failure:

• Listener is running and healthy
• Logs show "assigned job": 0, "decision": 1" so it sees a queued job and keeps scaling to 1, but JIT tokens produced by the controller are never picked up by runners

Tested across: v0.14.1 and v0.13.1 - same behaviour on both. Both official ghcr.io/actions/actions-runner:latest and custom runner images reproduce this

I will note that I'm deploying this in Helm using ArgoCD ApplicationSets, however the same issue occurs if I deploy without ArgoCD

EDIT: Looks related to #4307 (comment)

[nikola-jokic]

Hey @USBAkimbo,

Could you post the log of the runner? And for the controller as well. If it is exiting, it might be something unrelated. Please post the logs, I cannot investigate it otherwise.

[USBAkimbo]

@nikola-jokic see my comment on #4307 - I've got a workaround on there

That said, would you like the logs anyway?

[wesco-prathapmotupalli]

Hello @nikola-jokic

We added some resources and limits to the listener pod, but that didn't seem to help. Doesn't look like an OOM error. We did follow the pod logs until a crash and were able to catch the error in the listener pod logs:

2026/04/30 12:46:01 Application returned an error: failed to get message: failed to get next message: request GET https://broker.actions.githubusercontent.com/scalesets/message?lastMessageId=100003879 failed(status="200 OK", github_request_id="D217:2F2E2B:5E57863:6149E1D:69F34F06"): failed to parse message response: failed to decode runner scale set message response: EOF: unknown error

Also note, kubectl describe on the pod isn't giving us anything as the old pod is gone and a new pod is created. We tried changing the restartPolicy on the listener pod to 'OnFailure' to see if this would help, but the pod is still killed and a new one created.

[mcmarkj]

We're seeing the same thing - we need to uninstall and reinstall the runners in order to fix the issue.

time=2026-05-04T10:07:21.219Z level=INFO source=github.com/actions/scaleset@v0.3.0/client.go:971 msg="refreshing token" githubConfigUrl=https://github.com/<REDACTED_ORG>
2026/05/04 10:07:21 [DEBUG] POST https://api.github.com/app/installations/<REDACTED_INSTALLATION_ID>/access_tokens
time=2026-05-04T10:07:21.221Z level=INFO source=github.com/actions/scaleset@v0.3.0/client.go:789 msg="getting access token for GitHub App auth" accessTokenURL=https://api.github.com/app/installations/<REDACTED_INSTALLATION_ID>/access_tokens
time=2026-05-04T10:07:21.444Z level=INFO source=github.com/actions/scaleset@v0.3.0/client.go:748 msg="getting runner registration token" registrationTokenURL=https://api.github.com/orgs/<REDACTED_ORG>/actions/runners/registration-token
2026/05/04 10:07:21 [DEBUG] POST https://api.github.com/orgs/<REDACTED_ORG>/actions/runners/registration-token
time=2026-05-04T10:07:21.671Z level=INFO source=github.com/actions/scaleset@v0.3.0/client.go:841 msg="getting Actions tenant URL and JWT" registrationURL=https://api.github.com/actions/runner-registration
2026/05/04 10:07:21 [DEBUG] POST https://api.github.com/actions/runner-registration
2026/05/04 10:07:21 [DEBUG] POST https://broker.actions.githubusercontent.com/rest/_apis/runtime/runnerscalesets/<REDACTED_SCALESET_ID>/sessions?api-version=6.0-preview
time=2026-05-04T10:07:22.545Z level=INFO source=github.com/actions/actions-runner-controller/cmd/ghalistener/main.go:134 msg="Starting listener"
time=2026-05-04T10:07:22.545Z level=INFO source=github.com/actions/scaleset@v0.3.0/listener/listener.go:160 msg="Handling initial session statistics" component=listener totalAssignedJobs=0
time=2026-05-04T10:07:22.545Z level=INFO source=github.com/actions/actions-runner-controller/cmd/ghalistener/scaler/scaler.go:250 msg="Calculated target runner count" component=worker "assigned job"=0 decision=0 min=0 max=2147483647 currentRunnerCount=0
time=2026-05-04T10:07:22.547Z level=INFO source=github.com/actions/actions-runner-controller/cmd/ghalistener/scaler/scaler.go:197 msg=Compare component=worker original="{\"metadata\":{},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0,\"phase\":\"\"}}" patch="{\"metadata\":{},\"spec\":{\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0,\"phase\":\"\"}}"
time=2026-05-04T10:07:22.547Z level=INFO source=github.com/actions/actions-runner-controller/cmd/ghalistener/scaler/scaler.go:203 msg="Preparing EphemeralRunnerSet update" component=worker json="{\"spec\":{\"patchID\":0,\"replicas\":null}}"
2026/05/04 10:07:22 [DEBUG] DELETE https://broker.actions.githubusercontent.com/rest/_apis/runtime/runnerscalesets/<REDACTED_SCALESET_ID>/sessions/<REDACTED_SESSION_ID>?api-version=6.0-preview
2026/05/04 10:07:22 Application returned an error: handling initial message failed: could not patch ephemeral runner set , patch JSON: {"spec":{"patchID":0,"replicas":null}}, error: ephemeralrunnersets.actions.github.com "<REDACTED_RUNNERSET_NAME>" not found