GHES Runners at Enterprise Level support

[axel3rd]

Feature

GHES provides today Runners at enterprise level.

Having some pool usable by organizations (if enabled by orga owner in settings) can be helpful (project teams don't have to manage runners and scalability by them-self).

If it can be supported (as complement off enable_organization_runners parameter), it would be nice 😁.

---

Investigations

Currently if enable_organization_runners is used, the scale-up send as configuration:

{
    "environment": "github-runners-poc",
    "runnerServiceConfig": "--url https://github.company.com/some-org --token AAA[...] --labels ubuntu --runnergroup Default",
    "runnerOwner": "some-org",
    "runnerType": "Org"
}

When new Runner is added manually at Enterprise Level (https://github.company.com/enterprises/[enterprise-name]/settings/actions/runners), the configuration parameters are:

./config.sh --url https://github.company.com/enterprises/my-company-name --token BBB[...]

Even if userdata_template parameter is used with a full custom user-data.sh script where $CONFIG is not used and previous line ~hardcoded:

https://github.com/philips-labs/terraform-aws-github-runner/blob/e28ceab2467b4e068ba7efaa59c75b7f1f3b3111/modules/runners/templates/install-config-runner.sh#L20

https://github.com/philips-labs/terraform-aws-github-runner/blob/e28ceab2467b4e068ba7efaa59c75b7f1f3b3111/modules/runners/templates/install-config-runner.sh#L29

.... It doesn't work due to token validity during time, which provides after ~1 hour of usage:

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication

Http response code: Unauthorized from 'POST https://github.company.com/api/v3/actions/runner-registration'
{"message":"Token expired.","documentation_url":"https://docs.github.com/enterprise/3.2/rest"}
Response status code does not indicate success: 401 (Unauthorized).

---

Problems to solve

1° Parameter name:

A new enable_enterprise_runners, with company-name as value (=> string type), can be added.

It provides a config like:

{
    "environment": "github-runners-poc",
    "runnerServiceConfig": "--https://github.company.com/enterprises/my-company-name --token AAA[...] --labels ubuntu --runnergroup Default",
    "runnerOwner": "my-company-name",
    "runnerType": "Enterprise"
}

2° Retrieve token usable at enterprise level:

Retrieve a token usable for Runner at Enterprise Level (the one given when "New Runner" is clicked in Enterprise Settings) is perhaps not obvious, should be investigated.

[skyzyx]

I spent about 12 hours working through this same deployment, and ran into the exact same issue yesterday, trying to parse the differences between the current documentation and the GHES 3.2 docs.

Also to note: GHES 3.2 doesn't support the workflow_job event (had to fall back to check_run), nor the --ephemeral switch (although that's supposed to be coming in v3.3).

[axel3rd]

I will try to have info/help of GitHub team (version requirements, way to retrieve token, ...)

[npalm]

@jonico did you try to run the runners on enterprise level?

[jonico]

@npalm: I have tested other controllers with enterprise level scope but not your particular one (I have left GitHub and no longer have access to an enterprise account).

[npalm]

@npalm: I have tested other controllers with enterprise level scope but not your particular one (I have left GitHub and no longer have access to an enterprise account).

thx for the update

[axel3rd]

I have tested other controllers with enterprise level scope

@jonico : Off chance, do you remember which ones? (To deep dive the way how the token is created with manage_runners:enterprise scope). Many thanks.

[axel3rd]

NB1: When this issue will be solved, official GitHub documentation (Autoscaling with self-hosted runners > Recommended autoscaling solutions) should be updated 😁.

---

@jonico : Off chance, do you remember which ones?

NB2: actions-runner-controller/actions-runner-controller is probably a good sample 🤔

[jonico]

NB2: actions-runner-controller/actions-runner-controller is probably a good sample 🤔

This is the one where I tested the support.

[npalm]

@axel3rd did you got it all working, should we keep the issue open?

[axel3rd]

@axel3rd did you got it all working, should we keep the issue open?

@npalm no, it does not work. Issue should be kept open IMO.

Hard to find time from my side for the moment to investigate the way to:

• generate a token with manage_runners:enterprise scope
• find a convenient way to implement it from lambda

[npalm]

Are you using a service user? The module is set up to use an GitHub App. So no need to generate user tokens with scopes.

[axel3rd]

@npalm : I should have a try when time ; with service account or "personal" admin token ; to subscribe runner or generate a correct token for that.
Even if GitHub App is no more used for generating token at Enterprise level, the module mechanism to store tokens/credentials (SSM, ...) could be nice to adapt.

[axel3rd]

After find time and a exchange with GitHub support...

Autoscaling with self-hosted runners is only supported on GHES v3.3, so it is preferable to wait this release. (I suppose it is due to workflow_job:queued event support, better than check_run event only available in GHES v3.2.)

But both are supported by this scalability solution => can work 😁.

PoC which is working fine with GHES 3.2 :

• Read and understand create a registration token for an enterprise
• Generate an admin PAT with admin:enterprise permission (the only available in GHES v3.2, perhaps admin:enterprise/manage_runners:enterprise would be better when available in GHES ... v3.3?)
• In the user-data.sh, customize/hard-code the actions-runner ./config.sh call with:

echo "TMP - Get a runner registration token..."
ENTERPRISE=my-company
ADMIN_PAT=ghp_TheCorrectAdminPAT
TOKEN_REGISTRATION=$(curl -fs -X POST -u token:$ADMIN_PAT -H "Accept: application/vnd.github.v3+json" ${ghes_url}/api/v3/enterprises/$ENTERPRISE/actions/runners/registration-token | jq -r ".token")

echo "Configuring GitHub Action Runner..."
sudo -iu $USER_NAME bash -c "cd actions-runner && ./config.sh --unattended --name $INSTANCE_ID --labels ubuntu,ubuntu-latest,ubuntu-20.04 --runnergroup Default --url ${ghes_url}/enterprises/$ENTERPRISE --token $TOKEN_REGISTRATION"

Note: It is just for a test, a sustainable implementation should have new parameters (proposal):

• enable_enterprise_runners: Enable enterprise runner (same way as enable_organization_runners)
• enterprise_admin_pat: Admin PAT for runner token registration
• enterprise_id: Enterprise name, if cannot be retrieved automatically (to check)

With that, the original runner configuration (./config.sh --unattended --name "$instance_id" --work "_work" $${config}) can be reused, with config provided in SSM.