add service account with allow-app-sharing-role permissions #2917

Adam-D-Lewis · 2025-01-21T00:00:51Z

Reference Issues or PRs

In order for startup apps to be used by nebari, we need to create a service account with appropriate permissions to create the apps and share them with others. One issue this caused was the auth state for the service account doesn't get populated and it is needed in our custom Spawner code. This PR updates the service account auth_state during the pre-spawn-hook.

What does this implement/fix?

Put a x in the boxes that apply

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds a feature)
Breaking change (fix or feature that would cause existing features not to work as expected)
Documentation Update
Code style update (formatting, renaming)
Refactoring (no functional changes, no API changes)
Build related changes
Other (please describe):

Testing

Did you test the pull request locally?
Did you add new tests?

How to test this PR?

Do a nebari deployment with the following config defined in the nebari config file.

jhub_apps:
  enabled: true
  overrides: {
    startup_apps: [
      {
        "username": "service-account-jupyterhub",  # app will be created by this user
        "servername": "my-startup-server",  # specify a unique server name
        "user_options": {
            "display_name": "My Startup Server",
            "description": "description",
            "thumbnail": "data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mNkYPhfz0AEYBxVSF+FAP5FDvcfRYWgAAAAAElFTkSuQmCC",  # base64 encoded image data to use for thumbnail
            "filepath": "panel_basic.py",  # local file or path within git repo
            "framework": "panel",
            "public": False,  # Whether or not app is publicly accessible without authentication
            "keep_alive": False,  # Whether or not to shut down app after a period of idleness
            "env": {"MY_ENV_VAR": "MY_VALUE"},
            "repository": {"url": "https://github.com/nebari-dev/jhub-apps-from-git-repo-example.git"},  # specify if pulling app from git repo
            "conda_env": "global-panelenv",
            "profile": "small-instance",
            "share_with": {
              "users": [], 
              "groups": ["/admin"]
            },
        },
      },
    ]
  }

Then create this conda env after deployment in the global namespace

name: panelenv
channels:
  - conda-forge
dependencies:
  - panel
  - jhsingle-native-proxy
  - bokeh-root-cmd
  - ipykernel
variables: {}

Then create an admin user and give the user the jupyterhub client's "allow-app-sharing-role" role. Log in and make sure you can see my-startup-server listed as a shared app. Then open it and ensure it opens correctly.

Any other comments?

...i/stages/kubernetes_services/template/modules/kubernetes/services/keycloak-client/outputs.tf

Adam-D-Lewis · 2025-01-22T15:56:39Z

I'm seeing some issues right now. The issue is the "jhub-apps-sa" user which creates the startup_apps needs to have logged in before the app server can be started successfully. In the current design, the "jhub-apps-sa" user is meant to act more like a service account and not log in interactively as users do. The issues I've seen so far are in the code that is run by the Spawner to set preferred username and render profiles, but it's possible there are more. In those instances, the auth state for the "jhub-apps-sa" user is None before initial log in so an error is thrown once we try to access info in the auth_state object in those methods.

Some possible ideas on how to fix this:

if the auth_state is None, then go ask keycloak for the needed info? (login on behalf of the user maybe or some other way?)
log the "jhub-apps-sa" user in somehow on startup app creation?
I could add startup_app: True to the user options dict used by jhub_apps. I could then modify our problematic spawner code which requires auth_state to only run if the server starting up isn't a startup app.

@krassowski any thoughts on how best to do this?

krassowski · 2025-01-22T22:06:12Z

This might be too radical and not the kind of suggestion you are asking for, but could it be solved on the jhub-apps level? I mean jhub-apps is meant to be auth-provider agnostic so it should be possible to make this work without touching keycloak at all? This might be just my PTSD from figuring out keycloak piping last time speaking. I recall @aktech also had pleasure to work on keycloak integration - he may have better ideas.

krassowski · 2025-01-22T22:12:32Z

Some possible ideas on how to fix this:

if the auth_state is None, then go ask keycloak for the needed info? (login on behalf of the user maybe or some other way?)

log the "jhub-apps-sa" user in somehow on startup app creation?

I could add startup_app: True to the user options dict used by jhub_apps. I could then modify our problematic spawner code which requires auth_state to only run if the server starting up isn't a startup app.

If staying with this approach I would probably try the solutions in that exact order. I am not sure if the last one will work if you need to do anything beyond the configuration being set (i.e. whether server will actually spawn if you do not have auth_state).

aktech · 2025-01-24T18:30:02Z

This might be too radical and not the kind of suggestion you are asking for, but could it be solved on the jhub-apps level? I mean jhub-apps is meant to be auth-provider agnostic so it should be possible to make this work without touching keycloak at all?

That would be ideal, indeed. The way we are using the spawner here, expects the user to be logged in (hence populating auth_state) once before creating a server, which makes sense from jupyterhub pov as the servers are created by humans instead of robots.

This (startup apps) works in jhub-apps (without nebari) by default unless the spawner needs auth_state (which is the case here). Since its a feature of jhub-apps to provide the ability to create init apps on startup regardless of Authenticator or spawner, this should be handled in jhub-apps if possible (this is a big if though), you might need to pass in some kind of keycloak auth details in the JApps config, that might make it possible.

We can try to see if we can somehow call the authenticate method of the Authenticator in jhub-apps for the user, to populate the auth state before starting startup apps, but yes this might not be possible at all, in that case what you suggested in this comment #2917 (comment), sounds like a good approach, and I agree with @krassowski the last one won't work, as you need groups info to create nfs mounts.

Also, another thing to note here, is supporting this in jhub-apps might be tricky (if its possible), its probably ok to just go for your approach and we can tackle it in jhub-apps later, if time is of essence here.

Adam-D-Lewis · 2025-01-27T17:10:50Z

I think the ideal solution is to create the startup apps up using jupyterhub-service-account which is service account associated with the jupyterhub Keycloak client. We already do the authenticatation needed in Nebari's KeycloakOAuthenticator. We just need to set auth_state for that service account after authentication. I'm not clear on how to set auth state for jupyterhub-service-account, but I'll dig in to the jupyterhub and jupyterhub/OAuthentication code further.

import json
import urllib.parse
import requests
import jwt

# def get_token():
client_id = "jupyterhub"
client_secret = "<my-secret>"
token_url = "https://github-actions.nebari.dev/auth/realms/nebari/protocol/openid-connect/token"

body = urllib.parse.urlencode({
    "client_id": client_id,
    "client_secret": client_secret,
    "grant_type": "client_credentials",
})

headers = {
    'Content-Type': 'application/x-www-form-urlencoded'
}

response = requests.post(token_url, data=body, headers=headers, verify=False)
data = response.json()

# Get the token
token = data["access_token"]

# Decode and print token contents
decoded = jwt.decode(token, options={"verify_signature": False})
print(json.dumps(decoded, indent=2))

yields

{
  "exp": 1737998505,
  "iat": 1737998205,
  "jti": "9827b1e3-9612-4341-8076-22e99d2ccb04",
  "iss": "https://github-actions.nebari.dev/auth/realms/nebari",
  "aud": [
    "realm-management",
    "grafana",
    "argo-server-sso",
    "conda_store",
    "account"
  ],
  "sub": "c1da7cbd-3150-42ae-8b19-c9b4304f2054",
  "typ": "Bearer",
  "azp": "jupyterhub",
  "acr": "1",
  "realm_access": {
    "roles": [
      "offline_access",
      "default-roles-nebari",
      "uma_authorization"
    ]
  },
  "resource_access": {
    "realm-management": {
      "roles": [
        "view-realm",
        "view-users",
        "view-clients",
        "query-clients",
        "query-groups",
        "query-users"
      ]
    },
    "jupyterhub": {
      "roles": [
        "allow-read-access-to-services-role",
        "jupyterhub_developer",
        "allow-group-directory-creation-role"
      ]
    },
    "grafana": {
      "roles": [
        "grafana_viewer"
      ]
    },
    "argo-server-sso": {
      "roles": [
        "argo-viewer"
      ]
    },
    "conda_store": {
      "roles": [
        "conda_store_developer"
      ]
    },
    "account": {
      "roles": [
        "manage-account",
        "manage-account-links",
        "view-profile"
      ]
    }
  },
  "scope": "email profile",
  "clientHost": "10.244.0.1",
  "email_verified": false,
  "clientId": "jupyterhub",
  "roles": [
    "view-realm",
    "view-users",
    "view-clients",
    "query-clients",
    "query-groups",
    "query-users",
    "allow-read-access-to-services-role",
    "jupyterhub_developer",
    "allow-group-directory-creation-role",
    "grafana_viewer",
    "argo-viewer",
    "conda_store_developer",
    "manage-account",
    "manage-account-links",
    "view-profile"
  ],
  "groups": [
    "/analyst",
    "/users"
  ],
  "preferred_username": "service-account-jupyterhub",
  "clientAddress": "10.244.0.1"
}

which has the group membership for the service account, preferred_username, and permissions similar to a normal user.

Adam-D-Lewis · 2025-01-28T00:40:54Z

~~I haven't been able to successfully set auth state for the service account,~~ (Update: resolved by commit 110b0ee (next commit)) but just looking up the service account's info during the spawner code in this commit seems to work.

Adam-D-Lewis · 2025-01-28T20:55:40Z

...bari/stages/kubernetes_services/template/modules/kubernetes/services/keycloak-client/main.tf

-data "keycloak_role" "main-service" {
-  for_each = toset(var.service-account-roles)
+# Get client data for each service account client
+data "keycloak_openid_client" "service_clients" {


Before we only allowed service accounts to get roles from the realm-management client. This PR allows us to set roles by any client. This functionality was needed to be able to set the allow-app-sharing-role on the jupyterhub service account.

Adam-D-Lewis · 2025-01-28T20:58:57Z

...etes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/02-spawner.py

-@gen.coroutine
-def get_username_hook(spawner):
-    auth_state = yield spawner.user.get_auth_state()
+async def get_username_hook(spawner):


flyby: tornado coroutine -> native coroutine. We don't need to use a tornado coroutine.

...etes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/02-spawner.py

...tes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/03-profiles.py

This reverts commit 556661f.

This reverts commit 8cb0e63.

This reverts commit 2fb4fa8.

aktech

Looks good overall, I haven't had the chance to test it locally due to M1 issues, but will try to sort that out tomorrow.

How do you feel about adding a test here verifying the service account has correct roles?

...ernetes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/04-auth.py

aktech · 2025-01-30T20:03:13Z

...ernetes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/04-auth.py

+        token_info = await self._get_token_info()
+
+        # Get user info using the access token
+        user_info = await self.token_to_user(token_info)
+
+        # Get/set username
+        username = self.user_info_to_username(user_info)
+        username = self.normalize_username(username)
+
+        # Build auth model similar to OAuth flow
+        auth_model = {
+            "name": username,
+            "admin": True if username in self.admin_users else None,
+            "auth_state": self.build_auth_state_dict(token_info, user_info),
+        }
+
+        auth_model = await self.update_auth_model(auth_model)


We should add a note here and link to the JupyterHub code for posterity, incase something changes in JupyterHub, we can catch-up with that.

https://github.com/jupyterhub/oauthenticator/blob/d31bb193e84e7cda58b16f2f5d385c9b8affda4f/oauthenticator/oauth2.py#L1436

...ernetes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/04-auth.py

Adam-D-Lewis · 2025-02-05T18:36:27Z

I was originally planning on adding a test once I had a ragna distribution running that would test this, but I'll add a test for just this part (startup_apps).

Adam-D-Lewis · 2025-02-10T18:06:33Z

Looks good overall, I haven't had the chance to test it locally due to M1 issues, but will try to sort that out tomorrow.

How do you feel about adding a test here verifying the service account has correct roles?

@aktech, I added this test now in 80456c5, but it failed since test-user is not an admin. The test uses his jupyterhub token so I make the test-user an admin and it now works as expected.

aktech · 2025-02-11T14:44:58Z

Looks like its still failing: https://github.com/nebari-dev/nebari/actions/runs/13252428164/job/36993020527#step:13:220 (?)

Adam-D-Lewis · 2025-02-11T16:40:55Z

Looks like its still failing: nebari-dev/nebari/actions/runs/13252428164/job/36993020527#step:13:220 (?)

Thanks for pointing that out, @aktech. Yeah, this won't pass until we do a jhub_apps release and include the latest in the jupyterhub image used by nebari. I opened an issue about the jhub-apps release. I'll convert to draft for now.

Adam-D-Lewis · 2025-02-11T17:42:31Z

Amit reminded me that the startup app expects a global-mypanel env which we aren't creating and I'll need to try to switch it over to the nebari-git/dashboards one or build the global-mypanel env if we want to actually try to open the startup server.

viniciusdc · 2025-02-25T17:58:16Z

Amit reminded me that the startup app expects a global-mypanel env which we aren't creating and I'll need to try to switch it over to the nebari-git/dashboards one or build the global-mypanel env if we want to actually try to open the startup server.

I am curious on how that will work, let me know if you need any help.

Adam-D-Lewis · 2025-02-25T18:20:07Z

Amit reminded me that the startup app expects a global-mypanel env which we aren't creating and I'll need to try to switch it over to the nebari-git/dashboards one or build the global-mypanel env if we want to actually try to open the startup server.

I am curious on how that will work, let me know if you need any help.

According to @aktech, we would need to add:

jhsingle-native-proxy
bokeh-root-cmd
to the dashboard environment to match the app requirements.

so it should be straight forward.

Adam-D-Lewis added 4 commits January 20, 2025 17:58

add jhub apps service account with admin permissions

6bc13de

Merge branch 'main' into jhub_apps_user

a2e1620

reduce permissions

234baa2

cleanup

d609271

Adam-D-Lewis commented Jan 21, 2025

View reviewed changes

...i/stages/kubernetes_services/template/modules/kubernetes/services/keycloak-client/outputs.tf Show resolved Hide resolved

Adam-D-Lewis added 3 commits January 21, 2025 16:40

consolidate calls

01d1d5d

revert to non service account user for jhub apps startup apps

1bfe644

cleanup

a4943bb

Adam-D-Lewis added 3 commits January 27, 2025 17:51

hacky, but works

5f9834a

add role to service account + cleanup

7e6204a

try to set service account auth state, but I don't think it's working

2a3e49b

fix bug and set auth state for service account

110b0ee

Adam-D-Lewis changed the title ~~add jhub apps service account with admin permissions~~ add jhub apps service account with create-share-apps permissions Jan 28, 2025

Adam-D-Lewis changed the title ~~add jhub apps service account with create-share-apps permissions~~ add service account with allow-app-sharing-role permissions Jan 28, 2025

Adam-D-Lewis commented Jan 28, 2025

View reviewed changes

cleanup

a0f4efe

Adam-D-Lewis commented Jan 28, 2025

View reviewed changes

...etes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/02-spawner.py Outdated Show resolved Hide resolved

cleanup

f180f07

Adam-D-Lewis commented Jan 28, 2025

View reviewed changes

...tes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/03-profiles.py Outdated Show resolved Hide resolved

Adam-D-Lewis added 4 commits January 28, 2025 15:29

cleanup

6406e82

make service account name a variable

325a601

rename id to uuid for clarity

64d3e0b

remove unneeded code

cb775e0

Adam-D-Lewis added 5 commits January 29, 2025 16:10

ensure binary raw string

8cb0e63

strip all ansi formatting sequences

556661f

Revert "strip all ansi formatting sequences"

7e5c2b0

This reverts commit 556661f.

Revert "ensure binary raw string"

37bd636

This reverts commit 8cb0e63.

Revert "fix buffer full deadlock"

b6e75de

This reverts commit 2fb4fa8.

aktech requested changes Jan 30, 2025

View reviewed changes

Adam-D-Lewis added 3 commits February 3, 2025 10:10

fix fstring

1fce666

add comment with jupyter/oauth code we are mimicking

865c8d6

add keycloak service account name format comment

fad0155

Adam-D-Lewis requested a review from aktech February 3, 2025 16:27

merge with main

8569ee8

Adam-D-Lewis requested a review from a team as a code owner February 10, 2025 17:51

test that jupyterhub service account gets needed roles

80456c5

Adam-D-Lewis added 8 commits February 10, 2025 12:32

add a startup app to ci deployment

627c4aa

assert startup server is created

6de7c1d

fix test_startup_apps_created test

48eae29

remove breakpoint

fbaec09

refactor keycloak command cli

708f753

make test-user an admin

e7da0aa

fix test ids

de43a81

update tests since test-user is now an admin

9810fdb

Adam-D-Lewis marked this pull request as draft February 11, 2025 17:01

soapy1 mentioned this pull request Feb 21, 2025

Add workflow to run conda-store user journey tests #2895

Merged

8 tasks

add service account with allow-app-sharing-role permissions #2917

Are you sure you want to change the base?

add service account with allow-app-sharing-role permissions #2917

Uh oh!

Conversation

Adam-D-Lewis commented Jan 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues or PRs

What does this implement/fix?

Testing

How to test this PR?

Any other comments?

Uh oh!

Uh oh!

Adam-D-Lewis commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krassowski commented Jan 22, 2025

Uh oh!

krassowski commented Jan 22, 2025

Uh oh!

aktech commented Jan 24, 2025

Uh oh!

Adam-D-Lewis commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Adam-D-Lewis commented Jan 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Adam-D-Lewis Jan 28, 2025

Choose a reason for hiding this comment

Uh oh!

Adam-D-Lewis Jan 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aktech left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aktech Jan 30, 2025

Choose a reason for hiding this comment

Uh oh!

Adam-D-Lewis Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Adam-D-Lewis commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Adam-D-Lewis commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aktech commented Feb 11, 2025

Uh oh!

Adam-D-Lewis commented Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Adam-D-Lewis commented Feb 11, 2025

Uh oh!

viniciusdc commented Feb 25, 2025

Uh oh!

Adam-D-Lewis commented Feb 25, 2025

Uh oh!

Uh oh!

Adam-D-Lewis commented Jan 21, 2025 •

edited

Loading

Adam-D-Lewis commented Jan 22, 2025 •

edited

Loading

Adam-D-Lewis commented Jan 27, 2025 •

edited

Loading

Adam-D-Lewis commented Jan 28, 2025 •

edited

Loading

Adam-D-Lewis commented Feb 5, 2025 •

edited

Loading

Adam-D-Lewis commented Feb 10, 2025 •

edited

Loading

Adam-D-Lewis commented Feb 11, 2025 •

edited

Loading