JupyterHub not working #362
Replies: 3 comments
-
|
I notice you're using EESSI as a software stack, it may be related to problems with To get a working Jupyterhub you would need to register your cluster on a domain (I am not sure this is essential but I don't see how else you can have the necessary https). Here's a reasonably recent version of an OpenStack MC cluster that I successfully deployed that may give you some hints: terraform {
required_version = ">= 1.4.0"
}
variable "pool" {
description = "Slurm pool of compute nodes"
default = []
}
# This variable is the cluster private key and provided via Terraform Cloud
# (see https://github.com/ComputeCanada/magic_castle/tree/main/docs#415-eyaml_key-optional)
variable "tfc_eyaml_key" {}
module "openstack" {
source = "./openstack"
config_git_url = "https://github.com/ComputeCanada/puppet-magic_castle.git"
config_version = "14.1.2"
cluster_name = "jetstream2"
domain = "hpc-carpentry.org"
image = "Featured-RockyLinux9"
instances = {
mgmt = { type = "m3.quad", tags = ["puppet", "mgmt", "nfs"], count = 1 }
login = { type = "m3.large", tags = ["login", "public", "proxy"], count = 1 }
# node = { type = "m3.quad", tags = ["node", "pool"], count = 1 }
# Use a custom image once you have taken a snapshot
node = { type = "m3.quad", tags = ["node", "pool"], count = 20 , image = "rocky9_MC_node_snapshot"}
}
# var.pool is managed by Slurm through Terraform REST API.
# To let Slurm manage a type of nodes, add "pool" to its tag list.
# When using Terraform CLI, this parameter is ignored.
# Refer to Magic Castle Documentation - Enable Magic Castle Autoscaling
pool = var.pool
volumes = {
nfs = {
home = { size = 200 }
project = { size = 100 }
scratch = { size = 50 }
}
}
public_keys = [
...
]
hieradata = file("data.yaml")
eyaml_key = base64decode(var.tfc_eyaml_key)
nb_users = 0
# Shared password, randomly chosen if blank
guest_passwd = ""
# OpenStack specific
os_floating_ips = { login1 = "149.165.159.169"}
subnet_id = "00508328-3dcb-462f-9829-ab92ab5573af"
# Set stack
software_stack = "eessi"
}
output "accounts" {
value = module.openstack.accounts
}
output "public_ip" {
value = module.openstack.public_ip
}
output "hostnames" {
value = module.dns.hostnames
}
## Uncomment to register your domain name with CloudFlare
module "dns" {
source = "./dns/cloudflare"
name = module.openstack.cluster_name
domain = module.openstack.domain
public_instances = module.openstack.public_instances
} |
Beta Was this translation helpful? Give feedback.
-
|
Looking at your logs though I see and the 2.8.1 release looks relevant: cmd-ntrf/slurmformspawner@v2.8.0...v2.8.1 I'd suggest to bump your MC version to 14.2.1 ( |
Beta Was this translation helpful? Give feedback.
-
|
Ok, I
Then After 20 mins, JupyterHub seems to be up, and the DNS query gives the right IP (i.e. the MC login node): [admin@mc-login1 ~]$ ps -aux | grep jupyter
jupyter+ 8709 0.4 0.6 268848 101780 ? Ssl 15:47 0:04 /opt/jupyterhub/bin/python3 /opt/jupyterhub/bin/jupyterhub --config /etc/jupyterhub/jupyterhub_config.json
jupyter+ 8798 0.0 0.6 848336 101176 ? Sl 15:47 0:00 traefik --configfile /run/jupyterhub/traefik.toml
jupyter+ 8806 0.2 0.5 180132 85980 ? Ssl 15:47 0:02 /opt/jupyterhub/bin/python -m jupyterhub_announcement --AnnouncementService.config_file=/etc/jupyterhub/announcement_config.json
admin 10142 0.0 0.0 3872 2176 pts/0 S+ 16:03 0:00 grep --color=auto jupyter
[admin@mc-login1 ~]$ dig jupyter.magic-castle-cluster.mydomain.de
...
;; ANSWER SECTION:
jupyter.magic-castle-cluster.mydomain.de. 1 IN A 138.246.1.1
...and if I do a local forward However, nothing runs at the domain I set (i.e. [admin@mc-login1 ~]$ curl -kv https://localhost:8000 # here JupyterHub is reachable
* Trying ::1:8000...
* connect to ::1 port 8000 failed: Connection refused
* Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
...
< HTTP/2 302
< access-control-allow-headers: accept, content-type, authorization
< content-security-policy: frame-ancestors 'none'; report-uri /hub/security/csp-report
< content-type: text/html
< date: Fri, 09 May 2025 16:26:56 GMT
< location: /hub/
< server: TornadoServer/6.4.1
< x-jupyterhub-version: 4.1.6
< content-length: 0
<
* Connection #0 to host localhost left intact
*
[admin@mc-login1 ~]$ curl -kv jupyter.magic-castle-cluster.mydomain.de:8000
* Trying 138.246.235.80:8000...
* connect to 138.246.235.80 port 8000 failed: No route to host
* Failed to connect to jupyter.magic-castle-cluster.mydomain.de port 8000: No route to host
* Closing connection 0
curl: (7) Failed to connect to jupyter.magic-castle-cluster.mydomain.de port 8000: No route to host
[admin@mc-login1 ~]$ sudo cat /etc/jupyterhub/jupyterhub_config.json
{
"JupyterHub": {
"hub_ip": "0.0.0.0",
"ssl_key": "/etc/jupyterhub/ssl/key.pem",
"ssl_cert": "/etc/jupyterhub/ssl/cert.pem",
"cleanup_servers": false,
"shutdown_on_logout": true,
"template_paths": [
"/etc/jupyterhub/templates"
],
"proxy_class": "traefik_file",
"bind_url": "https://127.0.0.1:8000", # shouldn't this have the domain I set in the `main.tf`?
"authenticator_class": "pam",
"spawner_class": "slurmformspawner.SlurmFormSpawner",
"admin_access": false,
"services":
... |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I created a cluster with Magic Custle 14.2.1 on Openstack; slurm seems to work fine, but I don't find any active JupyterHub:
jupyteris not installed, port8080is not active and on443nothing runs.In the logs seems to show that the JupyterHub installation didn't go well:
"journalctl -u puppet | grep -i jupyter" on login node
```bash [admin@mc-login1 ~]$ journalctl -u puppet | grep -i jupyter May 08 16:04:18 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub::Base/File[/opt/uv]/ensure) created May 08 16:04:18 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub::Base/File[/opt/uv/bin]/ensure) created May 08 16:04:19 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub::Base/Archive[jh_install_uv]/ensure) download archive from https://github.com/astral-sh/uv/releases/download/0.4.22/uv-x86_64-unknown-linux-gnu.tar.gz to /tmp/uv and extracted in /opt/uv/bin with cleanup May 08 16:04:22 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub::Base/Exec[jupyterhub_venv]/returns) executed successfully May 08 16:04:22 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Profile::Jupyterhub::Hub/File[/opt/jupyterhub/bin/ipa_create_user.py]/ensure) defined content as '{sha256}efa74f870040d8c0704bc95e01899ebe87f3e11458ae56ef65bc047fdc324bf0' May 08 16:04:22 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Profile::Jupyterhub::Hub/File[/opt/jupyterhub/bin/kinit_wrapper]/ensure) defined content as '{sha256}893eec077eb5909a9093d3caecae415250842b155951ed120f30af8fa0704456' May 08 16:04:22 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Profile::Jupyterhub::Hub::Keytab/File[/opt/jupyterhub/bin/ipa_register_service.py]/ensure) defined content as '{sha256}fb4c531d9a42bc698f5b0dab7ba287163c689010ab6d3b7caee1e6b3ab1350c1' May 08 16:04:45 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Profile::Reverse_proxy/File[/etc/caddy/conf.d/jupyter.conf]/ensure) defined content as '{sha256}c51c445dacc6687971d25a1a05af8a28d5b20215322d3c44c06995e335bfcd25' May 08 16:05:45 mc-login1.int.magic-castle-cluster.calculquebec.cloud groupadd[8225]: group added to /etc/group: name=jupyterhub, GID=2003 May 08 16:05:45 mc-login1.int.magic-castle-cluster.calculquebec.cloud groupadd[8225]: group added to /etc/gshadow: name=jupyterhub May 08 16:05:45 mc-login1.int.magic-castle-cluster.calculquebec.cloud groupadd[8225]: new group: name=jupyterhub, GID=2003 May 08 16:05:45 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/Group[jupyterhub]/ensure) created May 08 16:05:45 mc-login1.int.magic-castle-cluster.calculquebec.cloud useradd[8231]: new user: name=jupyterhub, UID=984, GID=2003, home=/run/jupyterhub, shell=/sbin/nologin, from=none May 08 16:05:45 mc-login1.int.magic-castle-cluster.calculquebec.cloud useradd[8231]: add 'jupyterhub' to group 'jupyterhub' May 08 16:05:45 mc-login1.int.magic-castle-cluster.calculquebec.cloud useradd[8231]: add 'jupyterhub' to shadow group 'jupyterhub' May 08 16:05:45 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/User[jupyterhub]/ensure) created May 08 16:05:48 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/Archive[traefik]/ensure) download archive from https://github.com/traefik/traefik/releases/download/v2.10.4/traefik_v2.10.4_linux_amd64.tar.gz to /opt/puppetlabs/puppet/cache/puppet-archive/traefik_v2.10.4_linux_amd64.tar.gz and extracted in /usr/bin with cleanup May 08 16:05:48 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[jupyterhub.service]/ensure) defined content as '{sha256}209c829d10ec4eaf0ad4fa362de4ca275710d5b2f423d1488d0f463fed1bea17' May 08 16:05:48 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[/etc/sudoers.d/99-jupyterhub-user]/ensure) defined content as '{sha256}bbccd8209423f14724073124d966653e0a8374603139f76d456a6d782373c2b7' May 08 16:05:48 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[jupyterhub-auth]/ensure) defined content as '{sha256}0ccaf0d7d7a85389a6af74e476bcf5ab37e108b197ba2f737e888eacc6b5a2a0' May 08 16:05:48 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[jupyterhub-login]/ensure) defined content as '{sha256}cd5292ef8b059ce2c12d20b415732f7ed68c26deb8f7120840e9633de21590e4' May 08 16:05:48 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[/etc/jupyterhub]/ensure) created May 08 16:05:48 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[/etc/jupyterhub/ssl]/ensure) created May 08 16:05:48 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[/etc/jupyterhub/templates]/ensure) created May 08 16:05:48 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[/run/jupyterhub]/ensure) created May 08 16:05:48 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[/usr/lib/tmpfiles.d/jupyterhub.conf]/ensure) defined content as '{sha256}47efa6524f66accd174c753237b574a1849143423f7c30fe6b1bda3d76bf7e16' May 08 16:05:48 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[/etc/jupyterhub/templates/page.html]/ensure) defined content as '{sha256}10378ff9543a9d1ac3c6e81f80cc88fdcbd9a3ea3ee3a11eef211f03d358f8b4' May 08 16:05:48 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[jupyterhub_config.json]/ensure) defined content as '{sha256}0c6f00443ecdedc29158fe1798902e1a0d164c43aef6f966c43f915b35d9b622' May 08 16:05:49 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[announcement_config.json]/ensure) defined content as '{sha256}a1f6e409f627c179e7e5cd0c8e6e0d7e647ca19ac731e6f34825244fcec3e66d' May 08 16:05:49 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[submit.sh]/ensure) defined content as '{sha256}acce56c4ebd743857faa6e16a204790a07c54a24bd0ecd7bfa4f9e2dc4248ca4' May 08 16:05:49 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[/opt/jupyterhub/hub-requirements.txt]/ensure) defined content as '{sha256}63c0ea663a057ad0b48cc9901aba7bb4b551f6fec82f637449d22855426d01b3' May 08 16:05:57 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/Exec[hub_pip_install]/returns) Using Python 3.12.7 environment at /opt/jupyterhub May 08 16:05:57 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/Exec[hub_pip_install]/returns) × No solution found when resolving dependencies: May 08 16:05:57 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/Exec[hub_pip_install]/returns) ╰─▶ Because there is no version of slurmformspawner==2.8.0 and you require May 08 16:05:57 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/Exec[hub_pip_install]/returns) slurmformspawner==2.8.0, we can conclude that your requirements are May 08 16:05:57 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/Exec[hub_pip_install]/returns) unsatisfiable. May 08 16:05:57 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/Exec[hub_pip_install]) Failed to call refresh: 'uv pip install -r /opt/jupyterhub/hub-requirements.txt' returned 1 instead of one of [0] May 08 16:05:57 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/Exec[hub_pip_install]) 'uv pip install -r /opt/jupyterhub/hub-requirements.txt' returned 1 instead of one of [0] May 08 16:05:58 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/Exec[create_self_signed_sslcert]/returns) executed successfully May 08 16:05:58 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[/etc/jupyterhub/ssl/cert.pem]/mode) mode changed '0640' to '0644' May 08 16:05:58 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[/etc/jupyterhub/ssl/key.pem]/group) group changed 'root' to 'jupyterhub' May 08 16:05:58 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/File[/etc/jupyterhub/ssl/key.pem]/mode) mode changed '0600' to '0640' May 08 16:05:58 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/Service[jupyterhub]) Dependency Exec[hub_pip_install] has failures: true May 08 16:05:58 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Jupyterhub/Service[jupyterhub]) Skipping because of failed dependencies May 08 16:05:58 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Profile::Jupyterhub::Hub/File[/etc/jupyterhub/templates/login.html]/ensure) defined content as '{sha256}d4c07e484f250161498351eed0ae094d18a3b9e9dd06e876fd9ed6d5ebdd2154' May 08 16:06:29 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Profile::Jupyterhub::Hub/Consul::Service[jupyterhub]/File[/etc/consul/service_jupyterhub.json]/ensure) defined content as '{sha256}228c12847befee4403d1ac8d276e6f1f03a3b6f0365100c418cc8900f1e66449' May 08 16:11:18 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Profile::Jupyterhub::Hub::Keytab/Exec[jupyterhub_keytab]/returns) executed successfully May 08 16:11:18 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Profile::Jupyterhub::Hub::Keytab/Exec[jupyterhub_keytab]) Triggered 'refresh' from 1 event May 08 16:11:18 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Profile::Jupyterhub::Hub::Keytab/File[/etc/jupyterhub/jupyterhub.keytab]/group) group changed 'root' to 'jupyterhub' May 08 16:11:18 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[919]: (/Stage[main]/Profile::Jupyterhub::Hub::Keytab/File[/etc/jupyterhub/jupyterhub.keytab]/mode) mode changed '0600' to '0640' May 08 16:27:20 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[9587]: (/Stage[main]/Jupyterhub/Service[jupyterhub]/ensure) ensure changed 'stopped' to 'running' May 08 16:57:17 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[10022]: (/Stage[main]/Jupyterhub/Service[jupyterhub]/ensure) ensure changed 'stopped' to 'running' (corrective) May 08 17:27:17 mc-login1.int.magic-castle-cluster.calculquebec.cloud puppet-agent[10771]: (/Stage[main]/Jupyterhub/Service[jupyterhub]/ensure) ensure changed 'stopped' to 'running' (corrective) ```I paste here the
main.tffile I used to create the cluster (with the "sensitive" data changed WLOG):main.tf
I attach also the complete log of
$ journalctl -u puppetand$ cat /var/log/cloud-init-output.log(in the latter there is an "Invalid cloud-config provided", but i don't understand to what is related and if it's relevant)login_cloud-init-output.log
login_journalctl-puppet.log
Maybe I'm missing something obvious, in that case I apologise in advance.
Beta Was this translation helpful? Give feedback.
All reactions