Skip to content

tools: migrate to Toolforge buildservice#564

Open
lgelauff wants to merge 5 commits into
masterfrom
tools/buildservice
Open

tools: migrate to Toolforge buildservice#564
lgelauff wants to merge 5 commits into
masterfrom
tools/buildservice

Conversation

@lgelauff
Copy link
Copy Markdown
Collaborator

@lgelauff lgelauff commented May 4, 2026

Summary

Migrates Montage from toolforge webservice python3.13 to the Toolforge buildservice (Cloud Native Buildpacks). Adds Procfile, .python-version, and a root-level package.json shim so the Python and Node buildpacks both activate — baking the Vue frontend into the container image and eliminating the separate node20 frontend-build job.

Background and design rationale: #563.

Also incorporates the Python 3.13 dependency fixes from #512 (cffi 1.17.1, setuptools<82 pin).

What changed

  • Procfile: defines the gunicorn web process; binds to $PORT (Toolforge injects PORT=8000)
  • .python-version: pins Python 3.13 for the buildpack; removed from .gitignore
  • package.json (repo root): shim to trigger Node buildpack; build script delegates to frontend/
  • montage/utils.py: get_env_name() checks MONTAGE_ENV env var first (LDAP unavailable in containers); load_env_config() reads secrets from individual MONTAGE_* env vars when MONTAGE_OAUTH_CONSUMER_TOKEN is set, falls back to YAML for local dev
  • app.py: removes urllib3.contrib.pyopenssl shim (deprecated since urllib3 2.0, unnecessary on Python 3.13)
  • requirements.in: adds gunicorn; removes pyopenssl; unpins cryptography
  • requirements.txt: cffi 1.16.0 → 1.17.1 (adds cp313 wheels, no source compilation needed); setuptools pinned <82 (setuptools 82 removed pkg_resources, needed by python-graph-core — tracked in Automate setuptools installation #421)
  • dockerfile: Python 3.9 → 3.13 to match deployment; adds gcc libffi-dev build deps
  • deployment.md: rewritten for the buildservice workflow

Secrets

All credentials are now set via toolforge envvars (Kubernetes Secrets) — no config YAML file needed at runtime. See deployment.md step 3 for the full list of env vars and the warning about shell history on the shared bastion.

Test plan

  • npm run build from repo root (package.json shim) — assets land in montage/static/
  • cd frontend && npm run dev — Vite dev server starts
  • pytest montage/tests/ — 6 passed
  • Python config loads correctly with devtest (no env vars needed)
  • load_env_config() reads from MONTAGE_* env vars when set
  • gunicorn app:app starts and serves — / → 200, /meta/ → route list
  • CI passes
  • Node buildpack triggers from root package.json shim on Toolforge (needs working kube)
  • App is reachable via Toolforge ingress on port 8000 (needs working kube)
  • MONTAGE_ENV + secret envvars correctly configure the running service (needs working kube)
  • $TOOL_DATA_DIR is NFS-mounted in the webservice pod; log paths resolve (needs working kube)

Note: PR #534 (tools/build-frontend-script) is kept open as a fallback until the buildservice is confirmed working on Toolforge.

lgelauff and others added 5 commits May 4, 2026 15:47
Replace toolforge webservice python3.13 with the Toolforge buildservice
(Cloud Native Buildpacks). Adds Procfile, .python-version, and a root-level
package.json shim so both Python and Node buildpacks activate, baking the
Vue frontend into the container image and eliminating the separate node20
frontend-build job.

Config and env detection are updated in utils.py to work inside containers:
- get_env_name() checks MONTAGE_ENV first (LDAP/getpass.getuser unavailable
  in containers per Toolforge docs)
- load_env_config() loads secrets from individual MONTAGE_* environment
  variables (stored via toolforge envvars) when MONTAGE_OAUTH_CONSUMER_TOKEN
  is set; falls back to YAML for local dev

Removes the urllib3/pyopenssl SSL shim (deprecated since urllib3 2.0, no
longer needed on Python 3.13) and its dependency from requirements.in.
Removes the cryptography version pin, which was a uwsgi+PyO3 workaround
that does not apply to gunicorn. Adds gunicorn to requirements.in.

Rewrites deployment.md for the buildservice workflow.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Matches the Python version used on Toolforge (.python-version). Required
by gunicorn>=20 which dropped support for Python <3.10.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cffi 1.16.0 ships a cp39 manylinux wheel but not a cp313 one, so pip
falls back to compiling from source on Python 3.13. The slim image has
no C toolchain. gcc and libffi-dev are the standard build requirements
for cffi. The underlying fix (cffi>=1.17.1, which has cp313 wheels) is
tracked in #512.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cffi 1.17.1 adds cp313 wheels, eliminating source compilation on Python
3.13 (and the need for gcc in the CI image). setuptools is pinned <82
because setuptools 82 removed pkg_resources, which python-graph-core
depends on via namespace packages. Remove once python-graph-core is
replaced (#421).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant