tools: migrate to Toolforge buildservice#564
Open
lgelauff wants to merge 5 commits into
Open
Conversation
Replace toolforge webservice python3.13 with the Toolforge buildservice (Cloud Native Buildpacks). Adds Procfile, .python-version, and a root-level package.json shim so both Python and Node buildpacks activate, baking the Vue frontend into the container image and eliminating the separate node20 frontend-build job. Config and env detection are updated in utils.py to work inside containers: - get_env_name() checks MONTAGE_ENV first (LDAP/getpass.getuser unavailable in containers per Toolforge docs) - load_env_config() loads secrets from individual MONTAGE_* environment variables (stored via toolforge envvars) when MONTAGE_OAUTH_CONSUMER_TOKEN is set; falls back to YAML for local dev Removes the urllib3/pyopenssl SSL shim (deprecated since urllib3 2.0, no longer needed on Python 3.13) and its dependency from requirements.in. Removes the cryptography version pin, which was a uwsgi+PyO3 workaround that does not apply to gunicorn. Adds gunicorn to requirements.in. Rewrites deployment.md for the buildservice workflow. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Matches the Python version used on Toolforge (.python-version). Required by gunicorn>=20 which dropped support for Python <3.10. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cffi 1.16.0 ships a cp39 manylinux wheel but not a cp313 one, so pip falls back to compiling from source on Python 3.13. The slim image has no C toolchain. gcc and libffi-dev are the standard build requirements for cffi. The underlying fix (cffi>=1.17.1, which has cp313 wheels) is tracked in #512. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cffi 1.17.1 adds cp313 wheels, eliminating source compilation on Python 3.13 (and the need for gcc in the CI image). setuptools is pinned <82 because setuptools 82 removed pkg_resources, which python-graph-core depends on via namespace packages. Remove once python-graph-core is replaced (#421). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates Montage from
toolforge webservice python3.13to the Toolforge buildservice (Cloud Native Buildpacks). AddsProcfile,.python-version, and a root-levelpackage.jsonshim so the Python and Node buildpacks both activate — baking the Vue frontend into the container image and eliminating the separate node20 frontend-build job.Background and design rationale: #563.
Also incorporates the Python 3.13 dependency fixes from #512 (cffi 1.17.1, setuptools<82 pin).
What changed
Procfile: defines the gunicorn web process; binds to$PORT(Toolforge injectsPORT=8000).python-version: pins Python 3.13 for the buildpack; removed from.gitignorepackage.json(repo root): shim to trigger Node buildpack;buildscript delegates tofrontend/montage/utils.py:get_env_name()checksMONTAGE_ENVenv var first (LDAP unavailable in containers);load_env_config()reads secrets from individualMONTAGE_*env vars whenMONTAGE_OAUTH_CONSUMER_TOKENis set, falls back to YAML for local devapp.py: removesurllib3.contrib.pyopensslshim (deprecated since urllib3 2.0, unnecessary on Python 3.13)requirements.in: addsgunicorn; removespyopenssl; unpinscryptographyrequirements.txt:cffi1.16.0 → 1.17.1 (adds cp313 wheels, no source compilation needed);setuptoolspinned <82 (setuptools 82 removedpkg_resources, needed bypython-graph-core— tracked in Automate setuptools installation #421)dockerfile: Python 3.9 → 3.13 to match deployment; addsgcc libffi-devbuild depsdeployment.md: rewritten for the buildservice workflowSecrets
All credentials are now set via
toolforge envvars(Kubernetes Secrets) — no config YAML file needed at runtime. Seedeployment.mdstep 3 for the full list of env vars and the warning about shell history on the shared bastion.Test plan
npm run buildfrom repo root (package.json shim) — assets land inmontage/static/cd frontend && npm run dev— Vite dev server startspytest montage/tests/— 6 passedload_env_config()reads fromMONTAGE_*env vars when setgunicorn app:appstarts and serves —/→ 200,/meta/→ route listpackage.jsonshim on Toolforge (needs working kube)MONTAGE_ENV+ secret envvars correctly configure the running service (needs working kube)$TOOL_DATA_DIRis NFS-mounted in the webservice pod; log paths resolve (needs working kube)Note: PR #534 (
tools/build-frontend-script) is kept open as a fallback until the buildservice is confirmed working on Toolforge.