feat(deployment): add Xavier GPU offload for robot inference - #1556
Conversation
- add offload contract + remote.yaml schema specs - add parameterized Helm scaffolding and SO-101 example - vendor LeRobot ROS 2 bridge (lint-only reference) - add provenance, audit, and open-items docs 🚀 - Generated by Copilot
- move audit, carry-decision, and open-items notes into internal .tasks/ - strip provenance/porting narrative and attribution footers from docs - generalize SPDX attribution headers in source files - reframe scope and validation notes for end users 📝 - Generated by Copilot
🔄 - Generated by Copilot # Conflicts: # .cspell.json
- add safe MessagePack runtime and admission reconciliation - harden Helm deployment, TLS, RBAC, and workload opt-in - document upstream deviations and compatibility 🔒 - Generated by Copilot
🔒 - Generated by Copilot
* test * fix issue related to non-writeable location for config, fix issue related to dehydration in certain cases with multiple stages of servers, remove syncwithremote functionality since it does not work with messagepack * dockerfile/containerfile for source to include in workload * move * always pull to get latest image * Fix server deployment volume mounts Copy only mounts backed by volume definitions allowed on generated remote server deployments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * revert back to defaults so repository and tag for image file can be specified * change created server deployment's pull policy to match workload's * class2dict for cases where class not being converted to metaremoteduuid * allow hostPath to go to server deployment - otherwise model files are missing * special serialization for torch tensors * benchmark pickle vs messagepack * improve serialization of tensors * perf: optimize large remoter payloads - frame large byte values outside MessagePack metadata - preserve legacy packet decoding and codec limits 🚀 - Generated by Copilot * fix one potential race condition * reenable syncwithremote as class2dict offers serialization method * fix: prevent remoter serialization deadlocks - support UUID and arbitrary-precision integer payloads - return encoding failures and restore state without constructors 🔧 - Generated by Copilot * fix: preserve remote attribute fallback - reconstruct remote AttributeError as the native built-in - retain wrapped errors for other remote exception types 🔧 - Generated by Copilot * dockerignore to only copy source * remove importlib for safety * feat: restrict remote server hostPath volumes - allow configured host paths and deny unlisted hostPath volumes - inject the Helm allowlist and set the default mutator image repository 🔒 - Generated by Copilot * chore(settings): ignore gpu-offload uv lockfiles 🧹 - Generated by Copilot * chore(settings): restore gpu-offload uv lock tracking - revert the scoped ignore rule so project locks remain visible 🧹 - Generated by Copilot * fix: fall back for unavailable tensor devices - restore tensors on the source device with CPU fallback - cache unavailable advisory devices 🔧 - Generated by Copilot * security: bound tensor allocations during decode - enforce codec byte limits before tensor allocation - validate payload length and scalar tensor shapes - cover allocation limits with codec tests 🔒 - Generated by Copilot * security: harden codec error handling - validate call and result wire envelopes - preserve remote errors through serialization failures - add malformed payload regression coverage 🔒 - Generated by Copilot --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(deployment): support LeRobot GPU offload runtime Inherit runtime classes, avoid duplicate servers, and preserve enum, path, and scalar types across RPC. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef603d1e-d629-46cb-b210-38658d1a140e * fix(gpu-offload): resolve generated pod server labels Defer per-client label resolution when admission receives only generateName, allowing the runtime and generated server to use the final Job pod name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef603d1e-d629-46cb-b210-38658d1a140e --------- Co-authored-by: Thor <xavier@thor.corp.microsoft.com> Copilot-Session: ef603d1e-d629-46cb-b210-38658d1a140e
* running on baremetal wsl support validation mise helpers * fix(deployment): support LeRobot GPU offload runtime Inherit runtime classes, avoid duplicate servers, and preserve enum, path, and scalar types across RPC. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef603d1e-d629-46cb-b210-38658d1a140e * fix(gpu-offload): resolve generated pod server labels Defer per-client label resolution when admission receives only generateName, allowing the runtime and generated server to use the final Job pod name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef603d1e-d629-46cb-b210-38658d1a140e * init * docs(examples): clarify first-run demo must be uninstalled before pi05 GPU claim Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build(deployment): add missing uv.lock for gpu-offload controller and runtime Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * update todo * refactor(infrastructure): extract gpu-offload mise task scripts - move inline heredoc task bodies to scripts/ and examples/*/scripts/ - rename tasks with a-/b-/c-/d-/e-/f- prefixes so mise tasks sorts into the clone -> setup -> verify -> teardown workflow order - convert setup/verify/teardown orchestration to mise depends - update mise run references in docs and pi05 README Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * add ur10e policy example * feat(deployment): run ur10e-single Pi0.5 policy headless on the GPU offload stage - add headless mode driving the real UR10e with no display, homing on entry and exit - stop memoizing get_action: singleinstance returned the first action forever - add self-contained host registry with per-upstream pull-through caches - pass USB through to the control pod so RealSense cameras enumerate in cluster - wait for client deletion before replacing the stage to avoid a load() deadlock 🤖 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(deployment): add one-command ur10e-single policy demo task - add g-ur10e-52-demo: deploy headless with USB, then follow the run to homing - bound the demo to 80 steps so the arm homes instead of cycling indefinitely 🤖 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(deployment): surface the ur10e-single demo task in the gpu-offload docs - link the single-command hardware demo from the gpu-offload quick start - list demo.sh in the example file table and normalize table formatting 📝 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix docs --------- Co-authored-by: Thor <xavier@thor.corp.microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef603d1e-d629-46cb-b210-38658d1a140e
⚡ - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Shawn Cicoria <cicorias@users.noreply.github.com>
* feat(deployment): integrate SO-101 LeRobot example - replace the legacy example with the validated LeRobot workflows - source the offload runtime from the local BuildKit context - exclude benchmark and result artifacts 🤖 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef603d1e-d629-46cb-b210-38658d1a140e * feat(deployment): add SO-101 rollout timing - Add opt-in aggregated control-loop stage timing - Configure remoter log thresholds through Helm 🤖 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef603d1e-d629-46cb-b210-38658d1a140e * perf(deployment): reduce SO-101 offload payload - Move image preparation and policy processing to the GPU server - Document float32 and uint8 real-hardware measurements 🤖 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef603d1e-d629-46cb-b210-38658d1a140e * docs(deployment): clarify SO-101 offload configuration - Explain transparent and optimized inference modes - Replace host-specific values with a reusable example 📚 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef603d1e-d629-46cb-b210-38658d1a140e --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef603d1e-d629-46cb-b210-38658d1a140e
Added details about efficient inference and offloaded inference capabilities to the Physical AI Toolchain section.
…yering pattern, and prune the docs (#9) * chore(deployment): remove obsolete pi05 GPU offload example - delete examples/pi05 chart, control loop, scripts, and container image - drop the e-pi05-* mise tasks and PI05_* entries from .env.example - remove the Pi05 UR10e motion section and stale cspell words from docs/TODO.md - repoint the ur10e-single teardown note at an existing task 🧹 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(deployment): layer the remoter SDK into the first-run image - build from the pyremote payload image with a sitecustomize hook - drop the SDK import and autoremote.start call from client.py - fold Containerfile.local and Containerfile.gpu into two build targets - add the missing example README and a reference remote.yaml 🔌 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(deployment): prune and refresh the gpu-offload doc set - delete PR3-merge-issues.md; every recommendation is implemented - rewrite the docs index for all three platforms and link the diagrams - check off the completed mise task extraction in TODO.md - correct T0 plan rows for the replaced so101 example and torch codec 📚 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- apply ruff format, safe fixes, and taplo formatting - move module docstrings above __future__ imports - drop unreachable logger call in msgqueue - ignore vendored remoter style rules; add cspell terms - fix mise-tasks instructions frontmatter and headings 🎨 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- add title and description frontmatter required by validation - refresh stale ms.date on docs changed in this branch 📝 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1556 +/- ##
==========================================
+ Coverage 88.10% 88.13% +0.02%
==========================================
Files 279 279
Lines 23203 23245 +42
Branches 3199 3199
==========================================
+ Hits 20444 20486 +42
Misses 2071 2071
Partials 688 688
*This pull request uses carry forward flags. Click here to find out more.
🚀 New features to boost your workflow:
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
- raise kubernetes pin in gpu-offload controller and runtime - relock urllib3 2.3.0 -> 2.7.0 (GHSA-gm62, 2xpw, 38jv, qccp, pq67, 48p4) - drop google-auth/pyasn1 deps no longer required 🔒 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- split readonly declaration from command substitution (SC2155) - document GPU_OFFLOAD_DIR consumption by sourcing scripts (SC2034) 🐚 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- reject unsupported socket families instead of unbound ep - return generic errors from config server, log details - pass required callback in rmtconfig self-test 🔒 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- fix GHSA-6w46-j5rx-g56g (CVE-2025-71176) tmpdir handling - align with repo-wide pytest==9.1.1 pin 🔒 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- fix GHSA-r6ph, 537c, g6cj, jwv3, p423, m2h6 - align with repo-wide cryptography==50.0.1 pin 🔒 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- msgpack 1.2.2 (GHSA-6v7p-g79w-8964) - filelock 3.20.3 (GHSA-w853-jp5j-5j7f, GHSA-qmgc-5h2g-mvrw) - requests 2.34.2 (GHSA-gc5v-m9x4-r6x2) - flask 3.1.3 (GHSA-68rp-wp8r-4726) 🔒 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ength, DoS, registry/build tooling) controller/mutate.py: - Server Deployments no longer copy Secret volumes, or projected volumes that combine a Secret source, from the client container -- a client's mounted credentials were being exposed on a separately-specified server image/node. - The client container's environment is no longer copied onto the server by default. Only offload-protocol variables the controller itself injects and names explicitly listed in a new remoteableenv config field transfer; this closes a secret-exposure path via both literal env values and valueFrom references (secretKeyRef, configMapKeyRef). - _deployment_name now truncates and hashes names that would exceed Kubernetes' 63-character label-value limit (it's used as both the Deployment's own name and a label value), instead of letting a moderately-long client workload name fail Deployment/label validation. runtime/remoter/msgudp.py: - Reject a UDP chunk header whose totalchunks is zero/oversized or whose chunkindex is out of range, and reject a chunk whose totalchunks disagrees with an already-tracked message, before allocating [None] * totalchunks or indexing into it. A peer could previously force large allocations or an out-of-range index purely via header values, ahead of any codec-level size check. Registry/build tooling: - .env.example and the first-local-offload doc no longer claim GPU_OFFLOAD_REGISTRY_HOST accepts an Azure Container Registry endpoint -- the registry scripts only ever treat it as a local host:port; ACR caching already has its own, correct mechanism (GPU_OFFLOAD_ACR_NAME). - controller/Containerfile and build-controller-image.sh now pass PIP_INDEX_URL as a BuildKit secret (matching Containerfile.local), not a build arg, so a credential embedded in it can't land in image history. - install-controller.sh always installs the controller image as localhost/xavier-mutate:local with pullPolicy Never. Nothing in the documented flow (build-controller-image.sh, load-images.sh) ever pushes or tags that image under the host-local registry's host:port, on k3s or kind, so preferring the registry reference there always pointed at an image that didn't exist under that name. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
The current CI does not exercise the new Most of this coverage does not require Xavier hardware or a GPU. Please add a path-gated The PR description currently says these test suites are “relying on CI”, but no workflow references |
|
Could you please add dedicated CI coverage for the new 1. Existing unit testsThe PR adds 44 tests under:
Please consider adding a path-gated reusable workflow based on the existing A frozen controller test invocation currently fails because Hatch cannot infer the wheel contents. Could you clarify the intended packaging contract and either configure the build target or mark the project as script-only and test it with The new job should also be included in the PR and main validation summaries. 2. CPU smoke coveragePlease consider extending For container validation, could the existing path-gated runtime-image smoke pattern be reused to build and probe the controller, runtime payload and first-run CPU images? 3. CPU-only Kubernetes E2EThe PR already contains most of a suitable E2E scenario:
Please consider composing these assets into one path-gated workflow on a standard GitHub runner. Keeping the scenario logic in the existing scripts would avoid duplicating it in workflow YAML. This CPU path could verify chart installation, webhook TLS/RBAC, admission mutation, generated server reconciliation, packaging, networking and remote execution. GPU allocation, CUDA execution, latency measurements and robot motion could remain separate hardware-backed qualification checks. 4. Containerfile discovery
Could you please extend the existing discovery and change-trigger logic to include 5. Helm and
|
…ndency Address PR review feedback that the vendored LeRobot submodule costs ~403 MiB to initialize (git history + worktree) while the Docker build never patches the source. Since lerobot==0.6.1 is published on PyPI with all required entry points, depend on it directly via pyproject.toml/uv.lock instead of vendoring the repository. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…oject.toml - gpu-offload/examples/so101-real-hardware/pyproject.toml: reformat with taplo (4-space indent) to satisfy TOML Format Check. - .github/workflows/dependency-review.yml: allow GHSA-xrqw-3rrv-vx5w (transformers path traversal via save_pretrained chat_template keys, CVE-2026-9856). lerobot==0.6.1 constrains transformers<5.6.0 and the fix landed in 5.10.0, so no compatible patched version exists yet; the SO-101 example never calls save_pretrained() on a Hub-downloaded tokenizer or processor, only load()s its own pinned checkpoint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Address PR review feedback that the 44 tests added under
gpu-offload/{controller,runtime}/tests never run in CI.
- gpu-offload/controller/pyproject.toml: drop the unused [build-system]
(Hatchling) section and mark the project virtual (tool.uv.package = false).
mutate.py is a flat script, never installed as a wheel or built as an
editable install anywhere; Hatchling's wheel builder failed with "Unable
to determine which files to ship" because there is no gpu_offload_controller/
package directory. Re-locked uv.lock accordingly (source: editable -> virtual).
- .github/workflows/pytest-gpu-offload.yml: new reusable workflow, matrixed
over controller and runtime, each synced from its own uv.lock in an
isolated environment and run with pytest.
- pr-validation.yml: path-gate the new job on gpu-offload/{controller,runtime}/
changes and add it to the validation summary.
- main.yml: run the new job unconditionally and add it to the release gate.
- CONTRIBUTING.md: document the new component suite and its local-run command.
Coverage/Codecov integration is intentionally left out of scope; both
subprojects currently test without coverage tooling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Thanks for the detailed breakdown — this is a lot of good infra to add. To keep this PR reviewable I've scoped it to the highest-value, lowest-risk pieces from your list and will treat the rest as follow-up work:
Left for follow-up (would meaningfully grow this PR and each involves its own design decisions worth reviewing separately):
Coverage/Codecov integration for the new pytest job is also intentionally out of scope for now — neither subproject has coverage tooling wired up yet. Happy to open a tracking issue for the remaining items if that's useful. |
|
Replying here since this predates my more detailed reply on your follow-up comment (#issuecomment-5726472922):
Building/scanning the images, Helm lint/render, and a CPU-only smoke test are still open (tracked in my reply on the other comment) — those are more involved and I'd like to scope them as follow-up work rather than grow this PR further. |
Katrien De Graeve (katriendg)
left a comment
Katrien De Graeve (katriendg)
left a comment
There was a problem hiding this comment.
Added inline review comments for the current head.
|
Thanks for the substantial GPU-offload integration and for addressing earlier review rounds by Alexandre. Also adding some inline comments. The added pytest workflow and submodule removal are useful completed responses. The proposed follow-up split should be narrowed: hardware-only GPU qualification can be tracked separately, but CPU smoke, chart/config validation, image build discovery, Codecov wiring, and a validated fail-closed rollback path belong in this PR because they validate or govern the newly introduced merge surface. TODO.md should not remain the sole durable owner of known implementation defects; unresolved release-relevant items need explicit issue ownership and blocking/non-blocking disposition. Open question is even if a file named The customer-facing documentation should also establish the value before exposing the implementation machinery. State plainly that this is an optional deployment profile for running policies that exceed a robot's onboard compute, power, thermal, or battery budget on a nearby site GPU while robot I/O and control remain local. Add a decision guide covering when to use onboard inference, this GPU-offload profile, a conventional inference service, or cloud inference, together with explicit non-goals for disconnected and hard real-time operation. Clarify the intended audience and prerequisites so users without Kubernetes operations experience do not mistake this specialized capability for the default inference path. The naming also needs an explicit decision. The PR title and documentation use “Xavier,” but the implementation does not appear to require NVIDIA Jetson Xavier hardware; the term instead refers to another project), wonder if we keep that and if this will become clear for users. If “Xavier” remains in the title or user-facing terminology, confirm that this ambiguity is intentional and add a prominent clarification that it is not an NVIDIA Jetson Xavier requirement. Otherwise, prefer “GPU Offload” in the title and customer-facing navigation, and keep Xavier in provenance or implementation-history material. |
Katrien De Graeve (katriendg)
left a comment
Katrien De Graeve (katriendg)
left a comment
There was a problem hiding this comment.
Added three follow-up inline comments on dependency update coverage and pinning enforcement.
pr-validation.yml only gated the gpu-offload controller/runtime unit tests, so changes confined to the Helm chart, host scripts, local registry, or examples (including committed remote.yaml manifests and Containerfiles) could merge with zero GPU-offload validation. Add a new gpu_offload_e2e path filter covering those paths, and wire it to a new reusable workflow that runs the existing first-run example end to end against a CPU-only kind cluster: real admission (the webhook mutates the client and generates the server Deployment) and a real remote call (the function executes on the generated server pod, not the client). Every GPU-only step in the pipeline already self-skips on a GPU-less runner, so this reuses the existing mise tasks as-is. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extend the root Ruff config instead of redefining it, per PR review feedback. Fixes the small set of originally-suppressed E712/E731/F841 instances directly, and scopes the rest of the vendored remoter/ package's findings to narrowly-justified per-line noqa comments instead of a blanket per-file-ignore. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GitHub Actions in gpu-offload were already SHA-pinned, but the container
images referenced by its Dockerfiles/Containerfiles, shell-generated
manifests, and Helm values were not. The required pinning scanner missed
this because it explicitly excluded Dockerfile FROM lines (deferring to
OpenSSF Scorecard, which never actually runs on pull_request), only scanned
docker image YAML under workflow/infrastructure paths, and discovered base
images via a *Dockerfile* glob that never matches a file named Containerfile.
- Digest-pin every tag-only image under gpu-offload/: the controller,
first-run, ur10e-single, and so101 Python base images, and the CUDA base
image in cluster-gpu-check.sh. configure-k3s-nvidia.sh's device-plugin
image gets a paired --plugin-digest flag (alongside --plugin-version) and
a guard against silently pinning an overridden version to the old digest.
- Add Get-DockerfileFromViolations (new 'dockerfile-base-image' type) to
Test-DependencyPinning.ps1, handling multi-stage aliases, scratch, and
bare ${VAR} bases; add '**/*.sh' to the existing docker type so shell-
embedded image: manifests are covered too. Both wired into the actual
CI default type list.
- Fix discover-base-images.sh's glob to also match Containerfile, and its
digest-extraction regex to not truncate an ARG-templated FROM line into
a garbage partial match.
- Add dependabot docker ecosystem entries for the gpu-offload directories
so the new pins get kept up to date.
Verified: 266/266 Pester tests pass, zero pinning violations under the new
type or under gpu-offload, and all digests independently re-resolved
against their registries.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts: # docs/README.md # docs/contributing/ROADMAP.md # docs/getting-started/README.md # docs/infrastructure/cleanup.md # docs/infrastructure/cluster-setup-advanced.md # docs/training/isaac-lab-training.md
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
Splitting into two sentences keeps each line under the 500-char MD013 limit after combining upstream's Node.js/uv correction with this branch's optional-local-Kubernetes framing during the main merge. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
f-10-setup listed ten tasks in `depends`, but mise treats sibling depends entries as independent and runs them in parallel unless a task declares its own dependency on another. With no such edges, cluster- smoke-check.sh raced ahead of cluster-create.sh and failed immediately with "context was not found for specified context: kind-gpu-offload" (this workflow's first-ever CI run, since Actions required manual approval on every prior push to this fork). Chain each task to its true predecessor, matching the order the task IDs already encode: a-detect -> b-host-10-preflight -> c-cluster-20- create -> c-cluster-21-check -> c-cluster-30-gpu-enable -> c-cluster- 31-gpu-check -> d-offload-40-build-images -> d-offload-41-load-images -> d-offload-43-install-controller -> d-offload-50-deploy. Also chain c-cluster-90-delete after d-offload-90-teardown so teardown-all removes workloads before deleting the cluster they ran on. Verified with `mise tasks deps f-10-setup` (now a single linear chain, was ten unordered siblings) and `mise run f-10-setup` locally (tasks now execute strictly in order; it stops at the first missing local tool rather than racing ahead). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…Containerfile RUN --mount=type=secret,id=PIP_INDEX_URL,env=PIP_INDEX_URL fails under Podman/buildah with "secret should have syntax id=id[,target=path, required=bool,mode=uint,uid=uint,gid=uint]" - env is not a recognized attribute on the RUN --mount=type=secret directive itself in this buildah version (it belongs on the outer `podman build --secret id=...,env=...` flag, which build-controller-image.sh already sets correctly). Read the secret from its mounted file instead, which works both with and without the secret supplied. Reproduced and verified the fix locally with the same podman version (5.8.5) the CI runner uses: the original RUN line fails identically locally; the fixed line proceeds past the mount step in both the secret-provided and no-secret cases. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Required by the pr-validation-summary gate (the sole required check on main), which fails if msdate-freshness fails. Cross-checked every claim in the doc against the actual release workflow (main.yml) rather than blindly bumping the date: the sigstore==4.3.0 pin, the gitsign cert-identity string, the attest-build-provenance step, and all four SBOM files (SPDX/CycloneDX, source/dependencies) match exactly what's described. No content changes needed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The mutating webhook has failurePolicy: Fail, so any admission call that can't reach it blocks every offload-labeled pod/deployment/job. install-controller.sh only waited for `kubectl rollout status` (pod Ready), not for the Service's Endpoints to actually populate, leaving a race where the very next resource creation (deploy.sh) hit "connection refused" against the webhook Service before routing had propagated - reproduced in CI on commit 0faa279, ~2s after rollout success. Add a `kubectl wait --for=jsonpath=...` on the Endpoints object after rollout status. Verified against a real local kind cluster (same kubectl version CI uses): the wait resolves immediately once an Endpoints subset has a real address, and correctly times out (does not false-succeed) against an Endpoints object with none. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…s state
The Endpoints-populated wait from the previous commit was insufficient:
CI showed "endpoints/gpu-offload-mutate condition met" immediately
followed (~250ms later) by the same "connection refused" on the very
next admission call. The Endpoints API object reflects the pod's
address before kube-proxy has actually programmed the ClusterIP's
netfilter rule on the node - a control-plane/data-plane propagation
gap that no API object's state can promise against.
Replace it with a genuine reachability probe: retry
`kubectl apply --dry-run=server` against a minimal disposable Pod
carrying the webhook's trigger label. dry-run requests still invoke
mutating webhooks that declare sideEffects: None (this one does), so
this exercises the real webhook call without persisting anything.
Verified the full mechanism locally against a real kind cluster with a
throwaway webhook standing in for the real one: with no backing
listener it fails identically to CI ("connect: connection refused");
once a listener exists it fails differently ("connection reset by
peer" - proving reachability, not webhook logic, is what changes the
outcome). Also traced controller/mutate.py's DoMutate to confirm a
labeled-but-unannotated probe pod is legitimately allowed (`xaviercfg
is None -> return False -> allowed: true`), not rejected for missing
config, so a real, ready webhook will make this probe succeed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Katrien De Graeve (@katriendg) and Alexandre Gattiker (@algattik) , thank you for the feedback and the assistance in making this better. I have done my best to address all the issues raised. Some wil have to be put off for a review with the MS Research team. I will merge this and address the remainder of the issues in follow on work denoted here |
Pull Request
Description
Adds a
gpu-offload/domain that transparently offloads robot policy inference to a nearby on-premise GPU, based on the Xavier remote-execution runtime.gpu-offload/runtime): remoter library for opted-in functions and classes over TCP/UDP/Unix sockets, using a restricted MessagePack codec (no pickle) with size and type limitsgpu-offload/controller): mutating webhook that injects server/client configuration into workloads that explicitly opt ingpu-offload/helm): controller deployment with TLS, RBAC, and workload opt-in; node agent template retained but disabledremote.yamlschemaDraft while #1555 is in triage.
Closes #1555
Type of Change
Component(s) Affected
infrastructure/terraform/prerequisites/- Azure subscription setupinfrastructure/terraform/- Terraform infrastructureinfrastructure/setup/- OSMO control plane / Helmworkflows/- Training and evaluation workflowstraining/- Training pipelines and scriptsdocs/- Documentationgpu-offload/- New GPU offload domain (runtime, controller, Helm chart, examples, scripts)Testing Performed
planreviewed (no unexpected changes)applytested in dev environmentsmoke_test_azure.py)No Terraform, Azure, or OSMO changes; the checks above do not apply.
Validation performed on the final branch (local, macOS):
ruff checkandruff format --check(ruff 0.15.21, CI-pinned): passtaplo fmt --check,uv lock --check,npm run lint:hfpins: passcspell,markdownlint-cli2,markdown-table-formatter --check,npm run lint:links: pass on changed files-ChangedFilesOnly): 0 errors, 0 stale filespy_compileon all Python files modified during lint cleanup: pass__all__ordering; no removed import is referenced elsewhereEnvironment used: local workstation only; no Azure resources deployed for this sync.
Cost incurred: none.
Known limitations:
Update (2026-09-18):
gpu-offload/runtimeandgpu-offload/controllerpytest suites now runlocally (all passing) and in CI via the new path-gated
Pytest GPU Offloadjob(
.github/workflows/pytest-gpu-offload.yml), which also caught and fixed a Hatchpackaging bug in the controller's
pyproject.toml.shellcheckandactionlint(the tool behind the "YAML Lint" check) were also run locally and pass; both already
ran in CI prior to this update.
Documentation Impact
Bug Fix Checklist
Not applicable: feature PR.
Checklist
🤖 Generated with Claude Code