Repository navigation
CI: Add DL cluster (GB300) test pipeline - #11408
Conversation
2d8051f to
993e45b
Compare
yosefe
left a comment
There was a problem hiding this comment.
shall we exclude .ci from AZP PR test trigger?
Absolutely. A separate PR? |
993e45b to
7649710
Compare
Wraps now-correct svcnbu-swx-hpcx user (UID 149917). Header was stale after the user swap. Polish for review comment on Dockerfile.dl-base:13 in PR openucx#11408.
Aligns with the convention used by ucx-test-gpu-oss; makes the GPU nature explicit in the job name. Addresses review comment on proj_jjb_oss.yaml:91 in PR openucx#11408.
Adds Dockerfile.dl-base as a runs_on_dockers entry so ci-demo auto-builds and pushes it to harbor on tag bump. DOCKER_IMAGE points at the ci-demo URL, replacing the manual build+push step on swx-hpcx-arm01. Addresses review comment on test_dl_matrix.yaml:25 in PR openucx#11408.
17e6eca to
dbd3d34
Compare
|
/build |
|
Looks like build test failed to fetch the gpunetion submodule |
Seems like infra issue @Alexey-Rivkin |
|
/build |
0ac4fa7 to
1f33ed2
Compare
|
/build |
| mkdir -p config/m4 config/aux | ||
|
|
||
| git submodule update --init external/gpunetio || | ||
| GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=safe.directory GIT_CONFIG_VALUE_0='*' \ |
There was a problem hiding this comment.
is it really needed as part of UCX scripts?
There was a problem hiding this comment.
Seems the git submodule failure is caused by the K8s container running as root, while the workspace is owned by the Jenkins user.
This commit is to validate the theory. The actual fix will be implemented at the Jenkins level.
1f33ed2 to
ff772b7
Compare
|
/build |
Note that UCX PR is required it means we will need to bypass rules to merge for now |
Azure agents can't reach dlcluster.nvidia.com, so UCX has no aarch64 GB300 coverage in PR CI. Add a Jenkins matrix that allocates SLURM on dlcluster and runs contrib/test_jenkins.sh inside an aarch64 builder image - same shape as test_gpu_matrix.yaml, with dlcluster headnode + SSH credentials in place of scctl. Bumps Jenkinsfile.github to ci-demo@master because slurmCI lives there (stable_ucx fork doesn't carry it). Ref: HPCINFRA-4292 Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
Removes --no-container-mount-home extraArg and SHELL=/bin/bash slurmEnv. With tcsh purged from the dl-base image, pyxis cannot invoke /bin/csh, so the host /etc/passwd shell of svcnbu-swx-hpcx no longer matters. Addresses review comment on Dockerfile.dl-base:1 in PR openucx#11408.
Wraps now-correct svcnbu-swx-hpcx user (UID 149917). Header was stale after the user swap. Polish for review comment on Dockerfile.dl-base:13 in PR openucx#11408.
Aligns with the convention used by ucx-test-gpu-oss; makes the GPU nature explicit in the job name. Addresses review comment on proj_jjb_oss.yaml:91 in PR openucx#11408.
Adds Dockerfile.dl-base as a runs_on_dockers entry so ci-demo auto-builds and pushes it to harbor on tag bump. DOCKER_IMAGE points at the ci-demo URL, replacing the manual build+push step on swx-hpcx-arm01. Addresses review comment on test_dl_matrix.yaml:25 in PR openucx#11408.
Jenkins pods mount the workspace as the agent UID but run the container as root, so git's safe.directory check blocks `git submodule update` in autogen.sh. The `||` swallows it, distcheck then fails on the missing gpunetio submodule headers. Fix in the matrix step to keep autogen.sh and other CI paths untouched. Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
ff772b7 to
6edf6a1
Compare
|
/build |
|
@dpressle CI passed, pls review. |
What?
Add a Jenkins-based test pipeline that runs
contrib/test_jenkins.shon the DL cluster (GB300/aarch64) via SLURM.Why?
Azure CI agents can't reach
dlcluster.nvidia.com, so UCX has no aarch64 GB300 coverage in PR CI today. Tracked in HPCINFRA-4292.How?
.ci/pipeline/test_dl_matrix.yaml: allocate on dlcluster, sruntest_jenkins.shin a container, clean up..ci/dockerfiles/Dockerfile.dl-base: small wrapper over the existing hpcx aarch64 builder, adding thesvc-nixluser pyxis expects..ci/jenkins/proj_jjb_oss.yaml: registerucx-test-dl-ossin the dispatcher fanout.