Skip to content

CI: Add DL cluster (GB300) test pipeline - #11408

Merged
dpressle merged 8 commits into
openucx:masterfrom
Alexey-Rivkin:dl-test-matrix
May 14, 2026
Merged

dpressle merged 8 commits into
openucx:masterfrom
Alexey-Rivkin:dl-test-matrix

Conversation

@Alexey-Rivkin

@Alexey-Rivkin Alexey-Rivkin commented May 4, 2026 •

Copy link
Copy Markdown
Contributor

What?

Add a Jenkins-based test pipeline that runs contrib/test_jenkins.sh on the DL cluster (GB300/aarch64) via SLURM.

Why?

Azure CI agents can't reach dlcluster.nvidia.com, so UCX has no aarch64 GB300 coverage in PR CI today. Tracked in HPCINFRA-4292.

How?

  • .ci/pipeline/test_dl_matrix.yaml: allocate on dlcluster, srun test_jenkins.sh in a container, clean up.
  • .ci/dockerfiles/Dockerfile.dl-base: small wrapper over the existing hpcx aarch64 builder, adding the svc-nixl user pyxis expects.
  • .ci/jenkins/proj_jjb_oss.yaml: register ucx-test-dl-oss in the dispatcher fanout.

@Alexey-Rivkin
Alexey-Rivkin force-pushed the dl-test-matrix branch 27 times, most recently from 2d8051f to 993e45b Compare May 7, 2026 14:56

@yosefe yosefe left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we exclude .ci from AZP PR test trigger?

@Alexey-Rivkin

Copy link
Copy Markdown
Contributor Author

shall we exclude .ci from AZP PR test trigger?

Absolutely. A separate PR?

Alexey-Rivkin added a commit to Alexey-Rivkin/ucx that referenced this pull request May 10, 2026
Wraps now-correct svcnbu-swx-hpcx user (UID 149917). Header was stale
after the user swap.

Polish for review comment on Dockerfile.dl-base:13 in PR openucx#11408.
Alexey-Rivkin added a commit to Alexey-Rivkin/ucx that referenced this pull request May 10, 2026
Aligns with the convention used by ucx-test-gpu-oss; makes the GPU
nature explicit in the job name.

Addresses review comment on proj_jjb_oss.yaml:91 in PR openucx#11408.
Alexey-Rivkin added a commit to Alexey-Rivkin/ucx that referenced this pull request May 10, 2026
Adds Dockerfile.dl-base as a runs_on_dockers entry so ci-demo
auto-builds and pushes it to harbor on tag bump. DOCKER_IMAGE
points at the ci-demo URL, replacing the manual build+push step
on swx-hpcx-arm01.

Addresses review comment on test_dl_matrix.yaml:25 in PR openucx#11408.
@Alexey-Rivkin
Alexey-Rivkin requested a review from dpressle May 11, 2026 08:08
Comment thread .ci/pipeline/test_dl_matrix.yaml Resolved
Comment thread .ci/pipeline/test_dl_matrix.yaml Resolved
@yosefe

yosefe commented May 11, 2026

Copy link
Copy Markdown
Member

/build

@ofirfarjun7

ofirfarjun7 commented May 11, 2026 •

Copy link
Copy Markdown
Contributor

Looks like build test failed to fetch the gpunetion submodule

[2026-05-11T10:00:52.670Z] + ./autogen.sh
[2026-05-11T10:00:52.670Z] fatal: detected dubious ownership in repository at '/home/jenkins/agent/workspace/UCX-OSS/ucx-build-oss'
[2026-05-11T10:00:52.670Z] To add an exception for this directory, call:
[2026-05-11T10:00:52.670Z] 
[2026-05-11T10:00:52.670Z] 	git config --global --add safe.directory /home/jenkins/agent/workspace/UCX-OSS/ucx-build-oss
[2026-05-11T10:00:52.670Z] WARNING: Failed to update GPUNetIO submodule, continuing...

@yosefe

yosefe commented May 11, 2026

Copy link
Copy Markdown
Member

Build test failed to fetch the gpunetion submodule

[2026-05-11T10:00:52.670Z] + ./autogen.sh
[2026-05-11T10:00:52.670Z] fatal: detected dubious ownership in repository at '/home/jenkins/agent/workspace/UCX-OSS/ucx-build-oss'
[2026-05-11T10:00:52.670Z] To add an exception for this directory, call:
[2026-05-11T10:00:52.670Z] 
[2026-05-11T10:00:52.670Z] 	git config --global --add safe.directory /home/jenkins/agent/workspace/UCX-OSS/ucx-build-oss
[2026-05-11T10:00:52.670Z] WARNING: Failed to update GPUNetIO submodule, continuing...

Seems like infra issue @Alexey-Rivkin

@Alexey-Rivkin

Copy link
Copy Markdown
Contributor Author

/build

@Alexey-Rivkin

Copy link
Copy Markdown
Contributor Author

/build

Comment thread autogen.sh Outdated
mkdir -p config/m4 config/aux

git submodule update --init external/gpunetio ||
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=safe.directory GIT_CONFIG_VALUE_0='*' \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it really needed as part of UCX scripts?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems the git submodule failure is caused by the K8s container running as root, while the workspace is owned by the Jenkins user.
This commit is to validate the theory. The actual fix will be implemented at the Jenkins level.

@Alexey-Rivkin

Copy link
Copy Markdown
Contributor Author

/build

@dpressle

dpressle commented May 12, 2026 •

Copy link
Copy Markdown
Contributor

@Alexey-Rivkin maybe rebase this PR so AZP will not run anymore?

Note that UCX PR is required it means we will need to bypass rules to merge for now

Azure agents can't reach dlcluster.nvidia.com, so UCX has no aarch64
GB300 coverage in PR CI. Add a Jenkins matrix that allocates SLURM on
dlcluster and runs contrib/test_jenkins.sh inside an aarch64 builder
image - same shape as test_gpu_matrix.yaml, with dlcluster headnode +
SSH credentials in place of scctl.

Bumps Jenkinsfile.github to ci-demo@master because slurmCI lives there
(stable_ucx fork doesn't carry it).

Ref: HPCINFRA-4292
Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
Removes --no-container-mount-home extraArg and SHELL=/bin/bash slurmEnv.
With tcsh purged from the dl-base image, pyxis cannot invoke /bin/csh,
so the host /etc/passwd shell of svcnbu-swx-hpcx no longer matters.

Addresses review comment on Dockerfile.dl-base:1 in PR openucx#11408.
Wraps now-correct svcnbu-swx-hpcx user (UID 149917). Header was stale
after the user swap.

Polish for review comment on Dockerfile.dl-base:13 in PR openucx#11408.
Aligns with the convention used by ucx-test-gpu-oss; makes the GPU
nature explicit in the job name.

Addresses review comment on proj_jjb_oss.yaml:91 in PR openucx#11408.
Adds Dockerfile.dl-base as a runs_on_dockers entry so ci-demo
auto-builds and pushes it to harbor on tag bump. DOCKER_IMAGE
points at the ci-demo URL, replacing the manual build+push step
on swx-hpcx-arm01.

Addresses review comment on test_dl_matrix.yaml:25 in PR openucx#11408.
Jenkins pods mount the workspace as the agent UID but run the container
as root, so git's safe.directory check blocks `git submodule update` in
autogen.sh. The `||` swallows it, distcheck then fails on the missing
gpunetio submodule headers.

Fix in the matrix step to keep autogen.sh and other CI paths untouched.

Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
@Alexey-Rivkin

Copy link
Copy Markdown
Contributor Author

/build

@Alexey-Rivkin
Alexey-Rivkin requested a review from dpressle May 12, 2026 11:32
@Alexey-Rivkin

Copy link
Copy Markdown
Contributor Author

@dpressle CI passed, pls review.

@dpressle
dpressle merged commit 7c243e9 into openucx:master May 14, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants