Skip to content

ci(container): build aws images on cpu runners - #5610

Closed
ko3n1g wants to merge 1 commit into
mainfrom
ko3n1g/ci/cpu-image-build-20261104
Closed

ko3n1g wants to merge 1 commit into
mainfrom
ko3n1g/ci/cpu-image-build-20261104

Conversation

@ko3n1g

@ko3n1g ko3n1g commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Background

The protected-main Build Gym container job is queued behind H100 workloads on nemo-ci-aws-gpu-x2 (run 41861701599). The pool was at its 24-runner limit while the existing CPU pool had available capacity. Image construction and dependency prefetch require no GPU.

This leaves downstream CI using an older admissible image: CI-Events-Workflows run 41857672334, attempt 2 fails because that image has uv 0.11.29 while current Gym requires uv >=0.13.4. This focused infrastructure repair is tracked by those failing runs; no separate issue is needed.

What changed

Route the default AWS container build to the existing nemo-ci-aws-cpu-4-16 pool. Preserve nondefault preflight routes, Azure environment selection, attempt-qualified ephemeral labels, scheduled/manual triggers, upstream unit-test checks, and cancellation behavior.

Details

The existing CI contract test now checks the parsed container_build runner selection rather than matching a runner expression elsewhere in the workflow. The trigger policy from #2810 and ephemeral attempt routing from #3163 remain in force.

flowchart LR
    P[Preflight and unit checks] --> C[CPU image builder]
    C --> E[ECR image and source revision]
    E --> R[CI Events protected-main image resolver]
    R --> T[Gym cluster build test]
Loading

Compatibility and benchmark impact: none; the Dockerfile, image contents, runtime GPU jobs, and image admission rules are unchanged.

Tested

  • Head: aa7fe4cd3124d9ce94e77fe1a90bbab5d3fad042, signed and DCO signed off.
  • .venv/bin/python -m pytest tests/unit_tests/test_ci_environment.py -q: 81 passed.
  • .venv/bin/pre-commit run --all-files: passed, including guardwords and import layering. Staged checks and signed-commit hooks also passed; git diff --check passed.
  • bash /tmp/gym-cpu-image-pr-local-build.sh: built the canonical docker/Dockerfile release target from a clean archive of this exact head, using the official named nemo-gym source context, NEMO_GYM_PREFETCH_CONFIGS=tests/e2e/gpu_e2e.yaml, and a BuildKit container limited to 4 CPUs and 16 GiB. Dependency prefetch, Python 3.13 setup, and the nonroot CLI check completed.
  • Published the isolated local validation tag gym:ci-events-local-aa7fe4cd3124d9ce94e77fe1a90bbab5d3fad042; ECR independently confirms index sha256:16a316341daacad6e88dfb043fa2d788d1beb59bee2fb43a4dbd200f4791aa68 and image size 14,654,181,462 bytes. Local validation tags do not satisfy protected-main admission.
  • Verified the live CPU pool already provides Docker-in-Docker and linux/amd64 workers. IAM simulation allows all six Gym repository image pull/push actions for its existing ECR role.
  • Rollout: the local image build and registry publication are verified. Applying the CPU route to protected-main builds requires this change to merge and a new official image build to succeed. The existing GPU build can also restore image freshness when a runner becomes available. No production deployment is claimed by this PR.

Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ko3n1g requested a review from a team as a code owner November 4, 2026 17:40
@ko3n1g ko3n1g added ci CI, automation, test queue, or workflow infrastructure work area:ci CI workflows, test routing, and repository automation needs-review PR is ready for code review and waiting on a reviewer labels Nov 4, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team November 4, 2026 17:40
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

LGTM — no concerns found.

The runner selection moves the default AWS container build to CPU capacity while retaining nondefault routes, attempt-qualified ephemeral labels, Azure environment selection, and upstream/cancellation gates. The updated contract test now checks the container job specifically. I also traced the Dockerfile/prefetch path and downstream GPU job; I found no GPU requirement during image construction or changed publication contract.

Static review only: I didn't run tests or a container build, or independently verify the live runner configuration and registry permissions.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@ko3n1g ko3n1g closed this Nov 4, 2026

This branch was previously deployed

1 inactive deployment
public — aa7fe4cd Deployed Nov 4, 2026 by copy-pr-bot[bot] via release / finalize / notify #10070
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ci CI workflows, test routing, and repository automation ci CI, automation, test queue, or workflow infrastructure work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants