Skip to content

ci: check package layering with import-linter - #5168

Merged
ananthsub merged 1 commit into
mainfrom
ananthsub/layout-import-linter
Oct 30, 2026
Merged

ananthsub merged 1 commit into
mainfrom
ananthsub/layout-import-linter

Conversation

@ananthsub

@ananthsub ananthsub commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Adds an import-linter layers contract to pyproject.toml and runs it as a pre-commit hook, which the lint job already runs. Today 52 of the top-level modules and packages in nemo_gym/ sit in one import cycle; the contract stops new code from adding to it while the moves land.

Where this PR sits

The stack lands bottom-up; each PR's base is the branch of the PR before it, so each diff shows only that PR's change.

flowchart LR
  compat["#5167<br/>Old import paths<br/>keep working"]
  linter["#5168<br/>import-linter<br/>layer contract"]
  prep["#5169<br/>Remove path and<br/>name couplings"]
  config["#5189<br/>Move config/"]
  utils["#5195<br/>Move utils/"]
  core["#5203<br/>Move core/"]
  client["#5206<br/>Move client/,<br/>start telemetry/"]
  trajectory["#5209<br/>Move trajectory/"]
  verification["#5210<br/>Move verification/"]
  metrics["#5211<br/>Move metrics/"]
  artifacts["#5212<br/>Move artifacts/"]
  srmods["#5214<br/>Move server_runtime/<br/>modules"]
  su["#5215<br/>Split server_utils"]
  model["#5224<br/>Move models/<br/>modules"]
  modelbase["#5225<br/>Split model<br/>call capture"]
  agents["#5228<br/>Move agents/"]
  data["#5229<br/>Move data/"]
  harbor["#5230<br/>Split the<br/>Harbor task CLI"]
  rollouts["#5231<br/>Move rollouts/"]
  rsplit["#5232<br/>Split the<br/>rollout collector"]
  testing["#5233<br/>Dissolve testing/"]
  orch["#5234<br/>Move orchestration/<br/>modules"]
  sup["#5235<br/>Split the<br/>server supervisor"]
  eval["#5244<br/>Move eval/"]
  env["#5298<br/>Move environment/<br/>registries"]
  cli["#5300<br/>Retire the<br/>cli_setup_command shim"]
  docs["#5385<br/>Describe the<br/>package layout"]
  owners["next<br/>CODEOWNERS:<br/>per-package groups"]
  remove["0.10.0<br/>Remove the<br/>old paths"]
  compat --> linter
  linter --> prep
  prep --> config
  config --> utils
  utils --> core
  core --> client
  client --> trajectory
  trajectory --> verification
  verification --> metrics
  metrics --> artifacts
  artifacts --> srmods
  srmods --> su
  su --> model
  model --> modelbase
  modelbase --> agents
  agents --> data
  data --> harbor
  harbor --> rollouts
  rollouts --> rsplit
  rsplit --> testing
  testing --> orch
  orch --> sup
  sup --> eval
  eval --> env
  env --> cli
  cli --> docs
  docs --> owners
  owners --> remove
  classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937
  classDef later fill:#f3f4f6,stroke:#9ca3af,color:#4b5563,stroke-dasharray: 4 3
  class linter this
  class owners,remove later
Loading

What the contract enforces

The contract lists today's modules grouped by the package each one moves to, in this order. An import may only point down.

flowchart TD
  cli["cli/"]
  eval["eval/"]
  environment["environment/"]
  testing["testing/"]
  orchestration["orchestration/"]
  rollouts["rollouts/"]
  data["data/"]
  agents["agents/"]
  model["models/"]
  server_runtime["server_runtime/"]
  artifacts["artifacts/"]
  metrics["metrics/"]
  verification["verification/"]
  trajectory["trajectory/"]
  sandbox["sandbox/"]
  client["client/"]
  telemetry["telemetry/"]
  core["core/"]
  utils["utils/"]
  config["config/"]
  cli --> eval
  cli --> environment
  cli --> orchestration
  eval --> rollouts
  environment --> data
  testing --> rollouts
  orchestration --> data
  rollouts --> data
  rollouts --> agents
  rollouts --> model
  data --> server_runtime
  data --> artifacts
  agents --> server_runtime
  model --> server_runtime
  server_runtime --> metrics
  server_runtime --> verification
  server_runtime --> trajectory
  artifacts --> trajectory
  artifacts --> utils
  metrics --> utils
  verification --> sandbox
  trajectory --> core
  sandbox --> client
  client --> telemetry
  client --> utils
  telemetry --> core
  core --> config
  utils --> config
  classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937
  classDef done fill:#d1fae5,stroke:#047857,color:#1f2937
  classDef todo fill:#f3f4f6,stroke:#9ca3af,color:#4b5563
  class cli,eval,environment,testing,orchestration,rollouts,data,agents,model,server_runtime,artifacts,metrics,verification,trajectory,sandbox,client,telemetry,core,utils,config todo
Loading

An arrow means "imports"; each package imports only packages below it.

flowchart LR
  P["A PR adds an import"] --> L{"lint-imports"}
  L -->|"points down the order"| OK["Pass"]
  L -->|"points up the order"| E{"Listed in ignore_imports?"}
  E -->|"no"| F["Fail the lint job"]
  E -->|"yes"| OK
  M["Each move PR"] -->|"removes the entries it resolves"| E
  classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937
  class L,E this
Loading
  • Imports inside functions count too.
  • An ignore_imports entry that matches no import fails the check. A new entry does not, nor one whose import no longer points up, so review of the list is what keeps it from growing.
  • A package with a submodule that moves elsewhere is listed as its other submodules, so the __init__.py of cli, sandbox, _checkpoint, observability, and tasks is unchecked until their moves land. Each new module inside them is checked.

The 26 starting exceptions

Upward import today Resolved when
Responses API types from openai_utils (trajectory, server, and verifier modules) the types move down into core/
global_config imports checkpoint settings, exporters, the sidecar config, the sandbox registry, telemetry setup those move to feature-owned config, or their packages land
judge and two sandbox providers import the HTTP client from server_utils the client moves down into client/
Two sandbox modules import the Harbor digest rules the rules move to sandbox/
tasks/harbor/cli.py imports the CLI and rollout collection the Harbor handlers move into cli/
tasks/harbor/run_summary.py imports the health file names the names move down into artifacts/
rollout_correlation imports episode_types both move into core/episode.py
telemetry/setup.py imports sandbox.observability sandbox reads the rollout ID from core/
Exporters import rollout row keys and the aggregation error key the keys move down into artifacts/
comparison/loading.py imports did_you_mean from cli/utils.py it moves into utils/text.py (#5195)

Keeping the contract complete

A module outside every layer is not checked at all, and two kinds of import go around the contract. Unit tests read the contract from pyproject.toml and check three things:

  • Every module under nemo_gym/ is in a layer, or on a short list kept outside on purpose, each with its reason: the package root, package_info, the compatibility table _moved_paths, and verifiers, the standard-library-only package sandbox verifiers import by name. A new module, at the top level or inside a package the contract lists by submodule, has to join a layer before it merges.
  • Code inside nemo_gym/ imports no name through a package that is in no layer itself, such as the root, cli, or sandbox, unless the package defines the name. The contract records such an import as an import of the package alone. The test found four imports through nemo_gym.sandbox, in sandbox/shell.py, sandbox/transfer.py, and agent_utils/supervisor_client.py, which now import from the submodules that define the names.
  • Code inside nemo_gym/ imports nothing through an old path in _moved_paths.py, which has no file behind it. The table is empty here, so this check starts to apply in refactor(config): move config modules into nemo_gym/config #5189.

Validation

  • lint-imports and pre-commit run import-linter --all-files pass.
  • Adding an import of nemo_gym.rollout_collection inside a function in nemo_gym/failure_kinds.py fails the contract.
  • The coverage tests fail for a new module in each package the contract lists by submodule, for a new top-level module, for a directory without __init__.py, and for from nemo_gym import ServerClient or from nemo_gym.cli import RunHelper inside nemo_gym/.
  • pytest on the coverage tests and the sandbox tests, and pre-commit run --all-files.
  • In the move PRs above, scripts/migrate_imports.py renames the contract's modules and the contract keeps passing.

Rollout evidence: N/A. A CI check only.

Compatibility

No user-visible change. A contributor whose change adds an upward import gets a pre-commit failure.

Issue

Part of #5166.

@copy-pr-bot

copy-pr-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ananthsub
ananthsub added this pull request to stack #5170 October 28, 2026 00:08
@ananthsub ananthsub added the area:ci CI workflows, test routing, and repository automation label Oct 28, 2026
@ananthsub ananthsub self-assigned this Oct 28, 2026
@ananthsub ananthsub added the ci CI, automation, test queue, or workflow infrastructure work label Oct 28, 2026
@ananthsub
ananthsub force-pushed the ananthsub/layout-import-linter branch from 7736e18 to 73dd99f Compare October 28, 2026 14:38
@ananthsub
ananthsub force-pushed the ananthsub/layout-import-linter branch 2 times, most recently from 7fe8c49 to c046caa Compare October 28, 2026 16:54
@ananthsub
ananthsub force-pushed the ananthsub/layout-import-linter branch from c046caa to b912a64 Compare October 28, 2026 17:41
@ananthsub
ananthsub force-pushed the ananthsub/layout-import-linter branch from b912a64 to 8c0f809 Compare October 28, 2026 17:45
@ananthsub
ananthsub marked this pull request as ready for review October 28, 2026 17:47
@ananthsub
ananthsub requested a review from a team as a code owner October 28, 2026 17:47
@ananthsub
ananthsub requested a review from yaoyu-33 October 28, 2026 17:47
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 28, 2026 17:48
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Oct 28, 2026

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

LGTM — no concerns found.

Reviewed both changed files, including the hook configuration, layer contract and exceptions, and affected CI, package initialization, compatibility, and import paths. The existing lint entrypoint runs all hooks, and container hook-cache seeding includes the new dependency.

Static review only; tests and lint were not executed. Import-linter implementation source was unavailable, so its runtime semantics were not independently verified.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@ananthsub
ananthsub force-pushed the ananthsub/layout-import-linter branch from 8c0f809 to 9ed186e Compare October 28, 2026 19:01
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 28, 2026 19:03
@ananthsub
ananthsub force-pushed the ananthsub/layout-import-linter branch from 4099dc2 to 8515f8e Compare October 28, 2026 21:23
@ananthsub
ananthsub force-pushed the ananthsub/layout-import-linter branch from 8515f8e to 7199668 Compare October 28, 2026 21:56
@ananthsub
ananthsub force-pushed the ananthsub/layout-import-linter branch from 7199668 to 02b4520 Compare October 29, 2026 00:54
@ananthsub
ananthsub force-pushed the ananthsub/layout-import-linter branch from 02b4520 to 10df41f Compare October 29, 2026 14:04

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. A layering ratchet in front of the moves is the right way to keep the cycle from growing while the stack lands, and the contract reads clearly next to the plan in #5166.

What we checked and found working

  • lint-imports reports the contract KEPT at this head, and the Lint check on this head ran the hook ("Check package layering (import-linter)...Passed"). The offline image seeds hook environments with pre-commit install-hooks from the same config, so the new hook needs nothing extra there.
  • Upward imports are caught at module level, inside functions, under if TYPE_CHECKING:, and as relative imports. Indirect chains through unlayered modules are caught too.
  • A stale ignore_imports entry and a missing layer module both fail the check. : keeps modules in one group free to import each other. All 26 exceptions are exact module pairs that point up today, and they match the table in the description.
  • The new tests pass in the core unit-test job. The red unit-test checks on this head are Docker Hub pull-rate limits (HTTP 429 in the Harbor image tests and the skillsbench agent tests). #5167's head shows the same failures, so this PR did not cause them.

Inline (two NOTEs)

  • New modules inside the five packages that the contract lists by submodule (cli, sandbox, _checkpoint, observability, tasks) land in no layer, and the coverage test cannot see them. Each package's move PR would surface them later, but the description promises they are checked now. The inline comment has a replacement test that we ran.
  • Imports through the lazy exports of the unlayered nemo_gym and nemo_gym.cli roots never reach the graph.

Not blocking

  • Order of artifacts/: the contract puts it just below server_runtime/, while the plan in #5166 puts it just above core/. The contract's order is the one the code supports, because exporters import atif_export. It may be worth updating the issue so CODEOWNERS and reviewers work from the same order.
  • "This list may only shrink": the tool enforces this only for stale entries. Nothing stops a PR from adding an entry, and an entry that still matches an import but no longer points up stays silently. #5206 adds client.server_client -> _checkpoint.settings even as it removes others. Review of pyproject.toml is the real gate, which the comment could say.
  • Later in the stack (#5189 / #5195): once a module moves, an import through its old path has no file behind it, so grimp drops the edge. On #5195's head, from nemo_gym.path_utils import ... inside nemo_gym/config/secret_utils.py stays KEPT, while the same import through nemo_gym.utils.path_utils is flagged. A test that rejects _moved_paths keys inside nemo_gym/ would close this. It fits best in #5189, the first move, and could go with the lazy-name test from the inline NOTE.
  • Validation list: the description lists the lint-imports and hook runs. It could also list the new unit tests and the full pre-commit run --all-files, which CI ran green.

Verdict: SHIP. The contract works as described for every module that is in a layer. Both notes are about modules and import routes outside the layers. Taking the suggested test and correcting the description's coverage sentence would make the guarantee match what is checked.

yaoyu-33
yaoyu-33 previously approved these changes Oct 29, 2026

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The rebased head (1c15aac) carries the same commit as the one we reviewed (43c1b6b); only the #5167 base under it moved. The Lint check on this head runs the new hook and passes ("Check package layering (import-linter)...Passed").

The two inline notes from the earlier review are not blocking. The failing server suite shard 0 is the Docker Hub pull-rate limit (toomanyrequests), not this change.

@ananthsub

Copy link
Copy Markdown
Contributor Author

On the review's other notes:

yaoyu-33
yaoyu-33 previously approved these changes Oct 29, 2026

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving at 91efec2. The new coverage test closes the split-package gap: every module under nemo_gym/ must be in a layer, and an import through an unlayered package's re-export or through an old path now fails. The three sandbox imports now name the modules that define those names. The failing checks are the non-blocking harness conformance job.

Add an import-linter layers contract that orders today's nemo_gym
modules by the package each one moves to, so CI rejects any new import
that points up the target order, including imports inside functions.

The 26 imports that point up the order today are listed as exceptions.
Each goes away with the split or move that resolves it, and the list may
only shrink. The check runs as a pre-commit hook, which the lint job
already runs.

A unit test checks that every top-level module and package of nemo_gym
is in a layer of the contract, or listed with the reason it stays
outside, so a new top-level module cannot go unchecked.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — d937e34b Deployed Oct 30, 2026 by copy-pr-bot[bot] via release / finalize / notify #8624
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ci CI workflows, test routing, and repository automation ci CI, automation, test queue, or workflow infrastructure work ready-to-merge PR is approved, current, and waiting for final CI or merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants