Skip to content

fix(engine): honor ATOM_TORCH_PROFILER_DIR - #2925

Open
yelkhamra wants to merge 5 commits into
ROCm:mainfrom
yelkhamra:amd-automation/atom-profiler-dir-none
Open

yelkhamra wants to merge 5 commits into
ROCm:mainfrom
yelkhamra:amd-automation/atom-profiler-dir-none

Conversation

@yelkhamra

@yelkhamra yelkhamra commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #2911

Motivation

Summary

Let an omitted --torch-profiler-dir use ATOM_TORCH_PROFILER_DIR, while preserving explicit CLI values.

Rebased onto current main 004f342ab22dedbef6d476713eeabfe9dbdadcbb. Against that base, the environment-only case fails with the unmodified atom/model_engine/arg_utils.py, and the empty-environment case fails with the unmodified atom/utils/envs.py; both pass with this change. The rebase's only conflict was docs/environment_variables.md, where main had rewritten the Engram rows; the resolution keeps main's rows and wording and keeps this branch's single added sentence on the ATOM_ENGRAM_CACHE_DIR row. No rebase in the latest rounds: the base is still 004f342ab, the branch merges cleanly into upstream main ac9664173, and the latest round adds one comment-only commit (4954cac80) on top of the test-only ab725c857, itself on top of d9dd55bc0.

Review follow-up in the same pull request:

  • An empty ATOM_TORCH_PROFILER_DIR= now reads as unset in atom/utils/envs.py instead of reaching Config as "" and aborting startup on os.makedirs("").
  • The ATOMesh launcher no longer exports a profiler directory on every launch. The export existed for the online-quant dump, which got its own ATOM_ONLINE_QUANT_INFO_DIR knob so its artifacts stay where they were.
  • The regression test now drives the real CLI -> EngineArgs.create_engine() -> LLMEngine -> Config hand-off, with unset, empty and set environment values.
  • The offline profiling example now resolves its trace directory the same way: --torch-profiler-dir, else ATOM_TORCH_PROFILER_DIR, else its own ./profiler_traces. It always profiles, so the chain ends at a directory rather than None.
  • The empty-value rule now covers the sibling directory variables too: an empty ATOM_ENGRAM_CACHE_DIR reads as its default cache path instead of "" (which Path resolves to the working directory), and an empty ATOM_RTL_TRACE_DIR reads as None, like ATOM_TORCH_PROFILER_DIR.
  • Latest review round: ATOM_ONLINE_QUANT_INFO_DIR reports only its own variable and the fallback to ATOM_TORCH_PROFILER_DIR then the working directory is written once in the loader; the profiling example is a single or chain plus a print (an empty flag reads as unset there because "" is falsy, and the comment says that rule is example-specific); the no-op ATOM_V4_CG_INSTR_DIR edit is reverted.
  • Round after that (review 5908313797): the online-quant directory test loads the loader through aiter_stub.atom_without_aiter, like the other stand-in tests, instead of its own try/except ImportError fallback. That helper keeps the real AITER kernels only where a GPU can run them: an AITER build on a GPU-less host imports, because its arch probe is deferred, and then fails inside that probe, so the stand-ins are taken there too.
  • Latest round (review 5909240391): the ATOM_RTL_TRACE_DIR comment now describes its actual consumer instead of claiming an empty value disables tracing. No Python code consumes the setting; .github/scripts/atom_test.sh uses it as the rtl trace -o output directory under ENABLE_RTL_PROFILER=1, and an empty value falls back to /app/rtl_traces there. The _optional_path_env reader stays, and the empty-reads-as-unset note is now scoped to Python, like the other directory knobs. Comment-only change; no behavior changes.

Problem

On the base revision, a CLI launch with only ATOM_TORCH_PROFILER_DIR set produces Config.torch_profiler_dir=None, leaving the runner without a profiler output directory.

From source reading: in this state the runner skips profiling because its profiler directory is None, while /start_profile and /stop_profile still return success, so no trace is written.

Technical Details

Root cause

The EngineArgs field and CLI option default to None; _get_engine_kwargs() forwards that value, and LLMEngine passes it into Config, bypassing the field's environment-backed default_factory.

Evidence

  • The profiling documentation describes the environment variable as the default and allows it as an alternative to the CLI flag. Its offline-example section says the script defaults to ./profiler_traces, which is what the script did whether or not the variable was set.
  • In tests/test_arg_utils_profiler.py, the environment-only case fails against the unmodified atom/model_engine/arg_utils.py and passes with this patch; the other cases pass in both runs.
  • The empty-environment case fails against the unmodified atom/utils/envs.py with FileNotFoundError: [Errno 2] No such file or directory: '' and passes with the change.
  • atom/examples/profile_offline.py replaced an omitted flag with ./profiler_traces before EngineArgs saw it, so the environment variable could not apply there. tests/test_profile_offline.py drives main() over seven flag/environment combinations; with the environment step removed, its two environment cases fail.
  • With the pre-change envs.py, ATOM_ENGRAM_CACHE_DIR= reads as "" and Path("") / "compressed_vocab_<hash>.npz" is compressed_vocab_<hash>.npz in the working directory (atom/model_ops/engram/mapping.py:139).
  • ATOM_ONLINE_QUANT_INFO_DIR used to return ATOM_TORCH_PROFILER_DIR's value when unset, so the attribute did not name what it returned. It now returns its own variable, and tests/test_online_quant_info_dir.py pins the effective directory at the loader's call site instead.

Fix

  • atom/model_engine/arg_utils.py: remove only the torch_profiler_dir keyword when its EngineArgs value is None.
  • atom/utils/envs.py: add _optional_path_env (empty -> None) and _path_env (empty -> named default), and read the directory knobs through them: ATOM_TORCH_PROFILER_DIR, ATOM_ONLINE_QUANT_INFO_DIR and ATOM_RTL_TRACE_DIR -> None; ATOM_ENGRAM_CACHE_DIR -> ~/.cache/atom/engram. ATOM_V4_CG_INSTR_DIR is unchanged: it already read an empty value as None.
  • atom/utils/envs.py, atom/model_loader/loader.py: add ATOM_ONLINE_QUANT_INFO_DIR for the rank-0 online_quant_info_*.json dump. The knob is its own variable only; the loader's single call site reads ATOM_ONLINE_QUANT_INFO_DIR or ATOM_TORCH_PROFILER_DIR or os.getcwd().
  • atom/examples/profile_offline.py: resolve the trace directory as --torch-profiler-dir or ATOM_TORCH_PROFILER_DIR or DEFAULT_PROFILER_DIR, store it before EngineArgs is built, and print it. An empty flag reads as unset here because "" is falsy; that rule is specific to this example, since EngineArgs forwards "" to Config and os.makedirs("") aborts.
  • .github/scripts/atomesh/pd_server_atom.sh: stop exporting ATOM_TORCH_PROFILER_DIR on every launch; export ATOM_ONLINE_QUANT_INFO_DIR to the same ${RUN_DIR}/online_quant.../rank-N path instead. The profiling window still passes a per-worker --torch-profiler-dir.
  • tests/test_arg_utils_profiler.py, tests/test_profile_offline.py, tests/test_envs.py, tests/test_online_quant_info_dir.py: cover the hand-off, the example's resolution, the empty-value rule for the directory variables, and the online-quant dump's effective directory.
  • tests/aiter_stub.py: atom_without_aiter takes the stand-ins unless a GPU is available. An AITER build on a GPU-less host imports, because its arch probe is deferred, and then fails inside that probe, so "AITER imports" is not "AITER can run here"; a GPU host and a runner without AITER behave as before. tests/test_online_quant_info_dir.py loads the loader through it.

Why it works

Omitting the keyword lets Config evaluate its existing lazy environment default, giving precedence to an explicit CLI path, then the environment variable, then None. Reading the environment value with or keeps an empty value from being mistaken for a directory. The example applies the same order before it builds EngineArgs, so its own default applies only when neither the flag nor the variable is set. The online-quant dump keeps its historical home because its call site keeps the same fallback, without making the knob return another variable's value.

Risk / compatibility

Default or explicit None values supplied through EngineArgs now inherit the environment setting, including Config's existing directory creation and validation, per-rank directories, and prefill/decode directories when P/D disaggregation is enabled.

An empty ATOM_TORCH_PROFILER_DIR= now behaves exactly like an unset one. Before this change the CLI silently ignored it; without the envs.py part of the patch it aborts startup on os.makedirs("").

Direct Config and LLMEngine calls supplying torch_profiler_dir=None retain their current semantics.

With --mark-trace also set, the existing graph-capture profiler can now activate when the directory comes from the environment.

The offline profiling example now follows the engine's order and ends at ./profiler_traces: with the variable set it writes where the variable points instead of always to its own default. An explicitly empty --torch-profiler-dir counts as not given there, like an empty environment value, because the example always needs a directory. It prints the directory it resolved instead of a per-source notice.

The sibling directory variables change only when set to the empty string: ATOM_ENGRAM_CACHE_DIR= now uses the documented cache path rather than the working directory, and ATOM_RTL_TRACE_DIR= now reads as unset. The latter has no Python consumer today; .github/scripts/atom_test.sh reads the shell variable itself, where ${ATOM_RTL_TRACE_DIR:-/app/rtl_traces} already treated empty as unset. The debug-dump directories (ATOM_FWD_DUMP_DIR, ATOM_WEIGHT_DUMP_DIR, ATOM_DEBUG_TOPK_PATH) keep the empty string as their off switch and are unchanged.

Plugin configuration still supplies explicit None and is unchanged; plugin-mode profiling remains vLLM's profiler_config.

The ATOMesh launcher gives the engine a profiler directory only while its profiling window is on. Its online-quant dump now names ATOM_ONLINE_QUANT_INFO_DIR, but lands in the same directory as before, so collected artifacts do not move.

Related #2112 proposes profiler-window controls; coordinate its argument/config changes with this environment-default fix.

Test Plan

GPU validation plan

None needed for this argument/default fix: the CPU regression reproduces the failure and verifies the resolved configuration.

GPU trace emission was not tested. Heavy CI with ci:full ran at ab725c857:
23 of 24 accuracy cells pass at that head. The Accuracy (Kimi-K3 DSpark DCP8 TP8) cell failed on a vLLM readiness timeout (40 x 30 s) after weight loading
had finished, with no traceback; its product tree is identical to d9dd55bc0,
where the same cell passed, so a maintainer re-run is requested and merge is
gated on it and on ATOM vLLM Test Gate. ci:full does not run the ATOMesh
model benchmark on PRs; the pd_server_atom.sh change is covered by the
test_atomesh_* unit tests, which pass.

CPU: the regression tests and lint commands are listed under Test Result.

Test Result

tests/test_arg_utils_profiler.py drives the same hand-off as the server and the offline examples: EngineArgs.add_cli_args -> EngineArgs.from_cli_args -> EngineArgs.create_engine() -> LLMEngine -> Config, then asserts engine.config.torch_profiler_dir and that Config created the directory.

It covers the environment unset, empty and set, each without and with --torch-profiler-dir (six cases), plus an explicitly empty CLI string still being forwarded rather than replaced by the environment. Only Config's checkpoint and quantization lookups and the engine's tokenizer and CoreManager are mocked; argument forwarding, LLMEngine.__init__ and Config initialization run for real.

tests/test_profile_offline.py drives the example's main() up to EngineArgs.from_cli_args over seven combinations: flag and variable absent, variable set, flag set, both set, empty variable, empty flag, and empty flag with the variable set. It pins the directory handed to EngineArgs and the printed directory.

tests/test_envs.py covers the empty-value rule for ATOM_TORCH_PROFILER_DIR, ATOM_ENGRAM_CACHE_DIR and ATOM_RTL_TRACE_DIR, and pins that ATOM_ONLINE_QUANT_INFO_DIR reads only its own variable (unset and empty, with ATOM_TORCH_PROFILER_DIR set).

tests/test_online_quant_info_dir.py calls atom.model_loader.loader._save_online_quant_info, loaded through aiter_stub.atom_without_aiter (the real kernels on a GPU host, the stand-ins where AITER is absent or cannot run), and pins the effective output directory: own knob, else ATOM_TORCH_PROFILER_DIR (unset and empty own knob), else the working directory; and that a rank other than 0 writes nothing.

CPU environment: a local ATOM CPU container, Python 3.12.3, pytest 9.0.3.

python -m pytest -q tests/test_profile_offline.py tests/test_envs.py tests/test_online_quant_info_dir.py tests/test_arg_utils_profiler.py

With the patch: 136 passed.

The unit-test job's own condition is a runner with no AITER at all. This image has an editable AITER build, so that condition was reproduced by blocking AITER imports with a sitecustomize.py on PYTHONPATH: import aiter then raises ModuleNotFoundError, and tests/test_online_quant_info_dir.py passes (5 passed). On this image as it is, the same file passes through the new GPU check in atom_without_aiter; with the helper's old import aiter gate it errors on all five cases inside AITER's arch probe. Both runs are in CHECKS.md, together with the twelve other files that use the helper going from 189 errors to 0 on this image.

Unmodified atom/model_engine/arg_utils.py plus the new tests: only the environment-only case fails.

FAILED tests/test_arg_utils_profiler.py::test_torch_profiler_dir_handoff[env-None-env]

Unmodified atom/utils/envs.py: the empty-environment hand-off fails on os.makedirs(""), and the new env-var tests fail because the knob does not exist yet.

FAILED tests/test_arg_utils_profiler.py::test_torch_profiler_dir_handoff[-None-None]
FileNotFoundError: [Errno 2] No such file or directory: ''

The two sibling lambdas reverted to their pre-change spelling: exactly the two empty-value tests fail.

FAILED tests/test_envs.py::TestEnvsOverrides::test_empty_engram_cache_dir_reads_as_its_default
FAILED tests/test_envs.py::TestEnvsOverrides::test_empty_rtl_trace_dir_reads_as_unset

The example with its environment step removed: exactly the two environment cases fail.

FAILED tests/test_profile_offline.py::test_main_resolves_profiler_dir[None-/tmp/env-traces-...]
FAILED tests/test_profile_offline.py::test_main_resolves_profiler_dir[-/tmp/env-traces-...]

The loader without the two fallback terms: exactly the two fallback cases fail.

FAILED tests/test_online_quant_info_dir.py::test_unset_knob_falls_back_to_the_profiler_dir[None]
FAILED tests/test_online_quant_info_dir.py::test_unset_knob_falls_back_to_the_profiler_dir[]

envs.ATOM_ONLINE_QUANT_INFO_DIR restored to the old chained lambda: exactly the knob tests fail.

FAILED tests/test_envs.py::TestEnvsOverrides::test_online_quant_info_dir_reads_only_its_own_variable[None]
FAILED tests/test_envs.py::TestEnvsOverrides::test_online_quant_info_dir_reads_only_its_own_variable[]

Related suites:

python -B -m pytest -q tests/test_arg_utils_profiler.py tests/test_arg_utils_spec.py tests/test_envs.py tests/test_profile_offline.py tests/test_online_quant_info_dir.py tests/test_v4_indexer_defaults.py tests/entrypoints/test_api_server_helpers.py tests/test_pp.py

328 passed, 1 skipped.

Full native CPU suite, using the CI exclusions for plugin tests and GPU server integration, run on this branch and on a clean worktree of 004f342ab22dedbef6d476713eeabfe9dbdadcbb:

python -B -m pytest -q tests/ --ignore=tests/plugin --ignore=tests/entrypoints/test_openai_server.py --continue-on-collection-errors
patched: 117 failed, 11430 passed, 1460 skipped, 105 warnings,
         83 errors, 222 subtests passed in 674.22s (0:11:14)
base:    121 failed, 11351 passed, 1460 skipped, 105 warnings,
         128 errors, 222 subtests passed in 640.23s (0:10:40)

The suite is not green in this CPU-only image for pre-existing reasons
(torch.finfo() collection errors and No gfx arch found). The branch's
200-entry FAILED/ERROR set is a strict subset of the base's 249: nothing
fails on the branch that passes on the base. The 49 entries only the base has
are 45 in tests/test_deepseek_v41_mono_hi_gate.py and 4 in
tests/test_deepseek_v41_mono_hi_rope.py, both of which use
atom_without_aiter; on this GPU-less image with an editable AITER build, the
helper's old import aiter gate handed them the real kernels, and AITER's arch
probe failed, while the stand-ins load as they do on the unit-test runner. The branch also passes the 30 tests this PR
adds (11430 = 11351 + 49 + 30: 7 in tests/test_arg_utils_profiler.py, 7 in
tests/test_profile_offline.py, 5 in tests/test_online_quant_info_dir.py
and 11 in tests/test_envs.py). Collection with the same flags shows only that
delta, 12,934 tests against 12,904, with 76 collection errors on both.

Black and Ruff on the nine changed Python files (atom/examples/profile_offline.py, atom/model_engine/arg_utils.py, atom/model_loader/loader.py, atom/utils/envs.py, tests/aiter_stub.py, tests/test_arg_utils_profiler.py, tests/test_envs.py, tests/test_profile_offline.py, tests/test_online_quant_info_dir.py):

9 files would be left unchanged.
All checks passed!

Latest round (4954cac80, comment-only): tests/test_envs.py passes (117 passed), the eight related suites used in the previous round still pass (328 passed, 1 skipped), the test_atomesh_* unit tests pass (204 passed, 2 skipped), and Black and Ruff are clean on atom/utils/envs.py. The full native CPU suite reproduces the previous head's counts and its exact 200-entry FAILED/ERROR set (117 failed, 11430 passed, 1460 skipped, 83 errors). No test can distinguish a comment-only change, so the comment's claims were checked against its consumer instead: no Python module outside tests reads the variable, and both reads in atom_test.sh fall back to /app/rtl_traces for an unset or empty value (see CHECKS.md).

Submission Checklist

Omit an unspecified profiler directory from EngineArgs kwargs so Config
can resolve its existing environment default. Explicit CLI values retain
precedence, and direct Config callers keep their current semantics.

Read an empty ATOM_TORCH_PROFILER_DIR as unset in envs, so `VAR=` clears
it instead of aborting startup on os.makedirs(""). Give the online-quant
dump its own ATOM_ONLINE_QUANT_INFO_DIR, falling back to the profiler
directory, and stop the ATOMesh launcher exporting a profiler directory
on every launch now that the engine honors it.

Cover the hand-off through EngineArgs.create_engine() -> LLMEngine ->
Config with the environment unset, empty and set, and with and without
an explicit --torch-profiler-dir.

Signed-off-by: Yaakoub El Khamra <176948988+yelkhamra@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2925 --add-label <label>

@yelkhamra
yelkhamra marked this pull request as ready for review November 2, 2026 20:24

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed..

ci Offline inference smoke test failed — not sure yet (low): The captured log is empty (0 bytes), so the failing step cannot be identified. The job succeeded on main and on other recent runs, and this PR only changes arg_utils plus a new test, but that is not enough to attribute the failure.

next: Re-run the job and fetch its log, then classify it. https://github.com/ROCm/ATOM/actions/runs/41518846782/job/125529318605

f.name: getattr(self, f.name) for f in fields(self) if f.name != "model"
}
# Let Config resolve ATOM_TORCH_PROFILER_DIR when no path was supplied.
if self.torch_profiler_dir is None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empty ATOM_TORCH_PROFILER_DIR= now crashes startup, os.makedirs('').. read empty as unset in envs instead?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done as suggested: ATOM_TORCH_PROFILER_DIR now reads
os.getenv("ATOM_TORCH_PROFILER_DIR") or None in atom/utils/envs.py:771-774,
so ATOM_TORCH_PROFILER_DIR= is indistinguishable from unset and never reaches
Config as "". Covered by
tests/test_envs.py::TestEnvsOverrides::test_empty_torch_profiler_dir_reads_as_unset
and by test_torch_profiler_dir_handoff[-None-None], which drives the same
hand-off as the server. With the envs change reverted, that case fails with
FileNotFoundError: [Errno 2] No such file or directory: ''.

}
# Let Config resolve ATOM_TORCH_PROFILER_DIR when no path was supplied.
if self.torch_profiler_dir is None:
kwargs.pop("torch_profiler_dir")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the workaround comment in pd_server_atom.sh is stale now, and it exports a dir on every launch.. please sweep it here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Swept. .github/scripts/atomesh/pd_server_atom.sh no longer exports
ATOM_TORCH_PROFILER_DIR (lines 350-362), and the stale comment is replaced at
1948-1951 with what the per-worker flag does now.

The export had a second reader: _save_online_quant_info wrote rank 0's
online_quant_info_*.json under ATOM_TORCH_PROFILER_DIR when set. Removing
the export alone would have moved those dumps out of the results tree, so the
dump got its own knob, ATOM_ONLINE_QUANT_INFO_DIR (atom/utils/envs.py:891-900,
atom/model_loader/loader.py:180-182), which falls back to
ATOM_TORCH_PROFILER_DIR and then the working directory. The launcher exports it
to the same ${RUN_DIR}/online_quant[/${ATOMESH_EXECUTION_PHASE}]/rank-N path as
before, so the artifacts do not move.

The profiler itself is unchanged: only ATOMESH_TORCH_PROFILE_DURATION > 0 puts
--torch-profiler-dir on a launch, one directory per worker, and that flag still
outranks the environment default.

Comment thread tests/test_arg_utils_profiler.py Outdated
kwargs = _engine_kwargs(argv)

# Match LLMEngine's filtering and run Config's real directory validation.
config_fields = {f.name for f in fields(config_module.Config)}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this re-implements the Config filter, not the real hand-off.. can we drive LLMEngine path once and add the empty-env case?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworked. tests/test_arg_utils_profiler.py now goes
CLI -> EngineArgs.add_cli_args/from_cli_args -> EngineArgs.create_engine()
-> LLMEngine -> Config (the call openai_server and the examples use) and
asserts engine.config.torch_profiler_dir, plus that Config created the
directory. The field.name filtering is gone. Cases: env unset / empty / set,
each with no CLI path and with one; the empty-environment case is included. The
explicit empty CLI string is still checked at the kwargs boundary, since it does
not select the environment default.

@zufayu
zufayu requested review from valarLip and yitingw1 November 3, 2026 01:11
@yelkhamra
yelkhamra force-pushed the amd-automation/atom-profiler-dir-none branch from 90ac6b1 to 37b90a6 Compare November 5, 2026 19:36
@yelkhamra

Copy link
Copy Markdown
Contributor Author

@valar-cop thanks for the review. On the CI failure:

Classified as infra, not caused by this diff. The job's only annotation is
"The self-hosted runner lost communication with the server"; it got stuck in
the Start CI container step, which started at 20:26:55Z and never completed
(the job was killed at 20:39:53Z), so it never reached Run offline inference
and no ATOM code from the job ran. GitHub stored no log blob for it (the
job-log endpoint returns 404; the run's log archive holds only the
Check Heavy CI Gate job). The same commit had already passed this job on
...-runner-mc967 in the first run of the head (19:48Z), and passed it again on
...-runner-clsdw at 2026-11-03 01:59Z; the failure was a later run of the
same head, not the first. A different PR's identical job started at 20:26:53Z
on ...-runner-wrcp6 and passed at 20:31:35Z, while this one was still stuck.
I did not re-run it: none of the job's steps that would run this branch's code
ever started.

@valar-cop

valar-cop commented Nov 5, 2026 •

Copy link
Copy Markdown

@yitingw1

yitingw1 commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Read the production side in full (the 20 changed lines in arg_utils.py / envs.py / loader.py, plus the launcher diff) and ran the two test modules. The fix is right and the tests pin it: four mutations, each reddening exactly the cases it should, against a 120-passed baseline.

mutation red
drop the kwargs.pop test_torch_profiler_dir_handoff[env-None-env]
pop unconditionally (CLI value dropped too) the three [*-cli-cli] cases + test_empty_cli_directory_is_not_treated_as_none
revert envs.py to os.getenv(..., None) [-None-None] + test_empty_torch_profiler_dir_reads_as_unset
swap the ATOM_ONLINE_QUANT_INFO_DIR fallback order test_online_quant_info_dir_beats_profiler_dir

Two things, neither blocking. Not repeating valar-cop's three inline points — all look addressed.

1. atom/examples/profile_offline.py defeats this fix, in the one script whose purpose is to demonstrate profiling. :48 fills the default before from_cli_args:

if args.torch_profiler_dir is None:
    args.torch_profiler_dir = "./profiler_traces"
engine_args = EngineArgs.from_cli_args(args)

So on that path the precedence is CLI > ./profiler_traces, never CLI > env > None. Someone who sets ATOM_TORCH_PROFILER_DIR and runs the profiling example gets traces somewhere other than where every other entry point now puts them. The body covers it as "unchanged", but in a PR about the environment variable not being honored, leaving this one reads as an oversight rather than a decision.

→ Change: args.torch_profiler_dir = args.torch_profiler_dir or envs.ATOM_TORCH_PROFILER_DIR or "./profiler_traces" (and adjust the warning), or say in the description why this example deliberately overrides it.

2. The new "empty reads as unset" rule has two siblings that don't follow it. In envs.py:

variable read as empty value
ATOM_TORCH_PROFILER_DIR os.getenv(...) or None fixed here
ATOM_V4_CG_INSTR_DIR os.getenv(...) or None already correct
ATOM_ENGRAM_CACHE_DIR os.getenv(X, <default path>) ""
ATOM_RTL_TRACE_DIR os.getenv(X, None) ""

ATOM_ENGRAM_CACHE_DIR= does not crash the way the profiler one did — mapping.py:139 is Path(cache_dir or envs.X) / ... and Path("") is . — but it silently writes the compressed-vocab cache into the working directory instead of the cache directory. ATOM_RTL_TRACE_DIR has no consumer under atom/, so it's inert today.

→ Change: give both the same treatment (or None, or os.getenv(X) or <default> for the engram one) so the convention is one rule, or note in the description that it is deliberately scoped to the profiler variable.

Checked and found not to be issues, listing them so nobody re-derives them:

  • Is this one of N instances of the same bug? No — torch_profiler_dir is the only field in atom/config.py with an envs-backed default_factory (:2124), so the class is fully covered.
  • Does popping the key break other callers? No — _get_engine_kwargs has one production caller, arg_utils.py:966.
  • Does anything else in the repo export this variable, and would it now silently enable profiling? Only pd_server_atom.sh, which this PR removes — and had to, since after the fix that export would hand every ATOMesh launch a profiler directory. Good catch on the second reader (_save_online_quant_info).
  • Is the vLLM plugin's explicit torch_profiler_dir=None the same bug? No — plugin-mode profiling is owned by vLLM's own profiler_config (see the recipes/atom_vllm/*.md invocations), so ignoring ATOM's variable there is the right split.
  • Do people who already have the variable exported now pay a profiling cost? Essentially no: model_runner.py:805 only sets self.profiler_dir and creates the per-rank directories; sampling still waits for /start_profile. The --mark-trace graph-capture case is the exception and the body already calls it out.

Not verified: no GPU run, so I did not confirm that traces actually land in the resolved directory — as the body says, that is outside what this change touches.

Honor ATOM_TORCH_PROFILER_DIR in the offline profiling example, which
replaced an omitted flag with its own default before EngineArgs saw it.
Resolve the way the engine does -- flag, then environment, then the
example's ./profiler_traces -- and name the source that supplied the
directory in the notice.

Read an empty ATOM_ENGRAM_CACHE_DIR as its default cache path rather
than "", which Path resolves to the working directory, and an empty
ATOM_RTL_TRACE_DIR as None, like ATOM_TORCH_PROFILER_DIR. State the rule
once in _optional_path_env / _path_env, which the profiler, online-quant
and ATOM_V4_CG_INSTR_DIR knobs now use as well.

Cover the example's resolution over seven cases, including the notice,
and the two sibling variables with default, override and empty-value
tests.

Signed-off-by: Yaakoub El Khamra <176948988+yelkhamra@users.noreply.github.com>
@yelkhamra
yelkhamra force-pushed the amd-automation/atom-profiler-dir-none branch from 37b90a6 to 5247b8b Compare November 6, 2026 16:51
@yelkhamra

Copy link
Copy Markdown
Contributor Author

@yitingw1 Thanks for reading the production side and for the mutation table; both points
are addressed in commit 5247b8b2c (the branch is rebased onto current main
b117d4c2d).

1. atom/examples/profile_offline.py now honors the variable.
resolve_profiler_dir() (atom/examples/profile_offline.py:48-69) resolves
--torch-profiler-dir, then envs.ATOM_TORCH_PROFILER_DIR, then
./profiler_traces, and main() stores the result before it builds
EngineArgs (atom/examples/profile_offline.py:75-77), so this path now
resolves CLI > environment > example default rather than CLI > default. The
notice names the source: "using ATOM_TORCH_PROFILER_DIR:

" when the
variable supplied it, and the previous "using default: ./profiler_traces"
text only when neither is set. An explicitly empty flag counts as not given,
like an empty variable: the example always profiles, so it never leaves ""
for Config to reject the way EngineArgs still would.

tests/test_profile_offline.py:22-65 pins the seven combinations, directory
and notice. Removing the environment lookup reddens exactly the two
environment cases and leaves the flag and default cases green. A run of
main() with only the engine and tokenizer stubbed showed
ENGINE_DIR=/tmp/env-traces for the environment case, ./profiler_traces
when the variable is unset or empty, and the flag value when it is given.

2. The sibling variables follow the same rule.
ATOM_ENGRAM_CACHE_DIR= now reads as ~/.cache/atom/engram
(atom/utils/envs.py:1331-1333) instead of "", which
Path(cache_dir or envs.ATOM_ENGRAM_CACHE_DIR) at
atom/model_ops/engram/mapping.py:139 would resolve to the working directory.
ATOM_RTL_TRACE_DIR= now reads as None (atom/utils/envs.py:866). Both go
through _optional_path_env and _path_env (atom/utils/envs.py:112-123),
which ATOM_TORCH_PROFILER_DIR, ATOM_ONLINE_QUANT_INFO_DIR and
ATOM_V4_CG_INSTR_DIR now use as well, so the rule is stated once rather than
respelled per knob.

Tests: defaults at tests/test_envs.py:120-126, overrides and empty values at
:236-254. Reverting just those two lambdas reddens exactly
test_empty_engram_cache_dir_reads_as_its_default ('' against the cache
path) and test_empty_rtl_trace_dir_reads_as_unset ('' instead of None).

I left the debug-dump directories (ATOM_FWD_DUMP_DIR,
ATOM_WEIGHT_DUMP_DIR, ATOM_DEBUG_TOPK_PATH) alone: there the empty string
is the documented off switch, so redirecting it would change what VAR=
means rather than what it names.

Your "checked and found not to be issues" list matches what I re-derived for
this change — torch_profiler_dir is the only EngineArgs-forwarded field
whose Config environment default can be shadowed (the other envs-backed
default_factory, kv_events_config at atom/config.py:2164, has no
EngineArgs field, so nothing forwards over it), _get_engine_kwargs's one
production caller, and the vLLM plugin keeping profiler_config for its own
profiling. Nothing else needed changing.

Same limitation as you noted: no GPU run, so traces landing in the resolved
directory is still unverified; this change is about which directory is
resolved.

@zufayu
zufayu requested review from yhl-amd and removed request for valarLip November 9, 2026 02:00
@zufayu zufayu assigned yhl-amd and unassigned valarLip Nov 9, 2026

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (4)

The core fix is right and still needed on main (arg_utils.py:878-880, with main's own pd_server_atom.sh:1945 comment documenting the bug). valar-cop's three asks and yitingw1's two points are addressed at head.

  1. Rebase onto current main. docs/environment_variables.md conflicts with main (merge-tree), so heavy CI has not run on this code since 37b90a6. Re-run it after the rebase.
  2. Keep the online-quant fallback in the loader, not in the env knob. envs.py:932-935 makes envs.ATOM_ONLINE_QUANT_INFO_DIR return ATOM_TORCH_PROFILER_DIR's value when the knob is unset, so the attribute's name does not match what it returns. Make the knob plain _optional_path_env("ATOM_ONLINE_QUANT_INFO_DIR") and write envs.ATOM_ONLINE_QUANT_INFO_DIR or envs.ATOM_TORCH_PROFILER_DIR or os.getcwd() at loader.py:182; then drop the restated chain in the loader comment.
  3. Fix the resolve_profiler_dir docstring. profile_offline.py:51-56 says this is "the precedence every other entry point uses", but an empty --torch-profiler-dir is treated as unset only here. EngineArgs forwards "" to Config, which fails on os.makedirs(""), as your own test_empty_cli_directory_is_not_treated_as_none pins. Say that the empty-flag rule is specific to this example.
  4. Trim the extras. The ATOM_V4_CG_INSTR_DIR edit (envs.py:1009) changes nothing, so revert it. The example only needs args.torch_profiler_dir or envs.ATOM_TORCH_PROFILER_DIR or DEFAULT_PROFILER_DIR plus the print, not a (dir, notice) tuple with a 9-line docstring.

Reviewed at 5247b8b. Already fixed: empty env reads as unset (valar-cop), launcher export swept (valar-cop), real LLMEngine hand-off test (valar-cop), profile_offline honors env and sibling dir knobs (yitingw1).

…ample

ATOM_ONLINE_QUANT_INFO_DIR reported ATOM_TORCH_PROFILER_DIR's value when its
own variable was unset, so the attribute did not name what it returned. Make
it a plain _optional_path_env of its own variable and write the fallback once
at its only call site, in the loader: the dump's directory, else the profiler
directory, else the working directory. Pin that effective directory in a new
loader test, and have the envs test assert the knob reads only its own
variable.

Reduce the offline profiling example to the resolution it needs: the flag, or
ATOM_TORCH_PROFILER_DIR, or the example's own directory, then a print of the
directory. The empty-flag rule is now a comment where it applies and states
that it is specific to this example, since EngineArgs forwards "" to Config.

Revert the ATOM_V4_CG_INSTR_DIR edit, which changed nothing: the knob already
read an empty value as None.

Rebased onto 004f342; docs/environment_variables.md resolves to main's
Engram rows plus this branch's single added sentence.

Signed-off-by: Yaakoub El Khamra <176948988+yelkhamra@users.noreply.github.com>
@yelkhamra
yelkhamra force-pushed the amd-automation/atom-profiler-dir-none branch from 5247b8b to d9dd55b Compare November 14, 2026 23:04
@yelkhamra

Copy link
Copy Markdown
Contributor Author

@valarLip

Asks 2–4 are in commit d9dd55bc0; the rebase and the docs/environment_variables.md resolution (ask 1) are in the rebased commits it sits on.

1. Rebase onto current main.
Rebased onto 004f342ab (git rebase origin/main). The only conflict was
docs/environment_variables.md: main had rewritten the Engram rows
(ATOM_ENGRAM_UVA_COPY in 1 GiB pieces, the new ATOM_ENGRAM_REG_LOCK) while
this branch adds one sentence to the ATOM_ENGRAM_CACHE_DIR row. Resolution
keeps main's rows and wording and appends that sentence, so the file is main's
plus this branch's three doc edits (two rows and the new
ATOM_ONLINE_QUANT_INFO_DIR one), with only the engram sentence inside the
conflict region. Heavy CI has not been re-run from here (the branch is not
pushed).

2. Online-quant fallback back in the loader.
ATOM_ONLINE_QUANT_INFO_DIR is now a plain
_optional_path_env("ATOM_ONLINE_QUANT_INFO_DIR") (atom/utils/envs.py), so
the attribute returns exactly its own variable. The fallback is written once,
at its only call site:
envs.ATOM_ONLINE_QUANT_INFO_DIR or envs.ATOM_TORCH_PROFILER_DIR or os.getcwd()
(atom/model_loader/loader.py:180-182), and the loader comment that restated
the chain is gone. Tests: tests/test_envs.py now pins that the knob reads
only its own variable (unset and empty, with the profiler variable set), and
the new tests/test_online_quant_info_dir.py drives
_save_online_quant_info to pin the effective directory: own knob, else the
profiler directory (unset and empty own knob), else the working directory, and
no file from ranks other than 0. Removing the two fallback terms from the
loader reddens exactly the two fallback cases.

3. resolve_profiler_dir docstring.
The function is gone with the trim in item 4. The empty-flag rule is now stated
where it applies, as a comment on the resolution in main()
(atom/examples/profile_offline.py:51-53): an empty flag counts as unset only
in this example, because EngineArgs forwards "" to Config and
os.makedirs("") aborts. tests/test_arg_utils_profiler.py::test_empty_cli_directory_is_not_treated_as_none
still pins that engine behaviour.

4. Extras trimmed.
The ATOM_V4_CG_INSTR_DIR line is back to os.getenv(...) or None; no
behaviour change, _optional_path_env remains used by the profiler,
online-quant and RTL directory knobs and _path_env by the Engram one. The
example is just
args.torch_profiler_dir or envs.ATOM_TORCH_PROFILER_DIR or DEFAULT_PROFILER_DIR
plus a print of the resolved directory (atom/examples/profile_offline.py:54-57);
no tuple, no per-source notice. An empty flag still reads as unset because
"" is falsy, which the seven cases in tests/test_profile_offline.py pin by
driving main() up to EngineArgs.from_cli_args.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (2)

The fix itself is done: all four r20 asks are addressed at head, and the change still belongs in ATOM. With ATOM_ONLINE_QUANT_INFO_DIR and ATOM_TORCH_PROFILER_DIR set, the knob now returns None when its own variable is unset or empty (probe). The fallback chain is written only in loader.py:180-182. ATOM_V4_CG_INSTR_DIR is back to main's line (envs.py:1043). The example is the single or chain at profile_offline.py:54-56, with a comment saying the empty-flag rule applies only to that script.

  1. Run heavy CI on the rebased head. Pre Checkin, which includes the non-GPU unit tests, and the offline smoke test pass at d9dd55b. The ATOM, vLLM and SGLang accuracy jobs were skipped because the PR has no label, so heavy CI has not run since 37b90a6. Add ci:atom (or ci:full, since pd_server_atom.sh changed) and merge once they pass. The branch still merges cleanly into main 004f342.
  2. Reuse atom_without_aiter in the new loader test (nit). tests/test_online_quant_info_dir.py:10-29 writes its own try/except ImportError fallback to the stubs. tests/aiter_stub.py:295 atom_without_aiter() already does this, and tests/model_ops/test_sp_attention_dispatch.py:15 and tests/test_deepseek_v41_mono_hi_runner.py:28 use it. Replace the helper with with atom_without_aiter() as load: return load("atom.model_loader.loader").

Re-reviewed at d9dd55b. Already fixed: rebase / docs conflict (r20 #1, CI still pending), online-quant fallback moved to loader (r20 #2), example comment scoped to the example (r20 #3), V4_CG_INSTR_DIR revert + example trimmed (r20 #4); valar-cop's three asks and yitingw1's two points still hold.

Replace the test's hand-written try/except ImportError fallback with
aiter_stub.atom_without_aiter, the helper the other stand-in tests use, so
the loader is loaded exactly as theirs is.

That helper gated on `import aiter`, which succeeds on a host with an AITER
build and no GPU: AITER defers its arch probe, and the submodule that binds a
kernel -- reached from the loader's import chain -- then fails on the missing
device. Take the stand-ins unless a GPU is available, the line
tests/conftest.gpu_stack_import already draws. A GPU host and the CI unit
runner (no AITER at all) keep the paths they had.

Signed-off-by: Yaakoub El Khamra <176948988+yelkhamra@users.noreply.github.com>
@yelkhamra

Copy link
Copy Markdown
Contributor Author

@valarLip

The branch is still based on 004f342ab22dedbef6d476713eeabfe9dbdadcbb; this
round is one new commit on top of d9dd55bc0 and no rebase.

1. Heavy CI on the rebased head.

Nothing to change in code, and the label is already there: ci:full is on the
PR (checked through the issue API). We cannot trigger Actions runs from here, so
the pushed head is what has to produce them: Pre Checkin (Black, Ruff, catalog
validation, non-GPU unit tests) plus the ATOM, vLLM and SGLang accuracy jobs.
The branch still merges cleanly into 004f342ab and its base is unchanged, so
those jobs test the same source as d9dd55bc0 except for the test-only commit
below. Locally, the non-GPU suite was re-run on both this head and a clean
worktree of 004f342ab; the results are in CHECKS.md. Merge only after the
heavy jobs are green.

2. Reuse atom_without_aiter in the loader test.

Done exactly as suggested:

@pytest.fixture(scope="module")
def loader():
    with atom_without_aiter() as load:
        return load("atom.model_loader.loader")

tests/test_online_quant_info_dir.py:7-20. The hand-written
try/except ImportError helper and its importlib import are gone; this is the
same two-line shape as tests/model_ops/test_sp_attention_dispatch.py:15 and
tests/test_deepseek_v41_mono_hi_runner.py:28.

One adjunct was needed for that to load anything in a CPU image that has an
AITER build. atom_without_aiter gated on import aiter, which is true on a
host with no GPU: AITER defers its arch probe, the import succeeds, and the
submodule that binds a kernel (reached from atom/model_ops/utils.py, in the
loader's chain) then fails on the missing device. As written, the suggested
two-liner errors all five cases there. So tests/aiter_stub.py:294-315 now
takes the stand-ins unless a GPU is available — the same line
tests/conftest.py:40-63 gpu_stack_import already draws between an absent
stack and one that cannot run. A GPU host (torch.cuda.is_available() true) and
the CI unit runner (no AITER installed at all, ImportError on the import)
take the paths they took before.

Verified both ways, inside local-llm/atom:20261031 with no GPU:

loader fixture AITER installed, no GPU (this image) AITER absent (CI unit runner)
suggested two-liner alone 5 errors, ModuleNotFoundError: No module named 'jax' inside AITER's arch probe 5 passed
as committed 5 passed 5 passed

The same image also goes from 8 failed + 189 errors to 279 passed / 12 skipped
across the twelve test files that already use atom_without_aiter, so the gate
is not specific to this test.

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ci Accuracy (Kimi-K3 DSpark DCP8 TP8) failed — not sure yet (low): The captured log is 0 bytes, so the failing step cannot be identified. The head commit ab725c8 only changes tests/test_online_quant_info_dir.py and the same workflow passed at d9dd55b, so nothing ties the failure to this diff, but that cannot be confirmed without the log. The cell runs through the vLLM plugin, whose Config passes torch_profiler_dir=None explicitly and so does not read the changed env default; its online_quant_config path lands in os.getcwd() both before and after the loader change.

.valar/ci/job_132683815462.log is 0 bytes — no log lines available
next: Re-run the job and fetch its log, then classify; no code change is expected from this PR. https://github.com/ROCm/ATOM/actions/runs/43671008159/job/132683815462

ci ATOM vLLM Test Gate failed — not sure yet (low): Aggregation gate (evaluate-test-gate over check-signal, download_aiter_wheel, atom-vllm-oot); its log is also empty, but the gate's red follows from the failed accuracy matrix job above. It should turn green once that job does.

.valar/ci/job_132700690021.log is 0 bytes — no log lines available
next: No separate fix; re-run after the Accuracy (Kimi-K3 DSpark DCP8 TP8) job is re-run and passes. https://github.com/ROCm/ATOM/actions/runs/43671008159/job/132700690021

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (2)

Both r22 asks are done, and the change still belongs in ATOM. The loader test now uses with atom_without_aiter() as load (tests/test_online_quant_info_dir.py:10-20). The added GPU gate in tests/aiter_stub.py:305-315 leaves the CPU job (no AITER) and GPU hosts on the paths they took before. On a host with AITER and no GPU, the 12 files that use the helper went from 12 failed + 194 errors to 276 passed (2 unrelated topk_select failures). Heavy CI ran at ab725c8 with ci:full, and 23 of 24 accuracy cells pass.

  1. Re-run Accuracy (Kimi-K3 DSpark DCP8 TP8) and merge once it and ATOM vLLM Test Gate are green. Job 132683815462 failed with "vLLM server did not become ready in time" after 40×30 s. Weight loading had finished at about 03:48, and there is no traceback. The product code is identical to d9dd55b, where the same cell passed in 16.5 min (run 43660855824); the new commit only touches tests. So this looks like a startup timeout on the runner, but the gate is red until the re-run passes. Note that ci:full does not run the ATOMesh model benchmark on PRs (atomesh-benchmark.yaml:510 runs it only on workflow_dispatch). The pd_server_atom.sh change is covered only by the test_atomesh_* unit tests, which pass.
  2. Fix the ATOM_RTL_TRACE_DIR comment (nit). envs.py:897-899 says "VAR= disables it". Nothing in Python reads envs.ATOM_RTL_TRACE_DIR. The only consumer, .github/scripts/atom_test.sh:62,381, is gated by ENABLE_RTL_PROFILER=1 and reads ${ATOM_RTL_TRACE_DIR:-/app/rtl_traces}, so an empty value falls back to /app/rtl_traces and does not disable anything. Drop that sentence, or say the variable names atom_test.sh's output directory under ENABLE_RTL_PROFILER=1 (credit: comment-analyzer).

Re-reviewed at ab725c8. Already fixed: loader test reuses atom_without_aiter (r22 #2); ci:full heavy CI ran at head (r22 #1, one cell pending re-run); r20 #1-#4, valar-cop's three asks and yitingw1's two points still hold. Branch merges cleanly into main 7a79a14.

The comment said an empty value disables tracing, but no Python code reads
envs.ATOM_RTL_TRACE_DIR: the variable's only consumer is
.github/scripts/atom_test.sh, which uses it as the output directory for
`rtl trace -o` under ENABLE_RTL_PROFILER=1 and, there, an empty value falls
back to /app/rtl_traces. Say that instead, and note the empty-reads-as-unset
rule only for the Python reader, consistent with the other directory knobs.

Signed-off-by: Yaakoub El Khamra <176948988+yelkhamra@users.noreply.github.com>
@yelkhamra

Copy link
Copy Markdown
Contributor Author

@valarLip

No rebase this round: the branch is still based on 004f342ab, and one new
commit (4954cac80) sits on top of ab725c857.

1. Re-run Accuracy (Kimi-K3 DSpark DCP8 TP8).

We cannot re-run Actions jobs from here (no re-run rights on this repo), so
there is no code change to make. Could a maintainer please re-run that
job? The product code at ab725c857 is identical to d9dd55bc0,
where the same cell passed: the job stopped at the vLLM server readiness wait
after weight loading had finished, with no traceback, and
ab725c857 touches tests only. We cannot start ATOM vLLM Test Gate either,
so a green run of it at this head is needed too. We agree that ci:full does
not run the ATOMesh model benchmark on PRs, and that the pd_server_atom.sh
change is covered by the test_atomesh_* unit tests, which pass.

2. Fix the ATOM_RTL_TRACE_DIR comment.

Done in 4954cac80: atom/utils/envs.py:895-900. The sentence claiming that
ATOM_RTL_TRACE_DIR= disables tracing is gone. The comment now says that no
Python code outside tests consumes the value; that .github/scripts/atom_test.sh
uses it as the output directory for rtl trace -o under
ENABLE_RTL_PROFILER=1 and reads it back when the server stops; and that an
empty value falls back to /app/rtl_traces there. The _optional_path_env
reader is kept, and its empty-reads-as-unset note is now scoped to Python,
consistent with the other directory knobs. Verified against the script (reads
at lines 62 and 381) and with a repository grep for Python consumers; commands
and output are in CHECKS.md. Thanks to your comment-analyzer for spotting
it.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (1)

The code is done. The change still belongs in ATOM, and I found nothing left to change at 4954cac. Only CI is outstanding.

  1. Merge once heavy CI is green at this head. ATOM Test, ATOM vLLM Test and ATOM SGLang Test for 4954cac have been queued since 2026-11-16 15:03 UTC and have not started; 23 accuracy cells are pending. Pre Checkin, Benchmark artifact tests, Atomesh accuracy validation and the ATOMesh unit tests pass. The vLLM run includes Accuracy (Kimi-K3 DSpark DCP8 TP8), the cell that timed out at ab725c8, so it replaces the requested re-run. The product code is the same as at ab725c8 except the ATOM_RTL_TRACE_DIR comment, and the branch still merges cleanly into main 7762f29.

Re-reviewed at 4954cac (full diff). Already fixed: ATOM_RTL_TRACE_DIR comment now matches atom_test.sh:61-66,380-387 (r23 #2); loader test reuses atom_without_aiter (r22 #2); ATOM_ONLINE_QUANT_INFO_DIR reads only its own variable and the loader holds the fallback, profile_offline.py is a single or chain with the example-only empty-flag note, and the ATOM_V4_CG_INSTR_DIR edit is reverted (r20 #2-#4); rebased (r20 #1); valar-cop's three asks and yitingw1's two points still hold. The 136 PR tests pass locally; probes show empty ATOM_TORCH_PROFILER_DIR / ATOM_RTL_TRACE_DIR / ATOM_ONLINE_QUANT_INFO_DIR read as None and an empty ATOM_ENGRAM_CACHE_DIR reads as its default.

@valarLip valarLip added the LGTM waiting pass for CI label Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:full LGTM waiting pass for CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Issue]: ATOM_TORCH_PROFILER_DIR is ignored when the server is started without --torch-profiler-dir

5 participants