Skip to content

[Issue]: ATOM_TORCH_PROFILER_DIR is ignored when the server is started without --torch-profiler-dir #2911

Closed

Description

@yelkhamra

Problem Description

Symptom. Starting the ATOM server through the CLI with only ATOM_TORCH_PROFILER_DIR set (no --torch-profiler-dir flag) leaves Config.torch_profiler_dir at None. The profiling documentation describes the environment variable as the default and as an alternative to the CLI flag, so users following it get no profiler output directory.

From source reading (GPU trace emission was not tested): with the directory at None, the runner skips profiling, while /start_profile and /stop_profile still return success, so no trace is written and nothing reports why.

Root cause. The EngineArgs.torch_profiler_dir field and its CLI option default to None. _get_engine_kwargs() forwards that None, and LLMEngine passes it into Config as an explicit keyword, which bypasses the field's environment-backed default_factory. Direct Config callers that omit the keyword do get the environment value; only the CLI/EngineArgs path loses it.

Impact. Profiling configured through the documented environment variable is silently disabled for CLI launches. This is a configuration/correctness issue; no performance effect is claimed.

Expected. An omitted --torch-profiler-dir falls back to ATOM_TORCH_PROFILER_DIR; an explicit CLI value takes precedence.

Actual. The environment variable is ignored on the CLI path and the profiler directory stays None.

Operating System

Linux; the exact distribution version is not recorded in this report. The issue is in argument handling and does not depend on the OS version.

CPU

Not recorded in this report; the issue does not depend on the CPU model.

GPU

None needed to reproduce: the configuration result was reproduced on CPU (Python 3.10.12, torch 2.14.1+cpu, transformers 5.16.1).

ROCm Version

Not applicable to the CPU reproduction. The affected code path is the same on ROCm builds.

ROCm Component

ATOM (engine arguments, atom/model_engine/arg_utils.py, and atom/config.py)

Installation Method

Other: ATOM source checkout of main at 9a0a5202bc28e9fd092b39199f6698e459d1d77d, CPU-only Python environment.

Steps to Reproduce

  1. On main, set ATOM_TORCH_PROFILER_DIR to an existing directory and build the engine configuration the way the CLI does, without --torch-profiler-dir: create EngineArgs from the parsed CLI arguments (its torch_profiler_dir stays None), build the keywords with _get_engine_kwargs(), and initialize Config from them (model loading can be mocked).
  2. Read Config.torch_profiler_dir.

Observed on main: None. Expected: the directory from ATOM_TORCH_PROFILER_DIR. Setting --torch-profiler-dir explicitly works as expected.

The regression test in the linked pull request (tests/test_arg_utils_profiler.py) automates these steps on CPU; its environment-only case fails on unmodified main and passes with the fix.

End to end (proposed; not run in this exact form): start the server with ATOM_TORCH_PROFILER_DIR set and no --torch-profiler-dir, call /start_profile, send a request, call /stop_profile, and check that no trace appears in the directory.

Additional Information

A fix is proposed in the linked pull request.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions