Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions .github/scripts/atomesh/pd_server_atom.sh
Original file line number Diff line number Diff line change
Expand Up @@ -347,16 +347,19 @@ ATOMESH_TORCH_PROFILE_ROLES="${ATOMESH_TORCH_PROFILE_ROLES:-prefill,decode}"
PREFILL_KV_TRANSFER_CONFIG="${PREFILL_KV_TRANSFER_CONFIG:-}"
DECODE_KV_TRANSFER_CONFIG="${DECODE_KV_TRANSFER_CONFIG:-}"

default_profiler_dir="${RUN_DIR}/online_quant/rank-${NODE_RANK}"
# Rank 0's online_quant_info_*.json dump for this run. It has its own variable
# so that collecting it does not also hand the engine a profiler directory on
# every launch; the profiler window below supplies one per worker instead.
online_quant_dir="${RUN_DIR}/online_quant/rank-${NODE_RANK}"
if [[ "${ATOMESH_EXECUTION_PHASE}" != "combined" ]]; then
default_profiler_dir="${RUN_DIR}/online_quant/${ATOMESH_EXECUTION_PHASE}/rank-${NODE_RANK}"
online_quant_dir="${RUN_DIR}/online_quant/${ATOMESH_EXECUTION_PHASE}/rank-${NODE_RANK}"
fi
export ATOM_TORCH_PROFILER_DIR="${ATOM_TORCH_PROFILER_DIR:-${default_profiler_dir}}"
export ATOM_ONLINE_QUANT_INFO_DIR="${ATOM_ONLINE_QUANT_INFO_DIR:-${online_quant_dir}}"
RUNTIME_LOG_DIR="${RUN_DIR}/logs"
if [[ "${ATOMESH_EXECUTION_PHASE}" != "combined" ]]; then
RUNTIME_LOG_DIR="${RUNTIME_LOG_DIR}/${ATOMESH_EXECUTION_PHASE}"
fi
mkdir -p "${RUNTIME_LOG_DIR}" "${RUN_DIR}"/{benchmark_results,eval_results} "${ATOM_TORCH_PROFILER_DIR}"
mkdir -p "${RUNTIME_LOG_DIR}" "${RUN_DIR}"/{benchmark_results,eval_results} "${ATOM_ONLINE_QUANT_INFO_DIR}"

role_tp="${PREFILL_TP_SIZE}"
if [[ "${AGGREGATED}" == "1" ]]; then
Expand Down Expand Up @@ -1942,10 +1945,10 @@ write_metadata() {
EOF
}

# ATOM_TORCH_PROFILER_DIR above does not reach the engine: EngineArgs always
# passes its own torch_profiler_dir (None by default), which overrides the env
# default in Config, so the profiler is off unless the flag is on the command.
# One directory per worker keeps P/D ranks that share a node apart.
# Put a profiler directory on the command line only while the window is on:
# this script exports no ATOM_TORCH_PROFILER_DIR, so a launch that does not
# profile carries none. One directory per worker keeps P/D ranks that share a
# node apart; an explicit flag also outranks the environment default in Config.
build_torch_profiler_args() {
local log_name="$1"
local -n out_args="$2"
Expand Down
17 changes: 11 additions & 6 deletions atom/examples/profile_offline.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,16 @@

import argparse
import random

from transformers import AutoTokenizer

from atom import SamplingParams
from atom.model_engine.arg_utils import EngineArgs
from atom.utils import envs
from atom.utils.arg_parser import FlexibleArgumentParser

DEFAULT_PROFILER_DIR = "./profiler_traces"

parser = FlexibleArgumentParser(
formatter_class=argparse.RawTextHelpFormatter,
description="Offline profiling example for Atom LLM Engine",
Expand Down Expand Up @@ -44,12 +48,13 @@
def main():
args = parser.parse_args()

# Set default torch_profiler_dir to current directory if not provided
if args.torch_profiler_dir is None:
args.torch_profiler_dir = "./profiler_traces"
print(
"Warning: --torch-profiler-dir not specified, using default: ./profiler_traces"
)
# Flag, then the environment, then this example's own directory, the order
# the engine uses. An empty flag counts as unset only here; EngineArgs
# forwards "" to Config, which aborts on os.makedirs("").
args.torch_profiler_dir = (
args.torch_profiler_dir or envs.ATOM_TORCH_PROFILER_DIR or DEFAULT_PROFILER_DIR
)
print(f"Writing profiler traces to {args.torch_profiler_dir}")

print("\nInitializing LLM engine...")
engine_args = EngineArgs.from_cli_args(args)
Expand Down
3 changes: 3 additions & 0 deletions atom/model_engine/arg_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -902,6 +902,9 @@ def _get_engine_kwargs(self) -> dict:
kwargs = {
f.name: getattr(self, f.name) for f in fields(self) if f.name != "model"
}
# Let Config resolve ATOM_TORCH_PROFILER_DIR when no path was supplied.
if self.torch_profiler_dir is None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empty ATOM_TORCH_PROFILER_DIR= now crashes startup, os.makedirs('').. read empty as unset in envs instead?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done as suggested: ATOM_TORCH_PROFILER_DIR now reads
os.getenv("ATOM_TORCH_PROFILER_DIR") or None in atom/utils/envs.py:771-774,
so ATOM_TORCH_PROFILER_DIR= is indistinguishable from unset and never reaches
Config as "". Covered by
tests/test_envs.py::TestEnvsOverrides::test_empty_torch_profiler_dir_reads_as_unset
and by test_torch_profiler_dir_handoff[-None-None], which drives the same
hand-off as the server. With the envs change reverted, that case fails with
FileNotFoundError: [Errno 2] No such file or directory: ''.

kwargs.pop("torch_profiler_dir")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the workaround comment in pd_server_atom.sh is stale now, and it exports a dir on every launch.. please sweep it here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Swept. .github/scripts/atomesh/pd_server_atom.sh no longer exports
ATOM_TORCH_PROFILER_DIR (lines 350-362), and the stale comment is replaced at
1948-1951 with what the per-worker flag does now.

The export had a second reader: _save_online_quant_info wrote rank 0's
online_quant_info_*.json under ATOM_TORCH_PROFILER_DIR when set. Removing
the export alone would have moved those dumps out of the results tree, so the
dump got its own knob, ATOM_ONLINE_QUANT_INFO_DIR (atom/utils/envs.py:891-900,
atom/model_loader/loader.py:180-182), which falls back to
ATOM_TORCH_PROFILER_DIR and then the working directory. The launcher exports it
to the same ${RUN_DIR}/online_quant[/${ATOMESH_EXECUTION_PHASE}]/rank-N path as
before, so the artifacts do not move.

The profiler itself is unchanged: only ATOMESH_TORCH_PROFILE_DURATION > 0 puts
--torch-profiler-dir on a launch, one directory per worker, and that flag still
outranks the environment default.


# Handle special transformations
kwargs["kv_cache_block_size"] = kwargs.pop("block_size")
Expand Down
4 changes: 3 additions & 1 deletion atom/model_loader/loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,9 @@ def _save_online_quant_info(
"""Save online quantization info to a JSON file (rank 0 only)."""
if get_tp_group().rank_in_group != 0:
return
output_dir = envs.ATOM_TORCH_PROFILER_DIR or os.getcwd()
output_dir = (
envs.ATOM_ONLINE_QUANT_INFO_DIR or envs.ATOM_TORCH_PROFILER_DIR or os.getcwd()
)
os.makedirs(output_dir, exist_ok=True)
timestamp = time.strftime("%Y%m%d_%H%M%S")
timestamp_ns = time.time_ns() % 1_000_000_000
Expand Down
41 changes: 35 additions & 6 deletions atom/utils/envs.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,20 @@ def _nonnegative_int_env(name: str, default: int) -> int:
return value


def _optional_path_env(name: str) -> str | None:
"""Unset or empty reads as None: ``VAR=`` clears a path inline."""
return os.getenv(name) or None


def _path_env(name: str, default: str) -> str:
"""Unset or empty reads as ``default``, never as the empty path.

``Path("")`` is the working directory, so falling through to it would
silently write wherever the process happens to run.
"""
return os.getenv(name) or default


def _finite_float_env(name: str, default: float, *, allow_zero: bool) -> float:
"""Unset or empty reads as ``default``; a set value must be a valid float."""
raw = os.getenv(name)
Expand Down Expand Up @@ -822,7 +836,10 @@ def _finite_float_env(name: str, default: float, *, allow_zero: bool) -> float:
"ATOM_ENABLE_METRICS_DEVICE_TIMER": lambda: (
os.getenv("ATOM_ENABLE_METRICS_DEVICE_TIMER", "0") == "1"
),
"ATOM_TORCH_PROFILER_DIR": lambda: os.getenv("ATOM_TORCH_PROFILER_DIR", None),
# Unset reads as None, and so does empty: `ATOM_TORCH_PROFILER_DIR=` is
# how a launcher clears the variable inline, and Config would otherwise
# abort startup on os.makedirs("") for the empty path.
"ATOM_TORCH_PROFILER_DIR": lambda: _optional_path_env("ATOM_TORCH_PROFILER_DIR"),
# Move the startup heap (model, compiled graph, tokenizer, KV block pool)
# into CPython's permanent generation once warmup is done, so collections
# stop scanning it. On by default; set 0 to keep the old behaviour.
Expand Down Expand Up @@ -875,10 +892,13 @@ def _finite_float_env(name: str, default: float, *, allow_zero: bool) -> float:
# WARNING, ...); applied by atom.utils.set_atom_log_level(), which turns
# NOTSET and unknown names into INFO.
"ATOM_LOG_LEVEL": lambda: os.getenv("ATOM_LOG_LEVEL", "INFO"),
# RTL (rocm-trace-lite) GPU kernel tracing — set to output directory to enable.
# When set, the server launch is wrapped with `rtl trace` to collect per-kernel
# GPU timestamps for both prefill and decode phases.
"ATOM_RTL_TRACE_DIR": lambda: os.getenv("ATOM_RTL_TRACE_DIR", None),
# RTL (rocm-trace-lite) GPU kernel tracing. No Python code outside tests
# consumes this value: it names the output directory that
# .github/scripts/atom_test.sh passes to `rtl trace -o` when
# ENABLE_RTL_PROFILER=1, and reads back when the server stops; an empty
# value falls back to /app/rtl_traces there. Empty reads as unset here,
# like the other directory knobs.
"ATOM_RTL_TRACE_DIR": lambda: _optional_path_env("ATOM_RTL_TRACE_DIR"),
# --- Model Loading ---
"ATOM_DISABLE_MMAP": lambda: (
os.getenv("ATOM_DISABLE_MMAP", "false").lower() == "true"
Expand Down Expand Up @@ -939,6 +959,12 @@ def _finite_float_env(name: str, default: float, *, allow_zero: bool) -> float:
"ATOM_LOADER_STRICT_COVERAGE": lambda: (
os.getenv("ATOM_LOADER_STRICT_COVERAGE", "true").lower() == "true"
),
# Directory rank 0 writes the online_quant_info_*.json dump to, when set.
# Its own knob so that a launcher collecting the dump no longer has to
# hand the engine a profiler directory to get it.
"ATOM_ONLINE_QUANT_INFO_DIR": lambda: _optional_path_env(
"ATOM_ONLINE_QUANT_INFO_DIR"
),
# Quantize eligible modules as they load to reduce peak memory. Streaming
# quantizes local TP shards, so results may differ slightly from offline.
"ATOM_ONLINE_QUANT_STREAMING": lambda: (
Expand Down Expand Up @@ -1349,7 +1375,10 @@ def _finite_float_env(name: str, default: float, *, allow_zero: bool) -> float:
# Where the compressed-vocab table is cached between runs. The table is
# reproducible from the tokenizer, so this only trades startup time for
# disk; point it at shared storage to let several servers build it once.
"ATOM_ENGRAM_CACHE_DIR": lambda: os.getenv(
# Empty reads as the default, like the other directory knobs: Path("") is
# the working directory, so `ATOM_ENGRAM_CACHE_DIR=` would otherwise drop
# the cache beside whatever directory the server was launched from.
"ATOM_ENGRAM_CACHE_DIR": lambda: _path_env(
"ATOM_ENGRAM_CACHE_DIR", str(Path.home() / ".cache" / "atom" / "engram")
),
# Overlap hash/UVA lookup and TP reassembly with early layers, using private
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Defined in `atom/config.py`. The root dataclass that the engine consumes.
| `replay_history_tokens` | `int` | `0` | Set by the model, not a flag: the tokens before a forward that starts below where its prefill resumed which the backend reads (`Replay.history_ids`). DeepSeek-V4.1: its Engram n-gram lookback, `engram_max_ngram_size - 1` |
| `state_replay_tokens` | `int` | `0` | Set by the model, not a flag: how far below a prefix hit with no state checkpoint behind it a prefill starts to rebuild that state exactly (`BlockManager.replay_start`); 0 takes no such hit. DeepSeek-V4 under `ATOM_DSV4_STATE_REPLAY`: the window receptive field `W + (L-1)(W-1)`, 7,748 on V4-Pro and 7,875 with MTP or DSpark |
| `port` | `int` | `8006` | Engine internal communication port |
| `torch_profiler_dir` | `str \| None` | `os.getenv("ATOM_TORCH_PROFILER_DIR", None)` | Directory for saving PyTorch profiler traces; creates the directory if it does not exist |
| `torch_profiler_dir` | `str \| None` | `envs.ATOM_TORCH_PROFILER_DIR` (unset or empty → `None`) | Directory for saving PyTorch profiler traces; creates the directory if it does not exist. `--torch-profiler-dir` takes precedence over the environment default |
| `compilation_config` | `CompilationConfig` | `CompilationConfig()` | Compilation and CUDA graph settings (see Section 2) |
| `quant_config` | `QuantizationConfig` | *(auto-detected)* | Quantization settings; auto-detected from HuggingFace config during `__post_init__` via `QuantizationConfig(hf_config)` (see Section 3) |
| `asyncio_mode` | `bool` | `False` | Enable asyncio-based engine loop |
Expand Down Expand Up @@ -259,7 +259,7 @@ Notes:
- An empty JSON object (`--online_quant_config '{}'`) is treated the same as not passing the flag and does not enable online quantization.
- Online quantization currently applies when the source model is unquantized or uses supported FP8 block quantization (`QuantType.per_1x128`). The loader dequantizes FP8 block weights before applying the requested target format.
- Tensor-parallel weights are gathered before quantization only when local quantization would produce different scales than quantizing the full unpartitioned weight.
- Rank 0 writes an `online_quant_info_*.json` summary with elapsed time and per-layer target formats. The file is written under `ATOM_TORCH_PROFILER_DIR` when set, otherwise the current working directory.
- Rank 0 writes an `online_quant_info_*.json` summary with elapsed time and per-layer target formats. The file is written under `ATOM_ONLINE_QUANT_INFO_DIR` when set, otherwise under `ATOM_TORCH_PROFILER_DIR`, otherwise in the current working directory.

## Parallel configuration (`ParallelConfig`)

Expand Down
Loading
Loading