Skip to content

[mono] Keep FlyDSL's disk cache on for checked builds; no source locations for serving builds - #3348

Open
coderfeli wants to merge 2 commits into
mainfrom
fix/mono-single-trace
Open

coderfeli wants to merge 2 commits into
mainfrom
fix/mono-single-trace

Conversation

@coderfeli

@coderfeli coderfeli commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

FlyDSL 0.5.0 traces a @flyc.jit launcher on every call, before it looks the build up in its caches (the cache key is the traced module). Qwen3.8's compile_traced and DeepSeek-V4.1 hi's checked builds still forced FLYDSL_RUNTIME_ENABLE_CACHE=0 on top of that, so every process compiled their LLVM again. And the engine's builds captured a 5-frame source location for every traced op, which they never use.

Changes

  • The disk cache stays on for the checked builds. Qwen3.8 and the V4.1 hi compile_traced no longer turn it off, and neither do the V4.1 / MiniMax-M3 contract tests and the gfx1250 multi-thread trace test, so CI now runs the warm-cache path. A cached build is still traced, recorded and checked; it is only not compiled again. Stale comments and docs that described a cache which hides the trace are fixed (hi/emit/compile.py, hi_runner.py, CLAUDE.md, docs/environment_variables.md).
  • atom/mono/runtime/srcloc.py
    • Drops the 0.4-era location fingerprint and its warning.
    • scoped() keeps one frame per location for the checked builds.
    • unlocated() captures no locations for the engine's builds. Without debug info, FlyDSL leaves locations out of both the binary and its cache key, so the binary is the same. With debug info on, it falls back to scoped().

Not in this PR: the first launch still traces each program a second time. FlyDSL 0.5.0's COMPILE_ONLY=1 returns no callable, so aiter's _run_compiled → flyc.compile traces again. The fix belongs in aiter: _preload_compiled keeps the callable from flyc.compile(..., compile_only=True) (FlyDSL#1392, merged but not in the 0.5.0 release aiter pins). That will be a follow-up once aiter's FlyDSL pin includes it.

Measurements (compile-only builds for gfx950, ARCH=gfx950, MI308X host, one TP rank)

main this PR
V4.1 hi S=6, warm disk cache: serving build 44.5 s 28–29 s
V4.1 hi S=6, warm disk cache: build + first-launch retrace ~79 s ~63 s (retrace 34 s unchanged; see above)
Qwen3.8 target compile_traced, all 8 programs, warm cache LLVM recompiled every process (cache forced off) 34 s under cProfile (trace ~16 s), cache hit, checks pass

Testing

  • black and ruff are clean on the changed files.
  • New GPU-free tests in tests/test_mono_runtime.py:
    • srcloc.scoped() sets and restores FLYDSL_DEBUG_MAX_LOC_DEPTH, both when it was unset and when it was set;
    • srcloc.unlocated() swaps and restores capture_user_location in all three holders, also on an exception;
    • unlocated()'s debug-info branch.
  • Rebased on main (c9ed3ad). Against a fresh FLYDSL_RUNTIME_CACHE_DIR, cold then warm: test_mono_runtime.py, test_deepseek_v41_mono_hi_package.py, test_deepseek_v41_gfx1250.py, the V4.1 / MiniMax-M3 contract tests and test_deepseek_v41_mono_hi_contract.py (now every served TP × index layout) pass both times: 218 passed, 1 skipped.
  • test_qwen38_mono: its GPU-launch tests abort with hipErrorNoBinaryForGpu on the test box (gfx942, kernels built for gfx950); the same on the base commit.

Note: #3087 also touches the _env(..., ARCH=...) lines in qwen38/.../compile.py and hi/emit/compile.py. Whichever PR lands second drops ARCH= (FlyDSL#1410).

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 3348 --add-label <label>

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed..

aiter pr bumping flydsl to >=0.5.0? and a warm-disk-cache run of a checked build (qwen38 compile_traced / hi compile_traced) on the deployed flydsl?

ci green.

ARCH=arch,
FLYDSL_GPU_ARCH=arch,
),
_env(COMPILE_ONLY=1, ARCH=arch, FLYDSL_GPU_ARCH=arch),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dropping the cache=0 makes this depend on flydsl tracing a cached build, that's 0.5.0 behaviour but aiter still pins 0.4.1. where's the aiter bump? or add a version guard here..

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aiter main already pins flydsl==0.5.0 (setup.py, pyproject.toml), and since ca6bc0f851 it refuses an older FlyDSL at import (aiter/ops/flydsl/__init__.py, _MIN_FLYDSL_VERSION). Agreed it shouldn't rest on that alone, though, so I added atom.mono.runtime.compile.require_tracing_flydsl(). It raises ImportError on FlyDSL < 0.5.0, and it runs in trace(), Qwen3.8's compile_traced and V4.1 hi's compile_traced outside serving. An older FlyDSL now fails with a named error instead of the "nothing traced" a skipped trace would give.

Comment thread atom/mono/runtime/compile.py Outdated
return [launcher for launcher, _ in kernels]


def keep_compiled(launcher, *args, **kwargs):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nothing exercises keep_compiled, the only real launch test skips unless gfx950. add a fake-launcher test for each early return and the _cf set/skip?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to tests/test_mono_runtime.py, using a fake @flyc.jit launcher (no GPU). They cover the kept build (_cf set, arguments resolved, _make_compiled_function called with the one key) and each early return: no build or two builds, a Constexpr parameter, an unannotated parameter, and each missing FlyDSL internal. They also cover compile_kernels keeping the build, WidthBuilds not compiling a kept launcher a second time, and the version guard.

Comment thread atom/mono/runtime/compile.py Outdated
return None
bound = sig.bind(*args, **kwargs)
bound.apply_defaults()
launcher._resolve_arguments(bound.arguments)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you guard _make_compiled_function with getattr but then call _resolve_arguments and import private flydsl names unguarded, so the fallback is void. fence these too, or version gate..

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. _resolve_arguments is now looked up with getattr along with _make_compiled_function and _sig, and _runtime_argument returns False if FlyDSL's types can't be imported. Any missing internal leaves the launcher unchanged, so the first launch just traces again. The checked builds are also version-gated now (see the qwen comment).

them traced and checked there): a build FlyDSL has cached loads without tracing or compiling, and the ranks of a node
share one compile of a program (FlyDSL's compile lock). The resource gate and the code's hazard scan read the built
code object either way.
``check_memory``). FlyDSL traces a build before it looks it up in its disk cache (its key is the traced module), so a

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stale stuff left around: _ENV_IGNORED comment still says the cache switch sets itself, hi_runner still says a restart loads untraced, and the contract tests still force cache=0 so they never hit the warm path. please clean up..

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cleaned up:

  • _ENV_IGNORED now says it ignores whether the disk cache is on, because that decides where a build comes from, not what it is.
  • The hi_runner comment now says a restart traces the builds but compiles none.
  • CLAUDE.md's contract command no longer sets FLYDSL_RUNTIME_ENABLE_CACHE=0.
  • The V4.1 and MiniMax-M3 contract tests no longer force the cache off. I ran them twice against a fresh cache dir: 12 passed cold (64 s) and 12 passed warm (20 s), so the warm path traces, records and checks.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (4)

  1. Drop keep_compiled: don't build FlyDSL's CompiledFunction from its private members and write it into aiter's private _cf slot. atom/mono/runtime/compile.py:60-89 probes _make_compiled_function / _sig / _mem_cache with getattr and silently returns None when they are missing. It then calls _resolve_arguments and imports flydsl.compiler.jit_argument without a guard (valar-cop is right: the fallback does nothing), and it sets launcher._cf, which aiter's tensor_shim._run_compiled (L331) happens to read. That is the #3162 pattern again: the next FlyDSL or aiter change turns this into a silent second trace, or into a wrong callable. The double trace comes from FlyDSL 0.5.0 (it traces before any cache hit, and COMPILE_ONLY=1 returns None), and FlyDSL already fixed it upstream: flyc.compile(launcher, *args, compile_only=True) (FlyDSL#1392) never launches and returns the callable. This is the same call as #3276: put it in aiter, then use it from ATOM. aiter's _preload_compiled (tensor_shim.py L347) should return and keep that callable once aiter's FlyDSL pin includes #1392, and ATOM should call it instead of the 5 keep_compiled call sites and the getattr(..., "_cf", None) checks in widths.py:98 and gfx1250/runner.py:231. This also changes the first launch of every WidthBuilds user: MiniMax-M3, Kimi-K3 and the DSA indexer, none of which this PR tested. Until then, split it out: the cache and srcloc parts below don't depend on it.
  2. Finish removing the cache switch: leftovers still describe a cache that hides the trace. hi/emit/compile.py:58-61 (_ENV_IGNORED "the cache switch compile_traced sets itself"), hi_runner.py:1080 ("a restart loads them untraced"), CLAUDE.md:135, and tests/test_deepseek_v41_mono_contract.py:36,59 / tests/test_minimax_m3_mono_contract.py:34, which still set FLYDSL_RUNTIME_ENABLE_CACHE=0, so CI never runs the warm-cache path this PR depends on. Remove the setenv lines so the contract tests trace a cached build.
  3. Fix docs/environment_variables.md:208: FLYDSL_DEBUG_MAX_LOC_DEPTH=1 no longer does anything. Serving builds now run under srcloc.unlocated(), and checked builds run under scoped(), which overwrites the variable with DEPTH. Delete the sentence or say that the variable is overridden.
  4. Add CPU tests for the new branches. srcloc.unlocated() and scoped() should swap and restore the setting (the old test was deleted with no replacement). WidthBuilds._compiled should skip a kernel whose build is already kept. Also cover whatever replaces item 1. Nothing exercises them today; the only launch test is gated to gfx950.

Smaller: the compile_traced docstring at hi/emit/compile.py:159 is 160+ chars on one line, so rewrap it. This PR edits the same _env(...) / trace() docstring lines as #3087 (FLYDSL_GPU_ARCH only; FlyDSL#1410 removed ARCH), so whichever lands second has to drop ARCH= from qwen38/.../compile.py and hi/emit/compile.py. On valar-cop's version-pin question: aiter main already pins flydsl==0.5.0 (requirements.txt L25), and 0.5.0 traces before every cache lookup (jit_function.py: "Every ordinary JIT/compile request traces before ANY artifact hit"). Dropping FLYDSL_RUNTIME_ENABLE_CACHE=0 is therefore correct, and no version guard is needed.


Reviewed at 0d6a60c.

@valar-cop

valar-cop commented Nov 14, 2026 •

Copy link
Copy Markdown

ci triage for 7bf5237:

ATOM SGLang Test passed. https://github.com/ROCm/ATOM/actions/runs/43836680111

Atomesh Benchmark passed. https://github.com/ROCm/ATOM/actions/runs/43836679932

Atomesh Accuracy Validation passed. https://github.com/ROCm/ATOM/actions/runs/43836679926

ATOM vLLM Test passed. https://github.com/ROCm/ATOM/actions/runs/43836679977

ATOM Test passed. https://github.com/ROCm/ATOM/actions/runs/43836679934

Pre Checkin / Run non-GPU unit tests failed — flaky, not the pr (medium): The non-GPU unit test job is nondeterministic: the same job on main both failed and passed on 2026-11-16, and other recent runs are mostly successful. No job log was provided, so there is no evidence of a PR-specific failure in the changed files.

Same job on main: 2026-11-16T07:41 success, 2026-11-16T06:59 failure, 2026-11-16T04:29 failure, 2026-11-16T03:32 failure, 2026-11-16T02:28 success
Same job on other recent runs: {'cancelled': 2, 'success': 6}
Infra signatures matched by regex: none
next: Rerun the job; if it passes, mark it flaky and ping the CI owner to stabilize the non-GPU unit test suite. If it fails again, inspect the failing test output and fix the PR/test. https://github.com/ROCm/ATOM/actions/runs/43824046595/job/133153527854

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed..

ci green.

Comment thread atom/mono/runtime/compile.py Outdated
bound = sig.bind(*args, **kwargs)
bound.apply_defaults()
resolve(bound.arguments)
launcher._cf = make(next(iter(builds)), bound, None)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still the same design.. let's move the kept-callable path into aiter/FlyDSL, or split keep_compiled out and land the disk-cache/srcloc parts alone

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split out in 160e70f: keep_compiled, its call sites, the _cf reads in widths.py / gfx1250/runner.py and its tests are gone (the runners and WidthBuilds are back to main). This PR is now the disk-cache and srcloc parts only. The kept callable will go into aiter's _preload_compiled via flyc.compile(..., compile_only=True) (FlyDSL#1392) once aiter's FlyDSL pin includes it; 0.5.0 does not.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (4)

  1. Drop keep_compiled from ATOM. Fencing it did not fix the problem: it is now a silent probe of FlyDSL's private internals. atom/mono/runtime/compile.py:86-89 reads _make_compiled_function / _resolve_arguments / _sig / _mem_cache with getattr(..., None). _runtime_argument (L103-115) wraps FlyDSL's private jit_argument import in try/except ImportError: return False. This is the pattern #3162 and #3276 ruled out. With the new fences, any FlyDSL rename brings the double trace back with no error and no log. A probe built from FlyDSL's JitFunction shape, with FlyDSL's jit_argument module missing, returns None, sets no _cf and logs nothing (probe_r20.py). aiter already owns this step: tensor_shim._preload_compiled (L347) builds without dispatching and prefers FlyDSL's public JitFunction.preload(), and _run_compiled (L331) is the only reader of _cf. Make _preload_compiled keep the callable it gets from preload() / compile_only=True (FlyDSL#1392) in aiter. ATOM then calls it from compile_kernels and drops the 5 call sites plus the getattr(..., "_cf", None) reads in widths.py:98 and gfx1250/runner.py:232. Or split this part out and land the cache/srcloc change on its own (valar-cop asked for the same at L99).
  2. Remove require_tracing_flydsl: it is a FlyDSL version sniff, and aiter already enforces the version. compile.py:22,124-138 compares importlib.metadata.version("flydsl") against "0.5.0". aiter main pins flydsl==0.5.0 (requirements.txt) and already refuses an older FlyDSL at import (aiter/ops/flydsl/__init__.py:17, _MIN_FLYDSL_VERSION = Version("0.5.0")). Your own reply says so. Delete _TRACING_FLYDSL, the three calls (trace, both compile_traced) and test_the_hand_off_checks_need_a_flydsl_that_traces_cached_builds.
  3. Fix docs/environment_variables.md:208 (r18 item 3, not addressed). It still says "FLYDSL_DEBUG_MAX_LOC_DEPTH=1 shortens the trace of a first start's builds". Serving builds run under srcloc.unlocated(), which captures no locations, and checked builds run under scoped(), which overwrites the variable, so it changes nothing. Delete that sentence.
  4. Add the srcloc tests that r18 item 4 asked for. Nothing under tests/ imports srcloc now. Test that scoped() sets and restores FLYDSL_DEBUG_MAX_LOC_DEPTH (both when it was unset and when it was set), and that unlocated() swaps and restores capture_user_location in all three _HOLDERS. Also cover its enable_debug_info branch.

Smaller: tests/test_deepseek_v41_mono_hi_contract.py:7 still says load-time builds skip the checks "(FlyDSL's disk cache)"; it should say serving(). #3087 is still open and also touches ARCH= at qwen38/.../compile.py:65 and hi/emit/compile.py:168, so whichever PR lands second has to drop ARCH=. CI is green.


Re-reviewed at 2eb404e. Already fixed: stale cache-switch comments, CLAUDE.md command and the contract tests' FLYDSL_RUNTIME_ENABLE_CACHE=0 (r18 item 2); fake-launcher tests for keep_compiled / WidthBuilds skip; compile_traced docstring wrap.

@coderfeli

Copy link
Copy Markdown
Collaborator Author

@valarLip @valar-cop thanks. I missed the review bodies last round. Addressed in 160e70f:

  1. keep_compiled is split out. It is removed together with its 5 call sites, the _cf reads in widths.py / gfx1250/runner.py and its fake-launcher tests; the runners and WidthBuilds are back to main. FlyDSL#1392 (flyc.compile(..., compile_only=True)) is not in the 0.5.0 release aiter pins, so the kept callable becomes an aiter follow-up (_preload_compiled keeps it, _run_compiled uses it) once the pin moves; ATOM then calls that from compile_kernels. Until then the first launch still retraces each program (~34 s for hi S=6), as on main.
  2. require_tracing_flydsl is removed, along with _TRACING_FLYDSL, its calls and its test; aiter's pin and import guard enforce 0.5.0.
  3. docs/environment_variables.md: the FLYDSL_DEBUG_MAX_LOC_DEPTH=1 sentence is deleted.
  4. srcloc tests in tests/test_mono_runtime.py:
    • scoped() sets and restores the variable, both when it was unset and when it was set;
    • unlocated() swaps and restores capture_user_location in all three _HOLDERS, also when the block raises;
    • the enable_debug_info branch falls back to scoped().

Smaller:

Warm-disk-cache runs of checked builds on FlyDSL 0.5.0 (valar-cop), against a fresh FLYDSL_RUNTIME_CACHE_DIR, cold then warm:

  • The V4.1 and MiniMax-M3 contract tests plus test_mono_runtime / hi package / gfx1250 pass both times: 204 passed, 76 s cold / 31 s warm.
  • test_deepseek_v41_mono_hi_contract (hi compile_traced) passes both times.
  • Qwen3.8 target compile_traced (8 programs) hits the cache when warm and its checks pass.
  • hi S=6 serving build, warm: 44.5 s on main → 28–29 s.

The numbers are in the updated PR description.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (3)

  1. Rebase onto main: the PR conflicts, so no Pre Checkin / ATOM Test has run at 160e70f. GitHub reports mergeable_state: dirty. docs/environment_variables.md (the hi paragraph, which #3302 rewrote for TP2/TP8) and tests/test_deepseek_v41_mono_hi_contract.py (#3302 parametrized it over SERVED_TPS × INDEX_LAYOUTS) both conflict. At this head only readthedocs ran. The green runs are for 2eb404e, which still had keep_compiled.
  2. While rebasing, drop the cache-off line that #3279 added on main. tests/test_deepseek_v41_gfx1250.py:612 (test_the_step_kernel_traces_on_several_threads_at_once) sets FLYDSL_RUNTIME_ENABLE_CACHE=0 with the comment "a cached build is not traced". That is the claim this PR removes everywhere else. FlyDSL 0.5.0 traces before any artifact hit (jit_function.py: "Every ordinary JIT/compile request traces before ANY artifact hit"). Delete the setenv and the comment so the gfx1250 trace test also runs warm.
  3. Retitle the PR: after the split it no longer traces each build once. keep_compiled is gone, and your description says the first launch still traces each program a second time. Something like "[mono] Keep FlyDSL's disk cache on for checked builds; no source locations for serving builds" matches the diff.

Not blocking: srcloc.unlocated() still swaps capture_user_location in three FlyDSL modules. If FlyDSL adds a fourth holder, the saving silently disappears; it only costs time, the binary does not change. A FlyDSL knob (e.g. FLYDSL_DEBUG_MAX_LOC_DEPTH=0 → Location.unknown()) would let ATOM drop the patch. Please don't add a hasattr or fingerprint guard instead. #3087 is still open: whichever PR lands second drops ARCH=.


Re-reviewed at 160e70f. Already fixed: keep_compiled and its call sites and _cf reads removed (r20 #1); require_tracing_flydsl removed (r20 #2); FLYDSL_DEBUG_MAX_LOC_DEPTH=1 docs sentence removed (r20 #3); srcloc tests for scoped/unlocated incl. the exception and debug-info branches (r20 #4); hi_contract test docstring names serving(); stale cache-switch comments, CLAUDE.md and contract tests (r18 #2). FlyDSL 0.5.0 trace-before-lookup and location-free cache key verified against its source; srcloc restore semantics checked with a CPU probe.

@coderfeli coderfeli changed the title [mono] Trace each FlyDSL build once and keep FlyDSL's disk cache on [mono] Keep FlyDSL's disk cache on for checked builds; no source locations for serving builds Nov 16, 2026
@coderfeli
coderfeli force-pushed the fix/mono-single-trace branch from 160e70f to c9ed3ad Compare November 16, 2026 01:55
@coderfeli

Copy link
Copy Markdown
Collaborator Author

@valarLip addressed in c9ed3ad:

  1. Rebased onto main, with the branch squashed to one commit. The docs/environment_variables.md conflict keeps [DeepSeek-V4.1] High-interactivity monokernel: TP2 and TP8, and a fix for a hang at 5-8 requests #3302's TP2/TP4/TP8 paragraph; test_deepseek_v41_mono_hi_contract.py keeps its SERVED_TPS × INDEX_LAYOUTS parametrization, with the docstring naming serving().
  2. Dropped FLYDSL_RUNTIME_ENABLE_CACHE=0 and its "a cached build is not traced" comment from test_the_step_kernel_traces_on_several_threads_at_once. git grep FLYDSL_RUNTIME_ENABLE_CACHE now only finds _ENV_IGNORED.
  3. Retitled as suggested.

Against a fresh cache dir, cold then warm: mono runtime, hi package, gfx1250, the V4.1 / MiniMax-M3 contract tests and the hi contract (every served TP × layout) all pass, 218 passed both times.

On srcloc: agreed, no guard. A FlyDSL knob for location-free tracing (e.g. FLYDSL_DEBUG_MAX_LOC_DEPTH=0 → Location.unknown()) would let ATOM drop the swap; I will raise it with FlyDSL. #3087: whichever lands second drops ARCH=.

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ci Accuracy (DeepSeek-V4.1-Flash DSpark, deepseek-ai/DeepSeek-V4.1-Flash, --kv_cache_dtype fp8 --ind...) failed — not sure yet (low): The harness log for this job is empty (0 lines), so it cannot be classified. It passes 7/7 on main in the same window; this PR only changes the source locations of hi builds and the disk cache of checked builds, neither of which changes the served binary with debug info off.

FAILED job Accuracy (DeepSeek-V4.1-Flash DSpark, ... (ATOM Test), runner linux-atom-do-mi350x-8-k65lt-runner-vmcmp
next: Re-run the job and attach the log; if it reproduces, compare the served hi binary/cache entry against main before attributing it to this PR. https://github.com/ROCm/ATOM/actions/runs/43789054856/job/133047502308

ci ATOM Test Gate failed — not sure yet (medium): The ATOM Test aggregator failed alongside the accuracy job above and carries no log of its own; it most likely reports that job's failure rather than failing on its own (it also failed once on main at 2026-11-15T16:08).

FAILED job ATOM Test Gate (ATOM Test), runner GitHub Actions 1011030627
same job on main: ['2026-11-16T00:13 success', '2026-11-16T00:13 success', '2026-11-15T16:08 failure', ...]
next: Nothing to fix in this PR; the gate should clear once the accuracy job above is re-run successfully. https://github.com/ROCm/ATOM/actions/runs/43789054856/job/133076055662

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (2)

  1. Re-run the red Accuracy (DeepSeek-V4.1-Flash DSpark) job; nothing in this PR needs to change for it. The job scored 0.9098 flexible-extract against a 0.91 bar (run 43789054856). The two main runs either side of your rebase scored 0.9212 and 0.9181. This job cannot reach the code you changed: it serves --kv_cache_dtype fp8, and hi_gate.py:114-116 turns the hi path off for anything but bf16. The catalog note for the model says "a single red is noise until a rerun agrees with it". ATOM Test Gate is red only because of this job.
  2. Say in the serving docstrings that debug info brings the locations back. hi/emit/compile.py:21 ("capture no source locations"), :157 ("no source locations") and the comment at :164 ("serving: none") all say serving builds have no locations. But srcloc.unlocated() falls back to scoped() when FLYDSL_DEBUG_ENABLE_DEBUG_INFO is on, as its own docstring says. Add "(unless FLYDSL_DEBUG_ENABLE_DEBUG_INFO is set)" or similar.

Not blocking: #3087 is still open, so whichever PR lands second drops ARCH= from qwen38/.../compile.py and hi/emit/compile.py. Optionally annotate compile_traced as -> Record | None.


Re-reviewed at c9ed3ad. Already fixed: rebased onto main and merges cleanly with main d2d5a6b; #3302's TP2/TP4/TP8 docs paragraph and SERVED_TPS × INDEX_LAYOUTS parametrization kept (r25 #1). The gfx1250 multi-thread test no longer sets FLYDSL_RUNTIME_ENABLE_CACHE=0; git grep only finds it in _ENV_IGNORED (r25 #2). PR retitled (r25 #3). Still fixed from earlier rounds: keep_compiled, _cf reads and require_tracing_flydsl (0 hits). Checked: FlyDSL 0.5.0 re-reads FLYDSL_DEBUG_MAX_LOC_DEPTH on every capture (meta.py:159, env.py:40-50), so scoped() works; aiter main pins flydsl==0.5.0, which matches the docs bump. CPU probes: unlocated()/scoped() restore on nesting, on exceptions and in the debug-info branch; the conditional with ... as rec binds rec in both branches.

@valarLip valarLip added the LGTM waiting pass for CI label Nov 16, 2026
@coderfeli

Copy link
Copy Markdown
Collaborator Author

@valarLip addressed in 7bf5237:

  1. Accuracy (DeepSeek-V4.1-Flash DSpark): the re-run (run 43789054856, attempt 2) passed, and ATOM Test Gate is green on it. The ATOM Test run 43808629493 is red only because it was cancelled: it was a duplicate run, triggered at the same second as 43808625625, which succeeded.
  2. Serving docstrings now say that debug info brings the source locations back: hi/emit/compile.py's module docstring, the compile_traced docstring and the comment at the with, and srcloc's module docstring, which said the same. compile_traced is annotated -> Record | None.

#3087: whichever PR lands second drops ARCH=.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (1)

  1. Merge main into the branch (or push any update) so Pre Checkin and ATOM Test run again; nothing in this PR needs to change. At 7bf5237, Run non-GPU unit tests fails 2 tests ('Mxfp4MoEMethod' object has no attribute 'use_flydsl_herd', atom/model_ops/moe.py:2350). Check Pre Checkin Signal and ATOM Test Gate are red for that reason, and the GPU accuracy jobs were skipped. This PR doesn't touch that code. #2713 (02905b7) broke the test on main, and main failed the same way (Pre Checkin run 43820641381 at 7e2f723). #3393 (0819cb3) fixed it. A plain re-run reuses the old merge ref, so it would fail again. The branch merges cleanly with main 0819cb3.

Not blocking: #3087 is still open, so whichever PR lands second drops ARCH= from qwen38/.../compile.py and hi/emit/compile.py.


Re-reviewed at 7bf5237. Already fixed: the serving docstrings now say debug info brings the source locations back (hi/emit/compile.py:21-22, :157-159, the comment at :166, and the srcloc.py module docstring) (r28 #2). compile_traced is annotated -> Record | None. The V4.1-Flash DSpark accuracy job passed on re-run with 0.9219 flexible-extract (run 43789054856 attempt 2; ATOM Test Gate green at c9ed3ad) (r28 #1). Still fixed from earlier rounds: 0 hits for keep_compiled and require_tracing_flydsl; FLYDSL_RUNTIME_ENABLE_CACHE appears only in _ENV_IGNORED. The only reader of a serving build's None record (test_deepseek_v41_mono_hi_contract.py:39) checks for it.

coderfeli and others added 2 commits November 17, 2026 08:45
…tions for serving builds

FlyDSL 0.5.0 traces a build before it looks it up in its disk cache (the key
is the traced module), so a cached build is still traced, recorded and
checked. Stop forcing FLYDSL_RUNTIME_ENABLE_CACHE=0 in Qwen3.8's and the V4.1
hi checked builds and in the V4.1 / MiniMax-M3 contract and gfx1250 trace
tests, and fix the comments and docs that said a cached build is not traced.

srcloc: drop the 0.4-era fingerprinted location capture; scoped() keeps one
frame a location for the checked builds, unlocated() captures none for the
engine's builds (without debug info the binary and its cache key carry no
locations). CPU tests for both.

The first launch still traces each program again (COMPILE_ONLY returns no
callable); that belongs in aiter's _preload_compiled once its FlyDSL pin
includes FlyDSL#1392.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
srcloc.unlocated() falls back to scoped() when FLYDSL_DEBUG_ENABLE_DEBUG_INFO
is set; the hi compile_traced docstrings and the srcloc module docstring now
say so. compile_traced is annotated -> Record | None.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderfeli

Copy link
Copy Markdown
Collaborator Author

@valarLip rebased onto main 19c292f, which includes #3393 (0819cb3), so Pre Checkin and ATOM Test run again on the fixed main. The only conflict was the hi paragraph in docs/environment_variables.md; it now keeps main's updated TP8 note, with "they need FlyDSL 0.5.0". The code is unchanged. Locally, mono runtime, hi package, gfx1250 and the V4.1 / MiniMax-M3 contract tests pass (211 passed).

@coderfeli
coderfeli force-pushed the fix/mono-single-trace branch from 7bf5237 to 682b664 Compare November 17, 2026 08:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:atom LGTM waiting pass for CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants