Repository navigation
Conversation
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
valar-cop
left a comment
There was a problem hiding this comment.
some changes needed..
aiter pr bumping flydsl to >=0.5.0? and a warm-disk-cache run of a checked build (qwen38 compile_traced / hi compile_traced) on the deployed flydsl?
ci green.
| ARCH=arch, | ||
| FLYDSL_GPU_ARCH=arch, | ||
| ), | ||
| _env(COMPILE_ONLY=1, ARCH=arch, FLYDSL_GPU_ARCH=arch), |
There was a problem hiding this comment.
dropping the cache=0 makes this depend on flydsl tracing a cached build, that's 0.5.0 behaviour but aiter still pins 0.4.1. where's the aiter bump? or add a version guard here..
There was a problem hiding this comment.
aiter main already pins flydsl==0.5.0 (setup.py, pyproject.toml), and since ca6bc0f851 it refuses an older FlyDSL at import (aiter/ops/flydsl/__init__.py, _MIN_FLYDSL_VERSION). Agreed it shouldn't rest on that alone, though, so I added atom.mono.runtime.compile.require_tracing_flydsl(). It raises ImportError on FlyDSL < 0.5.0, and it runs in trace(), Qwen3.8's compile_traced and V4.1 hi's compile_traced outside serving. An older FlyDSL now fails with a named error instead of the "nothing traced" a skipped trace would give.
| return [launcher for launcher, _ in kernels] | ||
|
|
||
|
|
||
| def keep_compiled(launcher, *args, **kwargs): |
There was a problem hiding this comment.
nothing exercises keep_compiled, the only real launch test skips unless gfx950. add a fake-launcher test for each early return and the _cf set/skip?
There was a problem hiding this comment.
Added to tests/test_mono_runtime.py, using a fake @flyc.jit launcher (no GPU). They cover the kept build (_cf set, arguments resolved, _make_compiled_function called with the one key) and each early return: no build or two builds, a Constexpr parameter, an unannotated parameter, and each missing FlyDSL internal. They also cover compile_kernels keeping the build, WidthBuilds not compiling a kept launcher a second time, and the version guard.
| return None | ||
| bound = sig.bind(*args, **kwargs) | ||
| bound.apply_defaults() | ||
| launcher._resolve_arguments(bound.arguments) |
There was a problem hiding this comment.
you guard _make_compiled_function with getattr but then call _resolve_arguments and import private flydsl names unguarded, so the fallback is void. fence these too, or version gate..
There was a problem hiding this comment.
Fixed. _resolve_arguments is now looked up with getattr along with _make_compiled_function and _sig, and _runtime_argument returns False if FlyDSL's types can't be imported. Any missing internal leaves the launcher unchanged, so the first launch just traces again. The checked builds are also version-gated now (see the qwen comment).
| them traced and checked there): a build FlyDSL has cached loads without tracing or compiling, and the ranks of a node | ||
| share one compile of a program (FlyDSL's compile lock). The resource gate and the code's hazard scan read the built | ||
| code object either way. | ||
| ``check_memory``). FlyDSL traces a build before it looks it up in its disk cache (its key is the traced module), so a |
There was a problem hiding this comment.
stale stuff left around: _ENV_IGNORED comment still says the cache switch sets itself, hi_runner still says a restart loads untraced, and the contract tests still force cache=0 so they never hit the warm path. please clean up..
There was a problem hiding this comment.
Cleaned up:
_ENV_IGNOREDnow says it ignores whether the disk cache is on, because that decides where a build comes from, not what it is.- The
hi_runnercomment now says a restart traces the builds but compiles none. CLAUDE.md's contract command no longer setsFLYDSL_RUNTIME_ENABLE_CACHE=0.- The V4.1 and MiniMax-M3 contract tests no longer force the cache off. I ran them twice against a fresh cache dir: 12 passed cold (64 s) and 12 passed warm (20 s), so the warm path traces, records and checks.
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (4)
- Drop
keep_compiled: don't build FlyDSL'sCompiledFunctionfrom its private members and write it into aiter's private_cfslot.atom/mono/runtime/compile.py:60-89probes_make_compiled_function/_sig/_mem_cachewithgetattrand silently returns None when they are missing. It then calls_resolve_argumentsand importsflydsl.compiler.jit_argumentwithout a guard (valar-cop is right: the fallback does nothing), and it setslauncher._cf, which aiter'stensor_shim._run_compiled(L331) happens to read. That is the #3162 pattern again: the next FlyDSL or aiter change turns this into a silent second trace, or into a wrong callable. The double trace comes from FlyDSL 0.5.0 (it traces before any cache hit, andCOMPILE_ONLY=1returns None), and FlyDSL already fixed it upstream:flyc.compile(launcher, *args, compile_only=True)(FlyDSL#1392) never launches and returns the callable. This is the same call as #3276: put it in aiter, then use it from ATOM. aiter's_preload_compiled(tensor_shim.py L347) should return and keep that callable once aiter's FlyDSL pin includes #1392, and ATOM should call it instead of the 5keep_compiledcall sites and thegetattr(..., "_cf", None)checks inwidths.py:98andgfx1250/runner.py:231. This also changes the first launch of everyWidthBuildsuser: MiniMax-M3, Kimi-K3 and the DSA indexer, none of which this PR tested. Until then, split it out: the cache and srcloc parts below don't depend on it. - Finish removing the cache switch: leftovers still describe a cache that hides the trace.
hi/emit/compile.py:58-61(_ENV_IGNORED"the cache switchcompile_tracedsets itself"),hi_runner.py:1080("a restart loads them untraced"),CLAUDE.md:135, andtests/test_deepseek_v41_mono_contract.py:36,59/tests/test_minimax_m3_mono_contract.py:34, which still setFLYDSL_RUNTIME_ENABLE_CACHE=0, so CI never runs the warm-cache path this PR depends on. Remove the setenv lines so the contract tests trace a cached build. - Fix
docs/environment_variables.md:208:FLYDSL_DEBUG_MAX_LOC_DEPTH=1no longer does anything. Serving builds now run undersrcloc.unlocated(), and checked builds run underscoped(), which overwrites the variable withDEPTH. Delete the sentence or say that the variable is overridden. - Add CPU tests for the new branches.
srcloc.unlocated()andscoped()should swap and restore the setting (the old test was deleted with no replacement).WidthBuilds._compiledshould skip a kernel whose build is already kept. Also cover whatever replaces item 1. Nothing exercises them today; the only launch test is gated to gfx950.
Smaller: the compile_traced docstring at hi/emit/compile.py:159 is 160+ chars on one line, so rewrap it. This PR edits the same _env(...) / trace() docstring lines as #3087 (FLYDSL_GPU_ARCH only; FlyDSL#1410 removed ARCH), so whichever lands second has to drop ARCH= from qwen38/.../compile.py and hi/emit/compile.py. On valar-cop's version-pin question: aiter main already pins flydsl==0.5.0 (requirements.txt L25), and 0.5.0 traces before every cache lookup (jit_function.py: "Every ordinary JIT/compile request traces before ANY artifact hit"). Dropping FLYDSL_RUNTIME_ENABLE_CACHE=0 is therefore correct, and no version guard is needed.
Reviewed at 0d6a60c.
|
ci triage for 7bf5237:
|
| bound = sig.bind(*args, **kwargs) | ||
| bound.apply_defaults() | ||
| resolve(bound.arguments) | ||
| launcher._cf = make(next(iter(builds)), bound, None) |
There was a problem hiding this comment.
still the same design.. let's move the kept-callable path into aiter/FlyDSL, or split keep_compiled out and land the disk-cache/srcloc parts alone
There was a problem hiding this comment.
Split out in 160e70f: keep_compiled, its call sites, the _cf reads in widths.py / gfx1250/runner.py and its tests are gone (the runners and WidthBuilds are back to main). This PR is now the disk-cache and srcloc parts only. The kept callable will go into aiter's _preload_compiled via flyc.compile(..., compile_only=True) (FlyDSL#1392) once aiter's FlyDSL pin includes it; 0.5.0 does not.
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (4)
- Drop
keep_compiledfrom ATOM. Fencing it did not fix the problem: it is now a silent probe of FlyDSL's private internals.atom/mono/runtime/compile.py:86-89reads_make_compiled_function/_resolve_arguments/_sig/_mem_cachewithgetattr(..., None)._runtime_argument(L103-115) wraps FlyDSL's privatejit_argumentimport intry/except ImportError: return False. This is the pattern #3162 and #3276 ruled out. With the new fences, any FlyDSL rename brings the double trace back with no error and no log. A probe built from FlyDSL's JitFunction shape, with FlyDSL'sjit_argumentmodule missing, returns None, sets no_cfand logs nothing (probe_r20.py). aiter already owns this step:tensor_shim._preload_compiled(L347) builds without dispatching and prefers FlyDSL's publicJitFunction.preload(), and_run_compiled(L331) is the only reader of_cf. Make_preload_compiledkeep the callable it gets frompreload()/compile_only=True(FlyDSL#1392) in aiter. ATOM then calls it fromcompile_kernelsand drops the 5 call sites plus thegetattr(..., "_cf", None)reads inwidths.py:98andgfx1250/runner.py:232. Or split this part out and land the cache/srcloc change on its own (valar-cop asked for the same at L99). - Remove
require_tracing_flydsl: it is a FlyDSL version sniff, and aiter already enforces the version.compile.py:22,124-138comparesimportlib.metadata.version("flydsl")against"0.5.0". aiter main pinsflydsl==0.5.0(requirements.txt) and already refuses an older FlyDSL at import (aiter/ops/flydsl/__init__.py:17,_MIN_FLYDSL_VERSION = Version("0.5.0")). Your own reply says so. Delete_TRACING_FLYDSL, the three calls (trace, bothcompile_traced) andtest_the_hand_off_checks_need_a_flydsl_that_traces_cached_builds. - Fix
docs/environment_variables.md:208(r18 item 3, not addressed). It still says "FLYDSL_DEBUG_MAX_LOC_DEPTH=1shortens the trace of a first start's builds". Serving builds run undersrcloc.unlocated(), which captures no locations, and checked builds run underscoped(), which overwrites the variable, so it changes nothing. Delete that sentence. - Add the srcloc tests that r18 item 4 asked for. Nothing under
tests/importssrclocnow. Test thatscoped()sets and restoresFLYDSL_DEBUG_MAX_LOC_DEPTH(both when it was unset and when it was set), and thatunlocated()swaps and restorescapture_user_locationin all three_HOLDERS. Also cover itsenable_debug_infobranch.
Smaller: tests/test_deepseek_v41_mono_hi_contract.py:7 still says load-time builds skip the checks "(FlyDSL's disk cache)"; it should say serving(). #3087 is still open and also touches ARCH= at qwen38/.../compile.py:65 and hi/emit/compile.py:168, so whichever PR lands second has to drop ARCH=. CI is green.
Re-reviewed at 2eb404e. Already fixed: stale cache-switch comments, CLAUDE.md command and the contract tests' FLYDSL_RUNTIME_ENABLE_CACHE=0 (r18 item 2); fake-launcher tests for keep_compiled / WidthBuilds skip; compile_traced docstring wrap.
|
@valarLip @valar-cop thanks. I missed the review bodies last round. Addressed in 160e70f:
Smaller:
Warm-disk-cache runs of checked builds on FlyDSL 0.5.0 (valar-cop), against a fresh
The numbers are in the updated PR description. |
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (3)
- Rebase onto main: the PR conflicts, so no Pre Checkin / ATOM Test has run at 160e70f. GitHub reports
mergeable_state: dirty.docs/environment_variables.md(the hi paragraph, which #3302 rewrote for TP2/TP8) andtests/test_deepseek_v41_mono_hi_contract.py(#3302 parametrized it overSERVED_TPS×INDEX_LAYOUTS) both conflict. At this head only readthedocs ran. The green runs are for 2eb404e, which still hadkeep_compiled. - While rebasing, drop the cache-off line that #3279 added on main.
tests/test_deepseek_v41_gfx1250.py:612(test_the_step_kernel_traces_on_several_threads_at_once) setsFLYDSL_RUNTIME_ENABLE_CACHE=0with the comment "a cached build is not traced". That is the claim this PR removes everywhere else. FlyDSL 0.5.0 traces before any artifact hit (jit_function.py: "Every ordinary JIT/compile request traces before ANY artifact hit"). Delete the setenv and the comment so the gfx1250 trace test also runs warm. - Retitle the PR: after the split it no longer traces each build once.
keep_compiledis gone, and your description says the first launch still traces each program a second time. Something like "[mono] Keep FlyDSL's disk cache on for checked builds; no source locations for serving builds" matches the diff.
Not blocking: srcloc.unlocated() still swaps capture_user_location in three FlyDSL modules. If FlyDSL adds a fourth holder, the saving silently disappears; it only costs time, the binary does not change. A FlyDSL knob (e.g. FLYDSL_DEBUG_MAX_LOC_DEPTH=0 → Location.unknown()) would let ATOM drop the patch. Please don't add a hasattr or fingerprint guard instead. #3087 is still open: whichever PR lands second drops ARCH=.
Re-reviewed at 160e70f. Already fixed: keep_compiled and its call sites and _cf reads removed (r20 #1); require_tracing_flydsl removed (r20 #2); FLYDSL_DEBUG_MAX_LOC_DEPTH=1 docs sentence removed (r20 #3); srcloc tests for scoped/unlocated incl. the exception and debug-info branches (r20 #4); hi_contract test docstring names serving(); stale cache-switch comments, CLAUDE.md and contract tests (r18 #2). FlyDSL 0.5.0 trace-before-lookup and location-free cache key verified against its source; srcloc restore semantics checked with a CPU probe.
160e70f to
c9ed3ad
Compare
|
@valarLip addressed in c9ed3ad:
Against a fresh cache dir, cold then warm: mono runtime, hi package, gfx1250, the V4.1 / MiniMax-M3 contract tests and the hi contract (every served TP × layout) all pass, 218 passed both times. On srcloc: agreed, no guard. A FlyDSL knob for location-free tracing (e.g. |
valar-cop
left a comment
There was a problem hiding this comment.
LGTM
ci Accuracy (DeepSeek-V4.1-Flash DSpark, deepseek-ai/DeepSeek-V4.1-Flash, --kv_cache_dtype fp8 --ind...) failed — not sure yet (low): The harness log for this job is empty (0 lines), so it cannot be classified. It passes 7/7 on main in the same window; this PR only changes the source locations of hi builds and the disk cache of checked builds, neither of which changes the served binary with debug info off.
FAILED job
Accuracy (DeepSeek-V4.1-Flash DSpark, ...(ATOM Test), runner linux-atom-do-mi350x-8-k65lt-runner-vmcmp
next: Re-run the job and attach the log; if it reproduces, compare the served hi binary/cache entry against main before attributing it to this PR. https://github.com/ROCm/ATOM/actions/runs/43789054856/job/133047502308
ci ATOM Test Gate failed — not sure yet (medium): The ATOM Test aggregator failed alongside the accuracy job above and carries no log of its own; it most likely reports that job's failure rather than failing on its own (it also failed once on main at 2026-11-15T16:08).
FAILED job
ATOM Test Gate(ATOM Test), runner GitHub Actions 1011030627
same job on main: ['2026-11-16T00:13 success', '2026-11-16T00:13 success', '2026-11-15T16:08 failure', ...]
next: Nothing to fix in this PR; the gate should clear once the accuracy job above is re-run successfully. https://github.com/ROCm/ATOM/actions/runs/43789054856/job/133076055662
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (2)
- Re-run the red
Accuracy (DeepSeek-V4.1-Flash DSpark)job; nothing in this PR needs to change for it. The job scored 0.9098 flexible-extract against a 0.91 bar (run 43789054856). The two main runs either side of your rebase scored 0.9212 and 0.9181. This job cannot reach the code you changed: it serves--kv_cache_dtype fp8, andhi_gate.py:114-116turns the hi path off for anything but bf16. The catalog note for the model says "a single red is noise until a rerun agrees with it".ATOM Test Gateis red only because of this job. - Say in the serving docstrings that debug info brings the locations back.
hi/emit/compile.py:21("capture no source locations"),:157("no source locations") and the comment at:164("serving: none") all say serving builds have no locations. Butsrcloc.unlocated()falls back toscoped()whenFLYDSL_DEBUG_ENABLE_DEBUG_INFOis on, as its own docstring says. Add "(unless FLYDSL_DEBUG_ENABLE_DEBUG_INFO is set)" or similar.
Not blocking: #3087 is still open, so whichever PR lands second drops ARCH= from qwen38/.../compile.py and hi/emit/compile.py. Optionally annotate compile_traced as -> Record | None.
Re-reviewed at c9ed3ad. Already fixed: rebased onto main and merges cleanly with main d2d5a6b; #3302's TP2/TP4/TP8 docs paragraph and SERVED_TPS × INDEX_LAYOUTS parametrization kept (r25 #1). The gfx1250 multi-thread test no longer sets FLYDSL_RUNTIME_ENABLE_CACHE=0; git grep only finds it in _ENV_IGNORED (r25 #2). PR retitled (r25 #3). Still fixed from earlier rounds: keep_compiled, _cf reads and require_tracing_flydsl (0 hits). Checked: FlyDSL 0.5.0 re-reads FLYDSL_DEBUG_MAX_LOC_DEPTH on every capture (meta.py:159, env.py:40-50), so scoped() works; aiter main pins flydsl==0.5.0, which matches the docs bump. CPU probes: unlocated()/scoped() restore on nesting, on exceptions and in the debug-info branch; the conditional with ... as rec binds rec in both branches.
|
@valarLip addressed in 7bf5237:
#3087: whichever PR lands second drops |
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (1)
- Merge main into the branch (or push any update) so Pre Checkin and ATOM Test run again; nothing in this PR needs to change. At 7bf5237,
Run non-GPU unit testsfails 2 tests ('Mxfp4MoEMethod' object has no attribute 'use_flydsl_herd',atom/model_ops/moe.py:2350).Check Pre Checkin SignalandATOM Test Gateare red for that reason, and the GPU accuracy jobs were skipped. This PR doesn't touch that code. #2713 (02905b7) broke the test on main, and main failed the same way (Pre Checkin run 43820641381 at 7e2f723). #3393 (0819cb3) fixed it. A plain re-run reuses the old merge ref, so it would fail again. The branch merges cleanly with main 0819cb3.
Not blocking: #3087 is still open, so whichever PR lands second drops ARCH= from qwen38/.../compile.py and hi/emit/compile.py.
Re-reviewed at 7bf5237. Already fixed: the serving docstrings now say debug info brings the source locations back (hi/emit/compile.py:21-22, :157-159, the comment at :166, and the srcloc.py module docstring) (r28 #2). compile_traced is annotated -> Record | None. The V4.1-Flash DSpark accuracy job passed on re-run with 0.9219 flexible-extract (run 43789054856 attempt 2; ATOM Test Gate green at c9ed3ad) (r28 #1). Still fixed from earlier rounds: 0 hits for keep_compiled and require_tracing_flydsl; FLYDSL_RUNTIME_ENABLE_CACHE appears only in _ENV_IGNORED. The only reader of a serving build's None record (test_deepseek_v41_mono_hi_contract.py:39) checks for it.
…tions for serving builds FlyDSL 0.5.0 traces a build before it looks it up in its disk cache (the key is the traced module), so a cached build is still traced, recorded and checked. Stop forcing FLYDSL_RUNTIME_ENABLE_CACHE=0 in Qwen3.8's and the V4.1 hi checked builds and in the V4.1 / MiniMax-M3 contract and gfx1250 trace tests, and fix the comments and docs that said a cached build is not traced. srcloc: drop the 0.4-era fingerprinted location capture; scoped() keeps one frame a location for the checked builds, unlocated() captures none for the engine's builds (without debug info the binary and its cache key carry no locations). CPU tests for both. The first launch still traces each program again (COMPILE_ONLY returns no callable); that belongs in aiter's _preload_compiled once its FlyDSL pin includes FlyDSL#1392. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
srcloc.unlocated() falls back to scoped() when FLYDSL_DEBUG_ENABLE_DEBUG_INFO is set; the hi compile_traced docstrings and the srcloc module docstring now say so. compile_traced is annotated -> Record | None. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@valarLip rebased onto main 19c292f, which includes #3393 (0819cb3), so Pre Checkin and ATOM Test run again on the fixed main. The only conflict was the hi paragraph in |
7bf5237 to
682b664
Compare
Problem
FlyDSL 0.5.0 traces a
@flyc.jitlauncher on every call, before it looks the build up in its caches (the cache key is the traced module). Qwen3.8'scompile_tracedand DeepSeek-V4.1 hi's checked builds still forcedFLYDSL_RUNTIME_ENABLE_CACHE=0on top of that, so every process compiled their LLVM again. And the engine's builds captured a 5-frame source location for every traced op, which they never use.Changes
compile_tracedno longer turn it off, and neither do the V4.1 / MiniMax-M3 contract tests and the gfx1250 multi-thread trace test, so CI now runs the warm-cache path. A cached build is still traced, recorded and checked; it is only not compiled again. Stale comments and docs that described a cache which hides the trace are fixed (hi/emit/compile.py,hi_runner.py,CLAUDE.md,docs/environment_variables.md).atom/mono/runtime/srcloc.pyscoped()keeps one frame per location for the checked builds.unlocated()captures no locations for the engine's builds. Without debug info, FlyDSL leaves locations out of both the binary and its cache key, so the binary is the same. With debug info on, it falls back toscoped().Not in this PR: the first launch still traces each program a second time. FlyDSL 0.5.0's
COMPILE_ONLY=1returns no callable, so aiter's_run_compiled→flyc.compiletraces again. The fix belongs in aiter:_preload_compiledkeeps the callable fromflyc.compile(..., compile_only=True)(FlyDSL#1392, merged but not in the 0.5.0 release aiter pins). That will be a follow-up once aiter's FlyDSL pin includes it.Measurements (compile-only builds for gfx950,
ARCH=gfx950, MI308X host, one TP rank)compile_traced, all 8 programs, warm cacheTesting
blackandruffare clean on the changed files.tests/test_mono_runtime.py:srcloc.scoped()sets and restoresFLYDSL_DEBUG_MAX_LOC_DEPTH, both when it was unset and when it was set;srcloc.unlocated()swaps and restorescapture_user_locationin all three holders, also on an exception;unlocated()'s debug-info branch.FLYDSL_RUNTIME_CACHE_DIR, cold then warm:test_mono_runtime.py,test_deepseek_v41_mono_hi_package.py,test_deepseek_v41_gfx1250.py, the V4.1 / MiniMax-M3 contract tests andtest_deepseek_v41_mono_hi_contract.py(now every served TP × index layout) pass both times: 218 passed, 1 skipped.test_qwen38_mono: its GPU-launch tests abort withhipErrorNoBinaryForGpuon the test box (gfx942, kernels built for gfx950); the same on the base commit.Note: #3087 also touches the
_env(..., ARCH=...)lines inqwen38/.../compile.pyandhi/emit/compile.py. Whichever PR lands second dropsARCH=(FlyDSL#1410).🤖 Generated with Claude Code