Skip to content

[DSV4.1][gfx1250] single launch: thread-local trace slots for FlyDSL >= 0.5 - #3279

Open
yifehuan wants to merge 1 commit into
mainfrom
yifehuan/gfx1250-flydsl05-trace-slots
Open

yifehuan wants to merge 1 commit into
mainfrom
yifehuan/gfx1250-flydsl05-trace-slots

Conversation

@yifehuan

Copy link
Copy Markdown
Contributor

Motivation

The gfx1250 single-launch step kernel (DeepSeek-V4.1, PR #3109) does not build with FlyDSL 0.5: tracing fails with

'llvm.inttoptr' op operand #0 type from a different MLIRContext than this operation

The step kernel sets two trace-time slots while it traces -- the poll-timeout counter pointer (device.TOFLAG) and the stamp hook (stamps.HOOK) -- and clears them when its body ends. They are module-level [None] lists. FlyDSL 0.5 traces every kernel in its own MLIRContext, and a launcher that traces several ranks' kernels in parallel threads then hands one thread's IR value to another thread's trace. FlyDSL 0.4 used one context, which hid the sharing.

Change

device.TraceSlot: one trace-time value private to the tracing thread, with the same slot[0] interface. TOFLAG and HOOK use it. Serial tracing (FlyDSL 0.4) is unchanged.

Validation

On gfx1250 (CT10, 4 GPUs), the single-launch fixture (40 target + 3 draft layers, TP4, 6 rows) with the same thread-local slots:

  • FlyDSL 0.5.0 (the gfx1250-atom-k3-20261111 image's bundled FlyDSL): builds, runs all iterations, zero poll timeouts.
  • Logits vs the FlyDSL 0.4.1 run: relative L2 1.1e-4 (the kernel's known atomic-order nondeterminism), top-1 6/6.

Not covered here: the kernel's performance under FlyDSL 0.5 (the 0.5.0 run's median step was slower than 0.4.1 in this fixture; to be investigated separately).

🤖 Generated with Claude Code

…>= 0.5

The step kernel sets two trace-time slots while it traces -- the poll-timeout
counter pointer (device.TOFLAG) and the stamp hook (stamps.HOOK) -- and clears
them when its body ends. They were module-level `[None]` lists. FlyDSL 0.5
traces every kernel in its own MLIRContext, and a launcher that traces several
ranks' kernels in parallel threads then hands one thread's IR value to another
thread's trace; verification fails with "'llvm.inttoptr' op operand #0 type
from a different MLIRContext than this operation".

TraceSlot keeps the value per tracing thread behind the same `slot[0]`
interface, so serial tracing (FlyDSL 0.4) is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 3279 --add-label <label>

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ci green.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants