Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
131 commits
Select commit Hold shift + click to select a range
6696747
xxx
valarLip Aug 8, 2025
aaf83fa
update1
junhaha666 Aug 12, 2025
722a2b1
update for running qwen3
ZhangLirong-amd Aug 13, 2025
2923b6f
enable attention for aiter
ZhangLirong-amd Aug 13, 2025
65154e4
fix attention holding bug
ZhangLirong-amd Aug 13, 2025
855f671
enable aiter pa
junhaha666 Aug 15, 2025
af84ec7
revert self.to_dim
ZhangLirong-amd Aug 17, 2025
ad27df6
fix pa_asm acc issur
ZhangLirong-amd Aug 19, 2025
5a53f48
fix acc issue and support bf16 and fp8 quant
ZhangLirong-amd Aug 19, 2025
f962000
add download_weights_from_hf
junhaha666 Aug 19, 2025
3b7c16d
support rotary aiter
ZhangLirong-amd Aug 20, 2025
7d01684
use aiter get_rope
ZhangLirong-amd Aug 20, 2025
225b83d
minor refine
valarLip Aug 21, 2025
a17d4be
enable TP
junhaha666 Aug 22, 2025
5a4824a
update
valarLip Aug 22, 2025
b025dce
enable torch profiler, but only main process
junhaha666 Aug 22, 2025
535966a
refine
valarLip Aug 22, 2025
8c5d966
fix ressource release
valarLip Aug 22, 2025
571c81d
refine norm
valarLip Aug 22, 2025
183e9e0
fix recompilation errors in torch.compile and fullgraph=true
ZhangLirong-amd Aug 25, 2025
71dad51
impl cuda_graph capture size
ZhangLirong-amd Aug 26, 2025
b73657e
fix_allreduce_with_graph
valarLip Aug 26, 2025
783edbe
Enable llama and Simple support for quantification
junhaha666 Aug 26, 2025
d76687c
update graph cap info
valarLip Aug 27, 2025
a3c5ddd
enable fp8 quant linear (ptpc, per tensor, per block )
junhaha666 Sep 1, 2025
6164865
move outside cudagraph logic before compile
ZhangLirong-amd Sep 5, 2025
3872932
enable larger cuda capture graph
ZhangLirong-amd Sep 5, 2025
0a57dd5
support profiling for torch compile and tp
HaonanWang98 Sep 8, 2025
26fa369
match with new aiter code
valarLip Sep 15, 2025
f912c67
close to vllm v1
valarLip Sep 21, 2025
104cfcc
using aiter's softmax
valarLip Sep 21, 2025
2d5f75e
minor refine
valarLip Sep 21, 2025
1f7a993
support fused_moe and Mixtral
HaonanWang98 Sep 22, 2025
ca99a4f
remove vllm dependency
HaonanWang98 Sep 23, 2025
d078d35
filter duplicated safetensors file
HaonanWang98 Sep 23, 2025
b636e79
remove op register for topk
HaonanWang98 Sep 23, 2025
b3c66d2
fix eager
valarLip Sep 24, 2025
42f6bde
fix tp1
valarLip Sep 24, 2025
85882f0
Merge pull request #1 from valarLip/enable_aiter_kernels
valarLip Sep 24, 2025
cfe90ea
support torch compile in atom
ZhangLirong-amd Sep 19, 2025
b633ad3
support torch compile and fix acc by impl context
ZhangLirong-amd Sep 26, 2025
e18fb81
fix some bug
ZhangLirong-amd Sep 26, 2025
0671737
add level=1
ZhangLirong-amd Sep 26, 2025
2f356f6
add some copyright and remove some logger info
ZhangLirong-amd Sep 28, 2025
bb8544c
support other model
ZhangLirong-amd Sep 28, 2025
580f877
fix mixtral bug
ZhangLirong-amd Sep 28, 2025
7457615
merge output intead of ret one by one
valarLip Sep 28, 2025
e80d47e
clean
valarLip Oct 1, 2025
c63812f
add async_process
valarLip Oct 6, 2025
5037ee7
[draft] add deferred output
valarLip Oct 6, 2025
89a7e14
Merge pull request #2 from valarLip/enable_torch_compile
junhaha666 Oct 9, 2025
7b17aa6
add atom example confluence
ZhangLirong-amd Oct 9, 2025
16a01fd
rename
ZhangLirong-amd Oct 9, 2025
135a237
Merge pull request #5 from valarLip/example_atom
ZhangLirong-amd Oct 9, 2025
a73f7bc
support ds
jiayyu Oct 11, 2025
1879b15
example
jiayyu Oct 11, 2025
06e1862
fix slop mapping
jiayyu Oct 13, 2025
3dd24ba
mt weight load
valarLip Oct 13, 2025
dece197
check multiprocessing before initialization
Oct 14, 2025
50e7d31
Merge pull request #9 from valarLip/fix_mp
valarLip Oct 14, 2025
e00cdfd
Merge pull request #7 from valarLip/ds_load2
valarLip Oct 14, 2025
e569177
update readme
HaonanWang98 Oct 14, 2025
1dc1968
Merge pull request #10 from valarLip/update_readme
valarLip Oct 14, 2025
5e62de1
use aiter concat_and_cache_mla
junhaha666 Oct 14, 2025
4e1b810
Merge pull request #11 from valarLip/ds_load2
valarLip Oct 14, 2025
b893b57
update install flow; readme; (#13)
valarLip Oct 15, 2025
ba7ca0e
log cudagraph info
valarLip Oct 15, 2025
9dd7621
fix_minor_issue (#15)
amd-ruitang3 Oct 16, 2025
00d269c
add openai server for accuracy testing (#12)
HaonanWang98 Oct 17, 2025
7843c55
draft deffer output
ZhangLirong-amd Oct 15, 2025
3be94c4
draft deffer output 2
ZhangLirong-amd Oct 15, 2025
4d4a6d6
fix acc issue
ZhangLirong-amd Oct 16, 2025
29d2c49
commit on opt deffer
ZhangLirong-amd Oct 16, 2025
b2436c2
remove d2h in prepare decode
ZhangLirong-amd Oct 16, 2025
b1da4f6
use np
ZhangLirong-amd Oct 16, 2025
557f949
reduce prepare_input_ids overhead in prefill
ZhangLirong-amd Oct 17, 2025
721e7a3
rename inputbatch to prevbatch
ZhangLirong-amd Oct 17, 2025
d996e19
fix profile by push cover
ZhangLirong-amd Oct 17, 2025
374c30e
rename and remove some useless code
ZhangLirong-amd Oct 17, 2025
2b6d8a7
finalize deferred_out
valarLip Oct 19, 2025
0659952
support torch compile
jiayyu Oct 14, 2025
c18d8ca
Merge pull request #14 from valarLip/enable_deffer_out
ZhangLirong-amd Oct 20, 2025
84c1164
fix model_runner (#17)
junhaha666 Oct 21, 2025
b8fc207
support level 1 and 3
ZhangLirong-amd Oct 23, 2025
2fec5bf
fix for gfx950
valarLip Oct 23, 2025
b57255c
draft deffer output
ZhangLirong-amd Oct 15, 2025
35c960c
draft deffer output 2
ZhangLirong-amd Oct 15, 2025
70b80e8
fix acc issue
ZhangLirong-amd Oct 16, 2025
a6bf952
commit on opt deffer
ZhangLirong-amd Oct 16, 2025
f9df1dc
remove d2h in prepare decode
ZhangLirong-amd Oct 16, 2025
d37d75d
use np
ZhangLirong-amd Oct 16, 2025
cf53ec8
reduce prepare_input_ids overhead in prefill
ZhangLirong-amd Oct 17, 2025
405985b
rename inputbatch to prevbatch
ZhangLirong-amd Oct 17, 2025
d3bae13
fix profile by push cover
ZhangLirong-amd Oct 17, 2025
6214e82
rename and remove some useless code
ZhangLirong-amd Oct 17, 2025
123569a
finalize deferred_out
valarLip Oct 19, 2025
0ab11a6
fix model_runner (#17)
junhaha666 Oct 21, 2025
608216b
add barrier to avoid get_context thread error
ZhangLirong-amd Oct 23, 2025
84fe02b
fix level 1 acc issue
ZhangLirong-amd Oct 25, 2025
1d09c8e
replace SharedMemory by MessageQueue
valarLip Oct 26, 2025
db4ecdf
fix kv_indptr
valarLip Oct 26, 2025
73e5150
minor update
valarLip Oct 26, 2025
1301219
refine
valarLip Oct 26, 2025
8df1f6d
Merge branch 'main' into ds_comp
valarLip Oct 26, 2025
e319f7d
Merge pull request #8 from valarLip/ds_comp
valarLip Oct 26, 2025
f8f2049
Support piecewise deepseek and different attn backend (#19)
ZhangLirong-amd Oct 27, 2025
f58c89a
Support parents attention for mla and mha (#22)
ZhangLirong-amd Oct 28, 2025
0baea4f
add serving benchmark (#21)
HaonanWang98 Oct 29, 2025
5b583d0
use mixed_sample_outer_exponential (#20)
junhaha666 Oct 30, 2025
47bc0b5
Fix weak ref error in level 3 in torch 2.6 and 2.7 (#23)
ZhangLirong-amd Oct 30, 2025
cb802e2
Forward context support (#25)
ZhangLirong-amd Nov 3, 2025
42e104b
Ds v32 new 2 (#24)
junhaha666 Nov 4, 2025
a556796
Profile (#26)
HaonanWang98 Nov 5, 2025
19d5a2c
Fix some bug in atom (#28)
ZhangLirong-amd Nov 7, 2025
524d043
use aiter gemm_a8w8_blockscale_bpreshuffle (#29)
junhaha666 Nov 7, 2025
ba3339d
apply black format
valarLip Nov 8, 2025
c8ab77a
mla_qkv_proj_fused (#30)
amd-ruitang3 Nov 9, 2025
8fd0e73
use aiter persistent mla (#27)
junhaha666 Nov 10, 2025
7096453
broadcast rank 0 sampling result when using deferred_out (#33)
junhaha666 Nov 12, 2025
0d471e5
enable fused_qk_rope_cat_and_cache_mla (#35)
valarLip Nov 13, 2025
821b310
move attention meta data init to backend (#34)
junhaha666 Nov 14, 2025
7315f88
move block table (#37)
junhaha666 Nov 14, 2025
b5fdf42
Add async engine and stream mode (#38)
HaonanWang98 Nov 18, 2025
e1a306c
support dp in atom and dp attention+tp/ep moe
ZhangLirong-amd Nov 19, 2025
681e774
fix some bug
ZhangLirong-amd Nov 19, 2025
279a020
use real cuda graph size and remove RR counter
ZhangLirong-amd Nov 19, 2025
99daae4
use rr count in add_request
ZhangLirong-amd Nov 19, 2025
e399df0
format moe
ZhangLirong-amd Nov 19, 2025
90475db
Merge pull request #39 from valarLip/dp_attn_v1
ZhangLirong-amd Nov 21, 2025
b27db6c
merge README
valarLip Nov 21, 2025
f107d77
Merge branch 'tmp' of https://github.com/ROCm/ATOM into tmp
valarLip Nov 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# atom version
atom/_version.py

# Prerequisites
*.d

# Compiled Object files
*.slo
*.lo
*.o
*.obj

# Precompiled Headers
*.gch
*.pch

# Compiled Dynamic libraries
*.so
*.dylib
*.dll
build
blob

# Fortran module files
*.mod
*.smod

# Compiled Static libraries
*.lai
*.la
*.a
*.lib

# Executables
*.exe
*.out
*.app

# python cache
__pycache__
*.pyc
.eggs
*.egg-info

# editor
.vscode

# debug folder
debug
aiter_logs
*.log
149 changes: 148 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,149 @@
# ATOM
AiTer Optimized Model

**ATOM** (AiTer Optimized Model) is a lightweight vLLM-like implementation, focusing on integration and optimization based on [aiter](https://github.com/ROCm/aiter).

## 🚀 Features

- **ROCm Optimized**: Built on AMD's ROCm platform with torch compile support
- **Model Support**: Compatible with Qwen, Llama, and Mixtral models
- **Easy Integration**: Simple API for quick deployment

## 📋 Requirements

- AMD GPU with ROCm support
- Docker

## 🛠️ Installation

### 1. Pull Docker Image

```bash
docker pull rocm/pytorch:rocm7.0.2_ubuntu24.04_py3.12_pytorch_release_2.8.0
```

### 2. Run Docker Container

```bash
docker run -it --network=host \
--device=/dev/kfd \
--device=/dev/dri \
--group-add video \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v $HOME:/home/$USER \
-v /mnt:/mnt \
-v /data:/data \
--shm-size=16G \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
rocm/pytorch:rocm7.0.2_ubuntu24.04_py3.12_pytorch_release_2.8.0
```

### 3. Clone and Setup

```bash
pip install aiter -i https://mkmartifactory.amd.com/artifactory/api/pypi/hw-orc3pypi-prod-local/simple
git clone https://github.com/valarLip/atom.git
cd ./atom
pip install .
```

## 💡 Usage

### Basic Example

The default optimization level is 3 (running with torch compile). Supported models include **Qwen**, **Llama**, and **Mixtral**.

```bash
python -m atom.examples.simple_inference --model meta-llama/Meta-Llama-3-8B
```

> **Note:** First-time execution may take approximately 10 minutes for model compilation.

### Performance profiling

Profile offline inference
```bash
python -m atom.examples.profile_offline --model Qwen/Qwen3-0.6B
```
Or profile offline with custom input length
```bash
python -m atom.examples.profile_offline --model Qwen/Qwen3-0.6B --random-input --input-length 1024 --output-length 32
```

Profile online inference, after starting the server
```bash
python -m atom.examples.profile_online
```
Or profile online with custom input length
```bash
python -m atom.examples.profile_online --model Qwen/Qwen3-0.6B --random-input --input-length 1024 --output-length 32
```

Or directly send start profile and stop profile reuqest
```bash
curl -s -S -X POST http://127.0.0.1:8000/start_profile
```
```bash
curl -s -S -X POST http://127.0.0.1:8000/stop_profile
```

### Performance Benchmarking

Run online throughput benchmark:

start the server
```bash
python -m atom.entrypoints.openai_server --model Qwen/Qwen3-0.6B --cudagraph-capture-sizes [1,2,4,8,16,32,64,128]
```
run benchmark
```bash
python -m atom.bench.benchmark_throughput -n 128 -r 128 -c 64 -i 1020 -o 1024
```


## 📊 Performance Comparison

ATOM demonstrates significant performance improvements over vLLM:

| Model | Framework | Tokens | Time | Throughput |
|-------|-----------|--------|------|------------|
| **Qwen3-0.6B** | ATOM | 4096 | 0.25s | **16,643.74 tok/s** |
| Qwen3-0.6B | vLLM | 4096 | 0.63s | 6,543.06 tok/s |
| **Llama-3.1-8B-Instruct-FP8-KV** | ATOM | 4096 | 0.68s | **5,983.37 tok/s** |
| Llama-3.1-8B-Instruct-FP8-KV | vLLM | 4096 | 1.68s | 2,432.62 tok/s |


### Online serving throughput:

Deepseek-V3
| concurrency | IPS/QPS | prompts num | vLLM Throughput | ATOM Throughput |
|-------------|---------|-------------|-----------------|-----------------|
| 16 | 1024/1024 | 128 | 423.68 tok/s | **922.03 tok/s** |
| 32 | 1024/1024 | 128 | 629.06 tok/s | **1488.52 tok/s** |
| 64 | 1024/1024 | 128 | 760.22 tok/s | **2221.25 tok/s** |
| 128 | 1024/1024 | 128 | 1107.93 tok/s | **2254.88 tok/s** |

### Accuracy Benchmarking

First, install `lm-eval` to test model accuracy:

```bash
pip install lm-eval[api]
```

Next, start an OpenAI-compatible server using `openai_server.py`:

```bash
python -m atom.entrypoints.openai_server --model meta-llama/Meta-Llama-3-8B
```

Finally, run the evaluation by choosing your datasets:

```bash
lm_eval --model local-completions \
--model_args model=meta-llama/Meta-Llama-3-8B,base_url=http://localhost:8000/v1/completions,num_concurrent=8,max_retries=3,tokenized_requests=False \
--tasks gsm8k \
--num_fewshot 3 \
--batch_size 16
```
2 changes: 2 additions & 0 deletions atom/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
from atom.model_engine.llm_engine import LLMEngine, AsyncLLMEngine
from atom.sampling_params import SamplingParams
Loading