[serve] preset=dsv4 [serve] dpa=0 tbo=0 [serve] profiler dir: /traces/port8000_2026-07-15_12-45-00 [aiter] import [module_aiter_core] under /upstream/up-aiter/aiter/jit/module_aiter_core.so [aiter] import [module_mla_metadata] under /upstream/up-aiter/aiter/jit/module_mla_metadata.so [atom 04:45:04] Loading tokenizer from /shared/models/DeepSeek-V4-Pro... You are using a model of type `deepseek_v4` to instantiate a model of type ``. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. [atom 04:45:04] Loaded message encoder from /shared/models/DeepSeek-V4-Pro/encoding/encoding_dsv4.py [atom 04:45:04] Initializing engine with model /shared/models/DeepSeek-V4-Pro... [atom 04:45:04] Engine kwargs: {'trust_remote_code': True, 'tensor_parallel_size': 4, 'prefill_context_parallel_size': 1, 'data_parallel_size': 1, 'enforce_eager': False, 'enable_prefix_caching': False, 'port': 8006, 'kv_cache_dtype': 'fp8', 'max_model_len': 10240, 'max_num_batched_tokens': 16384, 'long_prefill_token_threshold': 0, 'attn_prefill_chunk_size': 16384, 'enable_chunked_prefill': False, 'scheduler_delay_factor': 10.0, 'max_num_seqs': 512, 'gpu_memory_utilization': 0.85, 'load_dummy': False, 'enable_expert_parallel': False, 'torch_profiler_dir': '/traces/port8000_2026-07-15_12-45-00', 'enable_dp_attention': False, 'kv_transfer_config': '{}', 'mark_trace': False, 'online_quant_config': None, 'hf_overrides': None, 'kv_cache_block_size': 16, 'compilation_config': CompilationConfig(level=3, use_cudagraph=True, local_cache_dir=None, cudagraph_capture_sizes=[1, 2, 4, 8, 16, 32, 48, 64, 128, 256, 512], cuda_graph_sizes=[512], debug_dump_path='', traced_files=set(), cache_dir='', use_inductor=True, cudagraph_mode=None, compilation_time=0.0, splitting_ops=None, cudagraph_copy_inputs=False, inductor_compile_config={}, compile_sizes=None, static_forward_context={}), 'speculative_config': None, 'enable_tbo': False, 'enable_tbo_decode': False, 'enable_low_latency': False} The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. You are using a model of type `deepseek_v4` to instantiate a model of type `deepseek_v3`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. `rope_parameters`'s factor field must be a float >= 1, got 16 `rope_parameters`'s beta_fast field must be a float, got 32 `rope_parameters`'s beta_slow field must be a float, got 1 `torch_dtype` is deprecated! Use `dtype` instead! [atom 04:45:04] Engine Core Mgr: Creating EngineCore for DP rank 0/1 [atom 04:45:04] Creating EngineCore process: DP rank 0, will use GPUs 0 to 3 [atom 04:45:04] Engine Core Mgr: Starting EngineCore for DP rank 0/1 [aiter] import [module_aiter_core] under /upstream/up-aiter/aiter/jit/module_aiter_core.so [aiter] import [module_mla_metadata] under /upstream/up-aiter/aiter/jit/module_mla_metadata.so [aiter] message queue communication handle: Handle(local_reader_ranks=[0, 1, 2, 3], buffer_handle=(4, 16777216, 10, 'psm_f473915e'), local_subscribe_addr='ipc:///tmp/b32062df-1ce1-446b-841d-b1f4a75617a5', remote_subscribe_addr=None, remote_addr_ipv6=False) [aiter] import [module_aiter_core] under /upstream/up-aiter/aiter/jit/module_aiter_core.so [aiter] import [module_mla_metadata] under /upstream/up-aiter/aiter/jit/module_mla_metadata.so [aiter] import [module_aiter_core] under /upstream/up-aiter/aiter/jit/module_aiter_core.so [aiter] import [module_mla_metadata] under /upstream/up-aiter/aiter/jit/module_mla_metadata.so [aiter] import [module_aiter_core] under /upstream/up-aiter/aiter/jit/module_aiter_core.so [aiter] import [module_mla_metadata] under /upstream/up-aiter/aiter/jit/module_mla_metadata.so [aiter] import [module_aiter_core] under /upstream/up-aiter/aiter/jit/module_aiter_core.so [aiter] import [module_mla_metadata] under /upstream/up-aiter/aiter/jit/module_mla_metadata.so [aiter] merge tuned file under model_configs/ and configs/ /upstream/up-aiter/aiter/configs/bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv4_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/kimi_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/glm47_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/llama70B_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen32B_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/llama405B_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv3_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m3_eagle_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3_5_397b_bf16_tuned_gemm.csv [atom 04:45:14] Create lazy wrapper for FusedMoE to change the naming [atom 04:45:14] ModelRunner rank=3, dp_rank_local=0, local_device_rank=3, device=cuda:3 [aiter] merge tuned file under model_configs/ and configs/ /upstream/up-aiter/aiter/configs/bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv4_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/kimi_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/glm47_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/llama70B_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen32B_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/llama405B_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv3_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m3_eagle_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3_5_397b_bf16_tuned_gemm.csv [atom 04:45:14] Create lazy wrapper for FusedMoE to change the naming [atom 04:45:14] ModelRunner rank=2, dp_rank_local=0, local_device_rank=2, device=cuda:2 [aiter] merge tuned file under model_configs/ and configs/ /upstream/up-aiter/aiter/configs/bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv4_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/kimi_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/glm47_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/llama70B_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen32B_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/llama405B_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv3_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m3_eagle_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3_5_397b_bf16_tuned_gemm.csv [aiter] merge tuned file under model_configs/ and configs/ /upstream/up-aiter/aiter/configs/bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv4_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/kimi_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/glm47_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/llama70B_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen32B_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/llama405B_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv3_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m3_eagle_bf16_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3_5_397b_bf16_tuned_gemm.csv [atom 04:45:14] Create lazy wrapper for FusedMoE to change the naming [atom 04:45:14] ModelRunner rank=0, dp_rank_local=0, local_device_rank=0, device=cuda:0 [atom 04:45:14] Create lazy wrapper for FusedMoE to change the naming [atom 04:45:14] ModelRunner rank=1, dp_rank_local=0, local_device_rank=1, device=cuda:1 [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [aiter] import [module_custom_all_reduce] under /upstream/up-aiter/aiter/jit/module_custom_all_reduce.so [aiter] import [module_custom_all_reduce] under /upstream/up-aiter/aiter/jit/module_custom_all_reduce.so [aiter] import [module_custom_all_reduce] under /upstream/up-aiter/aiter/jit/module_custom_all_reduce.so [aiter] import [module_custom_all_reduce] under /upstream/up-aiter/aiter/jit/module_custom_all_reduce.so [aiter] load RCCL version: 2.27.7 [aiter] import [module_quick_all_reduce] under /upstream/up-aiter/aiter/jit/module_quick_all_reduce.so [aiter] import [module_quick_all_reduce] under /upstream/up-aiter/aiter/jit/module_quick_all_reduce.so [aiter] import [module_quick_all_reduce] under /upstream/up-aiter/aiter/jit/module_quick_all_reduce.so [aiter] import [module_quick_all_reduce] under /upstream/up-aiter/aiter/jit/module_quick_all_reduce.so [aiter] message queue communication handle: Handle(local_reader_ranks=[1, 2, 3], buffer_handle=(3, 4194304, 6, 'psm_f9aa780f'), local_subscribe_addr='ipc:///tmp/c84c334b-c97b-4f0d-97b6-8c589988d065', remote_subscribe_addr=None, remote_addr_ipv6=False) [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3 [aiter] load RCCL version: 2.27.7 [aiter] rank 1 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1 [aiter] rank 3 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 3, EP rank 3 [aiter] rank 2 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 2, EP rank 2 [aiter] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0 `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! [atom 04:45:17] disable_mmap: False [atom 04:45:17] disable_mmap: False [atom 04:45:17] disable_mmap: False Loading safetensors shards[/shared/models/DeepSeek-V4-Pro]: 0%| | 0/64 [00:00/.cache/atom/torch_compile_cache/0e013f4943/rank_2/backbone for vLLM's torch.compile [atom 04:46:05] Dynamo bytecode transform time: 2.10 s [atom 04:46:05] Directly load the compiled graph(s) for dynamic shape from the cache, took 0.328 s [atom 04:46:05] Using cache directory: /.cache/atom/torch_compile_cache/0e013f4943/rank_1/backbone for vLLM's torch.compile [atom 04:46:05] Dynamo bytecode transform time: 2.09 s [aiter] import [module_mhc] under /upstream/up-aiter/aiter/jit/module_mhc.so [aiter] import [module_quant] under /upstream/up-aiter/aiter/jit/module_quant.so [aiter] merge tuned file under model_configs/ and configs/ /upstream/up-aiter/aiter/configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_dsv3.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_qwen3.5_397b.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_qwen3_235b.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv4_a8w8_blockscale_bpreshuffle_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_a8w8_blockscale_bpreshuffle_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_mm2.5.csv [aiter] import [module_gemm_a8w8_blockscale_bpreshuffle] under /upstream/up-aiter/aiter/jit/module_gemm_a8w8_blockscale_bpreshuffle.so [atom 04:46:05] Model load done: /shared/models/DeepSeek-V4-Pro [aiter] import [module_activation] under /upstream/up-aiter/aiter/jit/module_activation.so [aiter] shape is M:10240, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [atom 04:46:06] Directly load the compiled graph(s) for dynamic shape from the cache, took 0.320 s [aiter] merge tuned file under model_configs/ and configs/ /upstream/up-aiter/aiter/configs/tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_ds_v3.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_glm5.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv3_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_qwen3_235b.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_qwen3_5_397b.csv:/upstream/up-aiter/aiter/configs/model_configs/glm47_fp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_minimax-m2_5.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_fp8fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_fp8fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv4_fp8fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_i4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m25_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3_235b_bf16_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3_5_397b_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3next_80b_fp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m3_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m3_mxfp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_glm5_1.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_mxfp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_a16w4_tuned_fmoe.csv [aiter] [fused_moe] duplicate tuned rows after disabling act_type in /tmp/aiter_configs/tuned_fmoe.csv; keeping first match for 449 rows [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t128x256x256_bnt0_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t64x256x256_reduce_sbm128') for ('gfx950', 256, 16384, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] import [module_moe_sorting_opus] under /upstream/up-aiter/aiter/jit/module_moe_sorting_opus.so [aiter] import [module_mhc] under /upstream/up-aiter/aiter/jit/module_mhc.so [aiter] import [module_quant] under /upstream/up-aiter/aiter/jit/module_quant.so [aiter] merge tuned file under model_configs/ and configs/ /upstream/up-aiter/aiter/configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_dsv3.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_qwen3.5_397b.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_qwen3_235b.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv4_a8w8_blockscale_bpreshuffle_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_a8w8_blockscale_bpreshuffle_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_mm2.5.csv [aiter] import [module_gemm_a8w8_blockscale_bpreshuffle] under /upstream/up-aiter/aiter/jit/module_gemm_a8w8_blockscale_bpreshuffle.so [aiter] import [module_activation] under /upstream/up-aiter/aiter/jit/module_activation.so [aiter] shape is M:10240, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [atom 04:46:06] Using cache directory: /.cache/atom/torch_compile_cache/0e013f4943/rank_0/backbone for vLLM's torch.compile [atom 04:46:06] Dynamo bytecode transform time: 2.05 s [aiter] import [module_moe_topk] under /upstream/up-aiter/aiter/jit/module_moe_topk.so [aiter] merge tuned file under model_configs/ and configs/ /upstream/up-aiter/aiter/configs/tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_ds_v3.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_glm5.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv3_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_qwen3_235b.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_qwen3_5_397b.csv:/upstream/up-aiter/aiter/configs/model_configs/glm47_fp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_minimax-m2_5.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_fp8fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_fp8fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv4_fp8fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_i4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m25_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3_235b_bf16_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3_5_397b_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3next_80b_fp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m3_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m3_mxfp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_glm5_1.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_mxfp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_a16w4_tuned_fmoe.csv [aiter] import [module_rmsnorm_quant] under /upstream/up-aiter/aiter/jit/module_rmsnorm_quant.so [aiter] shape is M:1, N:32320, K:7168 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0 [aiter] [fused_moe] duplicate tuned rows after disabling act_type in /tmp/aiter_configs/tuned_fmoe.csv; keeping first match for 449 rows [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t128x256x256_bnt0_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t64x256x256_reduce_sbm128') for ('gfx950', 256, 16384, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] import [module_moe_sorting_opus] under /upstream/up-aiter/aiter/jit/module_moe_sorting_opus.so [aiter] import [module_moe_topk] under /upstream/up-aiter/aiter/jit/module_moe_topk.so [atom 04:46:06] Directly load the compiled graph(s) for dynamic shape from the cache, took 0.325 s [aiter] import [module_rmsnorm_quant] under /upstream/up-aiter/aiter/jit/module_rmsnorm_quant.so [aiter] shape is M:1, N:32320, K:7168 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0 [aiter] import [module_mhc] under /upstream/up-aiter/aiter/jit/module_mhc.so [aiter] import [module_quant] under /upstream/up-aiter/aiter/jit/module_quant.so [aiter] merge tuned file under model_configs/ and configs/ /upstream/up-aiter/aiter/configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_dsv3.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_qwen3.5_397b.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_qwen3_235b.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv4_a8w8_blockscale_bpreshuffle_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_a8w8_blockscale_bpreshuffle_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_mm2.5.csv [aiter] import [module_gemm_a8w8_blockscale_bpreshuffle] under /upstream/up-aiter/aiter/jit/module_gemm_a8w8_blockscale_bpreshuffle.so [aiter] import [module_activation] under /upstream/up-aiter/aiter/jit/module_activation.so [aiter] shape is M:10240, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] merge tuned file under model_configs/ and configs/ /upstream/up-aiter/aiter/configs/tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_ds_v3.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_glm5.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv3_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_qwen3_235b.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_qwen3_5_397b.csv:/upstream/up-aiter/aiter/configs/model_configs/glm47_fp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_minimax-m2_5.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_fp8fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_fp8fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv4_fp8fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_i4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m25_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3_235b_bf16_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3_5_397b_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3next_80b_fp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m3_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m3_mxfp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_glm5_1.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_mxfp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_a16w4_tuned_fmoe.csv [aiter] [fused_moe] duplicate tuned rows after disabling act_type in /tmp/aiter_configs/tuned_fmoe.csv; keeping first match for 449 rows [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t128x256x256_bnt0_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t64x256x256_reduce_sbm128') for ('gfx950', 256, 16384, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] import [module_moe_sorting_opus] under /upstream/up-aiter/aiter/jit/module_moe_sorting_opus.so [aiter] import [module_moe_topk] under /upstream/up-aiter/aiter/jit/module_moe_topk.so [aiter] import [module_rmsnorm_quant] under /upstream/up-aiter/aiter/jit/module_rmsnorm_quant.so [aiter] shape is M:1, N:32320, K:7168 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0 [atom 04:46:08] Using cache directory: /.cache/atom/torch_compile_cache/0e013f4943/rank_3/backbone for vLLM's torch.compile [atom 04:46:08] Dynamo bytecode transform time: 1.97 s [atom 04:46:08] Directly load the compiled graph(s) for dynamic shape from the cache, took 0.313 s [aiter] import [module_sample] under /upstream/up-aiter/aiter/jit/module_sample.so [aiter] import [module_mhc] under /upstream/up-aiter/aiter/jit/module_mhc.so [aiter] import [module_quant] under /upstream/up-aiter/aiter/jit/module_quant.so [aiter] merge tuned file under model_configs/ and configs/ /upstream/up-aiter/aiter/configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_dsv3.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_qwen3.5_397b.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_qwen3_235b.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv4_a8w8_blockscale_bpreshuffle_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_a8w8_blockscale_bpreshuffle_tuned_gemm.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_mm2.5.csv [aiter] import [module_gemm_a8w8_blockscale_bpreshuffle] under /upstream/up-aiter/aiter/jit/module_gemm_a8w8_blockscale_bpreshuffle.so [aiter] import [module_activation] under /upstream/up-aiter/aiter/jit/module_activation.so [aiter] shape is M:10240, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] merge tuned file under model_configs/ and configs/ /upstream/up-aiter/aiter/configs/tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_ds_v3.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_glm5.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv3_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_qwen3_235b.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_qwen3_5_397b.csv:/upstream/up-aiter/aiter/configs/model_configs/glm47_fp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_minimax-m2_5.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_fp8fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_fp8fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/dsv4_fp8fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/kimik2_i4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m25_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3_235b_bf16_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3_5_397b_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/qwen3next_80b_fp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m3_fp4_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/minimax_m3_mxfp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/a8w8_blockscale_tuned_fmoe_glm5_1.csv:/upstream/up-aiter/aiter/configs/model_configs/glm5_mxfp8_tuned_fmoe.csv:/upstream/up-aiter/aiter/configs/model_configs/gptoss_a16w4_tuned_fmoe.csv [aiter] [fused_moe] duplicate tuned rows after disabling act_type in /tmp/aiter_configs/tuned_fmoe.csv; keeping first match for 449 rows [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t128x256x256_bnt0_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t64x256x256_reduce_sbm128') for ('gfx950', 256, 16384, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] import [module_moe_sorting_opus] under /upstream/up-aiter/aiter/jit/module_moe_sorting_opus.so [aiter] import [module_sample] under /upstream/up-aiter/aiter/jit/module_sample.so [aiter] import [module_moe_topk] under /upstream/up-aiter/aiter/jit/module_moe_topk.so [aiter] import [module_rmsnorm_quant] under /upstream/up-aiter/aiter/jit/module_rmsnorm_quant.so [aiter] shape is M:1, N:32320, K:7168 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0 [aiter] import [module_sample] under /upstream/up-aiter/aiter/jit/module_sample.so [aiter] import [module_sample] under /upstream/up-aiter/aiter/jit/module_sample.so [atom 04:46:12] Model Runner3/4: warmup_model 5.89 seconds with 1 reqs 10240 tokens [atom 04:46:12] Model warmup done: /shared/models/DeepSeek-V4-Pro [atom 04:46:12] Memory budget: total_gpu=287.98GB, free=62.51GB, utilization=0.85, budget=244.79GB, peak_torch=209.38GB, cudagraph_est=0.51GB, safety=5.76GB, available_for_kv=29.14GB, block_bytes=1153024, num_kvcache_blocks=15222 [atom 04:46:12] Per-req cache pool: bytes_per_slot=25.59MB, max_slots=512, tensor_total=12.80GB, equiv_blocks_per_req=24, pool_blocks=15222 [atom 04:46:12] Concurrent capacity vs context length (max_model_len=10240, block_size=128, max_slots=512, pool_blocks=15222): 10% ( 1024 tok): 8 blk/req → max_concurrent=512 (bound by slots) 30% ( 3072 tok): 24 blk/req → max_concurrent=512 (bound by slots) 50% ( 5120 tok): 40 blk/req → max_concurrent=380 (bound by blocks) 70% ( 7168 tok): 56 blk/req → max_concurrent=271 (bound by blocks) 90% ( 9216 tok): 72 blk/req → max_concurrent=211 (bound by blocks) 100% ( 10240 tok): 80 blk/req → max_concurrent=190 (bound by blocks) [atom 04:46:12] Model Runner0/4: warmup_model 7.68 seconds with 1 reqs 10240 tokens [atom 04:46:12] Model warmup done: /shared/models/DeepSeek-V4-Pro [atom 04:46:12] Model Runner1/4: warmup_model 8.64 seconds with 1 reqs 10240 tokens [atom 04:46:12] Model warmup done: /shared/models/DeepSeek-V4-Pro [atom 04:46:12] Model Runner2/4: warmup_model 9.16 seconds with 1 reqs 10240 tokens [atom 04:46:12] Model warmup done: /shared/models/DeepSeek-V4-Pro [atom 04:46:12] Memory budget: total_gpu=287.98GB, free=62.22GB, utilization=0.85, budget=244.79GB, peak_torch=209.38GB, cudagraph_est=0.51GB, safety=5.76GB, available_for_kv=29.14GB, block_bytes=1153024, num_kvcache_blocks=15222 [atom 04:46:12] Per-req cache pool: bytes_per_slot=25.59MB, max_slots=512, tensor_total=12.80GB, equiv_blocks_per_req=24, pool_blocks=15222 [atom 04:46:12] Concurrent capacity vs context length (max_model_len=10240, block_size=128, max_slots=512, pool_blocks=15222): 10% ( 1024 tok): 8 blk/req → max_concurrent=512 (bound by slots) 30% ( 3072 tok): 24 blk/req → max_concurrent=512 (bound by slots) 50% ( 5120 tok): 40 blk/req → max_concurrent=380 (bound by blocks) 70% ( 7168 tok): 56 blk/req → max_concurrent=271 (bound by blocks) 90% ( 9216 tok): 72 blk/req → max_concurrent=211 (bound by blocks) 100% ( 10240 tok): 80 blk/req → max_concurrent=190 (bound by blocks) [atom 04:46:12] Memory budget: total_gpu=287.98GB, free=62.22GB, utilization=0.85, budget=244.79GB, peak_torch=209.38GB, cudagraph_est=0.51GB, safety=5.76GB, available_for_kv=29.14GB, block_bytes=1153024, num_kvcache_blocks=15222 [atom 04:46:12] Memory budget: total_gpu=287.98GB, free=62.13GB, utilization=0.85, budget=244.79GB, peak_torch=209.38GB, cudagraph_est=0.51GB, safety=5.76GB, available_for_kv=29.14GB, block_bytes=1153024, num_kvcache_blocks=15222 [atom 04:46:12] Per-req cache pool: bytes_per_slot=25.59MB, max_slots=512, tensor_total=12.80GB, equiv_blocks_per_req=24, pool_blocks=15222 [atom 04:46:12] Per-req cache pool: bytes_per_slot=25.59MB, max_slots=512, tensor_total=12.80GB, equiv_blocks_per_req=24, pool_blocks=15222 [atom 04:46:12] Concurrent capacity vs context length (max_model_len=10240, block_size=128, max_slots=512, pool_blocks=15222): 10% ( 1024 tok): 8 blk/req → max_concurrent=512 (bound by slots) 30% ( 3072 tok): 24 blk/req → max_concurrent=512 (bound by slots) 50% ( 5120 tok): 40 blk/req → max_concurrent=380 (bound by blocks) 70% ( 7168 tok): 56 blk/req → max_concurrent=271 (bound by blocks) 90% ( 9216 tok): 72 blk/req → max_concurrent=211 (bound by blocks) 100% ( 10240 tok): 80 blk/req → max_concurrent=190 (bound by blocks) [atom 04:46:12] Concurrent capacity vs context length (max_model_len=10240, block_size=128, max_slots=512, pool_blocks=15222): 10% ( 1024 tok): 8 blk/req → max_concurrent=512 (bound by slots) 30% ( 3072 tok): 24 blk/req → max_concurrent=512 (bound by slots) 50% ( 5120 tok): 40 blk/req → max_concurrent=380 (bound by blocks) 70% ( 7168 tok): 56 blk/req → max_concurrent=271 (bound by blocks) 90% ( 9216 tok): 72 blk/req → max_concurrent=211 (bound by blocks) 100% ( 10240 tok): 80 blk/req → max_concurrent=190 (bound by blocks) [atom 04:46:12] Binding KV cache for target model starting at layer_id=0 [atom 04:46:12] Binding KV cache for target model starting at layer_id=0 [atom 04:46:12] Binding KV cache for target model starting at layer_id=0 [atom 04:46:12] Binding KV cache for target model starting at layer_id=0 [atom 04:46:12] KV cache allocation mismatch: expected=29.143GB, actual=27.252GB, diff=6.5% [atom 04:46:12] KV cache allocation mismatch: expected=29.143GB, actual=27.252GB, diff=6.5% [atom 04:46:12] KV cache allocation mismatch: expected=29.143GB, actual=27.252GB, diff=6.5% [atom 04:46:12] KV cache allocation mismatch: expected=29.143GB, actual=27.252GB, diff=6.5% /up-venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. return func(*args, **kwargs) [rank0]:[W715 04:46:12.274126194 ProcessGroupNCCL.cpp:5140] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group() 0%| | 0/11 [00:00/upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_128x128.co [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_80x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_80x64_splitk_clean.co [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 512, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:512, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter43fp8gemm_bf16_blockscale_BpreShuffle_128x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_128x128.co [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_80x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_80x64_splitk_clean.co [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 512, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] LoadKernel: _ZN5aiter43fp8gemm_bf16_blockscale_BpreShuffle_128x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_128x128.co [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_80x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_80x64_splitk_clean.co [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 512, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] LoadKernel: _ZN5aiter43fp8gemm_bf16_blockscale_BpreShuffle_128x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_128x128.co [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_80x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_80x64_splitk_clean.co [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 512, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_80x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_80x128.co [aiter] shape is M:512, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_80x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_80x128.co [aiter] shape is M:512, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_80x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_80x128.co [aiter] shape is M:512, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_80x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_80x128.co [aiter] shape is M:512, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_96x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_96x64_splitk_clean.co [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_96x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_96x64_splitk_clean.co [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_96x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_96x64_splitk_clean.co [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_96x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_96x64_splitk_clean.co [aiter] import [module_dsv4_rotate_quant] under /upstream/up-aiter/aiter/jit/module_dsv4_rotate_quant.so [aiter] import [module_dsv4_rotate_quant] under /upstream/up-aiter/aiter/jit/module_dsv4_rotate_quant.so [aiter] import [module_dsv4_rotate_quant] under /upstream/up-aiter/aiter/jit/module_dsv4_rotate_quant.so [aiter] import [module_dsv4_rotate_quant] under /upstream/up-aiter/aiter/jit/module_dsv4_rotate_quant.so [aiter] import [module_top_k_per_row] under /upstream/up-aiter/aiter/jit/module_top_k_per_row.so [aiter] import [module_top_k_per_row] under /upstream/up-aiter/aiter/jit/module_top_k_per_row.so [aiter] import [module_top_k_per_row] under /upstream/up-aiter/aiter/jit/module_top_k_per_row.so [aiter] import [module_top_k_per_row] under /upstream/up-aiter/aiter/jit/module_top_k_per_row.so Capturing bs=512, max_q_len=1: 9%|▉ | 1/11 [00:00<00:09, 1.08it/s] Capturing bs=256, max_q_len=1: 9%|▉ | 1/11 [00:00<00:09, 1.08it/s][aiter] shape is M:256, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:256, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:256, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:256, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_64x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_64x128.co [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_64x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_64x128.co [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_64x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_64x128.co [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_64x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_64x128.co [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2') for ('gfx950', 256, 256, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2') for ('gfx950', 256, 256, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2') for ('gfx950', 256, 256, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2') for ('gfx950', 256, 256, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:256, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:256, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:256, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:256, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! Capturing bs=256, max_q_len=1: 18%|█▊ | 2/11 [00:01<00:06, 1.42it/s] Capturing bs=128, max_q_len=1: 18%|█▊ | 2/11 [00:01<00:06, 1.42it/s][aiter] shape is M:128, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:128, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:128, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:128, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 128, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:128, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 128, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_64x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_64x64_splitk_clean.co [aiter] shape is M:128, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 128, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:128, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 128, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_64x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_64x64_splitk_clean.co [aiter] shape is M:128, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_64x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_64x64_splitk_clean.co [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_64x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_64x64_splitk_clean.co Capturing bs=128, max_q_len=1: 27%|██▋ | 3/11 [00:01<00:04, 1.71it/s] Capturing bs=64, max_q_len=1: 27%|██▋ | 3/11 [00:01<00:04, 1.71it/s] [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_32x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_32x128.co [aiter] shape is M:64, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_32x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_32x128.co [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_32x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_32x128.co [aiter] shape is M:64, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:64, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_32x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_32x128.co [aiter] shape is M:64, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_xcd4_persist') for ('gfx950', 256, 64, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_xcd4_persist') for ('gfx950', 256, 64, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_xcd4_persist') for ('gfx950', 256, 64, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_xcd4_persist') for ('gfx950', 256, 64, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_48x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_48x128.co [aiter] shape is M:64, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_48x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_48x128.co [aiter] shape is M:64, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_48x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_48x128.co [aiter] shape is M:64, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter42fp8gemm_bf16_blockscale_BpreShuffle_48x128E hsaco: /upstream/up-aiter/hsa//gfx950/fp8gemm_blockscale/fp8gemm_bf16_blockscale_BpreShuffle_48x128.co [aiter] shape is M:64, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! Capturing bs=64, max_q_len=1: 36%|███▋ | 4/11 [00:02<00:04, 1.67it/s] Capturing bs=48, max_q_len=1: 36%|███▋ | 4/11 [00:02<00:04, 1.67it/s][aiter] shape is M:48, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:48, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:48, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:48, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:48, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:48, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:48, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:48, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! Capturing bs=48, max_q_len=1: 45%|████▌ | 5/11 [00:02<00:03, 1.85it/s] Capturing bs=32, max_q_len=1: 45%|████▌ | 5/11 [00:02<00:03, 1.85it/s][aiter] shape is M:32, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:32, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:32, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:32, N:16384, K:1536, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_xcd4') for ('gfx950', 256, 32, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:32, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_xcd4') for ('gfx950', 256, 32, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:32, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_xcd4') for ('gfx950', 256, 32, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:32, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_xcd4') for ('gfx950', 256, 32, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:32, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! Capturing bs=32, max_q_len=1: 55%|█████▍ | 6/11 [00:03<00:02, 1.96it/s] Capturing bs=16, max_q_len=1: 55%|█████▍ | 6/11 [00:03<00:02, 1.96it/s][aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 16, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:16, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 16, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:16, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 16, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:16, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 16, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:16, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! Capturing bs=16, max_q_len=1: 64%|██████▎ | 7/11 [00:03<00:02, 1.97it/s] Capturing bs=8, max_q_len=1: 64%|██████▎ | 7/11 [00:03<00:02, 1.97it/s] [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x256x256_atomic_bnt2_persist') for ('gfx950', 256, 8, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x256x256_atomic_bnt2_persist') for ('gfx950', 256, 8, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x256x256_atomic_bnt2_persist') for ('gfx950', 256, 8, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x256x256_atomic_bnt2_persist') for ('gfx950', 256, 8, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:8, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:8, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:8, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:8, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! Capturing bs=8, max_q_len=1: 73%|███████▎ | 8/11 [00:04<00:01, 2.01it/s] Capturing bs=4, max_q_len=1: 73%|███████▎ | 8/11 [00:04<00:01, 2.01it/s][aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x64x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x256x256_atomic_bnt2_persist') for ('gfx950', 256, 4, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x64x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x256x256_atomic_bnt2_persist') for ('gfx950', 256, 4, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x64x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x256x256_atomic_bnt2_persist') for ('gfx950', 256, 4, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x64x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x256x256_atomic_bnt2_persist') for ('gfx950', 256, 4, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:4, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:4, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:4, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:4, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_32x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_32x64_splitk_clean.co [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_32x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_32x64_splitk_clean.co [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_32x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_32x64_splitk_clean.co [aiter] LoadKernel: _ZN5aiter39bf16gemm_fp32bf16_tn_32x64_splitk_cleanE hsaco: /upstream/up-aiter/hsa//gfx950/bf16gemm/bf16gemm_fp32bf16_tn_32x64_splitk_clean.co Capturing bs=4, max_q_len=1: 82%|████████▏ | 9/11 [00:04<00:01, 1.98it/s] Capturing bs=2, max_q_len=1: 82%|████████▏ | 9/11 [00:04<00:01, 1.98it/s][aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 2, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:2, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 2, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:2, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 2, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:2, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 2, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:2, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! Capturing bs=2, max_q_len=1: 91%|█████████ | 10/11 [00:05<00:00, 2.11it/s] Capturing bs=1, max_q_len=1: 91%|█████████ | 10/11 [00:05<00:00, 2.11it/s][aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_xcd4_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 1, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_xcd4_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 1, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_xcd4_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 1, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] [fused_moe] using 2stage (kernelName1='flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_xcd4_fp8', kernelName2='flydsl_moe2_afp8_wfp4_bf16_t32x128x256_atomic_bnt2_persist') for ('gfx950', 256, 1, 7168, 768, 384, 6, , 'torch.bfloat16', 'torch.float8_e4m3fn', 'torch.float4_e2m1fn_x2', 'QuantType.per_1x32', True, False) [aiter] shape is M:1, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:1, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:1, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! [aiter] shape is M:1, N:7168, K:768, not found tuned config in /tmp/aiter_configs/a8w8_blockscale_bpreshuffle_tuned_gemm.csv, will use default config! Capturing bs=1, max_q_len=1: 100%|██████████| 11/11 [00:05<00:00, 2.26it/s] Capturing bs=1, max_q_len=1: 100%|██████████| 11/11 [00:05<00:00, 1.93it/s] [aiter] Registering 1107 cuda graph addresses [aiter] Registering 1107 cuda graph addresses [aiter] Registering 1107 cuda graph addresses [aiter] Registering 1107 cuda graph addresses [atom 04:46:18] Post-init memory: actual=254.87GB (88.5%), target=244.79GB (85%) [atom 04:46:18] Actual GPU memory usage (88.5%) exceeds target (85%) by 3.5%. Consider reducing gpu_memory_utilization. [atom 04:46:18] Post-init memory: actual=255.25GB (88.6%), target=244.79GB (85%) [atom 04:46:18] Actual GPU memory usage (88.6%) exceeds target (85%) by 3.6%. Consider reducing gpu_memory_utilization. [atom 04:46:18] Engine Core: cudagraph capture[1, 2, 4, 8, 16, 32, 48, 64, 128, 256, 512] cost: 6.72 seconds [atom 04:46:18] Engine Core: load model runner success [atom 04:46:19] Post-init memory: actual=255.25GB (88.6%), target=244.79GB (85%) [atom 04:46:19] Actual GPU memory usage (88.6%) exceeds target (85%) by 3.6%. Consider reducing gpu_memory_utilization. [atom 04:46:19] Post-init memory: actual=255.37GB (88.7%), target=244.79GB (85%) [atom 04:46:19] Actual GPU memory usage (88.7%) exceeds target (85%) by 3.7%. Consider reducing gpu_memory_utilization. [atom 04:46:19] Engine Core: EngineCore fully initialized and ready [atom 04:46:19] Engine Core Mgr: DP rank 0 is fully initialized and ready [atom 04:46:19] Engine Core Mgr: All EngineCores are fully initialized and ready [atom 04:46:19] Engine Core Mgr: All 1 EngineCores initialized and ready [atom 04:46:19] KV transfer config loaded: {} [atom 04:46:19] LLMEngine init with 1 data parallel ranks [atom 04:46:19] LLMEngine init with 1 data parallel ranks [atom 04:46:19] Starting server on 0.0.0.0:8000 (loop=uvloop)... INFO: Started server process [1939011] INFO: Waiting for application startup. [atom 04:46:19] Server started successfully and ready to accept requests INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)