Skip to content

GPU sum followed by concatenate is ~10x slower without optimization barrier #42556

Open

Description

@geng-r

Description

Concatenating a float64 vector with its sum takes ~0.27 ms per call, whereas adding a jax.lax.optimization_barrier around the sum reduces this to ~0.03 ms.

This reproduces in a fresh environment installed using pip install "jax[cuda12]==0.12.0.1", using a RTX 5090 GPU.

This was found while investigating a JAX-MD simulation slowdown, where the same simulation took ~25 ms/step with JAX 0.11.1 and ~71 ms/step with 0.12.0.1. A barrier around the relevant charge-equilibration sum restored the 25 ms/step.

The following minimal example reproduces the inefficient fusion in both versions of JAX, but does not show the version-to-version regression:

from timeit import timeit
import jax
import jax.numpy as jnp

jax.config.update("jax_enable_x64", True)
jax.config.update("jax_platforms", "cuda")

@jax.jit
def fused(x):
    return jnp.concatenate((x, x.sum(keepdims=True)))

@jax.jit
def barrier(x):
    total = jax.lax.optimization_barrier(x.sum(keepdims=True))
    return jnp.concatenate((x, total))


x = jnp.ones(10_000)
for fn in (fused, barrier):
    fn(x).block_until_ready()  # compile before timing
    seconds = timeit(lambda: fn(x).block_until_ready(), number=100)
    print(f"{fn.__name__}: {seconds / 100 * 1000:.3f} ms/call")

    # RTX 5090, JAX/jaxlib 0.12.0.1:
    # fused: ~0.27 ms/call
    # barrier: ~0.03 ms/call

Possibly related historical reports: #18424 and openxla/xla#7152.

System info (python version, jaxlib version, accelerator, etc.)

jax:    0.12.0.1
jaxlib: 0.12.0.1
numpy:  2.5.4
python: 3.12.16 | packaged by Anaconda, Inc. | (main, Oct 26 2026, 20:21:38) [GCC 14.3.0]
device info: NVIDIA GeForce RTX 5090-1, 1 local devices"
process_count: 1
platform: uname_result(system='Linux', node='hidden', release='5.14.0-611.54.3.el9_7.x86_64', version='#1 SMP PREEMPT_DYNAMIC Thu May 7 16:31:24 EDT 2026', machine='x86_64')
XLA_PYTHON_CLIENT_PREALLOCATE=false
JAX_PLATFORMS=cuda

$ nvidia-smi
Thu Nov 12 14:05:30 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08             Driver Version: 570.172.08     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        On  |   00000000:D1:00.0 Off |                  N/A |
|  0%   39C    P1             19W /  600W |    1038MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 5090        On  |   00000000:F1:00.0  On |                  N/A |
|  0%   46C    P8             36W /  600W |     943MiB /  32607MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Activity

  1. self-assigned this
    on Nov 13, 2026
  2. added
    GPUIssues that apply to both CUDA and ROCM GPUs
    on Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

GPUIssues that apply to both CUDA and ROCM GPUsXLAbugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions