Repository navigation
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Description
Switching a local semaphore signal to a remote signal adds two full copies of a 128 MiB donated HBM buffer. The kernel updates only 4 KiB and performs no remote data DMA.
Run in one process with at least two visible TPU chips:
import jax
import numpy as np
from jax.experimental import pallas as pl
from jax.experimental.pallas import tpu as pltpu
from jax.sharding import PartitionSpec as P
mesh = jax.make_mesh((2,), ('d',), devices=[d for d in jax.local_devices() if d.core_on_chip == 0][:2])
for remote in (False, True):
@jax.jit(donate_argnums=(0,))
@jax.shard_map(
mesh=mesh,
in_specs=P('d'),
out_specs=P('d'),
check_vma=False,
)
def run(x: jax.Array) -> jax.Array:
ref = jax.new_ref(x)
@pl.kernel(mesh=pltpu.TensorCoreMesh(axis_name='tc', num_cores=1), scratch_types=(pltpu.VMEM((8, 128), np.float32), pltpu.SemaphoreType.REGULAR))
def kernel(tile: jax.Ref, sem: jax.Ref) -> None:
pltpu.sync_copy(ref.at[:8], tile)
tile[...] += 1
pltpu.sync_copy(tile, ref.at[:8])
pl.semaphore_signal(sem, device_id={'d': 1 - jax.lax.axis_index('d')} if remote else None)
pl.semaphore_wait(sem)
kernel()
return jax.freeze(ref)
x = jax.device_put(np.zeros((524288, 128), np.float32), jax.NamedSharding(mesh, P('d')))
compiled = run.lower(x).compile()
print(f'{remote=}, copies={compiled.as_text().count(" copy(")}')
result = np.asarray(compiled(x)).reshape(2, 262144, 128)
np.testing.assert_array_equal(result[:, :8], 1)
np.testing.assert_array_equal(result[:, 8:], 0)Expected behavior
No full-buffer copies in either case.
Actual behavior
remote=False, copies=0
remote=True, copies=2
Both numerical checks pass. The two copies each cover the full 128 MiB buffer.
Notes
Related to #42263. Remote semaphore signaling alone is sufficient to reproduce the behavior with pl.kernel.
System info (python version, jaxlib version, accelerator, etc.)
jax: 0.12.1.dev20261111+fd48df9b6
jaxlib: 0.12.1.dev20261110
libtpu: 0.0.53.dev20261110+nightly
numpy: 2.5.4
python: 3.14.6 free-threading build (main, Jul 18 2026, 17:01:48) [Clang 22.1.3 ]
device info: TPU7x-8, 8 local devices"
process_count: 1
platform: uname_result(system='Linux', node='<redacted>', release='6.8.0-1063-gcp', version='#69-Ubuntu SMP Mon Jun 29 14:57:40 UTC 2026', machine='x86_64')
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity