Skip to content

[Pallas TPU] Remote semaphore signaling copies a local buffer twice #42508

Open

Description

@ayaka14732

Description

Switching a local semaphore signal to a remote signal adds two full copies of a 128 MiB donated HBM buffer. The kernel updates only 4 KiB and performs no remote data DMA.

Run in one process with at least two visible TPU chips:

import jax
import numpy as np
from jax.experimental import pallas as pl
from jax.experimental.pallas import tpu as pltpu
from jax.sharding import PartitionSpec as P

mesh = jax.make_mesh((2,), ('d',), devices=[d for d in jax.local_devices() if d.core_on_chip == 0][:2])

for remote in (False, True):
    @jax.jit(donate_argnums=(0,))
    @jax.shard_map(
        mesh=mesh,
        in_specs=P('d'),
        out_specs=P('d'),
        check_vma=False,
    )
    def run(x: jax.Array) -> jax.Array:
        ref = jax.new_ref(x)

        @pl.kernel(mesh=pltpu.TensorCoreMesh(axis_name='tc', num_cores=1), scratch_types=(pltpu.VMEM((8, 128), np.float32), pltpu.SemaphoreType.REGULAR))
        def kernel(tile: jax.Ref, sem: jax.Ref) -> None:
            pltpu.sync_copy(ref.at[:8], tile)
            tile[...] += 1
            pltpu.sync_copy(tile, ref.at[:8])
            pl.semaphore_signal(sem, device_id={'d': 1 - jax.lax.axis_index('d')} if remote else None)
            pl.semaphore_wait(sem)

        kernel()
        return jax.freeze(ref)

    x = jax.device_put(np.zeros((524288, 128), np.float32), jax.NamedSharding(mesh, P('d')))
    compiled = run.lower(x).compile()
    print(f'{remote=}, copies={compiled.as_text().count(" copy(")}')
    result = np.asarray(compiled(x)).reshape(2, 262144, 128)
    np.testing.assert_array_equal(result[:, :8], 1)
    np.testing.assert_array_equal(result[:, 8:], 0)

Expected behavior

No full-buffer copies in either case.

Actual behavior

remote=False, copies=0
remote=True, copies=2

Both numerical checks pass. The two copies each cover the full 128 MiB buffer.

Notes

Related to #42263. Remote semaphore signaling alone is sufficient to reproduce the behavior with pl.kernel.

System info (python version, jaxlib version, accelerator, etc.)

jax:    0.12.1.dev20261111+fd48df9b6
jaxlib: 0.12.1.dev20261110
libtpu: 0.0.53.dev20261110+nightly
numpy:  2.5.4
python: 3.14.6 free-threading build (main, Jul 18 2026, 17:01:48) [Clang 22.1.3 ]
device info: TPU7x-8, 8 local devices"
process_count: 1
platform: uname_result(system='Linux', node='<redacted>', release='6.8.0-1063-gcp', version='#69-Ubuntu SMP Mon Jun 29 14:57:40 UTC 2026', machine='x86_64')

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions