Skip to content

perf(checkpoint): pipeline packed prefix writes through a TQ actor - #5176

Open
macandro96 wants to merge 2 commits into
amahishi/ckpt-v2-prefix-batched-iofrom
amahishi/pr4508-prefix-write-pipeline-actor
Open

macandro96 wants to merge 2 commits into
amahishi/ckpt-v2-prefix-batched-iofrom
amahishi/pr4508-prefix-write-pipeline-actor

Conversation

@macandro96

Copy link
Copy Markdown
Contributor

What does this PR do?

Add opt-in packed generation-prefix writes, bounded preparation/write pipelining, and a colocated TransferQueue writer actor.

Stacked on #4508 (amahishi/ckpt-v2-prefix-batched-io, 788133a5545953dcdbd3a60af6b2a99d90482346). This PR does not change #4508's branch.

  • Direct column encoder and packing: build owned numeric/byte columns without intermediate per-row tensors. Pass dense/jagged columns across Ray as NumPy byte buffers, preserving exact dtype, shape, offsets, keys and tags. NumPy-supported dtypes use NumPy reshape/byte views to avoid repeated Torch dispatch; BF16 retains its exact-bit Torch path.
  • Bounded pipeline and writer actor: one preparation thread retains cut-generator/lifecycle ownership while one write stream runs. Compatible rows can merge across inventories within row/token/byte limits. The optional writer actor owns the TQ client in a separate CPU process on the generation node.

Issues

Follow-up to #4508. No issues closed.

Behavior and limits

  • All new options default off. The direct encoder, pipeline, and actor can be selected independently, except the pipeline requires direct column encoding.
  • No early acknowledgements: a cut seals only after the real TQ PUT finishes and acknowledgements are built.
  • Duplicate in-flight inventories share a future; inventories sharing model calls serialize. Cancellation of an HTTP waiter does not cancel an admitted write.
  • Retain per-call failures, deadline handling, rollback and ordered failed-write cleanup. Teardown rejects new work and closes the actor to unblock an in-flight remote PUT.
  • The writer actor currently supports TransferQueue/simple, text and router-replay tensors, not media. Terminal writes and restore GETs retain their existing client.
  • No Gym/submodule update, checkpoint-schema change, digest change, async data-plane checkpointing change, or cluster-specific launcher/profiler code.
  • Byte limits bound estimated buffered numeric payload, not total process RSS. A single row larger than the configured payload bound fails its cut and regenerates after restore.

Usage

Under the existing async-HTTP, text-only generation-prefix capture configuration:

policy:
  generation:
    vllm_cfg:
      prefix_writer_actor: true
      prefix_write_pipeline:
        enabled: true
        direct_column_encoding: true

The buffer defaults to 512 MiB, maximum write payload to 128 MiB, merged rows to 256, merged token positions to 4,194,304, and the coalescing window to 10 ms. Full settings and requirements are documented in docs/design-docs/single-controller-rollout-checkpointing.md.

Validation

On the isolated PR branch with Gym at the unchanged pin 5911d628:

  • 294 passed, 9 skipped across prefix pipeline/writer, vLLM config/hosting, and prefix-read-batcher tests. Skips require vLLM, absent from the CPU test environment.
  • Scoped Ruff checks/formatting passed. Scoped pyrefly passed with 0 errors (11 existing ignores) for the three new data-plane modules, TQ sink, and vLLM config.
  • Live CPU Ray/TQ smoke: stage two text-prefix chunks and a router tensor through the pipeline/actor, save the TQ checkpoint, clear the rows, restore through a fresh client, and verify all seven keys, reconstructed digests and route tensors.
  • Coverage includes exact bytes across all supported dtypes/BF16, empty and noncontiguous tensors, signed zero/NaN bit patterns, encoder parity, merged inventories, duplicate/overlapping calls, cancellation, bounded buffering, deadline/failure cleanup and actor teardown.
  • The existing test_tq_token_sink.py module could not collect in this CPU environment: Megatron is not installed; using the local source additionally requires Triton. The live TQ smoke and new encoder/parity tests passed. Full GPU training functional tests were not rerun from this isolated branch.

Benchmark observations

Four nodes: two TP4 generation engines, one TQ node and one load/driver node. Same pinned container and workload targets, router replay enabled, max_num_seqs=512, 630 outstanding calls and 39 inventories per engine. These are prefix-cut + TQ save/verification timings, not full Gym/trainer rollout-checkpoint latency.

Write path Snapshot 1 total Snapshot 2 total
#4508 baseline 52.44 s 107.60 s
Direct encoder + pipeline, local client 31.65 s 81.20 s
Direct encoder + pipeline + actor + NumPy packing 25.24 s 72.27 s

Single-run observations, not a statistical speedup claim: actual captured token positions can overshoot targets (baseline 6.60M / 14.28M versus full path 6.00M / 13.73M). Preparation and writes overlap, so summed phase times are not exclusive snapshot time.

Isolating the NumPy packing change reduced second-cut aggregate packing time by 85–87% across engines, but did not improve total snapshot latency in that comparison; preparation and handoff variability remain. This PR does not claim those bottlenecks are solved.

Before your PR is "Ready for review"

  • Followed contribution conventions; commit includes sign-off.
  • Added runtime/config/transport tests.
  • Ran targeted unit tests and live CPU save/restore validation.
  • Updated the existing checkpointing documentation.
  • Full GPU training functional suite / CI.

@macandro96 macandro96 added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Nov 12, 2026
@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test 981376a

@copy-pr-bot

copy-pr-bot Bot commented Nov 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Nov 12, 2026
@macandro96
macandro96 added this pull request to stack #5181 November 12, 2026 15:29
@macandro96
macandro96 requested a review from terrykong November 12, 2026 15:33

@macandro96 macandro96 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core design is sound, and several choices are worth keeping as they are:

  • One cut generator drives both the synchronous and the pipelined modes, so there is no second copy of the cut loop (the problem the #4508 review removed).
  • No receipt is sent before the rows are durable, and keys that no receipt advertised are cleared on interruption.
  • The writer client has the same put_samples / clear_samples interface as the data-plane client, so the sink does not need to know about actors.
  • The direct encoder writes exactly the same rows as the general one, and 3,600 contended stress runs of the pipeline tests showed no flaky failures.

Must fix before merge

  1. (C1) CI Lint is red: one pyrefly error and import order in 3 test files.
  2. (C2) The 78 new tests run in no CI lane, because they lack the nemo_gym mark.

How to organize this PR and remove the clutter

  1. (C3) Move the new keys into one block under rollout_recovery, next to their siblings. This removes the 2 copied limits, the hand-written bool check, and the 6 YAML hunks.
  2. (C6) Check each setup rule in one place instead of up to four (about 70 lines become about 25).
  3. (C7) Remove the benchmark timing code (about 110 lines) and keep prefix PUTs in the existing data-plane metrics.
  4. (C8) Move the writer actor into a stacked PR, so the encoder and pipeline can land first.

Together these remove about 200 lines and give each rule one home.

Other comments

  • Behavior: (C4) resume fails when the new keys change before the first trainer checkpoint; (C5) the pipeline holds calls frozen for longer; (C10) a dead writer actor is never replaced; (C24) with the pipeline on, a call whose row goes over the byte limit is never checkpointed again.
  • Smaller cleanups: (C9) writer startup timeout and error message, (C11) encoder split in the sink, (C12) copied key prefix, (C13) repeated column code, (C14) byte-estimate constants, (C15) module location.
  • Comments and docs: (C16) and (C17) restore deleted explanations; (C18) name the required keys in the docs.
  • Tests: (C19) expired prepared batch, (C20) byte limit and admission limit, (C21) encoder parity, (C22) writer startup failure, (C23) end-to-end coverage.

PR description

  • Write "Part of #3594" under Issues (not "Closes"; the stack is not complete).
  • Note that #4843 changes the same cut and route code on an older base, so whichever PR lands second needs to be ported onto the new generator.

Visual explainer: https://macandro96.github.io/gh-pages-poc/amahishi/pr-5176-prefix-writes.html

Generated by Claude Code

keys: list[str | None] = []
if past_deadline():
late.extend(prefix for _, _, _, _, prefix in batch)
return [None] * len(batch)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C1 · 2 action items, please fix in this PR. TL;DR — CI "Lint check" is red: pyrefly rejects this line (bad-return), and ruff's import-order check (I001) fails in 3 new test files.

The Lint check job reports Returned type list[None] is not assignable to declared return type list[str | None] [bad-return] here. All 4 errors are PR-introduced.

AI-1

Build the list with a comprehension. pyrefly 0.24.2 accepts it:

Suggested change
return [None] * len(batch)
return [None for _ in batch]

AI-2

Run ruff check --select I --fix on these 3 files (multi-file, so not a suggestion):

Typed-variable fix that still fails pyrefly

keys: list[str | None] = [None] * len(batch) fails with bad-assignment, so do not use it.

_assert_rolled_back,
_batch_worker_fixture,
_next_inventory,
)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C2 · 2 action items, please fix in this PR. TL;DR — the 78 new tests in this file and in test_prefix_writer_actor.py run in no CI lane: the Other lane skips both files at import, and the Nemo_Gym lane drops them because they have no nemo_gym mark.

  1. This file imports helpers from test_vllm_token_capture_hosting.py, which calls pytest.importorskip("nemo_gym.token_id_capture.staging") at import. Its pytestmark = pytest.mark.nemo_gym marks only its own tests, not the modules that import it.
  2. The Other lane has no nemo_gym installed, so both files are skipped at import.
  3. The Nemo_Gym lane runs with --nemo-gym-only, and conftest keeps only nemo_gym-marked items. Result: "78 tests collected … Running 0 items in this shard".

The "294 passed" in the PR description reproduces only with --noconftest.

AI-1

Mark this file as a Gym test file, as test_tq_token_sink.py does:

Suggested change
)
)
pytestmark = pytest.mark.nemo_gym

AI-2

Add the same line after the import block of test_prefix_writer_actor.py (other file, so not a suggestion here):

pytestmark = pytest.mark.nemo_gym

With both lines, the Nemo_Gym lane runs all 78 tests and they pass.

Alternative that keeps 60 writer tests in the Gym-free lane

Only 7 of the 67 cases in test_prefix_writer_actor.py need Gym: test_native_cut_uses_prefix_client_and_terminal_remains_local, test_worker_setup_installs_only_prefix_transport and test_worker_shutdown_closes_writer_once. Moving those 3 tests into the hosting module would let the other 60 run in the Other lane.

LOGGER = logging.getLogger(__name__)


class PrefixWritePipelineConfig(BaseModel, extra="forbid"):

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C3 · 2 action items. TL;DR — the new keys sit in policy.generation.vllm_cfg as 3 booleans plus 2 copies of existing limits, while every sibling prefix-write knob lives in rollout_recovery; this placement causes most of the clutter in the PR.

PR-introduced. The keys are new, so changing their shape is free now and a breaking change after merge.

AI-1

Replace this model and prefix_writer_actor with one block on RolloutRecoveryConfig (multi-file change, so a sketch, not a suggestion):

# nemo_rl/algorithms/single_controller_utils/config.py
class GenerationPrefixWritesConfig(BaseModel, extra="forbid"):
    # Direct column encoding plus bounded prepare/write overlap.
    pipeline: bool = False
    # Run prefix PUTs in one CPU actor on the generation node (TQ simple only).
    writer_actor: bool = False
    max_pending_inventories: Annotated[int, Field(gt=0, strict=True)] = 128
    max_buffered_payload_bytes: Annotated[int, Field(gt=0, strict=True)] = 536870912
    max_batch_payload_bytes: Annotated[int, Field(gt=0, strict=True)] = 134217728
    coalesce_wait_seconds: Annotated[float, Field(ge=0, allow_inf_nan=False)] = 0.01

    @model_validator(mode="after")
    def _buffer_holds_two_batches(self) -> "GenerationPrefixWritesConfig":
        # Room for one write in flight plus one preparation reservation.
        if self.max_buffered_payload_bytes < 2 * self.max_batch_payload_bytes:
            raise ValueError(
                "max_buffered_payload_bytes must be >= 2 * max_batch_payload_bytes"
            )
        return self


class RolloutRecoveryConfig(BaseModel, extra="allow"):
    ...
    generation_prefix_writes: GenerationPrefixWritesConfig = Field(
        default_factory=GenerationPrefixWritesConfig
    )

Then:

AI-2

In validate_single_controller_config, reject generation_prefix_writes.pipeline or .writer_actor when token_capture.enabled is false or target_level is not prefix, next to the existing top_logprobs rejection (line not in this PR's diff, so not a suggestion). Today the keys are silently ignored when token capture is off, because setup_token_capture is then never called.

Every place the new argument passes through

Make it a required argument with no default; #4508 removed the defaults on these signatures on purpose (review thread).

flat_logprobs: NotRequired[bool]
# Prefix-only TQ serialization/PUTs in one colocated CPU actor. Default off.
# Requires async HTTP capture, prefix cuts, and SimpleStorage; no media yet.
prefix_writer_actor: NotRequired[bool]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C4 · 1 action item. TL;DR — turning prefix_writer_actor or prefix_write_pipeline.* on or off between save and resume, before the first trainer checkpoint, makes setup raise ValueError, because these keys are not on the fingerprint exclusion list.

PR-introduced. These keys change how rows are written, not what is written: the direct encoder writes the same rows as the general one. When there is no trainer checkpoint yet and <checkpoint_dir>/bootstrap/ exists, setup calls validate_bootstrap_anchor, which raises:

rollout bootstrap anchor at <checkpoint_dir>/bootstrap/manifest.json is incompatible with the current rollout-semantic configuration. Changed compatibility fields:
  policy.generation.vllm_cfg.prefix_write_pipeline.enabled: False -> True
  policy.generation.vllm_cfg.prefix_writer_actor: False -> True

The sibling limits rollout_recovery.generation_prefix_batch_* are already excluded.

AI-1

Add both keys to _BOOTSTRAP_FINGERPRINT_EXCLUDED_PATHS (file not in this PR's diff, so not a suggestion):

        "policy.generation.vllm_cfg.prefix_write_pipeline",
        "policy.generation.vllm_cfg.prefix_writer_actor",

With these two entries the same resume succeeds. An entry excludes the whole subtree, so if the keys move into one block under rollout_recovery, this becomes one entry for that block.

),
None,
)
can_prepare = (

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C5 · 2 action items. TL;DR — with the pipeline on, every admitted inventory prepares a batch and keeps its calls frozen until every write queued ahead of it finishes, because can_prepare is limited only by bytes and text rows almost never reach the byte limit.

PR-introduced. This makes calls wait longer; it does not produce wrong results.

  1. Preparing a batch freezes its calls (cut_in_progress) until their rows are written and sealed.
  2. can_prepare checks only buffered + max_batch_payload_bytes <= max_buffered_payload_bytes. A text row is estimated at about 16 bytes per token + 512 bytes, so the 512 MiB buffer is not the limit, and up to max_pending_inventories (128) inventories prepare.
  3. They all wait behind the one write stream. On the base path at most one batch is frozen during one write. Here a call can stay frozen for the whole write queue (the PR reports 25-72 s per snapshot).
  4. A call that finishes or aborts while frozen waits that long for its response, inside a default-executor thread (finish, abort).

Measured with 64 calls, batches of 8 and a 50 ms PUT: base froze 8 calls at once with a 49 ms longest hold; the pipeline froze 64 calls with a 399 ms longest hold, in the same total time.

AI-1

Also limit preparation by rows, so at most one write is prepared ahead of the write in flight (not a suggestion: applied alone it fails the PR's own overlap test until that test changes):

                can_prepare = (
                    candidate is not None
                    and self._buffered + self._config.max_batch_payload_bytes
                    <= self._config.max_buffered_payload_bytes
                    # Prepare at most one write ahead of the write in flight, so
                    # calls stay frozen for about two writes, not the whole queue.
                    and sum(len(job.request.records) for job in self._prepared)
                    < self._config.max_batch_rows
                )

This keeps the overlap. With 40 ms of preparation per batch, it took 0.45 s, against 0.54 s at head and 0.73 s without the pipeline. If the row limits move to rollout_recovery, use generation_prefix_batch_size here.

AI-2

Update test_preparation_overlaps_put_but_receipts_and_locks_wait_for_durability (other file, so not a suggestion). It asserts that all 3 calls are frozen while the first PUT is blocked; with the row limit only 2 are. Assert instead that at least 2 calls are frozen while the first PUT is blocked, and that none is sealed or acknowledged before release.

pipeline = _pipeline(
recovery,
**(
{"max_batch_payload_bytes": 1, "max_buffered_payload_bytes": 2}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C20 · 3 action items. TL;DR — the "oversize" case asserts the same result as "deadline", so it never proves the payload limit caused the failure or that nothing was written; and no test splits a batch on the byte limit.

PR-introduced.

  • "oversize" checks only that every call is durable_failure and rolled back. It does not check the prefix_payload_limit reason, or that the sink received no write.
  • Removing the byte-limit flush leaves the 11 pipeline tests green.
  • The admission limit is not reached by any test.

AI-1

Remove "oversize" from the parametrize list and remove this **(...) argument, then add this test after the function (adds a test function, so not a suggestion). It needs import logging and _uncut_reasons from the hosting module:

def test_oversize_row_fails_its_call_without_a_write(
    caplog: pytest.LogCaptureFixture,
) -> None:
    recovery, sink, states, inventory = _batch_worker_fixture([2, 3])
    pipeline = _pipeline(
        recovery, max_batch_payload_bytes=1, max_buffered_payload_bytes=2
    )
    try:
        with caplog.at_level(logging.INFO):
            receipt = pipeline._pipeline.submit(inventory, deadline=None).result(5)
        receipt.validate_for(inventory)
        assert not sink.batches
        assert _uncut_reasons(caplog) == {"prefix_payload_limit": 2}
        for state, count in zip(states, [2, 3], strict=True):
            _assert_rolled_back(state, count)
    finally:
        pipeline._pipeline.close(wait=True)
        pipeline.shutdown()
        recovery.shutdown()

AI-2

Add a test where the byte limit splits a batch (new test function, so not a suggestion):

def test_byte_limit_splits_a_batch_and_seals_every_row() -> None:
    recovery, sink, states, inventory = _batch_worker_fixture([2, 3, 4])
    # Each row's estimate is about 600 bytes: two fit, a third does not.
    pipeline = _pipeline(
        recovery, max_batch_payload_bytes=1300, max_buffered_payload_bytes=2600
    )
    try:
        receipt = pipeline._pipeline.submit(inventory, deadline=None).result(5)
        receipt.validate_for(inventory)
        assert sink.batches == [[3, 4], [5]]
        assert all(ack.disposition == "durable_prefix" for ack in receipt.prefixes)
        assert not any(state.snapshot().cut_in_progress for state in states)
    finally:
        pipeline._pipeline.close(wait=True)
        pipeline.shutdown()
        recovery.shutdown()

AI-3

Add a test for the admission limit (new test function, so not a suggestion):

def test_admission_limit_rejects_new_inventories_but_shares_duplicates() -> None:
    release = threading.Event()
    sink = MagicMock()
    sink.stage_generation_prefix_batch.side_effect = lambda records, **kwargs: (
        release.wait(5),
        [record.key for record in records],
    )[1]

    def prepare(inventory: Any, *, deadline: float | None) -> CutSteps:
        record = SimpleNamespace(key=inventory.inventory_digest, delta_len=1)
        keys = yield PrefixWrite("cut", (record,), (0,), (None,), 1, lambda: [None])
        yield BatchReleased()
        return keys

    pipeline = PrefixWritePipeline(
        config=PrefixWritePipelineConfig(enabled=True, max_pending_inventories=1),
        sink=sink,
        prepare=prepare,
    )
    try:
        first = pipeline.submit(_inventory("one", ["a"]), deadline=None)
        assert pipeline.submit(_inventory("one", ["a"]), deadline=None) is first
        with pytest.raises(RuntimeError, match="admission limit"):
            pipeline.submit(_inventory("two", ["b"]), deadline=None)
        release.set()
        assert first.result(5) == ["one"]
    finally:
        release.set()
        pipeline.close(wait=True)



@pytest.mark.parametrize("routed", [False, True])
def test_direct_columns_match_general_sink_and_preserve_owned_buffers(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C21 · 2 action items. TL;DR — this parity test compares only the last PUT, never mixes routed and unrouted rows, and allows float tolerance, so reversing the row order in the dense token branch or the jagged byte branch leaves all 78 tests green.

PR-introduced.

  • call_args is only the last PUT. A batch that mixes routed and unrouted rows writes 2 groups in 2 PUTs, so one group would go unchecked; this test never mixes them.
  • Token rows here always have different lengths and byte fields always the same length, so the dense token branch and the jagged byte branch never run.
  • assert_close uses the default float tolerance, while the PR claims exact parity.

AI-1

Add this test after this function (new test function, so not a suggestion). It passes at head and fails with either row-order change above:

def test_direct_columns_match_general_sink_for_mixed_groups_and_optional_fields() -> (
    None
):
    """Dense token rows, ragged byte rows, mixed route groups, optional fields."""
    recovery, memory, _, inventory = _batch_worker_fixture([3, 3, 3, 3])
    recovery.checkpoint_cut(inventory)
    records = [record for _, record in memory.generation_prefix_records]
    recovery.shutdown()
    assert len({record.delta_len for record in records}) == 1
    attached: list[dict[str, torch.Tensor] | None] = []
    varied = []
    for index, record in enumerate(records):
        update: dict[str, Any] = {
            "model_call_id": record.model_call_id + "é" * index,
            "parent_call_id": None if index % 2 else f"parent-{'x' * index}",
            "extras": None if index == 0 else {"note": "ü" * index},
        }
        if index % 2:
            tensor = torch.arange(record.delta_len * 4, dtype=torch.int16).reshape(
                record.delta_len, 2, 2
            )
            update["extras"] = {
                **(update["extras"] or {}),
                "routed_experts": routed_experts_tensor_metadata(tensor),
            }
            attached.append({"routed_experts": tensor})
        else:
            attached.append(None)
        varied.append(record.model_copy(update=update))
    clients = [
        SimpleNamespace(put_samples=MagicMock(spec=lambda **kwargs: None))
        for _ in range(2)
    ]
    sinks = [TQTokenSink(client, staging_partition="staging") for client in clients]
    sinks[1].enable_direct_prefix_columns()
    for sink in sinks:
        sink.stage_generation_prefix_batch(
            varied,
            checkpoint_id="cut",
            chunk_sequences=[0] * len(varied),
            attachments=attached,
        )

    def rows(client: Any) -> dict[str, tuple[dict[str, Any], dict[str, torch.Tensor]]]:
        found = {}
        for call in client.put_samples.call_args_list:
            kwargs = call.kwargs
            for index, key in enumerate(kwargs["sample_ids"]):
                found[key] = (
                    kwargs["tags"][index],
                    {
                        name: kwargs["fields"][name][index]
                        for name in kwargs["fields"].keys()
                    },
                )
        return found

    old, new = rows(clients[0]), rows(clients[1])
    assert clients[0].put_samples.call_count == clients[1].put_samples.call_count == 2
    assert old.keys() == new.keys() and len(old) == len(varied)
    for key, (old_tags, old_fields) in old.items():
        new_tags, new_fields = new[key]
        assert old_tags == new_tags
        assert old_fields.keys() == new_fields.keys()
        for name, value in old_fields.items():
            torch.testing.assert_close(new_fields[name], value, rtol=0, atol=0)

AI-2

In this test's comparison at L98-L100, require exact equality (not the anchored line, so not a suggestion):

            torch.testing.assert_close(
                old["fields"][name][index], new["fields"][name][index], rtol=0, atol=0
            )

"writer_enabled,pipeline_enabled",
[(True, False), (False, False), (False, True), (True, True)],
)
def test_worker_setup_installs_only_prefix_transport(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C22 · 2 action items. TL;DR — no test covers worker setup when create_prefix_writer fails, so a change that leaves the pipeline thread running would go unnoticed; and this test never checks that direct columns were turned on.

PR-introduced. When writer startup fails, worker setup calls self._prefix_recovery.shutdown() to stop the pipeline thread. No test reaches those lines.

AI-1

Add this test after this function (new test function, so not a suggestion). It passes at head:

def test_worker_setup_stops_the_pipeline_when_the_writer_fails_to_start(
    monkeypatch: pytest.MonkeyPatch,
) -> None:
    worker = _fake_worker()
    worker.cfg = {
        "vllm_cfg": {
            "prefix_writer_actor": True,
            "prefix_write_pipeline": {"enabled": True, "direct_column_encoding": True},
        }
    }
    worker._http_engine_client = object()
    worker._progress_reader = MagicMock()
    worker.llm = SimpleNamespace(
        renderer=SimpleNamespace(get_tokenizer=lambda: object())
    )
    monkeypatch.setattr(
        "nemo_rl.data_plane.build_data_plane_client",
        lambda *args, **kwargs: SimpleNamespace(),
    )
    monkeypatch.setattr(
        "nemo_rl.models.generation.vllm.vllm_worker_async.create_prefix_writer",
        AsyncMock(side_effect=RuntimeError("writer startup failed")),
    )
    with pytest.raises(RuntimeError, match="writer startup failed"):
        asyncio.run(
            worker.setup_token_capture(
                {"impl": "transfer_queue", "backend": "simple"},
                "staging",
                generation_prefix_cuts_enabled=True,
                generation_cut_control_token="secret",
                generation_prefix_batch_size=256,
                generation_prefix_batch_max_tokens=4096,
            )
        )
    pipeline = worker._prefix_recovery._pipeline
    try:
        assert worker._prefix_writer is None
        pipeline._thread.join(5)
        assert not pipeline._thread.is_alive()
        with pytest.raises(RuntimeError, match="closed"):
            pipeline.submit(
                SimpleNamespace(checkpoint_id="c", inventory_digest="d"), deadline=None
            )
    finally:
        asyncio.run(worker._prefix_recovery.aclose())

AI-2

After the _pipeline assertion at L425, add (not the anchored line, so not a suggestion):

        assert worker._capture_sink._direct_prefix_columns is (
            not writer_enabled or pipeline_enabled
        )

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Durability, overlap, encoding parity and recovery tests for prefix writes."""

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C23 · 1 action item. TL;DR — no config, recipe or functional test turns on any of the new options, so the writer actor, the pipeline thread and the direct encoder never run end to end in CI; the PR also leaves the GPU functional suite unchecked.

PR-introduced. Every shipped config sets the new keys to false, and the unit tests use fakes for the actor and the TQ client.

AI-1

Add a phase or variant with the pipeline (and the writer actor) turned on to the existing grpo_async_gym_single_controller_prefix_recovery.sh, which already sets rollout_recovery.target_level=prefix, instead of adding a new script (other file, so not a suggestion).

value.numel() * value.element_size()
for value in attachments.values()
)
if byte_limit is not None and size > byte_limit:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C24 · 2 action items. TL;DR — with the pipeline on, once a call's prefix row is over max_batch_payload_bytes (128 MiB), every later cut of that call is refused too, so the call is never checkpointed again; with the pipeline off, the same row is written alone.

PR-introduced, and intended: the design doc says "a larger single row fails its cut and regenerates", and the oversize test case checks it. So this is a trade-off, not a bug: nothing wrong is written, but the cost is bigger than one failed cut.

  1. A refused cut rolls back, so the call's next row holds everything since its last saved cut. The row only grows, and this line refuses it at every later cut.
  2. A call's first row is its whole prefix, prompt included, and the estimate counts route bytes. With router replay at, for example, 90 MoE layers and top-8 (about 1.4 KB of routes per token), the limit is near 92k tokens. A long agentic call can be over it from its first cut, and is then never checkpointed. These are the most expensive calls to regenerate.
  3. Repro with two calls of 2 and 3 tokens and the limit between their sizes: cut 1 saves the first call and refuses the second. After 2 more tokens each, cut 2 saves the first call (4 tokens) and refuses the second again. With the pipeline off, both rows go in one put.

With the pipeline off (from #4508) there is no byte limit: byte_limit is None, and "one row larger than the token limit is processed alone". Route bytes are bounded only through the token count: a full 4,194,304-token put carries about 3.2 GB of routes at 48 layers and top-8 (#4508 author). 128 MiB is not a TransferQueue limit either: its ZMQ sockets set no queue limit (HWM 0) and no maximum message size. Nothing in the code, the doc or the PR body says where the number comes from.

AI-1

Add a single-row ceiling max_row_payload_bytes to PrefixWritePipelineConfig, and write one row between max_batch_payload_bytes and the ceiling as its own put (multi-file, so a sketch, not a suggestion):

  • In the pipeline, reserve max(max_batch_payload_bytes, max_row_payload_bytes) per preparation instead of max_batch_payload_bytes. Use it in the 2× buffer check, the _advance reservation, the request bound (allow more than max_batch_payload_bytes only for a single record) and can_prepare. _reserve stays as it is, and so does _collect, which already keeps a big first row alone.
  • Here, refuse only above the ceiling, and add or pending_bytes >= byte_limit to the flush check, so the big row is written alone at once.

With the ceiling unset (today's behavior), test_prefix_pipeline.py and test_vllm_config.py pass unchanged (54 passed). With it set, the repro saves both calls at both cuts, each big row in its own put, and the buffer returns to 0. The cost: a larger reservation means fewer batches prepared ahead, which also shortens the freezes in C5; and one big put holds the single write stream longer, so calls frozen behind it wait longer (C5).

Tested patch for AI-1 (+22/−6, in the pipeline config and this file)
diff --git a/nemo_rl/data_plane/prefix_pipeline.py b/nemo_rl/data_plane/prefix_pipeline.py
index 3fa251b554..6f2c8c921a 100644
--- a/nemo_rl/data_plane/prefix_pipeline.py
+++ b/nemo_rl/data_plane/prefix_pipeline.py
@@ -37,6 +37,12 @@ class PrefixWritePipelineConfig(BaseModel, extra="forbid"):
     max_pending_inventories: Annotated[int, Field(gt=0, strict=True)] = 128
     max_buffered_payload_bytes: Annotated[int, Field(gt=0, strict=True)] = 536870912
     max_batch_payload_bytes: Annotated[int, Field(gt=0, strict=True)] = 134217728
+    # One row above max_batch_payload_bytes is written alone up to this size.
+    max_row_payload_bytes: Annotated[int, Field(gt=0, strict=True)] | None = None
+
+    @property
+    def preparation_reservation(self) -> int:
+        return max(self.max_batch_payload_bytes, self.max_row_payload_bytes or 0)
     max_batch_rows: Annotated[int, Field(gt=0, strict=True)] = 256
     max_batch_tokens: Annotated[int, Field(gt=0, strict=True)] = 4194304
     coalesce_wait_seconds: Annotated[float, Field(ge=0, allow_inf_nan=False)] = 0.01
@@ -92,7 +98,7 @@ class PrefixWritePipeline:
         sink: Any,
         prepare: Callable[..., CutSteps],
     ) -> None:
-        if config.max_buffered_payload_bytes < 2 * config.max_batch_payload_bytes:
+        if config.max_buffered_payload_bytes < 2 * config.preparation_reservation:
             raise ValueError(
                 "Prefix pipeline buffer must hold a write and one preparation reservation"
             )
@@ -189,7 +195,7 @@ class PrefixWritePipeline:
 
     def _advance(self, job: _Job) -> None:
         started, cpu = time.perf_counter(), time.thread_time()
-        self._reserve(job, self._config.max_batch_payload_bytes)
+        self._reserve(job, self._config.preparation_reservation)
         try:
             if job.iterator is None:
                 job.iterator = self._prepare(job.inventory, deadline=job.deadline)
@@ -201,8 +207,13 @@ class PrefixWritePipeline:
                 raise TypeError("Prefix preparation must yield a write request")
             if request.checkpoint_id != job.inventory.checkpoint_id:
                 raise ValueError("Prefix write changed checkpoint identity")
-            if not 0 < request.payload_bytes <= self._config.max_batch_payload_bytes:
+            if not 0 < request.payload_bytes <= self._config.preparation_reservation:
                 raise ValueError("Prefix write exceeds its byte reservation")
+            if (
+                request.payload_bytes > self._config.max_batch_payload_bytes
+                and len(request.records) != 1
+            ):
+                raise ValueError("Only a single row may exceed the batch byte limit")
             if (
                 not len(request.records)
                 == len(request.sequences)
@@ -362,7 +373,7 @@ class PrefixWritePipeline:
                 )
                 can_prepare = (
                     candidate is not None
-                    and self._buffered + self._config.max_batch_payload_bytes
+                    and self._buffered + self._config.preparation_reservation
                     <= self._config.max_buffered_payload_bytes
                 )
                 should_write = self._prepared and (
diff --git a/nemo_rl/models/generation/prefix_recovery/recovery.py b/nemo_rl/models/generation/prefix_recovery/recovery.py
index 0164c36c7e..eca3793630 100644
--- a/nemo_rl/models/generation/prefix_recovery/recovery.py
+++ b/nemo_rl/models/generation/prefix_recovery/recovery.py
@@ -841,6 +841,7 @@ class PrefixRecovery:
             if config is None or not config.enabled
             else config.max_batch_payload_bytes
         )
+        row_limit = None if byte_limit is None else config.preparation_reservation
         late: list[Any] = []
 
         def past_deadline() -> bool:
@@ -1020,7 +1021,7 @@ class PrefixRecovery:
                         value.numel() * value.element_size()
                         for value in attachments.values()
                     )
-                if byte_limit is not None and size > byte_limit:
+                if row_limit is not None and size > row_limit:
                     transaction.close()
                     acknowledgements.append(
                         self._failed_ack(
@@ -1040,7 +1041,11 @@ class PrefixRecovery:
                 pending.append((transaction, record, sequence, attachments, prefix))
                 pending_tokens += tokens
                 pending_bytes += size
-                if len(pending) >= max_rows or pending_tokens >= max_tokens:
+                if (
+                    len(pending) >= max_rows
+                    or pending_tokens >= max_tokens
+                    or (byte_limit is not None and pending_bytes >= byte_limit)
+                ):
                     yield from flush()
             yield from flush()
         finally:

AI-2

Pick the ceiling's default from the largest put with routes on, which the #4508 author has not measured yet; where the keys live follows C3.

Context — no action.

  • Do not just raise max_batch_payload_bytes: the same number also sizes every merged write, the 2× buffer and each preparation's reservation.
  • Gym already counts the refusal as a durable_failure, but only the per-cut INFO line says it was the byte limit.

terrykong added a commit to terrykong/RL that referenced this pull request Nov 16, 2026
…ctors

The class-per-everything version cost more than it saved: the prefix writer
differs in owner, count, placement, transport and concurrency, and a pool
and a common process() method had no generic caller. Keep only what the
advantage actor and the rollout reassembler really share:

- DataPlaneDelegate: owns the storage-lending client, names the actor in Ray
  logs, and holds the checkpoint and metrics hooks once.
- start_cpu_actors: create, wait for a ready method, kill all on failure.
- spread_off_head: the advantage actor's placement, now reusable.

run/finalize keep their names, the controller keeps its queues, and the
prefix writer is as in NVIDIA-NeMo#5176.

Signed-off-by: Terry Kong <terryk@nvidia.com>
macandro96 and others added 2 commits November 16, 2026 21:40
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
The second round of prefix batched I/O review dropped `fail_multi_row`
from the shared `_BatchPrefixSink` fake, but the write-pipeline test still
sets it to force a batch failure and a one-row-at-a-time retry. The flag
was silently ignored and `[batch]` failed on `len(sink.batches) > 1`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant