Repository navigation
perf(checkpoint): pipeline packed prefix writes through a TQ actor - #5176
macandro96 wants to merge 2 commits into
Conversation
|
/ok to test 981376a |
macandro96
left a comment
There was a problem hiding this comment.
The core design is sound, and several choices are worth keeping as they are:
- One cut generator drives both the synchronous and the pipelined modes, so there is no second copy of the cut loop (the problem the #4508 review removed).
- No receipt is sent before the rows are durable, and keys that no receipt advertised are cleared on interruption.
- The writer client has the same
put_samples/clear_samplesinterface as the data-plane client, so the sink does not need to know about actors. - The direct encoder writes exactly the same rows as the general one, and 3,600 contended stress runs of the pipeline tests showed no flaky failures.
Must fix before merge
- (C1) CI Lint is red: one pyrefly error and import order in 3 test files.
- (C2) The 78 new tests run in no CI lane, because they lack the
nemo_gymmark.
How to organize this PR and remove the clutter
- (C3) Move the new keys into one block under
rollout_recovery, next to their siblings. This removes the 2 copied limits, the hand-written bool check, and the 6 YAML hunks. - (C6) Check each setup rule in one place instead of up to four (about 70 lines become about 25).
- (C7) Remove the benchmark timing code (about 110 lines) and keep prefix PUTs in the existing data-plane metrics.
- (C8) Move the writer actor into a stacked PR, so the encoder and pipeline can land first.
Together these remove about 200 lines and give each rule one home.
Other comments
- Behavior: (C4) resume fails when the new keys change before the first trainer checkpoint; (C5) the pipeline holds calls frozen for longer; (C10) a dead writer actor is never replaced; (C24) with the pipeline on, a call whose row goes over the byte limit is never checkpointed again.
- Smaller cleanups: (C9) writer startup timeout and error message, (C11) encoder split in the sink, (C12) copied key prefix, (C13) repeated column code, (C14) byte-estimate constants, (C15) module location.
- Comments and docs: (C16) and (C17) restore deleted explanations; (C18) name the required keys in the docs.
- Tests: (C19) expired prepared batch, (C20) byte limit and admission limit, (C21) encoder parity, (C22) writer startup failure, (C23) end-to-end coverage.
PR description
- Write "Part of #3594" under Issues (not "Closes"; the stack is not complete).
- Note that #4843 changes the same cut and route code on an older base, so whichever PR lands second needs to be ported onto the new generator.
Visual explainer: https://macandro96.github.io/gh-pages-poc/amahishi/pr-5176-prefix-writes.html
Generated by Claude Code
| keys: list[str | None] = [] | ||
| if past_deadline(): | ||
| late.extend(prefix for _, _, _, _, prefix in batch) | ||
| return [None] * len(batch) |
There was a problem hiding this comment.
C1 · 2 action items, please fix in this PR. TL;DR — CI "Lint check" is red: pyrefly rejects this line (bad-return), and ruff's import-order check (I001) fails in 3 new test files.
The Lint check job reports Returned type list[None] is not assignable to declared return type list[str | None] [bad-return] here. All 4 errors are PR-introduced.
AI-1
Build the list with a comprehension. pyrefly 0.24.2 accepts it:
| return [None] * len(batch) | |
| return [None for _ in batch] |
AI-2
Run ruff check --select I --fix on these 3 files (multi-file, so not a suggestion):
- test_prefix_pipeline.py:
from typing import Anygoes beforeunittest.mock, andBatchReleasedbeforeCutSteps. - test_prefix_writer_actor.py:
PrefixWriteReplygoes beforePrefixWriterService. - test_vllm_config.py: remove the extra blank line the PR added.
Typed-variable fix that still fails pyrefly
keys: list[str | None] = [None] * len(batch) fails with bad-assignment, so do not use it.
| _assert_rolled_back, | ||
| _batch_worker_fixture, | ||
| _next_inventory, | ||
| ) |
There was a problem hiding this comment.
C2 · 2 action items, please fix in this PR. TL;DR — the 78 new tests in this file and in test_prefix_writer_actor.py run in no CI lane: the Other lane skips both files at import, and the Nemo_Gym lane drops them because they have no nemo_gym mark.
- This file imports helpers from
test_vllm_token_capture_hosting.py, which callspytest.importorskip("nemo_gym.token_id_capture.staging")at import. Itspytestmark = pytest.mark.nemo_gymmarks only its own tests, not the modules that import it. - The Other lane has no
nemo_gyminstalled, so both files are skipped at import. - The Nemo_Gym lane runs with
--nemo-gym-only, and conftest keeps onlynemo_gym-marked items. Result: "78 tests collected … Running 0 items in this shard".
The "294 passed" in the PR description reproduces only with --noconftest.
AI-1
Mark this file as a Gym test file, as test_tq_token_sink.py does:
| ) | |
| ) | |
| pytestmark = pytest.mark.nemo_gym |
AI-2
Add the same line after the import block of test_prefix_writer_actor.py (other file, so not a suggestion here):
pytestmark = pytest.mark.nemo_gymWith both lines, the Nemo_Gym lane runs all 78 tests and they pass.
Alternative that keeps 60 writer tests in the Gym-free lane
Only 7 of the 67 cases in test_prefix_writer_actor.py need Gym: test_native_cut_uses_prefix_client_and_terminal_remains_local, test_worker_setup_installs_only_prefix_transport and test_worker_shutdown_closes_writer_once. Moving those 3 tests into the hosting module would let the other 60 run in the Other lane.
| LOGGER = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| class PrefixWritePipelineConfig(BaseModel, extra="forbid"): |
There was a problem hiding this comment.
C3 · 2 action items. TL;DR — the new keys sit in policy.generation.vllm_cfg as 3 booleans plus 2 copies of existing limits, while every sibling prefix-write knob lives in rollout_recovery; this placement causes most of the clutter in the PR.
PR-introduced. The keys are new, so changing their shape is free now and a breaking change after merge.
max_batch_rows/max_batch_tokensrepeatgeneration_prefix_batch_size/generation_prefix_batch_max_tokenswith the same defaults (256 / 4194304). The #4508 review madeRolloutRecoveryConfigthe only owner of these two limits. With the pipeline on,_cut_batchedtakesmin()of both pairs, sogeneration_prefix_batch_size: 512is silently lowered to 256.enabledis valid only withdirect_column_encoding: 2 booleans for 3 states, and no benchmark row measures "direct encoding only".prefix_writer_actoris outside this model, so it needs a hand-written bool check.- The sibling knobs reach the worker as
setup_token_capturearguments. This block is instead read back fromself.cfg["vllm_cfg"]in the worker, whererollout_recoveryandtoken_captureare not visible. That is why the rules are repeated in the worker, and why 3 exemplar and 3 reference configs gained keys their entrypoints never read.
AI-1
Replace this model and prefix_writer_actor with one block on RolloutRecoveryConfig (multi-file change, so a sketch, not a suggestion):
# nemo_rl/algorithms/single_controller_utils/config.py
class GenerationPrefixWritesConfig(BaseModel, extra="forbid"):
# Direct column encoding plus bounded prepare/write overlap.
pipeline: bool = False
# Run prefix PUTs in one CPU actor on the generation node (TQ simple only).
writer_actor: bool = False
max_pending_inventories: Annotated[int, Field(gt=0, strict=True)] = 128
max_buffered_payload_bytes: Annotated[int, Field(gt=0, strict=True)] = 536870912
max_batch_payload_bytes: Annotated[int, Field(gt=0, strict=True)] = 134217728
coalesce_wait_seconds: Annotated[float, Field(ge=0, allow_inf_nan=False)] = 0.01
@model_validator(mode="after")
def _buffer_holds_two_batches(self) -> "GenerationPrefixWritesConfig":
# Room for one write in flight plus one preparation reservation.
if self.max_buffered_payload_bytes < 2 * self.max_batch_payload_bytes:
raise ValueError(
"max_buffered_payload_bytes must be >= 2 * max_batch_payload_bytes"
)
return self
class RolloutRecoveryConfig(BaseModel, extra="allow"):
...
generation_prefix_writes: GenerationPrefixWritesConfig = Field(
default_factory=GenerationPrefixWritesConfig
)Then:
- Delete
direct_column_encoding,max_batch_rowsandmax_batch_tokens. Give the pipeline the row and token limits fromparts, asPrefixReadBatchergets them, and delete themin()lines. - Pass the block to
setup_token_captureas a required argument next togeneration_prefix_batch_size, and add it to_TokenCaptureSetupso a restarted shard gets it again. - Delete the
vllm_cfgkeys, the bool check and its test, and the 6 YAML hunks. Add a commented block beside the sibling keys in grpo_math_1B_megatron_single_controller.yaml and ppo_math_1B_megatron_single_controller.yaml.
AI-2
In validate_single_controller_config, reject generation_prefix_writes.pipeline or .writer_actor when token_capture.enabled is false or target_level is not prefix, next to the existing top_logprobs rejection (line not in this PR's diff, so not a suggestion). Today the keys are silently ignored when token capture is off, because setup_token_capture is then never called.
Every place the new argument passes through
Make it a required argument with no default; #4508 removed the defaults on these signatures on purpose (review thread).
GenerationInterface.setup_token_captureVllmGeneration.setup_token_capture,_TokenCaptureSetupand its construction- the vLLM worker's
setup_token_capture, which replaces theself.cfgreads on L842-L843 megatron_generation.pyandmegatron_worker.py: accept it and ignore it- the SC setup call
| flat_logprobs: NotRequired[bool] | ||
| # Prefix-only TQ serialization/PUTs in one colocated CPU actor. Default off. | ||
| # Requires async HTTP capture, prefix cuts, and SimpleStorage; no media yet. | ||
| prefix_writer_actor: NotRequired[bool] |
There was a problem hiding this comment.
C4 · 1 action item. TL;DR — turning prefix_writer_actor or prefix_write_pipeline.* on or off between save and resume, before the first trainer checkpoint, makes setup raise ValueError, because these keys are not on the fingerprint exclusion list.
PR-introduced. These keys change how rows are written, not what is written: the direct encoder writes the same rows as the general one. When there is no trainer checkpoint yet and <checkpoint_dir>/bootstrap/ exists, setup calls validate_bootstrap_anchor, which raises:
rollout bootstrap anchor at <checkpoint_dir>/bootstrap/manifest.json is incompatible with the current rollout-semantic configuration. Changed compatibility fields:
policy.generation.vllm_cfg.prefix_write_pipeline.enabled: False -> True
policy.generation.vllm_cfg.prefix_writer_actor: False -> True
The sibling limits rollout_recovery.generation_prefix_batch_* are already excluded.
AI-1
Add both keys to _BOOTSTRAP_FINGERPRINT_EXCLUDED_PATHS (file not in this PR's diff, so not a suggestion):
"policy.generation.vllm_cfg.prefix_write_pipeline",
"policy.generation.vllm_cfg.prefix_writer_actor",With these two entries the same resume succeeds. An entry excludes the whole subtree, so if the keys move into one block under rollout_recovery, this becomes one entry for that block.
| ), | ||
| None, | ||
| ) | ||
| can_prepare = ( |
There was a problem hiding this comment.
C5 · 2 action items. TL;DR — with the pipeline on, every admitted inventory prepares a batch and keeps its calls frozen until every write queued ahead of it finishes, because can_prepare is limited only by bytes and text rows almost never reach the byte limit.
PR-introduced. This makes calls wait longer; it does not produce wrong results.
- Preparing a batch freezes its calls (
cut_in_progress) until their rows are written and sealed. can_preparechecks onlybuffered + max_batch_payload_bytes <= max_buffered_payload_bytes. A text row is estimated at about 16 bytes per token + 512 bytes, so the 512 MiB buffer is not the limit, and up tomax_pending_inventories(128) inventories prepare.- They all wait behind the one write stream. On the base path at most one batch is frozen during one write. Here a call can stay frozen for the whole write queue (the PR reports 25-72 s per snapshot).
- A call that finishes or aborts while frozen waits that long for its response, inside a default-executor thread (finish, abort).
Measured with 64 calls, batches of 8 and a 50 ms PUT: base froze 8 calls at once with a 49 ms longest hold; the pipeline froze 64 calls with a 399 ms longest hold, in the same total time.
AI-1
Also limit preparation by rows, so at most one write is prepared ahead of the write in flight (not a suggestion: applied alone it fails the PR's own overlap test until that test changes):
can_prepare = (
candidate is not None
and self._buffered + self._config.max_batch_payload_bytes
<= self._config.max_buffered_payload_bytes
# Prepare at most one write ahead of the write in flight, so
# calls stay frozen for about two writes, not the whole queue.
and sum(len(job.request.records) for job in self._prepared)
< self._config.max_batch_rows
)This keeps the overlap. With 40 ms of preparation per batch, it took 0.45 s, against 0.54 s at head and 0.73 s without the pipeline. If the row limits move to rollout_recovery, use generation_prefix_batch_size here.
AI-2
Update test_preparation_overlaps_put_but_receipts_and_locks_wait_for_durability (other file, so not a suggestion). It asserts that all 3 calls are frozen while the first PUT is blocked; with the row limit only 2 are. Assert instead that at least 2 calls are frozen while the first PUT is blocked, and that none is sealed or acknowledged before release.
| pipeline = _pipeline( | ||
| recovery, | ||
| **( | ||
| {"max_batch_payload_bytes": 1, "max_buffered_payload_bytes": 2} |
There was a problem hiding this comment.
C20 · 3 action items. TL;DR — the "oversize" case asserts the same result as "deadline", so it never proves the payload limit caused the failure or that nothing was written; and no test splits a batch on the byte limit.
PR-introduced.
- "oversize" checks only that every call is
durable_failureand rolled back. It does not check theprefix_payload_limitreason, or that the sink received no write. - Removing the byte-limit flush leaves the 11 pipeline tests green.
- The admission limit is not reached by any test.
AI-1
Remove "oversize" from the parametrize list and remove this **(...) argument, then add this test after the function (adds a test function, so not a suggestion). It needs import logging and _uncut_reasons from the hosting module:
def test_oversize_row_fails_its_call_without_a_write(
caplog: pytest.LogCaptureFixture,
) -> None:
recovery, sink, states, inventory = _batch_worker_fixture([2, 3])
pipeline = _pipeline(
recovery, max_batch_payload_bytes=1, max_buffered_payload_bytes=2
)
try:
with caplog.at_level(logging.INFO):
receipt = pipeline._pipeline.submit(inventory, deadline=None).result(5)
receipt.validate_for(inventory)
assert not sink.batches
assert _uncut_reasons(caplog) == {"prefix_payload_limit": 2}
for state, count in zip(states, [2, 3], strict=True):
_assert_rolled_back(state, count)
finally:
pipeline._pipeline.close(wait=True)
pipeline.shutdown()
recovery.shutdown()AI-2
Add a test where the byte limit splits a batch (new test function, so not a suggestion):
def test_byte_limit_splits_a_batch_and_seals_every_row() -> None:
recovery, sink, states, inventory = _batch_worker_fixture([2, 3, 4])
# Each row's estimate is about 600 bytes: two fit, a third does not.
pipeline = _pipeline(
recovery, max_batch_payload_bytes=1300, max_buffered_payload_bytes=2600
)
try:
receipt = pipeline._pipeline.submit(inventory, deadline=None).result(5)
receipt.validate_for(inventory)
assert sink.batches == [[3, 4], [5]]
assert all(ack.disposition == "durable_prefix" for ack in receipt.prefixes)
assert not any(state.snapshot().cut_in_progress for state in states)
finally:
pipeline._pipeline.close(wait=True)
pipeline.shutdown()
recovery.shutdown()AI-3
Add a test for the admission limit (new test function, so not a suggestion):
def test_admission_limit_rejects_new_inventories_but_shares_duplicates() -> None:
release = threading.Event()
sink = MagicMock()
sink.stage_generation_prefix_batch.side_effect = lambda records, **kwargs: (
release.wait(5),
[record.key for record in records],
)[1]
def prepare(inventory: Any, *, deadline: float | None) -> CutSteps:
record = SimpleNamespace(key=inventory.inventory_digest, delta_len=1)
keys = yield PrefixWrite("cut", (record,), (0,), (None,), 1, lambda: [None])
yield BatchReleased()
return keys
pipeline = PrefixWritePipeline(
config=PrefixWritePipelineConfig(enabled=True, max_pending_inventories=1),
sink=sink,
prepare=prepare,
)
try:
first = pipeline.submit(_inventory("one", ["a"]), deadline=None)
assert pipeline.submit(_inventory("one", ["a"]), deadline=None) is first
with pytest.raises(RuntimeError, match="admission limit"):
pipeline.submit(_inventory("two", ["b"]), deadline=None)
release.set()
assert first.result(5) == ["one"]
finally:
release.set()
pipeline.close(wait=True)|
|
||
|
|
||
| @pytest.mark.parametrize("routed", [False, True]) | ||
| def test_direct_columns_match_general_sink_and_preserve_owned_buffers( |
There was a problem hiding this comment.
C21 · 2 action items. TL;DR — this parity test compares only the last PUT, never mixes routed and unrouted rows, and allows float tolerance, so reversing the row order in the dense token branch or the jagged byte branch leaves all 78 tests green.
PR-introduced.
call_argsis only the last PUT. A batch that mixes routed and unrouted rows writes 2 groups in 2 PUTs, so one group would go unchecked; this test never mixes them.- Token rows here always have different lengths and byte fields always the same length, so the dense token branch and the jagged byte branch never run.
assert_closeuses the default float tolerance, while the PR claims exact parity.
AI-1
Add this test after this function (new test function, so not a suggestion). It passes at head and fails with either row-order change above:
def test_direct_columns_match_general_sink_for_mixed_groups_and_optional_fields() -> (
None
):
"""Dense token rows, ragged byte rows, mixed route groups, optional fields."""
recovery, memory, _, inventory = _batch_worker_fixture([3, 3, 3, 3])
recovery.checkpoint_cut(inventory)
records = [record for _, record in memory.generation_prefix_records]
recovery.shutdown()
assert len({record.delta_len for record in records}) == 1
attached: list[dict[str, torch.Tensor] | None] = []
varied = []
for index, record in enumerate(records):
update: dict[str, Any] = {
"model_call_id": record.model_call_id + "é" * index,
"parent_call_id": None if index % 2 else f"parent-{'x' * index}",
"extras": None if index == 0 else {"note": "ü" * index},
}
if index % 2:
tensor = torch.arange(record.delta_len * 4, dtype=torch.int16).reshape(
record.delta_len, 2, 2
)
update["extras"] = {
**(update["extras"] or {}),
"routed_experts": routed_experts_tensor_metadata(tensor),
}
attached.append({"routed_experts": tensor})
else:
attached.append(None)
varied.append(record.model_copy(update=update))
clients = [
SimpleNamespace(put_samples=MagicMock(spec=lambda **kwargs: None))
for _ in range(2)
]
sinks = [TQTokenSink(client, staging_partition="staging") for client in clients]
sinks[1].enable_direct_prefix_columns()
for sink in sinks:
sink.stage_generation_prefix_batch(
varied,
checkpoint_id="cut",
chunk_sequences=[0] * len(varied),
attachments=attached,
)
def rows(client: Any) -> dict[str, tuple[dict[str, Any], dict[str, torch.Tensor]]]:
found = {}
for call in client.put_samples.call_args_list:
kwargs = call.kwargs
for index, key in enumerate(kwargs["sample_ids"]):
found[key] = (
kwargs["tags"][index],
{
name: kwargs["fields"][name][index]
for name in kwargs["fields"].keys()
},
)
return found
old, new = rows(clients[0]), rows(clients[1])
assert clients[0].put_samples.call_count == clients[1].put_samples.call_count == 2
assert old.keys() == new.keys() and len(old) == len(varied)
for key, (old_tags, old_fields) in old.items():
new_tags, new_fields = new[key]
assert old_tags == new_tags
assert old_fields.keys() == new_fields.keys()
for name, value in old_fields.items():
torch.testing.assert_close(new_fields[name], value, rtol=0, atol=0)AI-2
In this test's comparison at L98-L100, require exact equality (not the anchored line, so not a suggestion):
torch.testing.assert_close(
old["fields"][name][index], new["fields"][name][index], rtol=0, atol=0
)| "writer_enabled,pipeline_enabled", | ||
| [(True, False), (False, False), (False, True), (True, True)], | ||
| ) | ||
| def test_worker_setup_installs_only_prefix_transport( |
There was a problem hiding this comment.
C22 · 2 action items. TL;DR — no test covers worker setup when create_prefix_writer fails, so a change that leaves the pipeline thread running would go unnoticed; and this test never checks that direct columns were turned on.
PR-introduced. When writer startup fails, worker setup calls self._prefix_recovery.shutdown() to stop the pipeline thread. No test reaches those lines.
AI-1
Add this test after this function (new test function, so not a suggestion). It passes at head:
def test_worker_setup_stops_the_pipeline_when_the_writer_fails_to_start(
monkeypatch: pytest.MonkeyPatch,
) -> None:
worker = _fake_worker()
worker.cfg = {
"vllm_cfg": {
"prefix_writer_actor": True,
"prefix_write_pipeline": {"enabled": True, "direct_column_encoding": True},
}
}
worker._http_engine_client = object()
worker._progress_reader = MagicMock()
worker.llm = SimpleNamespace(
renderer=SimpleNamespace(get_tokenizer=lambda: object())
)
monkeypatch.setattr(
"nemo_rl.data_plane.build_data_plane_client",
lambda *args, **kwargs: SimpleNamespace(),
)
monkeypatch.setattr(
"nemo_rl.models.generation.vllm.vllm_worker_async.create_prefix_writer",
AsyncMock(side_effect=RuntimeError("writer startup failed")),
)
with pytest.raises(RuntimeError, match="writer startup failed"):
asyncio.run(
worker.setup_token_capture(
{"impl": "transfer_queue", "backend": "simple"},
"staging",
generation_prefix_cuts_enabled=True,
generation_cut_control_token="secret",
generation_prefix_batch_size=256,
generation_prefix_batch_max_tokens=4096,
)
)
pipeline = worker._prefix_recovery._pipeline
try:
assert worker._prefix_writer is None
pipeline._thread.join(5)
assert not pipeline._thread.is_alive()
with pytest.raises(RuntimeError, match="closed"):
pipeline.submit(
SimpleNamespace(checkpoint_id="c", inventory_digest="d"), deadline=None
)
finally:
asyncio.run(worker._prefix_recovery.aclose())AI-2
After the _pipeline assertion at L425, add (not the anchored line, so not a suggestion):
assert worker._capture_sink._direct_prefix_columns is (
not writer_enabled or pipeline_enabled
)| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| """Durability, overlap, encoding parity and recovery tests for prefix writes.""" |
There was a problem hiding this comment.
C23 · 1 action item. TL;DR — no config, recipe or functional test turns on any of the new options, so the writer actor, the pipeline thread and the direct encoder never run end to end in CI; the PR also leaves the GPU functional suite unchecked.
PR-introduced. Every shipped config sets the new keys to false, and the unit tests use fakes for the actor and the TQ client.
AI-1
Add a phase or variant with the pipeline (and the writer actor) turned on to the existing grpo_async_gym_single_controller_prefix_recovery.sh, which already sets rollout_recovery.target_level=prefix, instead of adding a new script (other file, so not a suggestion).
| value.numel() * value.element_size() | ||
| for value in attachments.values() | ||
| ) | ||
| if byte_limit is not None and size > byte_limit: |
There was a problem hiding this comment.
C24 · 2 action items. TL;DR — with the pipeline on, once a call's prefix row is over max_batch_payload_bytes (128 MiB), every later cut of that call is refused too, so the call is never checkpointed again; with the pipeline off, the same row is written alone.
PR-introduced, and intended: the design doc says "a larger single row fails its cut and regenerates", and the oversize test case checks it. So this is a trade-off, not a bug: nothing wrong is written, but the cost is bigger than one failed cut.
- A refused cut rolls back, so the call's next row holds everything since its last saved cut. The row only grows, and this line refuses it at every later cut.
- A call's first row is its whole prefix, prompt included, and the estimate counts route bytes. With router replay at, for example, 90 MoE layers and top-8 (about 1.4 KB of routes per token), the limit is near 92k tokens. A long agentic call can be over it from its first cut, and is then never checkpointed. These are the most expensive calls to regenerate.
- Repro with two calls of 2 and 3 tokens and the limit between their sizes: cut 1 saves the first call and refuses the second. After 2 more tokens each, cut 2 saves the first call (4 tokens) and refuses the second again. With the pipeline off, both rows go in one put.
With the pipeline off (from #4508) there is no byte limit: byte_limit is None, and "one row larger than the token limit is processed alone". Route bytes are bounded only through the token count: a full 4,194,304-token put carries about 3.2 GB of routes at 48 layers and top-8 (#4508 author). 128 MiB is not a TransferQueue limit either: its ZMQ sockets set no queue limit (HWM 0) and no maximum message size. Nothing in the code, the doc or the PR body says where the number comes from.
AI-1
Add a single-row ceiling max_row_payload_bytes to PrefixWritePipelineConfig, and write one row between max_batch_payload_bytes and the ceiling as its own put (multi-file, so a sketch, not a suggestion):
- In the pipeline, reserve
max(max_batch_payload_bytes, max_row_payload_bytes)per preparation instead ofmax_batch_payload_bytes. Use it in the 2× buffer check, the_advancereservation, the request bound (allow more thanmax_batch_payload_bytesonly for a single record) andcan_prepare._reservestays as it is, and so does_collect, which already keeps a big first row alone. - Here, refuse only above the ceiling, and add
or pending_bytes >= byte_limitto the flush check, so the big row is written alone at once.
With the ceiling unset (today's behavior), test_prefix_pipeline.py and test_vllm_config.py pass unchanged (54 passed). With it set, the repro saves both calls at both cuts, each big row in its own put, and the buffer returns to 0. The cost: a larger reservation means fewer batches prepared ahead, which also shortens the freezes in C5; and one big put holds the single write stream longer, so calls frozen behind it wait longer (C5).
Tested patch for AI-1 (+22/−6, in the pipeline config and this file)
diff --git a/nemo_rl/data_plane/prefix_pipeline.py b/nemo_rl/data_plane/prefix_pipeline.py
index 3fa251b554..6f2c8c921a 100644
--- a/nemo_rl/data_plane/prefix_pipeline.py
+++ b/nemo_rl/data_plane/prefix_pipeline.py
@@ -37,6 +37,12 @@ class PrefixWritePipelineConfig(BaseModel, extra="forbid"):
max_pending_inventories: Annotated[int, Field(gt=0, strict=True)] = 128
max_buffered_payload_bytes: Annotated[int, Field(gt=0, strict=True)] = 536870912
max_batch_payload_bytes: Annotated[int, Field(gt=0, strict=True)] = 134217728
+ # One row above max_batch_payload_bytes is written alone up to this size.
+ max_row_payload_bytes: Annotated[int, Field(gt=0, strict=True)] | None = None
+
+ @property
+ def preparation_reservation(self) -> int:
+ return max(self.max_batch_payload_bytes, self.max_row_payload_bytes or 0)
max_batch_rows: Annotated[int, Field(gt=0, strict=True)] = 256
max_batch_tokens: Annotated[int, Field(gt=0, strict=True)] = 4194304
coalesce_wait_seconds: Annotated[float, Field(ge=0, allow_inf_nan=False)] = 0.01
@@ -92,7 +98,7 @@ class PrefixWritePipeline:
sink: Any,
prepare: Callable[..., CutSteps],
) -> None:
- if config.max_buffered_payload_bytes < 2 * config.max_batch_payload_bytes:
+ if config.max_buffered_payload_bytes < 2 * config.preparation_reservation:
raise ValueError(
"Prefix pipeline buffer must hold a write and one preparation reservation"
)
@@ -189,7 +195,7 @@ class PrefixWritePipeline:
def _advance(self, job: _Job) -> None:
started, cpu = time.perf_counter(), time.thread_time()
- self._reserve(job, self._config.max_batch_payload_bytes)
+ self._reserve(job, self._config.preparation_reservation)
try:
if job.iterator is None:
job.iterator = self._prepare(job.inventory, deadline=job.deadline)
@@ -201,8 +207,13 @@ class PrefixWritePipeline:
raise TypeError("Prefix preparation must yield a write request")
if request.checkpoint_id != job.inventory.checkpoint_id:
raise ValueError("Prefix write changed checkpoint identity")
- if not 0 < request.payload_bytes <= self._config.max_batch_payload_bytes:
+ if not 0 < request.payload_bytes <= self._config.preparation_reservation:
raise ValueError("Prefix write exceeds its byte reservation")
+ if (
+ request.payload_bytes > self._config.max_batch_payload_bytes
+ and len(request.records) != 1
+ ):
+ raise ValueError("Only a single row may exceed the batch byte limit")
if (
not len(request.records)
== len(request.sequences)
@@ -362,7 +373,7 @@ class PrefixWritePipeline:
)
can_prepare = (
candidate is not None
- and self._buffered + self._config.max_batch_payload_bytes
+ and self._buffered + self._config.preparation_reservation
<= self._config.max_buffered_payload_bytes
)
should_write = self._prepared and (
diff --git a/nemo_rl/models/generation/prefix_recovery/recovery.py b/nemo_rl/models/generation/prefix_recovery/recovery.py
index 0164c36c7e..eca3793630 100644
--- a/nemo_rl/models/generation/prefix_recovery/recovery.py
+++ b/nemo_rl/models/generation/prefix_recovery/recovery.py
@@ -841,6 +841,7 @@ class PrefixRecovery:
if config is None or not config.enabled
else config.max_batch_payload_bytes
)
+ row_limit = None if byte_limit is None else config.preparation_reservation
late: list[Any] = []
def past_deadline() -> bool:
@@ -1020,7 +1021,7 @@ class PrefixRecovery:
value.numel() * value.element_size()
for value in attachments.values()
)
- if byte_limit is not None and size > byte_limit:
+ if row_limit is not None and size > row_limit:
transaction.close()
acknowledgements.append(
self._failed_ack(
@@ -1040,7 +1041,11 @@ class PrefixRecovery:
pending.append((transaction, record, sequence, attachments, prefix))
pending_tokens += tokens
pending_bytes += size
- if len(pending) >= max_rows or pending_tokens >= max_tokens:
+ if (
+ len(pending) >= max_rows
+ or pending_tokens >= max_tokens
+ or (byte_limit is not None and pending_bytes >= byte_limit)
+ ):
yield from flush()
yield from flush()
finally:AI-2
Pick the ceiling's default from the largest put with routes on, which the #4508 author has not measured yet; where the keys live follows C3.
Context — no action.
- Do not just raise
max_batch_payload_bytes: the same number also sizes every merged write, the 2× buffer and each preparation's reservation. - Gym already counts the refusal as a
durable_failure, but only the per-cut INFO line says it was the byte limit.
…ctors The class-per-everything version cost more than it saved: the prefix writer differs in owner, count, placement, transport and concurrency, and a pool and a common process() method had no generic caller. Keep only what the advantage actor and the rollout reassembler really share: - DataPlaneDelegate: owns the storage-lending client, names the actor in Ray logs, and holds the checkpoint and metrics hooks once. - start_cpu_actors: create, wait for a ready method, kill all on failure. - spread_off_head: the advantage actor's placement, now reusable. run/finalize keep their names, the controller keeps its queues, and the prefix writer is as in NVIDIA-NeMo#5176. Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
The second round of prefix batched I/O review dropped `fail_multi_row` from the shared `_BatchPrefixSink` fake, but the write-pipeline test still sets it to force a batch failure and a one-row-at-a-time retry. The flag was silently ignored and `[batch]` failed on `len(sink.batches) > 1`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
981376a to
d42cf30
Compare
What does this PR do?
Add opt-in packed generation-prefix writes, bounded preparation/write pipelining, and a colocated TransferQueue writer actor.
Stacked on #4508 (
amahishi/ckpt-v2-prefix-batched-io,788133a5545953dcdbd3a60af6b2a99d90482346). This PR does not change #4508's branch.Issues
Follow-up to #4508. No issues closed.
Behavior and limits
Usage
Under the existing async-HTTP, text-only generation-prefix capture configuration:
The buffer defaults to 512 MiB, maximum write payload to 128 MiB, merged rows to 256, merged token positions to 4,194,304, and the coalescing window to 10 ms. Full settings and requirements are documented in
docs/design-docs/single-controller-rollout-checkpointing.md.Validation
On the isolated PR branch with Gym at the unchanged pin
5911d628:test_tq_token_sink.pymodule could not collect in this CPU environment: Megatron is not installed; using the local source additionally requires Triton. The live TQ smoke and new encoder/parity tests passed. Full GPU training functional tests were not rerun from this isolated branch.Benchmark observations
Four nodes: two TP4 generation engines, one TQ node and one load/driver node. Same pinned container and workload targets, router replay enabled,
max_num_seqs=512, 630 outstanding calls and 39 inventories per engine. These are prefix-cut + TQ save/verification timings, not full Gym/trainer rollout-checkpoint latency.Single-run observations, not a statistical speedup claim: actual captured token positions can overshoot targets (baseline 6.60M / 14.28M versus full path 6.00M / 13.73M). Preparation and writes overlap, so summed phase times are not exclusive snapshot time.
Isolating the NumPy packing change reduced second-cut aggregate packing time by 85–87% across engines, but did not improve total snapshot latency in that comparison; preparation and handoff variability remain. This PR does not claim those bottlenecks are solved.
Before your PR is "Ready for review"