Repository navigation
fix(async): flush drained-refit caches and warn on stale prefixes - #4018
linj-glitch wants to merge 15 commits into
Conversation
cb396b0 to
8c0157b
Compare
…pdates With async rollouts and vLLM prefix caching enabled (the default on Ampere+), `recompute_kv_cache_after_weight_updates=false` did more than keep in-flight requests on their pre-update KV: it also left the prefix cache untouched across refits. Any later request sharing a prefix with an earlier one (same prompt, shared system prompt, groups spanning a refit) was served KV/state computed by whichever weights first filled the block, with no bound on how old. Since the trainer scores with current weights, the train-vs-rollout logprob mismatch grew step over step and reset only on engine restart. Observed on a Nemotron-H MoE async GRPO run with ~49k-token prompts that share ~85% of their context across same-instance rows recurring 3-20 steps apart: sequences masked by seq_logprob_error_threshold=2 went from 1-2/2048 on a fresh engine to 242/2048 after 17 steps, then back to ~8 on the next engine start. Changes: - Drained refits (no in-flight pause requested) now always invalidate the KV/ prefix cache after the weight update; nothing is using the cache at that point so this is free. The flag only governs what happens to requests that were in flight during an in-flight refit. - In-flight refits with the flag false and prefix caching on print a one-time warning explaining the unbounded staleness and how to fix it. - Default of recompute_kv_cache_after_weight_updates flipped to true (GRPO and PPO async configs). Recipes that set it explicitly are unchanged. - Docs: describe both effects of the flag; example snippets use true. - Tests: drained refit invalidates regardless of the flag; in-flight+false warns once and keeps the cache; no warning when prefix caching is off. Signed-off-by: Lin Jia <linj@nvidia.com>
…from kv_cache_management_mode The new default (true) targets vLLM, where the flag also controls prefix-cache invalidation. Megatron generation already encodes the choice engine-side, and its consistency check would now fail for recipes that leave the flag unset with kv_cache_management_mode=persist. Follow the engine mode when the user did not set the flag explicitly; explicit disagreement still errors. Signed-off-by: Lin Jia <linj@nvidia.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
The prefix-cache paragraph grouped "backends without a pause hook" with refits that drain generation first and promised that both always invalidate the caches after the update. That is not what the collector does: a backend that lacks the hook still goes through the pause request (pause-requested is set, the hook returns False), does not drain, and keeps the legacy behaviour of invalidating only when recompute_kv_cache_after_weight_updates is true. List only the paths that really drain (in_flight_weight_updates=false, or managed Dynamo) and spell out the no-pause-hook fallback in its own sentence. Runtime behaviour is unchanged. Signed-off-by: Lin Jia <linj@nvidia.com>
…ined refits `_prefix_caching_may_be_enabled` read only `vllm_cfg.enable_prefix_caching` and returned True whenever that block was absent, so every Megatron, TRT-LLM and async-SGLang in-flight refit with the flag false printed the vLLM-worded warning. On Megatron generation under persist/offload the flag is derived to false by setup itself, and both remedies in the message were wrong there: setting the flag true makes setup raise, and `vllm_cfg.enable_prefix_caching` is never read by that engine. The guard now returns False for every backend but vLLM, and the message names the flag without the `grpo.async_grpo.` prefix since the same collector serves `ppo.async_ppo`. The unconditional drained-refit invalidation made Megatron generation under persist/offload (whose `invalidate_kv_cache` truthfully returns False) print the "KV cache invalidation not supported" warning on every drained refit. When the user did not ask for a recompute, that is the backend following its own configuration, so report it as information; the warning stays when the flag is true. Tests: the legacy-fallback test is parametrized on the flag and now pins the no-pause-hook contract (invalidate iff the flag is true, no drain) and the absence of the stale-prefix warning; a direct guard test covers vllm (unset, true, false), megatron, sglang and trtllm; a drained Megatron-persist test checks the information line replaces the warning only when the flag is false. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
…rs are logged The unset-flag derivation ran at the end of setup(), after `logger.log_hyperparams(master_config.model_dump())`, so W&B/TensorBoard recorded `recompute_kv_cache_after_weight_updates: true` for a Megatron persist/offload run that executed with false, while the checkpoint `config.yaml` (written from the mutated config) recorded false. Move the derivation and the consistency check into `_reconcile_megatron_recompute_kv_cache_flag` and call it before the Logger is built, so the logged hyperparameters, the checkpoint config and the trajectory collector all see the same value and an explicit mismatch fails before any Ray resources are allocated. The helper's docstring notes that `model_fields_set` reflects the merged YAML (exemplar included), so only configs whose inheritance chain omits the key reach the derivation. Tests: direct helper tests for derivation (recompute/persist/offload), agreeing explicit values, and the three explicit mismatches; two setup() tests asserting the mismatch raises before Logger() is called and that the derived value is already in the logged hyperparameters; the mcore-marked noncolocated Megatron setup test now asserts the derived false. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
…t refits This is the only shipped config whose effective `recompute_kv_cache_after_weight_updates` changed with the new class default: it inherits `examples/nemo_gym/grpo_nanov3.yaml`, which omits the key, runs async vLLM with `in_flight_weight_updates: true`, and already disables `vllm_cfg.enable_prefix_caching`, so there was no stale-prefix problem to fix. Pin false so the convergence-tracked .v1 recipe keeps its behaviour instead of preempting and re-prefilling 24k-token video requests (and clearing the encoder cache it keeps via `reset_encoder_cache_after_weight_update: false`) on every refit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
…g scope The exemplars `grpo_math_1B.yaml` and `ppo_math_1B.yaml` still read "Set to true to recompute kv cache after weight updates", implying false is the default, and nearly every recipe inherits them through `defaults:`, so the flipped class default never reaches those recipes. Flipping the exemplars instead would change the in-flight behaviour of every performance recipe and make setup raise for the two sync megatron_generation recipes that inherit `persist`, so the exemplars deliberately keep false and their comments now say so: the class default is true, why false is kept, what false costs with in-flight updates and prefix caching on, and that drained refits invalidate regardless. The reference config, the research template, the workplace assistant example and docs/guides/ppo.md carry the same note. docs/guides/async-grpo.md item 5 is split into bullets: class default vs exemplar pin, the two in-flight behaviours, the drained path with the Megatron persist/offload caveat (it reports a kept cache; the collector logs it as information), the no-pause-hook fallback, and the Megatron derivation of an unset flag before the config is logged. The warning is stated to be vLLM-only. docs/guides/single-controller.md notes that `AsyncRLConfig` still defaults to false and does not warn. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
…on_async vLLM 0.29.0's `engine/protocol.py` marks `clear_cache` DEPRECATED and the RLHF HTTP router docstring says it is ignored in keep mode, while `EngineCoreProc.pause_scheduler` -> `_finish_pause(clear_cache)` preempts every running request and resets the prefix cache. The recompute-on-refit path depends on the latter, so record it where the call is made and ask for re-verification on a vLLM bump. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
8c0157b to
9dd13b3
Compare
|
/ok to test 9dd13b3 |
|
adding @mehraakash as reviewer as he is looking to fixing this globally and consolidating similar PRs doing the same thing |
terrykong
left a comment
There was a problem hiding this comment.
The drained-refit fix (trajectory_collector.py#L1175) is the right shape. It closes a real case where the vLLM prefix cache was never reset across refits, and it costs nothing because no request is running at that point. On vLLM 0.29.0 the reset also drops cached Mamba states for hybrid models, because one BlockPool backs every cache group.
Merge order. #3929 (pause generation during refit on the single-controller path) merges first, then #4647 (report a refused cache reset), then this PR. #3929 owns the default: true in AsyncGRPOConfig, AsyncPPOConfig, AsyncRLConfig and the exemplars. A test merge with #3929 conflicts in 8 files: docs/guides/async-grpo.md, docs/guides/ppo.md, docs/guides/single-controller.md, examples/configs/grpo_math_1B.yaml, examples/configs/ppo_math_1B.yaml, nemo_rl/algorithms/grpo.py, nemo_rl/algorithms/ppo.py and tests/unit/reference_configs/grpo_math_1B.yaml. The comments on grpo.py#L301 and on the Megatron derivation list what to drop. One more test merges cleanly but then fails (see the comment on test_grpo.py#L3912). After #3929, the single-controller path also has the stale-prefix-cache problem with the flag false (see the comment on the warning).
What stays after the rebase is worth landing, so please rebase rather than close:
- always invalidating after a drained refit (
trajectory_collector.py#L1175-L1176); - the one-time stale-prefix warning: 22 shipped in-flight vLLM configs still set the flag to
falseafter #3929; - the Megatron mismatch check, moved before the config is logged and before Ray starts;
- the new
async-grpo.mditem-5 bullets about the prefix cache; - the
pause_generation_asyncdocstring and the new tests.
Video recipe. The recompute_kv_cache_after_weight_updates: false line in the async-gym video recipe is right: prefix caching is off there and the video prefill is long. It is also what keeps nemotron_omni_gym_video_megatron_1n2g.sh passing once the default is true, so #3929's review asks #3929 to add the same line to that recipe. If #3929 lands with it, this hunk drops out on rebase; if not, keep it here.
PR title and description. Please update both after the rebase. This PR no longer flips the default, so "Reach of the class default" and "only config changed" become false. Suggested title: fix(async): flush the prefix cache on drained refits and warn when in-flight refits keep it.
Router. Not affected: this PR changes only the legacy loop, and the router is used only by the single controller and Gym.
PR description, "Remaining gaps":
- The dropped
reset_prefix_cacheresult is tracked in #4632, with a fix in #4647. Please link those instead of "a follow-up". - With in-flight updates and the flag
false, the warning does not limit how stale the cache gets. vLLM resets the prefix cache only when no block is in use (block_pool.py#L777-L784), so it cannot drop just the prefix cache while requests run. A per-requestcache_saltset to the weight version (kv_cache_utils.py#L604-L606) would stop reuse across weight versions while in-flight requests keep their KV. Worth a follow-up issue.
Visual explainer: https://terrykong.github.io/gh-pages-poc/terryk/pr-4018-stale-prefix.html
Generated by Claude Code
Signed-off-by: Lin Jia <linj@nvidia.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
…-prefix-cache # Conflicts: # docs/guides/async-grpo.md # examples/configs/recipes/vlm/vlm_grpo-nemotron-omni-30ba3b-16n8g-megatron-tp4ep4-async-gym-video.v1.yaml # nemo_rl/algorithms/grpo.py
- grpo.py: keep only the Megatron mismatch check, as _check_megatron_recompute_kv_cache_flag, called in setup() before the config is logged and gated on grpo.async_grpo.enabled; the later in-setup copy is removed and the message says to set the flag to false. - async-grpo.md: the prefix-cache, drained-refit and no-pause-hook bullets live under main's item 5; the derivation sentence is gone. - pause_generation_async docstring points at the vLLM-marked test that fails if a vLLM bump changes keep-mode clear_cache semantics. Signed-off-by: Lin Jia <linj@nvidia.com>
|
Rebased on #3929 (merge commit 49fd2f0 + 5468e03) and reworked per the review; title and description updated. What changed:
Local run: |
Signed-off-by: Lin Jia <linj@nvidia.com>
|
/ok to test 7963df3 |
What does this PR do ?
Includes merged #3929 and main through
0e36f2e6. Two changes remain in the async GRPO/PPO refit path, plus the Megatron check moved earlier:in_flight_weight_updates: false, a non-async engine, or managed Dynamo) now callsinvalidate_kv_cache()after the weight update regardless ofrecompute_kv_cache_after_weight_updates. Nothing holds the cache at that point, so this is free, and it closes the case where vLLM's prefix cache was never reset across refits. Megatron generation underkv_cache_management_mode: persist/offloadtruthfully reports that it kept its cache; with the flagfalsethe collector logs that as information (Megatron only) instead of warning on every refit.pause_generation_async(clear_cache=False)with prefix caching enabled warns once per engine process that later requests can reuse KV computed by older weights. It lives in the worker, so the SingleController path (which calls the same hook after feat: pause SingleController refits and enable KV recomputation by default #3929) gets it too, and the message names no config section.recompute_kv_cache_after_weight_updatesagrees withmcore_generation_config.kv_cache_management_modenow runs insetup()before the config is logged or Ray resources are allocated, gated ongrpo.async_grpo.enabled, so synchronous GRPO never trips over a flag it does not read. The later in-setup copy is removed.Issues
Related: #4632 (dropped
reset_prefix_cacheresult). Merge dependency: #4647 remains open; its reset-result fix must land before this PR.Usage
No configuration change. The class default of
recompute_kv_cache_after_weight_updatesistrue(#3929). Runs that set it tofalsewith prefix caching on will see the one-time warning.Before your PR is "Ready for review"
Pre checks:
test_keep_pause_clear_cache_resets_prefix_cacheis@pytest.mark.vllm)docs/guides/async-grpo.mditem 5)Additional Information
Review follow-ups addressed in this revision (thanks @terrykong):
recompute_kv_cache_after_weight_updates: falselines in the exemplars/reference config/video recipe, and thesingle-controller.mdrow are gone; the Megatron derivation block is deleted and only the mismatch check remains, with the "set the flag to false" wording.VllmAsyncGenerationWorkerImpl.pause_generation_async, using_resolve_enable_prefix_cachingdirectly;_prefix_caching_may_be_enabledis gone and the test moved totest_vllm_generation.py(it also checks thatclear_cache=Truestays quiet).async-grpo.md.pause_generation_asyncdocstring points attest_keep_pause_clear_cache_resets_prefix_cache, a vLLM-marked GPU test that drivesAsyncLLM.pause_generation(mode="keep", clear_cache=...)and readsnum_cached_tokenson a repeated prompt, so a vLLM bump that changes keep-mode semantics fails CI rather than a manual re-check.Remaining gap. With in-flight updates and the flag
false, the warning does not bound how stale the prefix cache gets: vLLM resets the prefix cache only when no block is in use, so it cannot drop just the prefix cache while requests run. A per-requestcache_saltset to the weight version would stop reuse across weight versions while in-flight requests keep their KV; worth a follow-up issue.Validation refreshed October 28 on
7963df34: 36 focused collector, setup, and worker tests passed. Both nativetest_keep_pause_clear_cache_resets_prefix_cacheGPU cases passed with Qwen3-0.6B, vLLM 0.29.0, Python 3.13.14, and one NVIDIA L4; the test verifies a warm cache hit before pause and bothclear_cache=TrueandFalsebehavior. The initial captured invocation hit a vLLM/pytest closed-stdout issue; the uncaptured spawn invocation passed both cases. All lint/format/pre-commit hooks except pyrefly passed. The full pyrefly run reports the same 44 diagnostics as pristine main in this local environment (including the missing transfer-queue dependency); no additional diagnostics. No distributed Megatron functional run was performed.