Skip to content

fix(async): flush drained-refit caches and warn on stale prefixes - #4018

Open
linj-glitch wants to merge 15 commits into
NVIDIA-NeMo:mainfrom
linj-glitch:fix/async-refit-stale-prefix-cache
Open

linj-glitch wants to merge 15 commits into
NVIDIA-NeMo:mainfrom
linj-glitch:fix/async-refit-stale-prefix-cache

Conversation

@linj-glitch

@linj-glitch linj-glitch commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Includes merged #3929 and main through 0e36f2e6. Two changes remain in the async GRPO/PPO refit path, plus the Megatron check moved earlier:

  • Drained refits always invalidate. A refit for which no in-flight pause was requested (in_flight_weight_updates: false, a non-async engine, or managed Dynamo) now calls invalidate_kv_cache() after the weight update regardless of recompute_kv_cache_after_weight_updates. Nothing holds the cache at that point, so this is free, and it closes the case where vLLM's prefix cache was never reset across refits. Megatron generation under kv_cache_management_mode: persist/offload truthfully reports that it kept its cache; with the flag false the collector logs that as information (Megatron only) instead of warning on every refit.
  • One-time stale-prefix warning in the vLLM worker. pause_generation_async(clear_cache=False) with prefix caching enabled warns once per engine process that later requests can reuse KV computed by older weights. It lives in the worker, so the SingleController path (which calls the same hook after feat: pause SingleController refits and enable KV recomputation by default #3929) gets it too, and the message names no config section.
  • Megatron mismatch check before logging. The check that recompute_kv_cache_after_weight_updates agrees with mcore_generation_config.kv_cache_management_mode now runs in setup() before the config is logged or Ray resources are allocated, gated on grpo.async_grpo.enabled, so synchronous GRPO never trips over a flag it does not read. The later in-setup copy is removed.

Issues

Related: #4632 (dropped reset_prefix_cache result). Merge dependency: #4647 remains open; its reset-result fix must land before this PR.

Usage

No configuration change. The class default of recompute_kv_cache_after_weight_updates is true (#3929). Runs that set it to false with prefix caching on will see the one-time warning.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests (unit tests; the GPU test test_keep_pause_clear_cache_resets_prefix_cache is @pytest.mark.vllm)
  • Did you add or update any necessary documentation? (docs/guides/async-grpo.md item 5)

Additional Information

Review follow-ups addressed in this revision (thanks @terrykong):

  • Rebased on feat: pause SingleController refits and enable KV recomputation by default #3929: the default flip, the recompute_kv_cache_after_weight_updates: false lines in the exemplars/reference config/video recipe, and the single-controller.md row are gone; the Megatron derivation block is deleted and only the mismatch check remains, with the "set the flag to false" wording.
  • The stale-prefix warning moved from the collector into VllmAsyncGenerationWorkerImpl.pause_generation_async, using _resolve_enable_prefix_caching directly; _prefix_caching_may_be_enabled is gone and the test moved to test_vllm_generation.py (it also checks that clear_cache=True stays quiet).
  • The "kept its KV caches" line prints only for the Megatron backend, so a failed vLLM/SGLang reset still reads as a warning.
  • The in-body comment on the drained-refit branch is two lines; the reasoning lives in async-grpo.md.
  • pause_generation_async docstring points at test_keep_pause_clear_cache_resets_prefix_cache, a vLLM-marked GPU test that drives AsyncLLM.pause_generation(mode="keep", clear_cache=...) and reads num_cached_tokens on a repeated prompt, so a vLLM bump that changes keep-mode semantics fails CI rather than a manual re-check.

Remaining gap. With in-flight updates and the flag false, the warning does not bound how stale the prefix cache gets: vLLM resets the prefix cache only when no block is in use, so it cannot drop just the prefix cache while requests run. A per-request cache_salt set to the weight version would stop reuse across weight versions while in-flight requests keep their KV; worth a follow-up issue.

Validation refreshed October 28 on 7963df34: 36 focused collector, setup, and worker tests passed. Both native test_keep_pause_clear_cache_resets_prefix_cache GPU cases passed with Qwen3-0.6B, vLLM 0.29.0, Python 3.13.14, and one NVIDIA L4; the test verifies a warm cache hit before pause and both clear_cache=True and False behavior. The initial captured invocation hit a vLLM/pytest closed-stdout issue; the uncaptured spawn invocation passed both cases. All lint/format/pre-commit hooks except pyrefly passed. The full pyrefly run reports the same 44 diagnostics as pristine main in this local environment (including the missing transfer-queue dependency); no additional diagnostics. No distributed Megatron functional run was performed.

@linj-glitch
linj-glitch requested review from a team as code owners September 5, 2026 08:18
@copy-pr-bot

copy-pr-bot Bot commented Sep 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Sep 5, 2026
@linj-glitch
linj-glitch marked this pull request as draft October 5, 2026 06:10
@linj-glitch
linj-glitch force-pushed the fix/async-refit-stale-prefix-cache branch from cb396b0 to 8c0157b Compare October 8, 2026 00:15
@linj-glitch
linj-glitch marked this pull request as ready for review October 13, 2026 05:02
linj-glitch and others added 10 commits October 13, 2026 02:59
…pdates

With async rollouts and vLLM prefix caching enabled (the default on Ampere+),
`recompute_kv_cache_after_weight_updates=false` did more than keep in-flight
requests on their pre-update KV: it also left the prefix cache untouched across
refits. Any later request sharing a prefix with an earlier one (same prompt,
shared system prompt, groups spanning a refit) was served KV/state computed by
whichever weights first filled the block, with no bound on how old. Since the
trainer scores with current weights, the train-vs-rollout logprob mismatch grew
step over step and reset only on engine restart.

Observed on a Nemotron-H MoE async GRPO run with ~49k-token prompts that share
~85% of their context across same-instance rows recurring 3-20 steps apart:
sequences masked by seq_logprob_error_threshold=2 went from 1-2/2048 on a fresh
engine to 242/2048 after 17 steps, then back to ~8 on the next engine start.

Changes:
- Drained refits (no in-flight pause requested) now always invalidate the KV/
  prefix cache after the weight update; nothing is using the cache at that point
  so this is free. The flag only governs what happens to requests that were in
  flight during an in-flight refit.
- In-flight refits with the flag false and prefix caching on print a one-time
  warning explaining the unbounded staleness and how to fix it.
- Default of recompute_kv_cache_after_weight_updates flipped to true (GRPO and
  PPO async configs). Recipes that set it explicitly are unchanged.
- Docs: describe both effects of the flag; example snippets use true.
- Tests: drained refit invalidates regardless of the flag; in-flight+false warns
  once and keeps the cache; no warning when prefix caching is off.

Signed-off-by: Lin Jia <linj@nvidia.com>
…from kv_cache_management_mode

The new default (true) targets vLLM, where the flag also controls prefix-cache
invalidation. Megatron generation already encodes the choice engine-side, and
its consistency check would now fail for recipes that leave the flag unset with
kv_cache_management_mode=persist. Follow the engine mode when the user did not
set the flag explicitly; explicit disagreement still errors.

Signed-off-by: Lin Jia <linj@nvidia.com>
The prefix-cache paragraph grouped "backends without a pause hook" with
refits that drain generation first and promised that both always invalidate
the caches after the update. That is not what the collector does: a backend
that lacks the hook still goes through the pause request (pause-requested is
set, the hook returns False), does not drain, and keeps the legacy behaviour
of invalidating only when recompute_kv_cache_after_weight_updates is true.

List only the paths that really drain (in_flight_weight_updates=false, or
managed Dynamo) and spell out the no-pause-hook fallback in its own sentence.
Runtime behaviour is unchanged.

Signed-off-by: Lin Jia <linj@nvidia.com>
…ined refits

`_prefix_caching_may_be_enabled` read only `vllm_cfg.enable_prefix_caching`
and returned True whenever that block was absent, so every Megatron, TRT-LLM
and async-SGLang in-flight refit with the flag false printed the vLLM-worded
warning. On Megatron generation under persist/offload the flag is derived to
false by setup itself, and both remedies in the message were wrong there:
setting the flag true makes setup raise, and `vllm_cfg.enable_prefix_caching`
is never read by that engine. The guard now returns False for every backend
but vLLM, and the message names the flag without the `grpo.async_grpo.`
prefix since the same collector serves `ppo.async_ppo`.

The unconditional drained-refit invalidation made Megatron generation under
persist/offload (whose `invalidate_kv_cache` truthfully returns False) print
the "KV cache invalidation not supported" warning on every drained refit.
When the user did not ask for a recompute, that is the backend following its
own configuration, so report it as information; the warning stays when the
flag is true.

Tests: the legacy-fallback test is parametrized on the flag and now pins the
no-pause-hook contract (invalidate iff the flag is true, no drain) and the
absence of the stale-prefix warning; a direct guard test covers vllm (unset,
true, false), megatron, sglang and trtllm; a drained Megatron-persist test
checks the information line replaces the warning only when the flag is
false.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
…rs are logged

The unset-flag derivation ran at the end of setup(), after
`logger.log_hyperparams(master_config.model_dump())`, so W&B/TensorBoard
recorded `recompute_kv_cache_after_weight_updates: true` for a Megatron
persist/offload run that executed with false, while the checkpoint
`config.yaml` (written from the mutated config) recorded false.

Move the derivation and the consistency check into
`_reconcile_megatron_recompute_kv_cache_flag` and call it before the Logger
is built, so the logged hyperparameters, the checkpoint config and the
trajectory collector all see the same value and an explicit mismatch fails
before any Ray resources are allocated. The helper's docstring notes that
`model_fields_set` reflects the merged YAML (exemplar included), so only
configs whose inheritance chain omits the key reach the derivation.

Tests: direct helper tests for derivation (recompute/persist/offload),
agreeing explicit values, and the three explicit mismatches; two setup()
tests asserting the mismatch raises before Logger() is called and that the
derived value is already in the logged hyperparameters; the mcore-marked
noncolocated Megatron setup test now asserts the derived false.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
…t refits

This is the only shipped config whose effective
`recompute_kv_cache_after_weight_updates` changed with the new class default:
it inherits `examples/nemo_gym/grpo_nanov3.yaml`, which omits the key, runs
async vLLM with `in_flight_weight_updates: true`, and already disables
`vllm_cfg.enable_prefix_caching`, so there was no stale-prefix problem to fix.
Pin false so the convergence-tracked .v1 recipe keeps its behaviour instead
of preempting and re-prefilling 24k-token video requests (and clearing the
encoder cache it keeps via `reset_encoder_cache_after_weight_update: false`)
on every refit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
…g scope

The exemplars `grpo_math_1B.yaml` and `ppo_math_1B.yaml` still read "Set to
true to recompute kv cache after weight updates", implying false is the
default, and nearly every recipe inherits them through `defaults:`, so the
flipped class default never reaches those recipes. Flipping the exemplars
instead would change the in-flight behaviour of every performance recipe and
make setup raise for the two sync megatron_generation recipes that inherit
`persist`, so the exemplars deliberately keep false and their comments now
say so: the class default is true, why false is kept, what false costs with
in-flight updates and prefix caching on, and that drained refits invalidate
regardless. The reference config, the research template, the workplace
assistant example and docs/guides/ppo.md carry the same note.

docs/guides/async-grpo.md item 5 is split into bullets: class default vs
exemplar pin, the two in-flight behaviours, the drained path with the Megatron
persist/offload caveat (it reports a kept cache; the collector logs it as
information), the no-pause-hook fallback, and the Megatron derivation of an
unset flag before the config is logged. The warning is stated to be
vLLM-only. docs/guides/single-controller.md notes that `AsyncRLConfig` still
defaults to false and does not warn.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
…on_async

vLLM 0.29.0's `engine/protocol.py` marks `clear_cache` DEPRECATED and the
RLHF HTTP router docstring says it is ignored in keep mode, while
`EngineCoreProc.pause_scheduler` -> `_finish_pause(clear_cache)` preempts
every running request and resets the prefix cache. The recompute-on-refit
path depends on the latter, so record it where the call is made and ask for
re-verification on a vLLM bump.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch
linj-glitch force-pushed the fix/async-refit-stale-prefix-cache branch from 8c0157b to 9dd13b3 Compare October 13, 2026 16:36
@linj-glitch

Copy link
Copy Markdown
Contributor Author

/ok to test 9dd13b3

@terrykong
terrykong requested a review from mehraakash October 15, 2026 20:18
@terrykong

Copy link
Copy Markdown
Collaborator

adding @mehraakash as reviewer as he is looking to fixing this globally and consolidating similar PRs doing the same thing

@terrykong terrykong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The drained-refit fix (trajectory_collector.py#L1175) is the right shape. It closes a real case where the vLLM prefix cache was never reset across refits, and it costs nothing because no request is running at that point. On vLLM 0.29.0 the reset also drops cached Mamba states for hybrid models, because one BlockPool backs every cache group.

Merge order. #3929 (pause generation during refit on the single-controller path) merges first, then #4647 (report a refused cache reset), then this PR. #3929 owns the default: true in AsyncGRPOConfig, AsyncPPOConfig, AsyncRLConfig and the exemplars. A test merge with #3929 conflicts in 8 files: docs/guides/async-grpo.md, docs/guides/ppo.md, docs/guides/single-controller.md, examples/configs/grpo_math_1B.yaml, examples/configs/ppo_math_1B.yaml, nemo_rl/algorithms/grpo.py, nemo_rl/algorithms/ppo.py and tests/unit/reference_configs/grpo_math_1B.yaml. The comments on grpo.py#L301 and on the Megatron derivation list what to drop. One more test merges cleanly but then fails (see the comment on test_grpo.py#L3912). After #3929, the single-controller path also has the stale-prefix-cache problem with the flag false (see the comment on the warning).

What stays after the rebase is worth landing, so please rebase rather than close:

  • always invalidating after a drained refit (trajectory_collector.py#L1175-L1176);
  • the one-time stale-prefix warning: 22 shipped in-flight vLLM configs still set the flag to false after #3929;
  • the Megatron mismatch check, moved before the config is logged and before Ray starts;
  • the new async-grpo.md item-5 bullets about the prefix cache;
  • the pause_generation_async docstring and the new tests.

Video recipe. The recompute_kv_cache_after_weight_updates: false line in the async-gym video recipe is right: prefix caching is off there and the video prefill is long. It is also what keeps nemotron_omni_gym_video_megatron_1n2g.sh passing once the default is true, so #3929's review asks #3929 to add the same line to that recipe. If #3929 lands with it, this hunk drops out on rebase; if not, keep it here.

PR title and description. Please update both after the rebase. This PR no longer flips the default, so "Reach of the class default" and "only config changed" become false. Suggested title: fix(async): flush the prefix cache on drained refits and warn when in-flight refits keep it.

Router. Not affected: this PR changes only the legacy loop, and the router is used only by the single controller and Gym.

PR description, "Remaining gaps":

  • The dropped reset_prefix_cache result is tracked in #4632, with a fix in #4647. Please link those instead of "a follow-up".
  • With in-flight updates and the flag false, the warning does not limit how stale the cache gets. vLLM resets the prefix cache only when no block is in use (block_pool.py#L777-L784), so it cannot drop just the prefix cache while requests run. A per-request cache_salt set to the weight version (kv_cache_utils.py#L604-L606) would stop reuse across weight versions while in-flight requests keep their KV. Worth a follow-up issue.

Visual explainer: https://terrykong.github.io/gh-pages-poc/terryk/pr-4018-stale-prefix.html

Generated by Claude Code

Comment thread nemo_rl/algorithms/grpo.py Resolved
Comment thread docs/guides/single-controller.md Resolved
Comment thread nemo_rl/algorithms/grpo.py Resolved
Comment thread nemo_rl/algorithms/grpo.py Resolved
Comment thread nemo_rl/algorithms/grpo.py Resolved
Signed-off-by: Lin Jia <linj@nvidia.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch linj-glitch added the CI:L2 Run doctests, unit tests, functional tests, and convergence tests label Oct 18, 2026
…-prefix-cache

# Conflicts:
#	docs/guides/async-grpo.md
#	examples/configs/recipes/vlm/vlm_grpo-nemotron-omni-30ba3b-16n8g-megatron-tp4ep4-async-gym-video.v1.yaml
#	nemo_rl/algorithms/grpo.py
- grpo.py: keep only the Megatron mismatch check, as
  _check_megatron_recompute_kv_cache_flag, called in setup() before the
  config is logged and gated on grpo.async_grpo.enabled; the later
  in-setup copy is removed and the message says to set the flag to false.
- async-grpo.md: the prefix-cache, drained-refit and no-pause-hook bullets
  live under main's item 5; the derivation sentence is gone.
- pause_generation_async docstring points at the vLLM-marked test that
  fails if a vLLM bump changes keep-mode clear_cache semantics.

Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch linj-glitch changed the title fix(async): do not let the vLLM prefix cache go stale across weight updates fix(async): flush the prefix cache on drained refits and warn when in-flight refits keep it Oct 24, 2026
@linj-glitch

Copy link
Copy Markdown
Contributor Author

Rebased on #3929 (merge commit 49fd2f0 + 5468e03) and reworked per the review; title and description updated.

What changed:

  • Dropped everything feat: pause SingleController refits and enable KV recomputation by default #3929 now owns: the default flip in AsyncGRPOConfig/AsyncPPOConfig, the recompute_kv_cache_after_weight_updates: false lines in grpo_math_1B.yaml, ppo_math_1B.yaml, the reference config and the video recipe, the ppo.md/single-controller.md edits, and the derivation block (grpo.py:499 thread). The failing assert in test_grpo.py:3915 is gone with it.
  • Megatron mismatch check: _check_megatron_recompute_kv_cache_flag (check only, "set the flag to false" wording) runs in setup() before Logger is built, gated on grpo.async_grpo.enabled (grpo.py:764 thread); main's later in-setup copy is removed so there is one check.
  • Stale-prefix warning: lives in VllmAsyncGenerationWorkerImpl.pause_generation_async via _resolve_enable_prefix_caching, names no config section, and the test moved to test_vllm_generation.py with a clear_cache=True stays-quiet case (trajectory_collector.py:1121 and test_async_utils.py:2851 threads).
  • "kept its KV caches" line only for the Megatron backend (trajectory_collector.py:1145 thread); the in-body comment is two lines (:1174 thread).
  • pause_generation_async docstring points at test_keep_pause_clear_cache_resets_prefix_cache (vLLM-marked GPU test, not run here) (vllm_worker_async.py:2295 thread).
  • async-grpo.md: the prefix-cache / drained-refit / no-pause-hook bullets sit under main's item 5; the derivation sentence is gone.

Local run: test_async_utils.py -k "refit or prefix or cache or invalidat" 12 passed; test_grpo.py -k "recompute or kv_cache or megatron" + test_config_v2.py 53 passed; the worker warning tests in test_vllm_generation.py pass (7; the pre-existing GPU test test_vllm_weight_update_and_prefix_cache_reset cannot run on this box). Pre-commit (ruff, ruff-format) clean on the touched files.

@linj-glitch linj-glitch changed the title fix(async): flush the prefix cache on drained refits and warn when in-flight refits keep it fix(async): flush drained-refit caches and warn on stale prefixes Oct 28, 2026
Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch

Copy link
Copy Markdown
Contributor Author

/ok to test 7963df3

@terrykong terrykong added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L2 Run doctests, unit tests, functional tests, and convergence tests labels Nov 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants