Skip to content
Open
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
c28c022
feat(integrations): add TP-aware Reward/Critic scoring and synchronou…
fusheng-ji Oct 11, 2026
7d6dba8
test(integrations): record PPO validation after the main rebase
fusheng-ji Oct 11, 2026
b01c6a2
feat(ppo): add pretrained Qwen3 BF16 adapters and reference qualifica…
fusheng-ji Oct 12, 2026
e538d55
fix(ppo): encode optional microbatch setting in rank agreement
fusheng-ji Oct 12, 2026
955f317
fix(ppo): reject incomplete pretrained reference loads
fusheng-ji Oct 12, 2026
2e2bf6f
fix(ppo): address review validation and Qwen3 export findings
fusheng-ji Oct 12, 2026
5144159
fix(ppo): serialize reference checkpoint loading diagnostics
fusheng-ji Oct 13, 2026
0bb76c8
feat(ppo): add immutable real-rollout precision and TP qualification …
fusheng-ji Oct 13, 2026
df48198
test(ppo): verify replicated TP tensors and FP32 optimizer states
fusheng-ji Oct 13, 2026
c36c995
docs(ppo): distinguish fixed-rollout qualification from historical ca…
fusheng-ji Oct 13, 2026
38079e2
fix(ppo): require update_observer on all TP ranks or none
fusheng-ji Oct 13, 2026
4c6eec3
Merge branch 'main' into feat/issue-44-tp-ppo
fusheng-ji Oct 13, 2026
17d1924
fix(ppo): collect vLLM dependencies without requiring pip
fusheng-ji Oct 13, 2026
4585cd3
docs(ppo): move the raw measurement outputs of the TP PPO checks to t…
fusheng-ji Oct 24, 2026
de3dcb6
Merge main into feat/issue-44-tp-ppo
fusheng-ji Oct 26, 2026
c7d34cc
fix(vllm): name the vLLM parameter that has no checkpoint source
fusheng-ji Oct 26, 2026
251e12f
example(tp_ppo): reject a rollout row without action tokens in the HF…
fusheng-ji Oct 26, 2026
258c701
docs: document the functions this change touches
fusheng-ji Oct 27, 2026
f0b58af
Merge main into feat/issue-44-tp-ppo
fusheng-ji Oct 27, 2026
6c78c7b
fix(ppo): address CodeRabbit review on grad_norm, GPU count and insta…
fusheng-ji Oct 30, 2026
2202486
Merge branch 'main' into feat/issue-44-tp-ppo
Flink-ddd Nov 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ jobs:
- name: Check refactor packaging and compatibility
run: python ci/run.py layout -q

- name: Run PPO BF16 buffer and checkpoint contracts
run: python -m pytest -q tests/integrations/engines/train/test_ppo_bf16.py tests/integrations/engines/train/test_qwen3_ppo_checkpoint.py tests/integrations/engines/train/test_qwen3_vllm_install.py tests/integrations/engines/train/test_ppo_adapters.py tests/integrations/common/test_ppo.py

- name: Run Mocked Hardware Discovery Tests
run: |
python -m pytest tests/runtime/test_dispatch.py -v
Expand Down
1 change: 1 addition & 0 deletions docs/.nav.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ nav:
- User Guide:
- usage/README.md
- usage/weight-sync-bridge.md
- TP Reward and Critic models: usage/tp-reward-critic.md
- usage/grpo-single-gpu-example.md
- Qwen3-8B VIME Consistency: usage/qwen3-vime-consistency.md
- Getting Started:
Expand Down
74 changes: 74 additions & 0 deletions docs/usage/tp-reward-critic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# TP-sharded Reward and Critic models

This page describes how rollout token IDs reach a tensor-parallel (TP) Reward or
Critic model and how scalar rewards return to the rollout ranks (issue #44). The
building blocks are in `rl_engine/integrations/common/weights.py` and
`rl_engine/integrations/common/tp_scalar.py`. Import `TensorPacketBridge` from
`integrations.common.weights` and `TPModelInputs` and `TPScalarExecutor` from
`integrations.common.tp_scalar`, as shown below.

## Data path

1. **Rollout tokens to the TP group.** `TensorPacketBridge` broadcasts an agreed,
ordered tensor schema (for example `input_ids`, `attention_mask`,
`position_ids`) with dynamic shapes from one source rank. Use a group that
contains the rollout leader and every rank of the Reward TP group.
2. **TP forward.** Each TP rank runs `TPScalarExecutor.rewards()` on the same
tokens. Rewards are pooled at the last valid token **before** any
communication, so a `local_sum` model needs one reduce of `B` scalars to the
TP leader. No hidden states or vocabulary logits are gathered.
3. **Rewards back to rollout.** The TP leader broadcasts the `[B]` rewards with
another `TensorPacketBridge` whose group contains the TP leader and the
rollout ranks.

```python
from rl_engine.integrations.common.tp_scalar import TPModelInputs, TPScalarExecutor
from rl_engine.integrations.common.weights import TensorPacketBridge

fields = ("input_ids", "attention_mask", "position_ids")
tokens = TensorPacketBridge(group=to_reward, source=rollout_leader, device=device)
packet = tokens.broadcast(fields, batch if rank == rollout_leader else None)

executor = TPScalarExecutor(model, group=tp_group, leader=reward_leader, output_layout="local_sum")
rewards = executor.rewards(TPModelInputs(*(packet[name] for name in fields)))

back = TensorPacketBridge(group=to_rollout, source=reward_leader, device=device)
rewards = back.broadcast(("rewards",), {"rewards": rewards} if rank == reward_leader else None)
```

## Contracts

- **`TensorPacketBridge`:**
- The source is a **global rank**, and the process group must be passed explicitly.
- All group members call `broadcast` in the same order with the same field tuple.
- Only the source supplies tensors, and they must already be on the configured device.
- Transported tensors are detached snapshots, not distributed autograd edges.
- Schema errors are raised on every member before any payload is sent.
- **`TPScalarExecutor` model adapters:**
- The adapter receives `input_ids`, `attention_mask`, `position_ids` and `use_cache=False`.
- It returns `[B, S]` values, or `[B]` or `[B, S]` rewards.
- `output_layout="replicated"` means the model has already completed its own head communication.
- `output_layout="local_sum"` means each rank returns an additive contribution. Include any bias on exactly one rank.
- The layout must be the same on every rank of a TP group.
- **`rewards()`:**
- Runs without gradients, in eval mode, and restores the caller's module modes afterwards.
- Only the leader receives the tensor; other ranks get `None`.
- **`values()`:**
- Returns differentiable `[B, S]` Critic values on every TP rank.
- Its `local_sum` reduction (`sum_tp_contributions`) is SUM in forward and identity in backward, because every TP rank evaluates the same loss. It is not a general distributed-autograd SUM.
- **`clip_tp_grad_norm_`:**
- Clips Critic gradients by the TP-global norm.
- Takes a layout per parameter: `sharded`, or `replicated` if the gradient is already synchronized.
- Replicated gradients count once in the norm.
- **Equivalence:** native PyTorch reductions give numerical, not bitwise, equivalence across TP sizes.

`TPScalarExecutor.head_metrics` and `TensorPacketBridge.metrics` count reduce
calls and payload bytes, excluding the small validation collectives.

## Tests

```bash
OMP_NUM_THREADS=1 python -m pytest -q tests/integrations/common/test_tp_scalar.py
# Additional NCCL check on two GPUs:
RUN_TP_CUDA=1 OMP_NUM_THREADS=1 python -m pytest -q tests/integrations/common/test_tp_scalar.py -k nccl
```
64 changes: 64 additions & 0 deletions examples/post_training/tp_ppo/QWEN3_ACCEPTANCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Qwen3-8B acceptance (in progress)

The previous random-GPT FP32 four-GPU result remains a smoke test. No production
model, BF16 or vLLM TP2 acceptance is claimed yet.

Pinned checkpoint: Qwen/Qwen3-8B at b968826d9c46dd6066d109eabc6255188de91218.
The manifest identifies weights, config and tokenizer files. The native mapping
preserves GQA Q/K/V grouping, replicated Q/K RMSNorm, unscaled RoPE, gate/up
partition stride and vocabulary padding. Scalar heads use seed 144 and scale
0.01 on independently loaded pretrained backbones. This does not validate a
learned reward model or PPO convergence.

BF16 training clips the optimizer's actual FP32 gradient buffers. Megatron uses
FP32 masters and accumulates microbatches into FP32 before clipping; DeepSpeed
uses its BF16 optimizer's FP32 accumulation views. DP=1 and ZeRO=0 are required;
FP16 loss scaling remains rejected. Microbatch loss weights count valid action
tokens, so unequal completion lengths do not bias gradient accumulation.

The historical teacher-forced HF calibration completed in job 1106655. It is
not sampler or PPO evidence. The formal fixed-rollout reference below collects
real vLLM output and independently computes HF PPO targets and updates. Its
artifacts and thresholds must be published before formal candidate execution;
they cannot be widened after observing candidates.

Formal matrix, still awaiting hardware qualification:

| Engine | Training precision | Training TP | Rollout |
|---|---|---|---|
| Megatron and DeepSpeed | FP32/BF16 | 1/2 | Same frozen vLLM TP2 rollout |
| Megatron and DeepSpeed | BF16 | Actor 1, Critic 1 | Actual vLLM TP2, two PPO rounds |
| Megatron and DeepSpeed | BF16 | Actor 2, Critic 1 | Actual vLLM TP1, two PPO rounds |
| Megatron and DeepSpeed | FP32 tiny GPT | Actor 2, Critic 1 | Existing vLLM TP1 regression |

The pretrained four-GPU entry is `tests.distributed.tp.check_qwen3_ppo_rollout`.
It requires frozen FP32-reference thresholds, uses real safetensors and sampler
logprobs, preserves reference/reward weights, and requires versions 0/1/2.
vLLM checks each rank's loaded QKV/gate-up/vocabulary shard against the actor
weights, rejects stale versions/missing rank confirmations, resets prefix cache
and checks cold/replay cache behavior. Rank memory and round timing are recorded.

Remaining: actual native checkpoint load validation; fixed-rollout HF controls;
FP32/BF16 TP1/TP2 fixed-rollout loss/gradient/update comparison; actual two-round
Megatron/DeepSpeed/vLLM TP2 and TP1 runs; GPU negative controls; immutable raw
artifacts and final threshold/acceptance report. CPU mapping and buffer tests
alone do not resolve PR #548's experiment gap.

The fixed-rollout qualification now has two separate jobs. `qualify_reference.sbatch`
uses actual vLLM TP2 sampling and independent HF scoring/GAE/PPO updates. Both HF
precisions repeat on the same immutable rollout and targets. `freeze_fixed_reference`
produces an acceptance manifest and per-tensor error bounds, which must be committed
before submitting `qualify_candidate.sbatch`. The candidate job checks all eight
engine/precision/TP combinations for both roles, then runs both four-GPU topologies
and the tiny-GPT regression. AdamW settings match in every precision. Evidence uses
actual FP32 optimizer gradients, measured clipping norms and parameter deltas.

The rule for max-absolute errors is four times the independent control difference
plus 1e-5 for scores, 1e-6 for losses/gradients/norms, or 1e-7 for parameter updates.
Relative L2 errors are also reported. Same-precision TP comparisons use independent
same-precision repeat bounds. No candidate result may establish or widen a bound.
Large per-parameter evidence stays outside Git; the committed manifests anchor its
hashes. Missing inputs, tampering, invalid tensor inventories and skipped GPU stages
are failures, not completed qualification. All jobs use `overflow`, `test`, 4 B200,
512 GB host memory and a 24-hour limit. Threshold publication is a stage boundary;
no formal PPO GPU acceptance is claimed until that candidate job actually passes.
Loading
Loading