Repository navigation
Commit a53e2eb
committed
Merge upstream/main into feat/sc-resiliency-04-restart-readmission
PR3 (#3591) was SQUASH-merged into main as b3b6713, so none of its commits are
ancestors of main while PR4 still carries all of them. Git therefore sees PR3's whole
diff as independently added on both sides, which is why all 16 conflicts name b3b6713
and why the PR showed CONFLICTING despite the content being identical.
That made the classification, not the content, the work. For each conflicted file: is
main's version byte-identical to PR3's head (3d9ce21), and does PR4 add anything beyond
it? Three groups fell out.
GROUP A -- pure squash artefacts, resolved by taking OURS (10 files)
fleet_health.py, collective_weight_synchronizer.py, membership.py,
nccl_reshard_weight_synchronizer.py, grpo_sc_generation_shard_recovery.sh,
test_watchdog_pump.py, test_membership.py, test_reconcile_communicator.py,
test_reshard_rebuild.py, test_weight_synchronizer.py
main == PR3 exactly and no other PR touched them, so PR4's side is main's content plus
PR4's delta. Taking ours loses nothing.
GROUP B -- PR4 contributes nothing, resolved by taking THEIRS (2 files)
single_controller_utils/setup.py (#3480, #3727, #3821 on top of PR3)
tests/unit/single_controller/test_refit_recovery.py (#3480 on top of PR3)
GROUP C -- genuine merges (4 files), one per upstream PR below.
The six upstream PRs that contributed real content, and what each needed:
#3480 recover replay buffer from native TQ checkpoints
single_controller.py: rollout_recovery imports. Kept alongside ours.
setup.py, test_refit_recovery.py, L1 harness: group B / additive.
#3765 log toolcall and thinktag violation rate
single_controller.py: VIOLATION_TAG_KEYS. Auto-merged, verified present.
#3727 support non-colocated MInf
single_controller.py: MegatronGeneration import, kept alongside ours.
L1 harness: grpo_megatron_generation_gym_single_controller.sh entry.
#3821 warm-start the value model from a critic-pretrain checkpoint
config.py: the max_num_epochs validator. Ours only adds restart_dead_shards to
FleetHealthConfig, so both survive; verified the field landed in the right class
and the validator is intact.
#3655 nemo-lens telemetry
vllm_generation.py: the @trace_fn decorator on generate. Ours adds restart_shard
in a different region; both kept.
#3839 pause generation during in-flight refit
vllm_generation.py: pause_generation_for_refit / resume_generation_after_refit.
Auto-merged, verified present -- worth knowing it exists, since it pauses engines
around a refit and this PR restarts them.
Verified after resolving: no conflict markers; all four lint hooks clean (the single
pyrefly error is the pre-existing unrelated transfer_queue import); 1122 unit tests pass;
both submodule pointers and uv.lock/pyproject byte-identical to main.
Both sides' work was checked individually rather than assumed: EngineSupervisor wiring,
restart_dead_shards, restart_shard, recreate_worker, desired_membership and the report_refit
call on our side; the six items above on main's.
Note for anyone reproducing locally: #3655 adds a nemo-lens dependency that the pre-merge
container image does not carry, so tests fail at import with ModuleNotFoundError: nemo
until the venv is refreshed. Plain upstream/main fails the same way in that image; it is
not a merge defect.
Signed-off-by: asolergibert <asolergibert@nvidia.com>187 files changed
Lines changed: 20717 additions & 3341 deletions
File tree
- .github/workflows
- 3rdparty/Automodel-workspace
- docker
- docs
- about
- assets
- design-docs
- guides
- models
- glm
- observability
- examples
- configs
- recipes/llm
- nemo_gym
- nemo_rl
- algorithms
- async_utils
- loss
- single_controller_utils
- x_token
- data_plane
- adapters
- distributed
- experience
- models
- automodel
- generation
- megatron
- sglang/utils
- vllm
- megatron
- policy
- workers
- value/workers
- telemetry
- utils
- tests
- functional
- test_suites
- llm
- unit
- algorithms
- x_token
- data_plane
- experience
- models
- automodel
- generation
- sglang
- megatron
- policy
- value
- reference_configs
- single_controller
- telemetry
- tools
- utils
- tools
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
284 | 284 | | |
285 | 285 | | |
286 | 286 | | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
287 | 295 | | |
288 | | - | |
| 296 | + | |
| 297 | + | |
289 | 298 | | |
290 | 299 | | |
291 | | - | |
| 300 | + | |
292 | 301 | | |
293 | 302 | | |
294 | 303 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| |||
Submodule Automodel updated 2191 files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
186 | 186 | | |
187 | 187 | | |
188 | 188 | | |
189 | | - | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
190 | 195 | | |
191 | 196 | | |
192 | 197 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
| 18 | + | |
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
19 | 21 | | |
20 | 22 | | |
21 | 23 | | |
| |||
33 | 35 | | |
34 | 36 | | |
35 | 37 | | |
36 | | - | |
| 38 | + | |
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
| |||
43 | 45 | | |
44 | 46 | | |
45 | 47 | | |
46 | | - | |
| 48 | + | |
47 | 49 | | |
48 | 50 | | |
49 | | - | |
| 51 | + | |
50 | 52 | | |
51 | 53 | | |
52 | 54 | | |
53 | 55 | | |
54 | 56 | | |
55 | 57 | | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
66 | 72 | | |
67 | 73 | | |
68 | 74 | | |
| |||
72 | 78 | | |
73 | 79 | | |
74 | 80 | | |
75 | | - | |
| 81 | + | |
76 | 82 | | |
77 | 83 | | |
78 | 84 | | |
| |||
82 | 88 | | |
83 | 89 | | |
84 | 90 | | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
93 | 103 | | |
94 | 104 | | |
95 | 105 | | |
96 | 106 | | |
97 | 107 | | |
98 | | - | |
| |||
Loading
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
0 commit comments