Repository navigation

FSDP + TP & native save/load distributed - #45028

Merged

3outeille merged 133 commits into

mainfrom

refactor-tp-dtensor

May 19, 2026

Merged

FSDP + TP & native save/load distributed#45028
3outeille merged 133 commits into
mainfrom
refactor-tp-dtensor

Commits on Mar 25, 2026
- init
  3outeille
  committed
  7c84339
- Merge branch 'main' into distributed_api
  3outeille
  authored
  69bc48e
Commits on Apr 13, 2026
- Merge branch 'main' into distributed_api
  3outeille
  authored
  b7ec958
- Merge remote-tracking branch 'origin/main' into distributed_api
  3outeille
  committed
  45a01a5
- FSDP2 (fully_shard) integration
  3outeille
  committed
  a5c2554
- DistributedConfig + shard-on-read loading
  3outeille
  committed
  739332c
- TPStyle API + dense model tensor parallelism
  3outeille
  committed
  11b55a2
- Merge branch 'main' into distributed_api
  3outeille
  authored
  eeefc9e
- Merge branch 'distributed_api' into fsdp-vs-ddp
  3outeille
  authored
  9038475
- revert some files
  3outeille
  committed
  abfd57e
- Merge branch 'fsdp-vs-ddp' of https://github.com/huggingface/transformers into fsdp-vs-ddp
  3outeille
  committed
  23a2c05
- Add distributed training scripts
  3outeille
  committed
  c33873e
- Merge branch 'distributed_api' of https://github.com/huggingface/transformers into distributed_api
  3outeille
  committed
  e783231
- Remove train_fsdp_tp_torchtitan_style.py
  3outeille
  committed
  34db840
- unify the utils for fsdp
  3outeille
  committed
  6f9e2b6
Commits on Apr 14, 2026
- Merge branch 'distributed_api' into fsdp-vs-ddp
  3outeille
  authored
  5e017cf
- Fix CI: re-export moved FSDP utils + remove stale type: ignore
  3outeille
  committed
  37dcc14
- Merge branch 'fsdp-vs-ddp' into fsdp-core-model-loading
  3outeille
  committed
  c1dab9e
- Merge branch 'fsdp-core-model-loading' into refactor-tp-dtensor
  3outeille
  committed
  e0c4e06
- Fix ruff formatting in core_model_loading.py
  3outeille
  committed
  21f0561
- Fix ruff linting and formatting
  3outeille
  committed
  cd45107
- Merge branch 'fsdp-core-model-loading' into refactor-tp-dtensor
  3outeille
  committed
  52c390f
- Backport new TP/FSDP API from orchestration-save-load branch
  3outeille
  committed
  ba3990f
- Fix DTensor imports in Copied-from model files
  3outeille
  committed
  92a3491
- MoE expert parallelism + sequence parallelism (#45408)
  
  3outeille
  and
  claude
  authored
  7ca7911
- do monkey patching for rotary
  3outeille
  committed
  d4400d5
- Revert modeling file diffs to match fsdp-core-model-loading base
  
  3outeille
  and
  claude
  committed
  6793503
- Migrate all model TP plans from strings to TPStyle
  3outeille
  committed
  b943512
- Restore mxfp4.py to match base branch
  3outeille
  committed
  5ce6faa
- Drop mla_kv_a_proj and moe_identity_expert from TP plans
  3outeille
  committed
  b694f36
Commits on Apr 15, 2026
- more comments
  3outeille
  committed
  1b82460
- fix tp for most models. PyTorch doesn't implement all placement conversions (e.g. _StridedShard↔Shard). We force replicate beforehand
  3outeille
  committed
  48f8d6f
Commits on Apr 16, 2026
- fix tp through _replicate_dtensor
  3outeille
  committed
  91b4824
- revert small change
  3outeille
  committed
  44706eb
- push temporary fix for TP and strided shard for backward
  3outeille
  committed
  aa45f5b
- refactor a bit
  3outeille
  committed
  0a566c5
- patches for rotary
  3outeille
  committed
  11a55d4
- refactor MoEExpertsParallel
  3outeille
  committed
  53490d9
Commits on Apr 18, 2026
- fix tp for last models
  3outeille
  committed
  0c09915
- refactor moe expert parallels
  3outeille
  committed
  ebd03ec
- linting
  3outeille
  committed
  c08c071
- add sp plan for models
  3outeille
  committed
  4804d0d
- add deepseek v2 sp plan
  3outeille
  committed
  1a51928
- undo sp plan for some tricky models
  3outeille
  committed
  fd3a722
Commits on Apr 20, 2026
- remove lm_head from config
  3outeille
  committed
  253b89e
- first pass of refactoring dtensor shard operator
  3outeille
  committed
  3ff1fee
- better refacto
  3outeille
  committed
  4d96b2d
Commits on Apr 22, 2026
- batter explanation of DtensorShardOperation
  3outeille
  committed
  04521bf
- refactor dtensor test to reflect real world scenario
  3outeille
  committed
  f710f0d
- more comments
  3outeille
  committed
  a35993c
- fix tp olmo hybrid and exaone
  3outeille
  committed
  8529d7c
- Enhance tensor parallel weight tying logic to prevent clobbering of lm_head when embed_tokens is not in the plan.
  3outeille
  committed
  43b792b
Commits on Apr 28, 2026
- fix fsdp mixin test due to missing args
  3outeille
  committed
  0dbef90
- fix test non model
  3outeille
  committed
  da83f32
- skip sp plan for exaone and olmo hybrid
  3outeille
  committed
  3903757
- linting
  3outeille
  committed
  e51f663
- fix import for ci
  3outeille
  committed
  96f3f29
- test distributed config
  3outeille
  committed
  dfb448e
- attempt to fix guarding import ci
  3outeille
  committed
  0a74b7d
- fix ci check repro
  3outeille
  committed
  c50e49c
- add ALL_PARALLEL_STYLES registry alongside TPStyle
  3outeille
  committed
  f9daf7b
- route apply_tensor_parallel through ALL_PARALLEL_STYLES
  3outeille
  committed
  8a1a9e5
- migrate modular files to string-based TP plans
  3outeille
  committed
  7819783
- migrate standalone configs and modelings to string-based TP plans
  3outeille
  committed
  e70ac37
- delete TPStyle dataclass
  3outeille
  committed
  061d4e6
Commits on Apr 30, 2026
- fix use_local_output defaults for SequenceParallel and PrepareModuleInput in registry
  3outeille
  committed
  8e0f60c
- use parallel style from torch
  3outeille
  committed
  5b336bd
Commits on May 1, 2026
- revert changes in weight converter
  3outeille
  committed
  465d029
- remove dead code in set_param_for_module
  3outeille
  committed
  bc6d6f9
- remove dead code
  3outeille
  committed
  f305f92
- cleaning again
  3outeille
  committed
  39db8c1
- cleaning
  3outeille
  committed
  951d4ae
- revert change
  3outeille
  committed
  1b040ef
- linting
  3outeille
  committed
  85ef27c
Commits on May 4, 2026
- refactor dtensor shard ops
  3outeille
  committed
  1fd7b1d
- revert some stuff in core model loading
  3outeille
  committed
  4547eb3
Commits on May 5, 2026
- core model loading clean
  3outeille
  committed
  43086d3
Commits on May 10, 2026
- guarding import
  3outeille
  committed
  1b7ebe1
- better separation tensor parall and generic utils
  3outeille
  committed
  6d86746
- isolate DtensorShardOperation into a separate file
  3outeille
  committed
  ff49346
- no need to patch rotary
  3outeille
  committed
  a806b3d
Commits on May 11, 2026
- better seperation
  3outeille
  committed
  98d2dc5
- simplify gather_full_state_dict
  3outeille
  committed
  14e02aa
- simplify _replicate_dtensor
  3outeille
  committed
  9acf944
- fix and clean _replicate_dtensor
  3outeille
  committed
  20cf4e8
- better doc for DtensorShardOperation
  3outeille
  committed
  ca6d06b
Commits on May 12, 2026
- fix saving optimizer with DCP for fused weights
  3outeille
  committed
  7e2115f
Commits on May 13, 2026
- save_pretrained(distributed_checkpoint=true)
  3outeille
  committed
  1c6f848
- linting
  3outeille
  committed
  41bc6eb
- refactor into a single function _dtensor_from_local_like
  3outeille
  committed
  27fc8a9
- zeros_like instead of empty_like
  3outeille
  committed
  1e25f1f
- move tp and fsdp under distributed
  3outeille
  committed
  7405892
- distribute_model
  3outeille
  committed
  ed45c91
- fix deadlock when saving
  3outeille
  committed
  f97c3a4
- clip grad norm function
  3outeille
  committed
  b59c4bf
- maybe_disable_foreach_and_fused_for_mixed_dtensor_groups
  3outeille
  committed
  242e814
Commits on May 14, 2026
- better TP api for ease of understanding
  3outeille
  committed
  8fe831f
- remove shard_param to make it easier
  3outeille
  committed
  bf30f0a
- fix import in test
  3outeille
  committed
  261c59b
- _swap_dtensor_params_for_local
  3outeille
  committed
  f0f5f67
- fix qwen3 nanochat dots1
  3outeille
  committed
  c135d0e
- Merge branch 'main' into fsdp-vs-ddp
  3outeille
  committed
  10c6563
- Merge branch 'fsdp-vs-ddp' into refactor-tp-dtensor
  3outeille
  committed
  5db32b8
- add tpu
  3outeille
  committed
  920ade5
- move TP refactor experimentation scripts to backup branch
  
  3outeille
  and
  claude
  committed
  13646c8
- linting
  3outeille
  committed
  c49e9ab
- register distributed sharding_utils and utils in __init__
  
  3outeille
  and
  claude
  committed
  a4c6ba8
- rename TP plan styles to match new ALL_PARALLEL_STYLES registry
  
  3outeille
  and
  claude
  committed
  65b0311
- enable EP
  3outeille
  committed
  dbf0c60
- Add enable_expert_parallel configuration option in test_distributed_config
  3outeille
  committed
  51068ca
Commits on May 16, 2026
- no more auto mode
  3outeille
  committed
  bf0696f
- edit fsdp plan to every other models
  3outeille
  committed
  d13ab2e
- update fsdp mixin tests
  3outeille
  committed
  4165209
Commits on May 17, 2026
- linting
  3outeille
  committed
  6f5dbfb
Commits on May 18, 2026
- fix test fsdp
  3outeille
  committed
  d075668
- fsdp linting
  3outeille
  committed
  9472436
Commits on May 19, 2026
- revert gitignore
  3outeille
  committed
  9a835eb
- _apply within for loop
  3outeille
  committed
  86dc7b4
- rename
  3outeille
  committed
  9378ccb
- doc sp plan
  3outeille
  committed
  e878594
- fix
  3outeille
  committed
  8caa870
- unified settattr + torch no grad + _local_tensor
  3outeille
  committed
  e0e787b
- revert
  3outeille
  committed
  9158d99
- linting
  3outeille
  committed
  1c8203a
- Merge branch 'main' into refactor-tp-dtensor
  3outeille
  authored
  7fb37af
- Merge branch 'main' into refactor-tp-dtensor
  3outeille
  authored
  0fdc61f
- fix ruff
  3outeille
  committed
  7e2f686
- make check-repository-consistency
  3outeille
  committed
  e6484d3
- trigger fsdp mixin test in CI
  3outeille
  committed
  4b649ed
- Merge branch 'main' into refactor-tp-dtensor
  3outeille
  authored
  5d673d3
- fix fsdp ci
  3outeille
  committed
  6b835cb
- Merge branch 'refactor-tp-dtensor' of https://github.com/huggingface/transformers into refactor-tp-dtensor
  3outeille
  committed
  a38bc32
- Reset tests/test_modeling_common.py to main
  3outeille
  committed
  d563a9c

Commits

Commits on Mar 25, 2026

Commits on Apr 13, 2026

Commits on Apr 14, 2026

Commits on Apr 15, 2026

Commits on Apr 16, 2026

Commits on Apr 18, 2026

Commits on Apr 20, 2026

Commits on Apr 22, 2026

Commits on Apr 28, 2026

Commits on Apr 30, 2026

Commits on May 1, 2026

Commits on May 4, 2026

Commits on May 5, 2026

Commits on May 10, 2026

Commits on May 11, 2026

Commits on May 12, 2026

Commits on May 13, 2026

Commits on May 14, 2026

Commits on May 16, 2026

Commits on May 17, 2026

Commits on May 18, 2026

Commits on May 19, 2026