Log config only on rank zero at torchtune/recipes/full_finetune_distributed.py
#2711
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/2711
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New FailureAs of commit 6ace7c2 with merge base e5ee1b2 ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
recipes/full_finetune_distributed.py
Outdated
| @@ -1056,7 +1057,9 @@ def recipe_main(cfg: DictConfig) -> None: | |||
| - Parameters specified in config (see available configs through ``tune ls``) | |||
| - Overwritten by arguments from the command-line | |||
| """ | |||
| config.log_config(recipe_name="FullFinetuneRecipeDistributed", cfg=cfg) | |||
| if torch.distributed.get_rank() == 0: | |||
There was a problem hiding this comment.
Could we move this inside the recipe's setup and use if self._is_rank_zero?
There was a problem hiding this comment.
Thanks for the PR @kabir2505! In addition to @SalmanMohammadi's comment, if you want to go a bit further here we can also make the same changes to our other distributed recipes: full_dpo_distributed.py, lora_dpo_distributed.py, knowledge_distillation_distributed.py, qat_distributed.py, qat_lora_finetune_distributed.py, early_exit_finetune_distributed.py, generate_v2_distributed.py, grpo_full_finetune_distributed.py, lora_finetune_distributed_multi_dataset.py
|
Hey @SalmanMohammadi @ebsmothers — just to confirm, should |
Correct! |
recipes/full_finetune_distributed.py
Outdated
| @@ -13,6 +13,7 @@ | |||
| from warnings import warn | |||
|
|
|||
| import torch | |||
| import torch.distributed | |||
There was a problem hiding this comment.
it was for checking rank outside setup earlier, i've fixed it now
| @@ -287,9 +287,9 @@ def setup(self, cfg: DictConfig) -> None: | |||
| """ | |||
| if self._is_rank_zero: | |||
| self._metric_logger = config.instantiate(cfg.metric_logger) | |||
|
|
|||
| # log config with parameter override | |||
There was a problem hiding this comment.
I think this still needs to log config here. The naming is bad, but one logs it to the metric logger (meaning it's saved into the WandB or Tensorboard run), and the other just prints it to disk
| @@ -1077,7 +1078,16 @@ def recipe_main(cfg: DictConfig) -> None: | |||
| - Parameters specified in config (see available configs through ``tune ls``) | |||
| - Overwritten by arguments from the command-line | |||
| """ | |||
| config.log_config(recipe_name="QATRecipeDistributed", cfg=cfg) | |||
| if not training.is_distributed(): | |||
There was a problem hiding this comment.
Were you running into errors here? Our tune run code should catch this.
There was a problem hiding this comment.
Hey! Nope, I wasn’t running into any errors. That part was already present in the code—I just added a line/space there, so it shows up as a diff. Nothing was changed functionally.
Context
What is the purpose of this PR? Is it to
This PR ensures that log_config is only called on rank zero in distributed runs to avoid redundant logging across devices.
Address: #2700
Please link to any issues this PR addresses.
Changelog
What are the changes made in this PR?
- Added a rank check to only log config on rank zero
Test plan
Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.
pre-commit install)pytest testspytest tests -m integration_testUX
If your function changed a public API, please add a dummy example of what the user experience will look like when calling it.
Here is a docstring example
and a tutorial example