[train] Simplify ray.train.xgboost/lightgbm (1/n): Align frequency-based and checkpoint_at_end checkpoint formats

[justinvyu]

Why are these changes needed?

This PR fixes XGBoostTrainer and LightGBMTrainer checkpointing:

1. Centralizes checkpoint saving and loading implementations around the utility callbacks ray.train.xgboost/lightgbm.RayTrainReportCallback as the standard utilities to define checkpoints save/load format.
   • Previously, we had 3 separate implementations of xgb/lgbm checkpointing: (1) XGBoostCheckpoint, (2) ray.tune.integration.xgboost.TuneReportCheckpointCallback, and (3) XGBoostTrainer._save_model.
     • (1) was only used by (3) to get the XGBoostCheckpoint.MODEL_FILENAME constant in some places. But, we re-implemented the from_model and get_model logic for some reason.
     • (2) was used for saving frequency-based checkpoints configured via CheckpointConfig(checkpoint_frequency)
     • (3) was only used for saving the final checkpoint at the end of training
     • These diverging codepaths is what resulted in the confusing bug described here.
   • Now, we have a single codepath (the ray.train.*.RayTrainReportCallback) that handles both checkpoint_frequency and checkpoint_at_end. This codepath standardizes on the framework specific checkpoint implementation of checkpoint saving.
2. Deletes a lot of the legacy code that was needed during Ray 2.7 to 2.9 for migration purposes.
3. Hard deprecates legacy APIs (TuneReportCallback). The migration is simple: TuneReportCallback() -> TuneReportCheckpointCallback(frequency=0).
4. Removes the circular dependency of ray.tune and xgboost_ray/lightgbm_ray.
   • The circular dependency: xgboost_ray -> ray.tune.* -> ray.train.* -> ray.train.xgboost -> xgboost_ray
   • This caused an ImportError which xgboost_ray incorrectly used to determine whether Ray Train/Tune were installed.
   • The long term fix here is to fully remove the xgboost_ray and lightgbm_ray dependencies by re-implementing simple versions of these trainers as DataParallelTrainers. See: [train] Simplify ray.train.xgboost/lightgbm (2/n): Re-implement XGBoostTrainer as a lightweight DataParallelTrainer #42767.

API Change Summary

API	Change	Notes
ray.train.xgboost.RayTrainReportCallback	🆕	This is the utility callback that we can recommend users to build off of, similar to ray.train.lightning.RayTrainReportCallback. This will be exposed to users if they have full control over the training loop in the new simplified XGBoostTrainer.
ray.train.xgboost.RayTrainReportCallback.get_model(filename)	🆕	This is a utility method to load a model from a checkpoint saved by this callback. This should be recommended over XGBoostTrainer.get_model in the future.
ray.tune.integration.xgboost.TuneReportCheckpointCallback	↔️	This callback is now just an alias of the one above, and will be usable by Tune users who want to use the callback in Tune without distributed training. We can consider deprecating it in the future.

The same APIs are introduced for the lightgbm counterparts.

TODOs left for followups

• Revamp release tests, as most of them are just testing xgboost_ray right now.
• Fix the checkpoint_at_end vs. checkpoint_frequency overlap logic for the test case with a TODO in test_xgboost_trainer after switching to the simplified xgboost trainer.

Related issue number

Closes #41608

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[matthewdeng]

Do we still need these changes if we're centralizing on the Callbacks?

[justinvyu]

Nope I can get rid of it. If anybody does use this, specifying your own temp dir might be useful though if you want it to be cleaned up after.

[matthewdeng]

nit: For consistency with this, should we update the training example to use the LightGBMTrainer? Same for xgboost.

[justinvyu]

I want to add the *Trainer examples once I add in a v2 xgboost/lightgbm trainer, since then it'll actually show the callback usage in the training func. Right now the user doesn't need to create the callback themselves.

[woshiyyya]

Are we going to add them back later?

[justinvyu]

This used to be a code-block that didn't run 😅 I just wanted to show a mock xgboost.train call with the callback inside, without needing to specify the dataset and everything.

[justinvyu]

TuneCallback for lgbm was originally an empty class that wasn't referenced anywhere else so I just removed it.