Add an example using Optuna and Transformers

[ParagEkbote]

What does this PR do?

In this end-to-end tutorial, we are going to utilize the optuna library to perform hyperparameter optimization on a BERT model using the IMDB dataset.

Firstly, we will load and preprocess the dataset and define the model we want to perform HPO on. Then, we shall set the metrics and wrap it inside the trainer class along with a search space that will search the best set of hyperparameters for the learning rate, weight decay and batch size. Lastly, we will visualize the results as well.

Please let me know if any modifications are required and I will make the necessary changes.

Who can review?

@stevhliu.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[stevhliu]

Thanks for your work! However, I don't think its all that different from the current hyperparameter search docs in Transformers except its a more complete example.

@merveenoyan @sergiopaniego what do you think?

[ParagEkbote]

Just for the record, I'd actually wanted to include support for the transformer's library in their optuna-integration package. But since there is backend support provided by the transformers library, I contributed an starting example to their repo.

This PR builds on that example and provides a more hands-on approach for users to understand how to apply HPO to transformer models 🙂

[merveenoyan]

@ParagEkbote cookbook mostly contains end-to-end applied AI recipes where library integrations shine 💫 rather than minimal examples. it would be great to make it a more applied ML type of recipe

[ParagEkbote]

I have now added the following improvements to the recipe to make it more applied:

1. The successful trials are now saved in sqlite using RDBStorage.
2. Observability is now available using Weight & Biases to track and analyze the HPO trials.
3. Also, we can perform the final training from the optimized parameters and push the model to HF Hub.

Could you please review the changes?

cc: @stevhliu, @merveenoyan

[sergiopaniego]

Thanks so much for the contribution, @ParagEkbote! 🙌

To make it more aligned with the rest of the recipes, it would be great to add a bit more context about the problem this recipe is aiming to solve. Including visuals or showcasing the results would also go a long way in making the explanation clearer and more engaging. I’d recommend checking out the other recipes for inspiration 😄 Also, make sure it runs as it is in Colab

What do you think?

[ParagEkbote]

Thanks so much for the contribution, @ParagEkbote! 🙌

To make it more aligned with the rest of the recipes, it would be great to add a bit more context about the problem this recipe is aiming to solve. Including visuals or showcasing the results would also go a long way in making the explanation clearer and more engaging. I’d recommend checking out the other recipes for inspiration 😄 Also, make sure it runs as it is in Colab

What do you think?

What type of visuals or results can help for the recipe. Also, are there any specific sections do you think additional context is required?

cc: @sergiopaniego

[sergiopaniego]

What type of visuals or results can help for the recipe. Also, are there any specific sections do you think additional context is required?

cc: @sergiopaniego

For example, we currently don't have any output throughout the notebook so we are not displaying any results 😄

[ParagEkbote]

The visualizations which can be created in the example:

[ParagEkbote]

I have updated the tutorial to add context about the problem and executed the notebook as well. You can find the model on the HF Hub: https://huggingface.co/AINovice2005/bert-imdb-optuna-hpo

Could you please review the changes?

[sergiopaniego]

Great improvement!!!

Could we include those visualizations in the notebook?
For example, there is a section specifically called Visualize the results but no results are displayed 😅

Additionally, we could still add additional context and make it more applied (explaining why is this important for the chosen use case in detail, providing outputs/visualization...) For example, showing the dataset, why the column selection, testing the trained model in the end to showcase its capabilities, compare the trained model to another one not trained using this approach... The idea of the recipe is to be and end-to-end applied notebook for a specific use case, rather than a documentation page 😄

[ParagEkbote]

@sergiopaniego I've fixed the visualization errors and the made the cookbook language less instructional. Could you please review?

[sergiopaniego]

Can we adjust the titles, subtitles... in markdown format to follow:

\# Main title

\## Subtitle

....

?

Currently, the main tittle is \###. This will be really helpful for generating the index afterwards :)

Reply via ReviewNB

[sergiopaniego]

Great improvement! Thanks!
Let's adjust the depth of the titles, subtitles... (#, ##, ###...) and we're ready :)

[ParagEkbote]

@sergiopaniego Thank you for being a patient reviewer and providing me with detailed feedback for this recipe :)

I've updated the titles and subtitles with the appropriate markdown format as well.

Could you please review?

[sergiopaniego]

Nice!

We finally need to add it to the index page as part of the latest notebooks. To do so, you can remove the oldest one (the one at the bottom) and add yours to the first position. You can look for example in previous PRs.

[ParagEkbote]

It's done. Could you please review?

cc: @sergiopaniego

[sergiopaniego]

LGTM!
Let's wait for @stevhliu or @merveenoyan comments 😄

[stevhliu]

Thanks both for iterating on this!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.