Whisper model support in Lite by oyilmaz-nvidia · Pull Request #11464 · NVIDIA-NeMo/NeMo

oyilmaz-nvidia · 2024-12-03T18:58:27Z

What does this PR do ?

Adds AutoModelSeq2Seq HF support

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

nemo/collections/llm/gpt/data/hf_dataset.py

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

nemo/collections/asr/models/hf_auto_model_for_speech_seq2seq.py

nemo/collections/llm/gpt/data/hf_dataset.py

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

…ur/whisper

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

pzelasko · 2024-12-11T17:14:51Z

I made some changes to enable the training to converge (commit c605d12):

Preserve HF tokenizer's Whisper special tokens (prompt)
Change the precision to bf16 (fp16 results in nan after the first weight update)
Fix decoder_input_ids and labels for training step

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

nemo/collections/speechlm/api.py

+
+from typing_extensions import Annotated
+
+import nemo.lightning as nl


Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

nemo/collections/speechlm/recipes/optim/adam.py

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

…ur/whisper

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

examples/speechlm/sft/hf2.py

examples/speechlm/sft/hf_hindi.py

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

…ur/whisper

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

titu1994

Looks good !

github-actions · 2024-12-24T05:28:21Z

[🤖]: Hi @oyilmaz-nvidia 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

* Data loader * another dataset * preprocessed audio dataset Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * seq2seq support Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * remove any update Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * fixing validation errors Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Modify training step and tokenizer to achieve correct Whisper training Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com> * Apply isort and black reformatting Signed-off-by: pzelasko <pzelasko@users.noreply.github.com> * Moved files into speechlm collection Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * revert changes Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * create recipes folder Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * generalize forward Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * example update Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * address codeql reviews Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * remove examples Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com> Signed-off-by: pzelasko <pzelasko@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: Piotr Żelasko <pzelasko@nvidia.com> Co-authored-by: pzelasko <pzelasko@users.noreply.github.com>

* Data loader * another dataset * preprocessed audio dataset Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * seq2seq support Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * remove any update Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * fixing validation errors Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Modify training step and tokenizer to achieve correct Whisper training Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com> * Apply isort and black reformatting Signed-off-by: pzelasko <pzelasko@users.noreply.github.com> * Moved files into speechlm collection Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * revert changes Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * create recipes folder Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * generalize forward Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * example update Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * address codeql reviews Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * remove examples Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com> Signed-off-by: pzelasko <pzelasko@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: Piotr Żelasko <pzelasko@nvidia.com> Co-authored-by: pzelasko <pzelasko@users.noreply.github.com> Signed-off-by: Abhinav Garg <abhgarg@nvidia.com>

* Data loader * another dataset * preprocessed audio dataset Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * seq2seq support Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * remove any update Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * fixing validation errors Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Modify training step and tokenizer to achieve correct Whisper training Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com> * Apply isort and black reformatting Signed-off-by: pzelasko <pzelasko@users.noreply.github.com> * Moved files into speechlm collection Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * revert changes Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * create recipes folder Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * generalize forward Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * example update Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * address codeql reviews Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * remove examples Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com> Signed-off-by: pzelasko <pzelasko@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: Piotr Żelasko <pzelasko@nvidia.com> Co-authored-by: pzelasko <pzelasko@users.noreply.github.com> Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>

oyilmaz-nvidia added 4 commits November 26, 2024 12:47

Data loader

8633771

another dataset

dd35d7e

preprocessed audio dataset

80cef7c

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

seq2seq support

9661008

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

oyilmaz-nvidia requested a review from pzelasko December 3, 2024 18:58

github-actions bot added the ASR label Dec 3, 2024

oyilmaz-nvidia and others added 3 commits December 3, 2024 18:59

Apply isort and black reformatting

089ea81

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

remove any update

0fd8069

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

fix merge conflicts

e9f5388

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

akoumpa reviewed Dec 3, 2024

View reviewed changes

nemo/collections/llm/gpt/data/hf_dataset.py Outdated Show resolved Hide resolved

Apply isort and black reformatting

1ae30ab

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

github-advanced-security bot found potential problems Dec 3, 2024

View reviewed changes

nemo/collections/asr/models/hf_auto_model_for_speech_seq2seq.py Fixed Show fixed Hide fixed

oyilmaz-nvidia changed the title ~~Onur/whisper~~ Whisper model support in Lite Dec 3, 2024

pzelasko reviewed Dec 10, 2024

View reviewed changes

nemo/collections/llm/gpt/data/hf_dataset.py Show resolved Hide resolved

oyilmaz-nvidia and others added 4 commits December 10, 2024 15:09

Merge branch 'main' of https://github.com/NVIDIA/NeMo into onur/whisper

56771de

fixing validation errors

358f78f

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Merge branch 'onur/whisper' of https://github.com/NVIDIA/NeMo into on…

12945fb

…ur/whisper

Modify training step and tokenizer to achieve correct Whisper training

c605d12

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

github-actions bot added the common label Dec 11, 2024

Apply isort and black reformatting

cc65fc5

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

Moved files into speechlm collection

945c447

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

github-actions bot removed the ASR label Dec 12, 2024

github-advanced-security bot found potential problems Dec 12, 2024

View reviewed changes

nemo/collections/speechlm/api.py

from typing_extensions import Annotated

import nemo.lightning as nl

Check notice

Code scanning / CodeQL

Module is imported with 'import' and 'import from'

Module 'nemo.lightning' is imported with both 'import' and 'import from'.

oyilmaz-nvidia added 2 commits December 12, 2024 18:10

revert changes

08a5cee

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

create recipes folder

ee9325a

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

github-advanced-security bot found potential problems Dec 12, 2024

View reviewed changes

nemo/collections/speechlm/recipes/optim/adam.py Fixed Show fixed Hide fixed

oyilmaz-nvidia and others added 3 commits December 16, 2024 12:25

generalize forward

18c16ed

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Apply isort and black reformatting

dbc1904

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

example update

6636b65

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

oyilmaz-nvidia and others added 2 commits December 16, 2024 16:16

Merge branch 'onur/whisper' of https://github.com/NVIDIA/NeMo into on…

a0fe933

…ur/whisper

Apply isort and black reformatting

beff019

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

github-advanced-security bot found potential problems Dec 16, 2024

View reviewed changes

oyilmaz-nvidia added 3 commits December 16, 2024 16:24

address codeql reviews

212d8c3

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Merge branch 'onur/whisper' of https://github.com/NVIDIA/NeMo into on…

41f37c7

…ur/whisper

remove examples

969e109

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

oyilmaz-nvidia marked this pull request as ready for review December 16, 2024 21:27

oyilmaz-nvidia requested a review from pzelasko December 16, 2024 21:27

titu1994 approved these changes Dec 18, 2024

View reviewed changes

oyilmaz-nvidia added Run CICD skip-docs labels Dec 18, 2024

pzelasko approved these changes Dec 18, 2024

View reviewed changes

Merge branch 'main' into onur/whisper

72abe0a

oyilmaz-nvidia enabled auto-merge (squash) December 19, 2024 16:44

oyilmaz-nvidia added Run CICD skip-docs and removed Run CICD common skip-docs labels Dec 19, 2024

akoumpa added Run CICD and removed Run CICD labels Dec 24, 2024

oyilmaz-nvidia merged commit 6224655 into main Dec 24, 2024

oyilmaz-nvidia deleted the onur/whisper branch December 24, 2024 05:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Whisper model support in Lite#11464

Whisper model support in Lite#11464
oyilmaz-nvidia merged 25 commits intomainfrom
onur/whisper

oyilmaz-nvidia commented Dec 3, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pzelasko commented Dec 11, 2024

Uh oh!

Check notice

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

titu1994 left a comment

Uh oh!

github-actions bot commented Dec 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		from typing_extensions import Annotated

		import nemo.lightning as nl

Conversation

oyilmaz-nvidia commented Dec 3, 2024

What does this PR do ?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pzelasko commented Dec 11, 2024

Uh oh!

Check notice

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

titu1994 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants