This repository was archived by the owner on Aug 1, 2024. It is now read-only.
Pretraining resources of ESM2 #414
Answered
by
halilakin
GanjinZero
asked this question in
Q&A
-
|
I am interesting to know the pretraining resources for different sizes of ESM2 (GPU count and training time). Thank you. |
Beta Was this translation helpful? Give feedback.
Answered by
halilakin
Dec 6, 2022
Replies: 1 comment
-
|
ESM2 700M had 1.6e22 TFLOPs compute ESM2 3B had 6e22 TFLOPs compute (~4x over 700M) ESM2 15B had 1.2e23 TFLOPs compute (2x over 3B) |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
halilakin
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
ESM2 700M had 1.6e22 TFLOPs compute
512 GPUs * 45TFLOPs per sec * (606024) * 8 days on V100s
ESM2 3B had 6e22 TFLOPs compute (~4x over 700M)
512 GPUs * 45TFLOPs per sec * (606024) * 30 days on V100s
ESM2 15B had 1.2e23 TFLOPs compute (2x over 3B)