Skip to content

try to use 48xlarge runners for llava-runner and eval_llama-mmlu #8181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

swolchok
Copy link
Contributor

@swolchok swolchok commented Feb 4, 2025

These jobs are timing out. I found big gaps in the timestamps of the logs even before the blame PR, which to me suggests swapping. Let's try bigger runners and see what happens on this PR.

Fixes #8180 (if it lands)

[ghstack-poisoned]
@swolchok
Copy link
Contributor Author

swolchok commented Feb 4, 2025

Copy link

pytorch-bot bot commented Feb 4, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8181

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

❌ 1 New Failure, 2 Pending

As of commit ae0eb57 with merge base 58c725c (image):

NEW FAILURE - The following job has failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 4, 2025
swolchok added a commit that referenced this pull request Feb 4, 2025
These jobs are timing out. I found big gaps in the timestamps of the logs even before the blame PR, which to me suggests swapping. Let's try bigger runner and see what happens on this PR.

Fixes #8180 (if it lands)

ghstack-source-id: 1648a66a3ed17701b2049481dffeb342d5c6f6c6
ghstack-comment-id: 2634545665
Pull Request resolved: #8181
Copy link

github-actions bot commented Feb 4, 2025

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@swolchok
Copy link
Contributor Author

swolchok commented Feb 4, 2025

  1. looks like there may be no such thing as a 48xlarge runner. The runners appear to use c5 AWS instances, and 24xlarge is the largest in that category.
  2. c5.24xlarge instances have 192 GB of RAM. If we're swapping on one of those, something is probably wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI timeout (test-llava-runner-linux) since #7922
2 participants