Skip to content

Update int4pack related in torchchat gguf #1404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Dec 17, 2024
Merged

Update int4pack related in torchchat gguf #1404

merged 5 commits into from
Dec 17, 2024

Conversation

yanbing-j
Copy link
Contributor

@yanbing-j yanbing-j commented Dec 9, 2024 •
edited
Loading

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Dec 9, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1404

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

❌ 4 New Failures

As of commit e7b6f14 with merge base bb72b09 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Dec 9, 2024
@yanbing-j yanbing-j mentioned this pull request Dec 9, 2024
Closed
@Jack-Khuu
Copy link
Contributor

Jack-Khuu commented Dec 9, 2024 •
edited
Loading

I'll bump the pin real quick

#1407

@Jack-Khuu Jack-Khuu mentioned this pull request Dec 9, 2024
Merged
Jack-Khuu
Jack-Khuu approved these changes Dec 9, 2024
View reviewed changes
torchchat/utils/gguf_loader.py Outdated
@@ -24,6 +24,9 @@
pack_scales_and_zeros,
)

from torchao.dtypes.utils import is_device
from torchao.utils import TORCH_VERSION_AT_LEAST_2_6
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

torchchat locks onto a specific torch version, so we don't need to check

Assume > 2.6

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI failures seem that torchao version is not that new, because TORCH_VERSION_AT_LEAST_2_6 is a new one. And I saw you pin pytorch nightly to 20241013, which is also not new, and this nightly does not have pytorch/pytorch#139611 inside. This is my question, because the nightly used in the CI is 20241126.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, working on the bump here: #1367

We'll test your fixes on there

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

torchchat/utils/gguf_loader.py
weight = torch.empty(
(
out_features,
in_features // 2,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

torchchat/utils/gguf_loader.py
@@ -623,7 +655,7 @@ def load_model_and_state_dict(
in_features=in_features,
out_features=out_features,
bias=False,
device="meta",
device="cpu",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep this as a meta device as long as we can

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now only CPU acts different from cuda and meta. https://github.com/pytorch/torchchat/pull/1404/files/b884e295a164fa0b8cd172196e4409e51315567b#diff-28cab20c48af32e561f6e95cec7d029fa076708223a00d64afa80ad62b9b52a4R192 Use device meta here cannot tell the right shape of weight.

@Jack-Khuu Jack-Khuu added the Quantization Issues related to Quantization or torchao label Dec 10, 2024
@mikekgfb mikekgfb mentioned this pull request Dec 11, 2024
Open
@Jack-Khuu Jack-Khuu mentioned this pull request Dec 14, 2024
Closed
@Jack-Khuu
Copy link
Contributor

I did a quick rebase for you; feel free to change as needed

@yanbing-j
Copy link
Contributor Author

Hi @Jack-Khuu , thanks for the rebase! The remaining 4 CI failures seem related to cuda device and I cannot get the obvious errors related to Int4 code change. Could you please help me find out the simple reproducer? Thanks!

@Jack-Khuu
Copy link
Contributor

The cuda failures are known issues

thanks for the fix

@Jack-Khuu Jack-Khuu merged commit 56be609 into pytorch:main Dec 17, 2024
49 of 53 checks passed
@yanbing-j yanbing-j deleted the yanbing/fix_1389 branch December 17, 2024 05:03
vmpuri pushed a commit that referenced this pull request Feb 4, 2025
@yanbing-j @Jack-Khuu
Update int4pack related in torchchat gguf (#1404)
902542d 
* Update int4pack related for gguf

* Update gguf_loader.py

---------

Co-authored-by: Jack-Khuu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Jack-Khuu Jack-Khuu Jack-Khuu approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Meta Open Source bot. Quantization Issues related to Quantization or torchao
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Working around new int4wo weight packing
3 participants
@yanbing-j @Jack-Khuu @facebook-github-bot