Fix gptq device_map = "cpu"#1662
Conversation
fxmarty
left a comment
fxmarty
left a comment
There was a problem hiding this comment.
In that case, we still need to move each block to the gpu 0 on our own since we didn't add hooks.
Do you mean that accelerate does not add hooks in case we have a single device in device_map?
What if the user does not have a GPU?
| cache_block_outputs = True | ||
| modules_to_quantize_inside_block = None | ||
|
|
||
| device_map_for_quantization = {"": 0} |
There was a problem hiding this comment.
Setting default dictionary is dangerous: https://stackoverflow.com/questions/26320899/why-is-the-empty-dictionary-a-dangerous-default-value-in-python
There was a problem hiding this comment.
wow, thanks for the advice.
Yes, that's right. We added initially them, but we removed them since it was confusing for most users.
You need a GPU to quantize the model. It will trigger an error |
* fix gptq cpu device_map * fix test * remove default dict
2 participants
What does this do ?
This PR fixes the case where the user passes device_map = "cpu". In that case, we still need to move each block to the gpu 0 on our own since we didn't add hooks.
Fixes 28632