Skip to content

Pin pt nightly CPU version #1061

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 12, 2024
Merged

Pin pt nightly CPU version #1061

merged 1 commit into from
Oct 12, 2024

Conversation

msaroufim
Copy link
Member

This just makes https://github.com/pytorch/ao/actions/runs/11293700323/job/31428383947?pr=962 green instead

There's dozens of these failures but the main one with a repro is

  f = <built-in function create_dynamic>
  args = (ModuleSpec(name='czbwqbrpps7k557gwtx5ach55zcy5nlo4hkgesymkc7ywznzhsf5.kernel', loader=<_frozen_importlib_external.Ext...ject at 0x7f2ec81a6d90>, origin='/tmp/torchinductor_root/zb/czbwqbrpps7k557gwtx5ach55zcy5nlo4hkgesymkc7ywznzhsf5.so'),)
  kwds = {}
  
  >   ???
  E   torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
  E   ImportError: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /tmp/torchinductor_root/zb/czbwqbrpps7k557gwtx5ach55zcy5nlo4hkgesymkc7ywznzhsf5.so)
  E   
  E   Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
  E   
  E   
  E   You can suppress this exception and fall back to eager by setting:
  E       import torch._dynamo
  E       torch._dynamo.config.suppress_errors = True
  E   
  E   
  E   To execute this test, run the following from the base repo dir:
  E       python test/prototype/test_low_bit_optim.py TestOptim.test_optim_smoke_optim_name_AdamWFp8_bfloat16_device_cpu

This just makes https://github.com/pytorch/ao/actions/runs/11293700323/job/31428383947?pr=962 green instead

There's dozens of these failures but the main one with a repro is 


```
  f = <built-in function create_dynamic>
  args = (ModuleSpec(name='czbwqbrpps7k557gwtx5ach55zcy5nlo4hkgesymkc7ywznzhsf5.kernel', loader=<_frozen_importlib_external.Ext...ject at 0x7f2ec81a6d90>, origin='/tmp/torchinductor_root/zb/czbwqbrpps7k557gwtx5ach55zcy5nlo4hkgesymkc7ywznzhsf5.so'),)
  kwds = {}
  
  >   ???
  E   torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
  E   ImportError: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /tmp/torchinductor_root/zb/czbwqbrpps7k557gwtx5ach55zcy5nlo4hkgesymkc7ywznzhsf5.so)
  E   
  E   Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
  E   
  E   
  E   You can suppress this exception and fall back to eager by setting:
  E       import torch._dynamo
  E       torch._dynamo.config.suppress_errors = True
  E   
  E   
  E   To execute this test, run the following from the base repo dir:
  E       python test/prototype/test_low_bit_optim.py TestOptim.test_optim_smoke_optim_name_AdamWFp8_bfloat16_device_cpu
```
Copy link

pytorch-bot bot commented Oct 11, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1061

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 4e1142b with merge base 9cd0da6 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 11, 2024
@msaroufim msaroufim merged commit 4734eda into main Oct 12, 2024
17 checks passed
@msaroufim msaroufim deleted the msaroufim-patch-23 branch October 12, 2024 00:07
HDCharles added a commit that referenced this pull request Oct 14, 2024
jainapurva pushed a commit that referenced this pull request Oct 15, 2024
This just makes https://github.com/pytorch/ao/actions/runs/11293700323/job/31428383947?pr=962 green instead

There's dozens of these failures but the main one with a repro is 


```
  f = <built-in function create_dynamic>
  args = (ModuleSpec(name='czbwqbrpps7k557gwtx5ach55zcy5nlo4hkgesymkc7ywznzhsf5.kernel', loader=<_frozen_importlib_external.Ext...ject at 0x7f2ec81a6d90>, origin='/tmp/torchinductor_root/zb/czbwqbrpps7k557gwtx5ach55zcy5nlo4hkgesymkc7ywznzhsf5.so'),)
  kwds = {}
  
  >   ???
  E   torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
  E   ImportError: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /tmp/torchinductor_root/zb/czbwqbrpps7k557gwtx5ach55zcy5nlo4hkgesymkc7ywznzhsf5.so)
  E   
  E   Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
  E   
  E   
  E   You can suppress this exception and fall back to eager by setting:
  E       import torch._dynamo
  E       torch._dynamo.config.suppress_errors = True
  E   
  E   
  E   To execute this test, run the following from the base repo dir:
  E       python test/prototype/test_low_bit_optim.py TestOptim.test_optim_smoke_optim_name_AdamWFp8_bfloat16_device_cpu
```
msaroufim added a commit that referenced this pull request Oct 15, 2024
* Revert "Pin pt nightly CPU version (#1061)"

This reverts commit 4734eda.

* Set LD_LIBRARY_PATH

* Update regression_test.yml

---------

Co-authored-by: Mark Saroufim <[email protected]>
yanbing-j pushed a commit to yanbing-j/ao that referenced this pull request Dec 9, 2024
This fixes the following assert that is easy to repro in any chat session:

```
Traceback (most recent call last):
    File "/home/ubuntu/cali/torchchat/torchchat.py", line 69, in <module>
        generate_main(args)
    File "/home/ubuntu/cali/torchchat/generate.py", line 896, in main
        for _ in gen.chat(generator_args):
    File "/home/ubuntu/cali/torchchat/generate.py", line 748, in chat
        self.chat_formatter.encode_header(
    File "/home/ubuntu/cali/torchchat/generate.py", line 53, in encode_header
        tokens.extend(self.tokenizer.encode(role, bos=False, eos=False))
    File "/home/ubuntu/cali/torchchat/tokenizer/tiktoken.py", line 133, in encode
        assert type(s) is str
```

I believe this regressed with pytorch/torchchat#1035.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants