[WIP] gpt_fused #189
[WIP] gpt_fused #189
Conversation
facebook-github-bot
added
the
CLA Signed
| For example | ||
|
|
||
| ``` | ||
| PYTHONPATH=/home/cpuhrsch/local/ao/torchao/prototype/models/gpt_fused CUDA_VISIBLE_DEVICES=0 numactl --membind 0 --cpubind 0 python generate.py --compile --checkpoint_path checkpoints/$MODEL_REPO/model.pth --prompt "Hello, my name is" |
There was a problem hiding this comment.
what's going on here lol, why do i need to set the python path?
There was a problem hiding this comment.
So that the import statements in gpt-fast pick up on the location of model.py in torchao
| @@ -0,0 +1,13 @@ | |||
| ## gpt-fused | |||
|
|
|||
| A more handwritten version of [gpt-fast](https://github.com/pytorch-labs/gpt-fast)'s model.py for us to experiment with. | |||
There was a problem hiding this comment.
We could use this to try various fused kernels (Triton or CUDA).
| @@ -0,0 +1,255 @@ | |||
| # Copyright (c) Meta Platforms, Inc. and affiliates. | |||
There was a problem hiding this comment.
could you add a file in test or benchmark script that would just sanity check that the script works with real and random weights
Also why prototype namespace? I think torchao.models.gpt is better, i expect a lot of people will use this as is
There was a problem hiding this comment.
Using prototype just to get started. Yes, we can add a benchmark script. I'll work on that next.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/189
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New FailuresAs of commit de8400d with merge base e7bbbd2 ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
* convert group_size to groupsize * group_size to groupsize in REAADME.md
A torchao version of gpt-fast's model.py for experimentation.
Currently just a copy-paste of gpt-fast's model.py to get feedback on the idea.