Add export_llama performance regression test using expected ops #9158
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9158
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 54cd638 with merge base 306b649 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
jackzhxng
requested review from
iseeyuan,
larryliu0820 and
swolchok
as code owners
jackzhxng
added
the
release notes: examples
| # Ops expected to be found in the default exported llama_transformer. Obtained through | ||
| # print_delegation_info from the backend_debug module, which is displayed with | ||
| # export_llama under --verbose. | ||
| BASE_EXPECTED_OPS = { |
There was a problem hiding this comment.
I think this might be a little risky to test. If the IR changes which is a common possibility this count will change and we'll have to keep fixing this.
| # we cannot test quantization args in this way | ||
| # since quantization requires promoting meta tensors | ||
| # to the cpu device, which requires real weights. | ||
| export_args_str = """ |
There was a problem hiding this comment.
Also why do this? Why not just generate the args directly and use them.
There was a problem hiding this comment.
Oh I feel like it's more clear to read?
There was a problem hiding this comment.
Hmm generally it's not good practice to hard code strings like this as they'll tend to change. The underlying args they refer to tend to be more stable so i'd say switch over to the args directly.
|
@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
jackzhxng
force-pushed
the
jz/regression-test-llama-perf
branch
from
738cdf0 to
d3d8e7d
Compare
jackzhxng
force-pushed
the
jz/regression-test-llama-perf
branch
from
d3d8e7d to
785904b
Compare
|
@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
jackzhxng
force-pushed
the
jz/regression-test-llama-perf
branch
from
c963335 to
8ca7e05
Compare
|
@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
### Summary Add a proxy for an `export_llama` performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base `ModelArgs` values for `llama_transformer` will be used. ### Test plan N/A
Summary
Add a proxy for an
export_llamaperformance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the baseModelArgsvalues forllama_transformerwill be used.Test plan
N/A