Skip to content

Incorporate update_*_test_checks.py into llvm-lit #81320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jroelofs opened this issue Feb 9, 2024 · 1 comment · Fixed by #108425
Closed

Incorporate update_*_test_checks.py into llvm-lit #81320

jroelofs opened this issue Feb 9, 2024 · 1 comment · Fixed by #108425
Labels
enhancement Improving things as opposed to bug fixing, e.g. new or missing feature llvm-lit testing-tools

Comments

@jroelofs
Copy link
Contributor

jroelofs commented Feb 9, 2024

Capturing the idea here from a discussion on Discord:

  • It is annoying to have to specify tool paths to UTC scripts, but this is something llvm-lit has first-class knowledge of, so no more awkward --llc-binary/--clang
  • llvm-lit knows how to do test discovery, parallel job dispatch, and gives a nice progress bar when you use -sv
@jroelofs jroelofs added enhancement Improving things as opposed to bug fixing, e.g. new or missing feature testing-tools labels Feb 9, 2024
@jroelofs
Copy link
Contributor Author

jroelofs commented Feb 9, 2024

cc @arsenm @nhaehnle @nikic

@hnrklssn hnrklssn mentioned this issue Sep 12, 2024
Merged
hnrklssn added a commit that referenced this issue Oct 1, 2024
@hnrklssn
[Utils] Add new --update-tests flag to llvm-lit (#108425)
bb8b9ac 
This adds a flag to lit for detecting and updating failing tests when
possible to do so automatically. The flag uses a plugin architecture
where config files can add additional auto-updaters for the types of
tests in the test suite. When a test fails with `--update-tests` enabled
lit passes the test RUN invocation and output to each registered test
updater until one of them signals that it updated the test (or all test
updaters have been run). As such it is the responsibility of the test
updater to only update tests where it is reasonably certain that it will
actually fix the test, or come close to doing so.

Initially adds support for UpdateVerifyTests and UpdateTestChecks. The
flag is currently only implemented for lit's internal shell, so
`--update-tests` implies `LIT_USE_INTERNAL_SHELL=1`.

Builds on work in #97369
Fixes #81320
Sterling-Augustine pushed a commit to Sterling-Augustine/llvm-project that referenced this issue Oct 3, 2024
@hnrklssn @Sterling-Augustine
[Utils] Add new --update-tests flag to llvm-lit (llvm#108425)
fff8477 
This adds a flag to lit for detecting and updating failing tests when
possible to do so automatically. The flag uses a plugin architecture
where config files can add additional auto-updaters for the types of
tests in the test suite. When a test fails with `--update-tests` enabled
lit passes the test RUN invocation and output to each registered test
updater until one of them signals that it updated the test (or all test
updaters have been run). As such it is the responsibility of the test
updater to only update tests where it is reasonably certain that it will
actually fix the test, or come close to doing so.

Initially adds support for UpdateVerifyTests and UpdateTestChecks. The
flag is currently only implemented for lit's internal shell, so
`--update-tests` implies `LIT_USE_INTERNAL_SHELL=1`.

Builds on work in llvm#97369
Fixes llvm#81320
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Improving things as opposed to bug fixing, e.g. new or missing feature llvm-lit testing-tools
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Utils] Add new --update-tests flag to llvm-lit hnrklssn/llvm-project
1 participant
@jroelofs