[fuzzer] Relax the FileCheck string for test fuzzer-custommutator.test #66343
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@llvm/pr-subscribers-compiler-rt-sanitizer ChangesThe FileCheck string `LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS: {{[0-9]*}} {{(([a-zA-Z]*-){11,})}} {{.*}}` is too restrictive and may fail the test in some case.If we look at the commit that added this check(66df989), This check is for printing out the long mutation sequence, such as this one But if we look at the code doing the printing void MutationDispatcher::PrintMutationSequence(bool Verbose) {
Printf("MS: %zd ", CurrentMutatorSequence.size());
size_t EntriesToPrint =
Verbose ? CurrentMutatorSequence.size()
: std::min(kMaxMutationsToPrint, CurrentMutatorSequence.size());
for (size_t i = 0; i < EntriesToPrint; i++)
Printf("%s-", CurrentMutatorSequence[i].Name);
if (!CurrentDictionaryEntrySequence.empty()) {
Printf(" DE: ");
EntriesToPrint = Verbose ? CurrentDictionaryEntrySequence.size()
: std::min(kMaxMutationsToPrint,
CurrentDictionaryEntrySequence.size());
for (size_t i = 0; i < EntriesToPrint; i++) {
Printf("\"");
PrintASCII(CurrentDictionaryEntrySequence[i]->GetW(), "\"-");
}
}
}We can see that the This output is totally legit and will fail that check. So I remove the check for the following strings, I think -- 1 Files Affected:
diff --git a/compiler-rt/test/fuzzer/fuzzer-custommutator.test b/compiler-rt/test/fuzzer/fuzzer-custommutator.test
index 7d94ae064bf9698..314530415746c3d 100644
--- a/compiler-rt/test/fuzzer/fuzzer-custommutator.test
+++ b/compiler-rt/test/fuzzer/fuzzer-custommutator.test
@@ -16,7 +16,7 @@ LLVMFuzzerCustomMutatorWithLenControl: BINGO
RUN: %cpp_compiler %S/CustomMutatorWithLongSequencesTest.cpp -o %t-CustomMutatorWithLongSequencesTest
RUN: not %run %t-CustomMutatorWithLongSequencesTest -verbosity=2 2>&1 | FileCheck %s --check-prefix=LLVMFuzzerCustomMutatorLongSequence
LLVMFuzzerCustomMutatorLongSequence: Flag: verbosity 2
-LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS: {{[0-9]*}} {{(([a-zA-Z]*-){11,})}} {{.*}}
+LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS: {{[0-9]*}} {{(([a-zA-Z]*-){11,})}}
LLVMFuzzerCustomMutatorLongSequence: BINGO
# check a target that prints long mutation sequences and verifies the printed
|
|
Ping @vitalybuka @MaskRay . Could you please review this PR, I cannot add reviewers like Phabricator. |
kstoimenov
approved these changes
Sep 20, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The FileCheck string
LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS: {{[0-9]*}} {{(([a-zA-Z]*-){11,})}} {{.*}}is too restrictive and may fail the test in some case.If we look at the commit that added this check(66df989), This check is for printing out the long mutation sequence, such as this one
But if we look at the code doing the printing
We can see that the
DE: XXXis not always printed. So the following output is possible(and is from real-life failure), notince the missing ofDE: XXX.This output is totally legit and will fail that check.
So I remove the check for the following strings, I think
MS: {{[0-9]*}} {{(([a-zA-Z]*-){11,})}}is sufficient for checking the long mutation sequence. This should help resolve the flaky failure of fuzzer-custommutator.test.