Skip to content

[clang-doc] fix flaky test in clang-doc #101387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Aug 8, 2024

Conversation

PeterChou1
Copy link
Contributor

@PeterChou1 PeterChou1 commented Jul 31, 2024

Fixes #97507

#96809 introduce non-determinstic behaviour in clang-doc which caused the output to be randomly ordered which lead to randomly failing test. This patch modify clang-doc to behave deterministically again by sorting the output by location or USR.

@PeterChou1 PeterChou1 changed the title [clang-doc] uncomment unsupported [clang-doc] fix flaky test in clang-doc Jul 31, 2024
@ilovepi
Copy link
Contributor

ilovepi commented Jul 31, 2024

This isn't fixing anything. Its re-enabling it. I'd also suggest adding people from that bug as reviewers to help you get this re-enabled.

@PeterChou1
Copy link
Contributor Author

This isn't fixing anything. Its re-enabling it. I'd also suggest adding people from that bug as reviewers to help you get this re-enabled.

This was just meant as an experiment to check what was failing in the CI pipeline

@PeterChou1 PeterChou1 force-pushed the clang-doc-fix-basic-project-test branch from 4543ee0 to 114a572 Compare August 1, 2024 07:24
Copy link

github-actions bot commented Aug 1, 2024

✅ With the latest revision this PR passed the C/C++ code formatter.

@PeterChou1 PeterChou1 force-pushed the clang-doc-fix-basic-project-test branch from 399c51e to b88d3ea Compare August 1, 2024 09:51
@PeterChou1 PeterChou1 force-pushed the clang-doc-fix-basic-project-test branch from 1630f5a to 2539d74 Compare August 1, 2024 09:58
@PeterChou1 PeterChou1 requested review from cor3ntin and ilovepi August 1, 2024 10:33
@PeterChou1 PeterChou1 marked this pull request as ready for review August 1, 2024 10:41
@@ -104,6 +104,9 @@ struct Reference {

bool mergeable(const Reference &Other);
void merge(Reference &&I);
bool operator<(const Reference &Other) const {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How stable are usrs? Perhaps we should be sorting on another property?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern here is that we probably want clang-doc to output to be ordered consistently. While sorting by USRs may make it deterministic, it’s also likely that they won’t be ordered similarly run to run. For instance, I’d expect public class fields to be sorted by name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the main reason I used USR was because it was unique to every declaration std sort wouldn't deal with equal elements, but it seems from the doc llvm:;sort already deterministically sort equal elements So I changed it to sort by name

@PeterChou1 PeterChou1 force-pushed the clang-doc-fix-basic-project-test branch from 3b0c23b to 3de5ffe Compare August 1, 2024 19:39
@PeterChou1 PeterChou1 force-pushed the clang-doc-fix-basic-project-test branch from d4e62cb to fc2a139 Compare August 1, 2024 19:46
@PeterChou1 PeterChou1 requested a review from ilovepi August 1, 2024 19:47
@ilovepi
Copy link
Contributor

ilovepi commented Aug 1, 2024

In the description, its probably better to use the #<PR number> notation than linkify PR. Most of us will only see the raw text in the commit message, and while a full link is appreciated, it would be good to know the PR number at a glance.

@PeterChou1 PeterChou1 force-pushed the clang-doc-fix-basic-project-test branch from 1b3ec71 to 49b39e6 Compare August 1, 2024 20:04
Comment on lines 321 to 329
bool operator<(const SymbolInfo &Other) const {
if (DefLoc && Other.DefLoc) {
return *DefLoc < *Other.DefLoc;
}
if (Loc.size() > 0 && Other.Loc.size() > 0) {
return Loc[0] < Other.Loc[0];
}
return extractName() < Other.extractName();
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add some comments about this comparison. It seems you're ordering by DefLoc, then Loc, and lastly by Name, but why that's a good/reasonable ordering isn't obvious.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the logic up a bit, now it works by sorting by declaration location, location then name, I also added some comments explaining my logic

/// Make the output of clang-doc deterministic by sorting the children of
/// namespaces and records.
void sortUsrToInfo(llvm::StringMap<std::unique_ptr<doc::Info>> &USRToInfo) {
for (auto &I : USRToInfo) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does USRToInfo itself need to be in a sorted/stable order?

Copy link
Contributor Author

@PeterChou1 PeterChou1 Aug 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No since USRToInfo is generated on a per file basis so there's no need to sort it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't necessarily rule out its own iteration order affecting the output.

Copy link
Contributor Author

@PeterChou1 PeterChou1 Aug 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How so? StringMap iteration order is not guaranteed to be deterministic and I don't think it was affecting the tests before

if (Loc.size() > 0 && Other.Loc.size() > 0) {
return Loc[0] < Other.Loc[0];
}
return extractName() < Other.extractName();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the overall comparison, you could probably use std::tuple to make the logic simpler. The optional value may make that harder though, as I don't recall offhand how that's normally handled.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too familiar with what you are talking about can you give an example?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PeterChou1 PeterChou1 force-pushed the clang-doc-fix-basic-project-test branch from 61bf9a8 to 3768c86 Compare August 1, 2024 21:05
@PeterChou1 PeterChou1 force-pushed the clang-doc-fix-basic-project-test branch from 3768c86 to 7255f85 Compare August 1, 2024 21:10
Comment on lines +327 to +328
// Sort by declaration location since we want the doc to be
// generated in the order of the source code.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation often sorts methods by name, so its easy to find things. As an example, here's Rust's Vec documentation https://doc.rust-lang.org/std/vec/struct.Vec.html. Methods are sorted in the navigation window and within the body, despite not being that way in source.

Doxygen maintains source order, so its fine to do it that way for now, but we may want to consider making that a configurable option, since I think its more user friendly to do it that way (and arguably more modern).

Copy link
Contributor

@ilovepi ilovepi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but test locally w/ LLVM_ENABLE_EXPENSIVE_CHECKS=On https://llvm.org/docs/CMake.html. If that passes, then go ahead and land this, since I think the nondeterminism in output order will be accounted for.

@PeterChou1 PeterChou1 merged commit 3c9e345 into llvm:main Aug 8, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Test clang-tools-extra/test/clang-doc/basic-project.test failing when built using non-ninja generator
2 participants