Skip to content

Retry tests if msbuild crashes#6181

Merged
radical merged 3 commits into
microsoft:mainfrom
radical:retry-msbuild-failure
Oct 21, 2024
Merged

Retry tests if msbuild crashes#6181
radical merged 3 commits into
microsoft:mainfrom
radical:retry-msbuild-failure

Conversation

@radical

@radical radical commented Oct 8, 2024

Copy link
Copy Markdown
Member

And get the msbuild debug files from crashes to aid debugging.

Issue: #6163

Microsoft Reviewers: Open in CodeFlow

@radical radical added area-engineering-systems infrastructure helix infra engineering repo stuff testing ☑️ labels Oct 8, 2024
@joperezr

joperezr commented Oct 8, 2024

Copy link
Copy Markdown
Member

Do we understand why the crash is happening though? How confident are we that when in a failure state, retrying will make this pass?

@radical

radical commented Oct 8, 2024

Copy link
Copy Markdown
Member Author

It doesn't crash like this on every build, but it does crash sometimes. So, this is just to ease the pain of having to do retries or having red builds on internal pipeline. Meanwhile, we are tracking the occurrence of these with #6163 .

@radical

radical commented Oct 11, 2024

Copy link
Copy Markdown
Member Author

@joperezr I think this would be useful to take.

@joperezr joperezr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @radical. If there is a way of not only retrying, but also keeping trace of when that needed to happen it would be great, as that way we won't get false-negatives indicating that this is no longer a problem, but that shouldn't block this.

@radical

radical commented Oct 11, 2024

Copy link
Copy Markdown
Member Author

LGTM, thanks @radical. If there is a way of not only retrying, but also keeping trace of when that needed to happen it would be great, as that way we won't get false-negatives indicating that this is no longer a problem, but that shouldn't block this.

I'm setting MSBUILDDEBUGPATH here, so we should get some debug logs from msbuild which we can use to further debug this.

@radical

radical commented Oct 15, 2024

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@radical

radical commented Oct 21, 2024

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@radical radical merged commit cbc8773 into microsoft:main Oct 21, 2024
@radical radical deleted the retry-msbuild-failure branch October 21, 2024 17:15
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-engineering-systems infrastructure helix infra engineering repo stuff testing ☑️

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants