Provide clear instructions for implementing PSScriptAnalyzer in CI systems

[HowardWolosky]

Summary of the new feature

In microsoft/PowerShellForGitHub, we wanted to have PSScriptAnalyzer run in our CI system (we happen to be using Azure DevOps, but that's besides the point). This project is great, but its output doesn't lend itself well to CI reporting systems.

I ended up putting together a script that will take the results from an Invoke-ScriptAnalyzer run, and output an NUnit-style XML result file that can then be used to display clear UT-style results in any CI system that supports that file format (which is most of them).

You can see the change as it was implemented here: microsoft/PowerShellForGitHub#211.
It was intentionally written generically to make it easy for anyone to re-use it in their own project.

It creates a test for every Rule, and makes them pass unless there was a failure in that rule, and then it adds a test failure within that rule for each actual test failure. You can see an example build with a failure reported in the Tests tab here (but only if you have permission to the ms org for some reason).

My suggestion is to directly include that file along with some snippets of how to use it in various CI systems (I can easily include a snippet for using it in Azure DevOps, and I'm sure someone familiar with AppVeyor or other systems could adapt it to those systems too.
Proposed technical implementation details (optional)**

Example screenshots from build

What is the latest version of PSScriptAnalyzer at the point of writing
v1.19.0

[bergmeister]

The minimalistic way of using PSSA in CI is to use the -EnableExit switch, which will make it return a non-zero exit code, which CI providers will register as a failure. Details of the failure are then to be obtained from the command line log. Would it help if we described this in the main ReadMe of this repo? Get-Help Invoke-ScriptAnalyzer already mentions this switch and CI explicitly.

In terms of the ask for the fancy UI option: You are right that most CI providers accept NUnit style XML for visualisation of test results. This has long been asked for, we already have #1296 and #672 to track that, in fact the -EnableExit switch was added as a minimum viable feature to use PSSA in CI instead of that due to time resourcing limitations.
I see you already found the Export-NUnitXml.psm1 module from @MathieuBuisson . This solution seems to work well already and I am not sure PSSA has to ship such functionality out of the box because external modules can adapt easier/quicker if for example the module were to break in a new PowerShell version. Therefore my suggestion as the next step would be for @MathieuBuisson to publish it to the PSGallery for easier sharing. Since his repo has an MIT licence, in fact anyone could fork and publish. I am not denying, that it would be nice to have an out of the box solution but in order for it to be more stable/robust and continue to work for all future PS versions, we would probably implement it in C#, but this has not got priority atm. We always welcome PRs but it has to be said though that serious work on PSSA v2 is about to start and this might be the first time that there could be breaking changes to the DiagnosticRecord API or that some of the PSSA concepts get a re-think....
Another alternative that I have seen people do is use Pester tests to loop over an array of all files/rules and then have a test to check the results do not contain that specific rule/file. Since Pester ships with NUnit XML output support, it solves the output file problem.

[SydneyhSmith]

Thanks @bergmeister we can use #1296 to track the request for NUnit style XML.

This issue has been marked as duplicate and has not had any activity for 1 day. It will be closed for housekeeping purposes.