Feature: Separate check for policy/score evaluation

[laurentsimon]

We need to separate check and its policy evaluation. This will allow us to return 'raw' results for users to apply arbitrary policies.
The first step is to do the separation within the checks package. Then all checks are migrated, we will create a pkg.RunRawScorecards() and create our default policy (currently the scores) thru an additional call.
We will later expose this in the CLI

[naveensrinivasan]

We need to separate check and its policy evaluation. This will allow us to return 'raw' results for users to apply arbitrary policies.
The first step is to do the separation within the checks package. Then all checks are migrated, we will create a pkg.RunRawScorecards() and create our default policy (currently the scores) thru an additional call.
We will later expose this in the CLI

Cool! Is this part of v4 release?

[laurentsimon]

We need to separate check and its policy evaluation. This will allow us to return 'raw' results for users to apply arbitrary policies.
The first step is to do the separation within the checks package. Then all checks are migrated, we will create a pkg.RunRawScorecards() and create our default policy (currently the scores) thru an additional call.
We will later expose this in the CLI

Cool! Is this part of v4 release?

no. Just too many checks to migrate :-) v5 or v6.

[azeemshaikh38]

@laurentsimon assigning to you for now.

[laurentsimon]

cc @richsalz this is the large blocker that will help us provide more granularity in the results and give is the ability to provide remediation steps for each warning. (something that came up in #1500)

[richsalz]

Thanks for CC'ing me, @laurentsimon! Have you thought of assigning each check a unique ID, and then allowing command-line flags like --ignore filename:id as in --ignore .github/workflows/ci.yml:unpinned

[laurentsimon]

We thought about something along these lines early on. I think we'll achieve what you describe with policies. Once we have the separation of checks vs policy, we'll be able to apply --ignore .github/workflows/ci.yml:unpinned, and even more expressive policies such as ignore unpinned action org/name or ignore unpinned action only if permissions are not write, or ignore unpinned action in workflow path/file but not unpinned pip install. The file approach works well on simple examples, but is hard to generalize to other uses case like, say warn me if a reviewers is not in list X,Y,Z.

We feel the policies will allow flexibility for advanced users, without bloating the CLI arguments. We will have built-in policies, of course.

Wdut?

[richsalz]

Your ideas sound great. (I'm not surprised.)

[laurentsimon]

glad that it makes sense to you too. Everyone is asking for different exceptions and special cases, policies are the only solution that seemed viable in the end. Let's hope this works