Changing the workflow file from the "expected" contents should provide a more helpful error, be documented

[jakebailey]

I changed the default workflow defaults on the TS repo to enable pipefail; that included the scorecard workflow.

However, doing this caused our builds to fail with a 500 error: https://github.com/microsoft/TypeScript/actions/runs/5082801912/jobs/9132975534

tlog entry created with index: 21654547
MEUCIHzLk7hN1sf4Q/GdFCYqGijqiGl0rDkSn5XAFYAZUjvQAiEAuk0nG4gAJ5keCXea2s0O751IVv8AURSCji7Uv5lF670=
2023/05/25 17:23:56 error processing signature: http response 500, status: 500 Internal Server Error, error: {"code":500,"message":"something went wrong and we are looking into it."}

It turns out that this is intentional in some way: microsoft/TypeScript#54382 (comment)

If there's only one supported workflow format, can that be documented in the README (rather than listed as an "example") and provide a better error message when it does happen other than a 500 error?

[kaidokert]

We started getting an obscure failure because we needed to flip from "actions/checkout" to a forked version in another repo, waiting for upstream fix to be accepted.

[jakebailey]

Better would be to lift this restriction, of course!

[spencerschrock]

If there's only one supported workflow format, can that be documented in the README (rather than listed as an "example") and provide a better error message when it does happen other than a 500 error?

Thanks for the issue, this should 100% be in the README. I'll take a look at the HTTP response and see if we can't surface a reason and a better error code (probably 400?)

Better would be to lift this restriction, of course!

The restriction is only in-place if publish_results is true (which feeds into the API dataset and then the badge). It's a necessary part of trusting these crowd-sourced results. We can't be sure what other steps in the job will do:

• Modifying repository contents before analysis (forked checkout)
• Post self-produced scores, instead of being produced by ossf/scorecard-action.

[kaidokert]

Ideally the error at the end of the failed scorecard run would say something like Looks like you have modified your action code. This isn't trustworthy, so we blocked the run. If you want to publish results, keep the actions yaml unmodifed, please see https://foo.bar/explanation.md

[spencerschrock]

We have logic which tries to surface these failures as 400 errors, but due to a bug this wasn't happening (%v instead of a %w).
(link)

After fixing that and adding a link, the response would look something like:

{
"code":400,
"message":"Your request failed validation. See foo.bar/explanation.md for details. workflow verification failed: workflow could not be verified: workflow has no job that calls ossf/scorecard-action"
}