[ashley-cui]

None

[baude]

LGTM, couple of nitty questions around clarification; otherwise, this is excellent that we are doing this. tyvm!

[Luap99]

Thanks, design wise I just have the one point about the signing timing race condition. The rest are mostly questions, I do not know enough about the tooling to know if this will actually work e2e but it sounds reasonable.

[Luap99]

I wonder have you looked at actions/attest, https://github.com/actions/attest#container-image
I am not sure how it differs, I think it also uses sigstore but maybe a bit more github native to use?

[Luap99]

One large concern I have with this workflow is the push THEN sign. That means there is a small window where an unsigned image exists on registry. In cases of a podman patch release we just push the minor version tag which means an user on 6.1.4 might then pull the just published 6.1.5 image while the signature was not uploaded yet and thus the download fails "randomly". This may be a a big deal if we fail for whatever reason to sign that image, i.e. workflow failed.

I am not sure how this works on the registry level but I think what we need is more like push unamed image, sign that, only then tag that image on the registry.

cc @mtrmac

[ashley-cui]

I wonder have you looked at actions/attest, https://github.com/actions/attest#container-image I am not sure how it differs, I think it also uses sigstore but maybe a bit more github native to use?

I think attest would require c/image changes to verify the attestation, while cosign with fulcio works with just a policy.json change.

One large concern I have with this workflow is the push THEN sign. That means there is a small window where an unsigned image exists on registry. In cases of a podman patch release we just push the minor version tag which means an user on 6.1.4 might then pull the just published 6.1.5 image while the signature was not uploaded yet and thus the download fails "randomly". This may be a a big deal if we fail for whatever reason to sign that image, i.e. workflow failed.

I am not sure how this works on the registry level but I think what we need is more like push unamed image, sign that, only then tag that image on the registry.

Yeah, I think the way to do it is to push the image as quay.io/podman/machine-os:X.Y-staging, sign the image, then use skopeo copy to "promote" it to quay.io/podman/machine-os:X.Y? We could then delete the staging image, or just leave it.

[ashley-cui]

If that sounds okay to you and @mtrmac gives it an ack, I'll go ahead and update this design doc and re-push

[Luap99]

That sounds good

[mtrmac]

The in-toto attestations are not supported by c/image.

[mtrmac]

• Does this need to use cosign? I would hope that podman push --sign-by-sigstore= with oidcMode: "staticToken" might work. And podman push can do both in one step — admittedly there’s a currently small race where the tag is moved but the signatures are not uploaded yet. And I haven’t manually done that with Fulcio in … ages.
• I think cosign would nowadays need --new-bundle-format=false and perhaps more options. But there is --sign-container-identity so it is possible to upload to a staging environment, sign there, and then move the final image to the final location. (--sign-container-identity is typically required anyway for interoperability with c/image when signing tags.)

[ashley-cui]

• Does this need to use cosign? I would hope that podman push --sign-by-sigstore= with oidcMode: "staticToken" might work. And podman push can do both in one step — admittedly there’s a currently small race where the tag is moved but the signatures are not uploaded yet. And I haven’t manually done that with Fulcio in … ages.

I think the podman push --sign-by-sigstore= probably works, though the oidc flow with cosign was much better documented, so that's what I went with at first. I don't have too strong of an opinion on this, I think I'd defer to you for you prefer one way or the other.

• I think cosign would nowadays need --new-bundle-format=false and perhaps more options. But there is --sign-container-identity so it is possible to upload to a staging environment, sign there, and then move the final image to the final location. (--sign-container-identity is typically required anyway for interoperability with c/image when signing tags.)

ack, will do

[Luap99]

I like @mtrmac to have a look at this flow.

[mtrmac]

What do we do when we want to update the policy?

• Can we do that at all, i.e. do users run the installer on every Podman update?
• Presumably we want the policy to be user-editable; if so, we would need to be able to somehow determine whether the user edited it and we should leave it alone, or whether we can update it.
  • We might manage that state manually, with checksums or something, I suppose…
  • Alternatively, we might ship a global-default policy in one location, and let admins override in other locations. E.g. on Linux, we would ship a policy in /usr and let users override in /etc. But containers-config(5) says “There is no location for system defaults.” on Windows; and on macOS I worry we might find it difficult to modify the documented /usr/share location (but I didn’t try).

[ashley-cui]

Yeah, we assume users upgrade by using installers or pkg managers. I'd be okay with the default policy and then using location overrides.

[Luap99]

question: refs/tags/ thing basically means we only trust images build as part of a tag right?
I think that seems logical but we do currently publish the next image i.e. 6.2 today build of main which means our CI and local devs could not verify the newer image? So I guess we would want to allow the main image and the tags.

[ashley-cui]

Yeah, I'll change this bit to sign on main too.

[mtrmac]

This sort of hints at a core of the problems with workflow signing — what does the workflow data mean / guarantee.

The signature, basically, says, “I, the holder of $key, vouch that the image with $digest is what I intend to publish for registry.example/product-name/image-name:version”. And consumers have policy that say “when consuming registry.example/product-name/image-name:$any-version, I want the holder of $key to vouch for the image, with the specific :version that I refer to”. [But cosign generally ignores the :version part AFAIK during verification.]

One of the ways this fails is when a malicious registry substitutes a legitimately signed “image-name:v0.0.1-beta-known-vulnerable” for the user-requested “v10.1-latest”. So the version matching is important.

With Fulcio, “holder of $key” is replaced by “holder of an OIDC token with specific properties”, fine, but then which properties indicate that some code / commit is trusted to vouch for code in this repo / for a particular version?

It’s easy enough to define a policy like “this image was built from a GitHub action running from the main branch”, but that gives up on the version matching.

And for version matching: e.g. if we allow any arbitrary commit on main to sign a release, and the code existing as of that commit chooses the version number, can someone add a commit on main to sign with a v6.1.10 identity even when there is no such tag, or the tag points at a completely different code?

The policy can currently only enforce a static match (not even substrings, no wildcards). Even supposing we relax this somehow… “a commit exists in the repo’s history that contains code that can create a signature versioned v6.1.10” is rather different from “this workflow ran from exactly tag v6.1.10 and used the code as of that tag to create a signature naversioneded v6.1.10” (…and git tags can be moved I think, so even then it’s not too clear what it means).

Now maybe all of this has good answers and I am missing a trivial solution, I have never had too much time to think about this.

---

So I guess we would want to allow the main image and the tags.

policy.json(intentionally, although I suppose this is negotiable, especially as a single-purpose field as opposed to a generic mechanism) has no mechanism for “for $repo, allow either A or B”.

The basic recommendation is that if the user needs to express 2 different policies, they should exist for separate repos, i.e. have a policy for “builds from main” and have a policy for “tagged builds” — but that wouldn’t work if we use a build from main to create a release that would later be treated as a tagged build.

[mtrmac]

It’s easy enough to define a policy like “this image was built from a GitHub action running from the main branch”, but that gives up on the version matching.

It’s totally possible I’m overthinking this, and “we have a $workflow that is expected to checkout a provided tag, build an image based on that, and sign it with the same tag” could be a workflow on main and we would trust it to work essentially the same across all versions, and the policy would say “signed by $workflow” (and, ~hard-coded in c/image, the signed tag matches what the user asked for). E.g. something like that might be valuable — or required! — so that, running that $workflow on main, we create signatures for all relevant past versions.

OTOH we should have some plan for a successful compromise.

• With static key pairs, if the key is completely compromised, or if it just once signs something unwanted, we can give up on that key pair, generate a new one, re-sign all past versions with the new key, and publish an updated policy that trusts the new key and not the old one.
• With the public-instance Fulcio, we are not in control of the root key, so the plan above doesn’t work; and there isn’t any other revocation mechanism of individual signatures either.
  • At some point sigstore upstream suggested that a policy could contain a list of known-malicious signatures that should be rejected. That assumes we can even enumerate them, somehow. (We could look into the Rekor records to see what all was signed, but then we would need to enumerate the valid signatures… which is not obviously easier.)
  • I suppose we could give up on the $workflow name, re-sign all past versions with $workflow2 , and publish an updated policy that requires signatures from $workflow2 ?

I really haven’t thought this all through.

[mtrmac]

So I guess we would want to allow the main image and the tags.

We probably want the ability to create signatures for past releases/tags. Both as an initial bootstrapping step, and perhaps also for the compromise recovery scenarios.

[ashley-cui]

Okay, a lot that I haven't thought about, which is great that you bring these concerns up now!

Let me first clarify how we're building, tagging, and pushing images. I think the versioning is a little different than what you're thinking of. podman-machine-os images are only tagged as an x.y image. Every time we cut a Z release, we're just moving the x.y tag to the newest built version. Tags are always tagged on branches, so the build pipeline only runs on commits on main, or tags on branches.

I think the way we have things set up makes life a little easier - if something goes wrong, we just re-build, move the tag, and remove the old image. I think fundamentally though, your argument is that a signature should be tied to a X.Y.Z tag, which might means that the scope of this has increased. I think there are some gotchas in podman machine that make this tricky, but I don't remember off the top of my head

With that being said, under my understanding, I think your concerns boil down to:

1. If we decide to use workflow signing (ie, OIDC with Sigstore/Fulcio) the only thing the signature verifies is that the image is built by our GHA CI.
2. This means that we're not really vouching that a certain version is signed by us, just that the image the user is pulling, regardless of the version, was created by our workflow
3. How do we make sure that signatures on main are treated differently than tagged? Are they built differently?
4. How do we respond if there is a signed compromised image?
5. How do we sign for previous releases/tags?

1&2: I think the way we currently handle podman-machine-os tags means that 1&2 hold true. We don't really ship separate versions to begin with.
3: I think we can ship only https://github.com/podman-container-tools/podman-machine-os/.github/workflows/release.yml@refs/tags/, and not main, so images built off main, which are supposed to be for testing purposes only anway, aren't accepted by release versions of podman.
4: I think workflow-2 workflow 3 makes sense? To be honest, I haven't thought too much about this, perhaps naively. Maybe we could rename the workflow every Y release, and have the policy shipped with the release be like, workflow-x.y.z? Maybe this is also naive, but could we just, move the tag like we usually do to update the image, and pull the compromised image? Or are we talking something like a MITM attack, where there is a compromised server that is MITM-ing us with a signed image?
5. Do we need to sign previous images? We pretty much expect the X.Y version of podman to match the X.Y version of the image, maybe I'm missing something, but I feel like there wouldn't be a scenario where we ship a policy.json with a new version of podman that has an older version?

[Luap99]

5. Do we need to sign previous images? We pretty much expect the X.Y version of podman to match the X.Y version of the image, maybe I'm missing something, but I feel like there wouldn't be a scenario where we ship a policy.json with a new version of podman that has an older version?

Right I would say very strongly we should not sign any older images, the point of siging in the workflow is to prove the image come from the workflow. This is no longer possible for already published images.

4: I think workflow-2 workflow 3 makes sense? To be honest, I haven't thought too much about this, perhaps naively. Maybe we could rename the workflow every Y release, and have the policy shipped with the release be like, workflow-x.y.z? Maybe this is also naive, but could we just, move the tag like we usually do to update the image, and pull the compromised image? Or are we talking something like a MITM attack, where there is a compromised server that is MITM-ing us with a signed image?

I am not sure this would scale to well, it is kinda awkward to rename the workflows all the time.
In terms of revoking keys, either the user already has the compromised image downloaded in which case policy.json will never validate again or they do not have it downloaded in which case if we know something is compromised we must delete the images from our quay.io right away and then a user will fail to download them so while a proper revoke would be nice I am not sure it is strictly required.

The way I mostly look at this, today any image we push to quay.io is trusted. That means if I or someone else upload to our namespace and replaced the CI build image with their on malicious one nobody would notice. The only trusted source must be the CI workflow.

So if we now sign based on the CI workflow it means just access to quay.io is no longer enough. And the other way around even if the Rekor key is compromised the attacker would still need to put the image on quay.io or pull a MITM attack.
So the main attack entry is reduced to the Github CI and our repo for supply chain attacks which is an improvement.

If we think using our own keys is better we can do that as well but ultimately it means they still needs to be stored and used in the github workflow. To me the nice thing about the keyless signing is I do no have to trust anyone on the team to have generated the key securely and not leak it/keep it after they put it in the github secrets.

[ashley-cui]

Yeah that makes sense to me, I think I'm comfortable current design, since we can just pull images from quay if they're bad, and workflow signing increases the security to where the malicious actor has to gain access to both our workflow and our quay, which reduces the attack surface by a lot. Certainly checking versioned signatures is more secure, but I feel like this design doc allows us to do enough with a lot less effort, and with systems that already exist.

[mtrmac]

Let me first clarify how we're building, tagging, and pushing images. I think the versioning is a little different than what you're thinking of. podman-machine-os images are only tagged as an x.y image. Every time we cut a Z release, we're just moving the x.y tag to the newest built version.

OK; I don’t think that’s ideal (e.g. users might really want “the Podman-machine image after the eMudhra trust root removal”) but I think it’s fine; dealing with that is way out of scope of this design doc. We are designing a signing mechanism here.

I think the way we have things set up makes life a little easier - if something goes wrong, we just re-build, move the tag, and remove the old image. I think fundamentally though, your argument is that a signature should be tied to a X.Y.Z tag, which might means that the scope of this has increased. I think there are some gotchas in podman machine that make this tricky, but I don't remember off the top of my head

I’m not at all pushing for moving to X.Y.Z here. (I also think that once we have a signature mechanism for X.Y defined, moving to X.Y.Z should be possible without any design changes to the signing process.)

---

With that being said, under my understanding, I think your concerns boil down to:

The way I think about the starting point is “a signature is a claim that $something is true; A. what exactly is the property we are claiming, and then B. is our implementation achieving that at a high enough level of confidence?”.

And for A., I think the intuitive goal to aim for is “the user gets exactly the image they asked for [or things fail and the user gets nothing]”. I.e. users who ask for podman-machine-os:6.2 get podman-machine-os:6.2.

• “user” can be a human explicitly specifying a custom repo:tag/repo@digest, or a Podman-machine default asking for podman-machine-os:v6.2
• This does imply that users who want v6.2 do not get v5.8; a v5.8 signature should not validate for v6.2; but if users don’t differentiate between the October version of v6.2 and the November version of v6.2, it’s unnecessary for signatures to differentiate between them.

---

1. If we decide to use workflow signing (ie, OIDC with Sigstore/Fulcio) the only thing the signature verifies is that the image is built by our GHA CI.

Not entirely; the OIDC token does say what workflow file it came from, in a way that can be enforced by policy.json. (It also precisely identifies the workflow version, down to a commit, in a way that is not very useful for policy.json, because at the time of shipping Podman 6.2.0 we don’t know the exact commit that will going to be used to sign am updated 6.2 machine image 3 months later.)

And then it is fine to say “we trust that the workflow file only signs images that were built by our CI, and we trust that the workflow file will create a signature for v6.2 that really corresponds some version that humans decided to be a v6.2 image (and designated using some mechanism)” — because we designed the workflow file that way, and because we keep reviewing the workflow file that way.

(We should have a discussion about the details here, maybe imposing a stricter review requirements and louder change notifications on the workflow and its dependencies, up to and possibly including moving the workflow and the related secrets, if any, to a separate, much more locked down, repo.)

2. This means that we're not really vouching that a certain version is signed by us, just that the image the user is pulling, regardless of the version, was created by our workflow

I’m not strongly concerned about the difference between “a human signed v6.2” and “a workflow signed v6.2”, other than “how does the workflow know that an image based on $commit should be signed v6.2”.

3. How do we make sure that signatures on main are treated differently than tagged? Are they built differently?

I think this is an implementation detail… and with the previous suggestion that re-signing past images using a future changed key might be desirable, I think think that we might not want to have a strict enforcement of branching.

---

4. How do we respond if there is a signed compromised image?

In the past, we did have a suspicion of a security incident, that caused us to audit all previously-published container images; so yes, I do think we need a plan. It doesn’t need to be very detailed and it definitely doesn’t need to be productized to the point of automation.

4: I think workflow-2 workflow 3 makes sense? To be honest, I haven't thought too much about this, perhaps naively. Maybe we could rename the workflow every Y release, and have the policy shipped with the release be like, workflow-x.y.z?

I think this would work if workflow-0.1 were buggy in a way that allowed an attacker to sign something improper, and we moved to a fixed workflow-1.0.

But it wouldn’t work if the whole repo hosting the workflow were compromised (or if an inattentive reviewer let in a malicious PR), because an attacker with the ability to change the repo contents could publish a file named workflow-x.y.z for arbitrary x.y.z at any time. (I suppose we’d have to enforce that workflow-x.y.z comes from tag x.y.z, and have immutable tags? It gets complicated.)

So I don’t think regularly changing workflow names buys us enough to be worth the hassle. OTOH it’s also not that necessary — in the very worst case we can always abandon the whole repo and start using a workflow identity from a different one, as long as we have control over the policy.json published in Podman.

---

Maybe this is also naive, but could we just, move the tag like we usually do to update the image, and pull the compromised image? Or are we talking something like a MITM attack, where there is a compromised server that is MITM-ing us with a signed image?

I think once we publish a signed malicious image on the internet, a malicious registry can keep serving that image; we have (currently) no way to revoke a single signature (and if a key or a Fulcio server were compromised, we might not even notice that such a malicious signature exists)

And generally I don’t think we can trust registries that way: If our security depended on the registry not being malicious, and us being able to control what images are published under which tags … that would be a plausible design of a supply chain, but under that assumption we wouldn’t need signatures at all, we could just push to the registry to publish what we want to publish. IOW: I think the goal of adding signing implies that we don’t want to trust the registry that way.

---

5. Do we need to sign previous images? We pretty much expect the X.Y version of podman to match the X.Y version of the image, maybe I'm missing something, but I feel like there wouldn't be a scenario where we ship a policy.json with a new version of podman that has an older version?

I think that’s mostly true, apart, perhaps, from the scenario of recovering from a compromise. If we publish Podman 6.2.0, a podman-machine-os:6.2 image, and later our signing mechanism were compromised and we knew that the podman-machine-os:6.2 image were not attacked, we might want to re-do the signing setup, publish Podman 6.2.1 with an updated policy.json that accepts the new one, and re-sign the existing podman-machine-os:6.2 image with the new signing setup without changing it, to minimize risk. But it would also work well enough to re-generate an entirely new podman-machine-os:6.2 image using the standard workflow, and it would be probably easier for everyone.

[Luap99]

by installers you mean the macos/windows ones? I suppose we would want to patch container-common rpm as well to update our default policy.json on fedora/RHEL as well with that?

[TomSweeneyRedHat]

It would be good to add some examples of what the change would look like in policy.json

[mtrmac]

This plan requires c/image to be able to consume the certificates the public-instance Fulcio issues for GitHub actions, and I don’t think that works today.

At this point I don’t have much of an opinion on whether we should try to make it work, or whether we should give up Fulcio and manage keys manually.

[mtrmac]

Hum, this implies that we trust the public sigstore instance to sign our images. That’s not too bad but it is a choice, as opposed to having our own key pair (and having to somehow manage it and carefully (manually?) use it).

(One risk is that they have TUF set up to allow updating the root keys over time without client-side changes; policy.json requires providing the actual keys.)

---

Downstream vendors would probably want to use product-specific keys they own… but that doesn’t directly drive what we can / should do upstream.

[ashley-cui]

Honestly, I only chose to use Fulcio because in the sigstore docs, and looking around other projects, it seemed like Fulcio is built to be used in CI versus managing our own key pair. But you do seem to bring up a lot of concerns about this arch, so I could be convinced either way.

[ashley-cui]

There's also a world where we can manage our own Fulcio instance? But I'm not sure if that creates more or less problems..

[mtrmac]

Managing our own Fulcio instance means, first, hosting it somewhere where the CI jobs can contact it at arbitrary times = an ongoing cost and maintenance.

Operationally, I think keeping Fulcio running is strictly harder than managing a plain key pair (because managing Fulcio includes managing the root certificate of the Fulcio CA). But if it means avoiding manual signing operations and manually handling the secret signing keys, it might still be worth it — but then again, even with a plain key pair we might be be looking for a way to have the signing automated without the key material being handled by a human.

[mtrmac]

Does this work? I never tried using the GitHub actions tokens, but I’d expect (for the tokens to be meaningful) to not let callers arbitrarily set values in the certificate. So I’d be surprised if it were possible to set an e-mail field to an URL (at all), and also to an action-chosen URL (possibly indicating a different action name…)

I understand that this works by Fulcio having a mapping from GitHub-issued tokens, per https://github.com/sigstore/fulcio/blob/main/docs/oid-info.md ; there should not be a way for the client to interfere. (Also, looking at code, I don’t think GitHub workflow tokens cause the email field to be populated in generated certificates; and under the assumption that we use the public sigstore instance, we are not in control of the Fulcio configuration.)

And that runs into c/image not having any policy.json options for matching the workflow-identity fields (there are a few long-standing discussions in the repo).

Now, there is demand for supporting this, so if we can figure out how, that would be generally valuable, but I think it rather expands the scope.

[ashley-cui]

Whoops yeah, I think i thought c/image already supported this, but I was mistaken. There's an open PR and a lot of discussion though.

Under my understanding and looking at examples, I think this works? But could you clarify what you mean by "callers arbitrarily set values in the certificate"?

[mtrmac]

In the Fulcio-created certificate, all the security-relevant fields are set by Fulcio based on the OIDC token (where the issuer of the token is trusted to only issue “true” claims in their tokens).

Otherwise, if Fulcio created/signed certificates where security-relevant values were set by the otherwise-unauthenticated caller, the certificate would be ~meaningless: anyone could get a certificate issued for anything.

So, unless the OIDC token contains a claim for “subject email https://…”, Fulcio won’t issue a certificate saying “subject email https://…”.

[mtrmac]

… and, to finish the thought, GitHub won’t issue an OIDC token for “subject email https://…”, because, similarly, if it issued tokens with caller-chosen otherwise-unauthenticated values, the OIDC tokens would be meaningless for security.

[mtrmac]

This sort of hints at a core of the problems with workflow signing — what does the workflow data mean / guarantee.

The signature, basically, says, “I, the holder of $key, vouch that the image with $digest is what I intend to publish for registry.example/product-name/image-name:version”. And consumers have policy that say “when consuming registry.example/product-name/image-name:$any-version, I want the holder of $key to vouch for the image, with the specific :version that I refer to”. [But cosign generally ignores the :version part AFAIK during verification.]

One of the ways this fails is when a malicious registry substitutes a legitimately signed “image-name:v0.0.1-beta-known-vulnerable” for the user-requested “v10.1-latest”. So the version matching is important.

With Fulcio, “holder of $key” is replaced by “holder of an OIDC token with specific properties”, fine, but then which properties indicate that some code / commit is trusted to vouch for code in this repo / for a particular version?

It’s easy enough to define a policy like “this image was built from a GitHub action running from the main branch”, but that gives up on the version matching.

And for version matching: e.g. if we allow any arbitrary commit on main to sign a release, and the code existing as of that commit chooses the version number, can someone add a commit on main to sign with a v6.1.10 identity even when there is no such tag, or the tag points at a completely different code?

The policy can currently only enforce a static match (not even substrings, no wildcards). Even supposing we relax this somehow… “a commit exists in the repo’s history that contains code that can create a signature versioned v6.1.10” is rather different from “this workflow ran from exactly tag v6.1.10 and used the code as of that tag to create a signature naversioneded v6.1.10” (…and git tags can be moved I think, so even then it’s not too clear what it means).

Now maybe all of this has good answers and I am missing a trivial solution, I have never had too much time to think about this.

---

So I guess we would want to allow the main image and the tags.

policy.json(intentionally, although I suppose this is negotiable, especially as a single-purpose field as opposed to a generic mechanism) has no mechanism for “for $repo, allow either A or B”.

The basic recommendation is that if the user needs to express 2 different policies, they should exist for separate repos, i.e. have a policy for “builds from main” and have a policy for “tagged builds” — but that wouldn’t work if we use a build from main to create a release that would later be treated as a tagged build.

[mtrmac]

What do we do when we want to update the policy?

• Can we do that at all, i.e. do users run the installer on every Podman update?
• Presumably we want the policy to be user-editable; if so, we would need to be able to somehow determine whether the user edited it and we should leave it alone, or whether we can update it.
  • We might manage that state manually, with checksums or something, I suppose…
  • Alternatively, we might ship a global-default policy in one location, and let admins override in other locations. E.g. on Linux, we would ship a policy in /usr and let users override in /etc. But containers-config(5) says “There is no location for system defaults.” on Windows; and on macOS I worry we might find it difficult to modify the documented /usr/share location (but I didn’t try).

[ashley-cui]

@mtrmac Thanks for the review! Hopefully what I'm saying makes sense, but, let me know if I've misunderstood or I'm missing something - I'm trying to understand the complexity here, which I may have initially underestimated.

[mtrmac]

To re-word this into a hopefully more specific proposal / chain of thought:

The way I think about the starting point is “a signature is a claim that $something is true; A. what exactly is the property we are claiming, and then B. is our implementation achieving that at a high enough level of confidence?”.

And for A., I think the intuitive goal to aim for is “the user gets exactly the image they asked for [or things fail and the user gets nothing]”. I.e. users who ask for podman-machine-os:6.2 get podman-machine-os:6.2.

• So, we have software creating a signature as :6.2. How is the software confident that it is signing a legitimate object as :6.2?
  • “The podman-machine-os repo contains a tag for 6.2.X, that tag refers to a merged commit on the 6.2 branch (or on the main branch), and the repo has branch protection set up so that merges require N approvals, and adding tags is also locked down $that way”. Is that the rule? (I’m very unsure.)
  • Is it sufficient? Up to us.
    • I think it well might be; we should protect against individual mistakes but we shouldn’t over-complicate things.
    • I don’t think GitHub allows requiring multiple approvals to create a tag. We probably should have immutable tags, at least. Maybe we require the tagged commit to have specific contents signaling the intent for it to be tagged as 6.2.X?

---

For B.

And then it is fine to say “we trust that the workflow file only signs images that were built by our CI, and we trust that the workflow file will create a signature for v6.2 that really corresponds some version that humans decided to be a v6.2 image (and designated using some mechanism)” — because we designed the workflow file that way, and because we keep reviewing the workflow file that way.

(We should have a discussion about the details here, maybe imposing a stricter review requirements and louder change notifications on the workflow and its dependencies, up to and possibly including moving the workflow and the related secrets, if any, to a separate, much more locked down, repo.)

My hypothetical starting point here is “we have a separate repo that is very locked down; and within that repo a workflow that does the signing. That repo implements our logic for what is allowed to be signed as :6.2. The OIDC token issued for that workflow, and the Fulcio certificate, identify the workflow enough that we can define a policy.json rule requiring that the signature came form this workflow”.

• The repo/workflow must not be totally immutable: environments and CLIs change, and we need to be able to change it without the policy.json reference to it breaking.
• How does the workflow get triggered and how does it know what signature it was asked to create? That would probably be easier within the same repo, we could trigger on tag creation.
• It doesn’t have to be a separate repo. Maybe just a separate more locked-down branch in the podman-machine-os repo. (It probably should not follow the default branch layout of the repo, because “how things are signed” is independent of Podman versioning.)
• Maybe we don’t even need it “more locked-down” than the rest of the repo. The important thing is that we are confident enough it will not be maliciously modified, that changes to it will get noticed, and that we can identify it in the OIDC token / Fulcio certificate.

---

Another thing: The signing operations should produce an audit log — some kind of long-term (not disappearing after N months); and, if at all possible, immutable, record — of everything that was ever signed. (Maybe an add-only S3 bucket??? No idea.)

(Already Fulcio is reportedly creating a transparency log for issued certificates, and all signatures are uploaded to Rekor. But that would also include the hypothetical malicious signatures, so we should have a separate log of what we intended to sign, to be able to detect any completely unexpected signatures.)

[mtrmac]

• So, we have software creating a signature as :6.2. How is the software confident that it is signing a legitimate object as :6.2?

  • “The podman-machine-os repo contains a tag for 6.2.X, that tag refers to a merged commit on the 6.2 branch (or on the main branch), and the repo has branch protection set up so that merges require N approvals, and adding tags is also locked down $that way”. Is that the rule? (I’m very unsure.)

This is fine for identifying a commit, but not for identifying the binary artifact. IOW “build an OS image; then trigger a super-secure signing workflow” is at most as as strong as the “build OS image step” — actually weaker because the signing workflow has no way to pair the commit with the OS image binary.

So that implies for B., the build itself needs to be driven by the secure workflow. Either we have a trusted “build+sign” workflow (in the previously-suggested separate repo/branch) that is given a target tag/commit, validates it, triggers an “ordinary” build for that commit (however “a build” is defined in the podman-machine-os repo as of the target commit), and signs the outcome; or we give up on the separation, and just have a “build an OS image + sign” workflow in the primary podman-machine-os repo, and all of them are allowed to create signatures.

I think separating the “determine commit; trigger build; sign the outcome” part from the “build an image from commit” part could be valuable, because we could lock down the top-level+signing part without slowing down development of the the whole repo, but I’m not at all sure it is practical.

[Luap99]

“The podman-machine-os repo contains a tag for 6.2.X, that tag refers to a merged commit on the 6.2 branch (or on the main branch), and the repo has branch protection set up so that merges require N approvals, and adding tags is also locked down $that way”. Is that the rule? (I’m very unsure.)

It not fully configured that way today but I think we really need to configure "environments" and there we can require approvals.

https://docs.github.com/en/actions/reference/workflows-and-actions/deployments-and-environments#deployment-branches-and-tags

I don’t think GitHub allows requiring multiple approvals to create a tag. We probably should have immutable tags, at least. Maybe we require the tagged commit to have specific contents signaling the intent for it to be tagged as 6.2.X?

We have immutable releases already setup there. As for preventing tag pushes nor from a branch I am not sure if github allows that directly but with the environments mentioned above we can prevent the CI from triggering on it I think.

Overall I am not sure how well this all works together but I think the ability to lock the workflows down tightly exists. Ultimately if you are an org/repo admin you can disable all protection settings so this can never prevent full account takeover compromises but if somebody just has my ssh key to push to the repo they could not bypass that so it helps a bit.

I don't know if this exists but if we want to protect against the single maintainer compromise I suppose we would need multiple signatures from multiple people? Like maintainer A and B must have signed this and they store those keys locally but then we loose the automated aspect again so that is not the direction we like to go here I think.

[mtrmac]

I don't know if this exists but if we want to protect against the single maintainer compromise I suppose we would need multiple signatures from multiple people? Like maintainer A and B must have signed this and they store those keys locally but then we loose the automated aspect again so that is not the direction we like to go here I think.

If we trust GitHub to issue correct OIDC tokens for the signing workflow, we trust GitHub period — so I think we could check whether the commit comes from a PR with 2 GitHub PR approvals, and that would be ~equivalent. Or “only” document that the repo should have 2 approvals, and assume that in the workflow, without checking the specific commit.

Same for “does the commit come from the intended branch”, we can check that ourselves in code, or we can trust GitHub configuration.

I’d slightly prefer the script enforcing policy, because it is visible and it isn’t so easy to disable/override (like we do for merging private branches) without forgetting to re-enable, but that’s not a hard opinion.

[Luap99]

I’d slightly prefer the script enforcing policy, because it is visible and it isn’t so easy to disable/override (like we do for merging private branches) without forgetting to re-enable, but that’s not a hard opinion.

Well I guess we need both then, a script does not help if somebody can push a commit/tag they can remove the check in the script as part of a commit they tagged. Now I agree that such malicious action would be more visible as part of the git history not some private setting that can be turned-off at any time.

[mtrmac]

That’s why I think the signing workflow would be better separate from, and more protected than, the full podman-machine-os repo (and why the very start of my comments was about “did we use version 6.1.10 to sign 6.1.10”, although I no longer think that kind of matching is preferable).