Need clarification: KEP stages and transitions

[thockin]

From slack (summarized):

Is the KEP freeze "you must have all KEP changes merged" or "You must have all intended-to-change KEPs in the sheet"?

For example: I have some KEPs which we WANT to move from Alpha->Beta or Beta->GA, but the code to do those things is not done yet and MIGHT slip. I don't want to change KEP stage text until we know whether it is in or not, and that won't be until well after the enhancement freeze.

That's not an exception - that's the norm, I think.

I think the only sane interpretation is that the KEP deadline indicates which KEPs are being tracked, but every KEP should be considered "at risk" until stage changes, and when stage changes, that should be authoritative. What I DON'T want is to have to remember to revert changes to a KEP if the code misses the release. That will fail and leave ambiguous results.

Do we need a 2-phase commit - a distinction between target and actual?

We need clarity on the enhancement freeze - what EXACTLY is the deadline for? What sorts of KEP changes are allowed after the enhancement freeze?

Rey said "Enhancements Freeze indicates which KEPs are being tracked ... What the release team looks for in Enhancements Freeze is for the KEP to be marked as "implementable", has the correct stage for the release, ... and an approved PRR"

This says to me that if I want to move to GA in 1.23, I update the KEP to say GA for stage. If the code doesn't make it, we have a KEP that is ambiguous (or flat out lying). Expecting people to come back and revert the stage is unrealistic.

IMO - stage should be a trailing indicator. Updating stage should be on the success path, not on the failure path.

Some SIGs/reviewers are resetting the status to implemented/implementable for each stage. I don't think that was the intention.

Between stage, milestone, and status there's overlap and ambiguity. For example, let's look at a new KEP.

It starts as:

stage: alpha
status: provisional
alpha: 
beta: 
ga: 

We iterate on that and it ultimately gets approved:

stage: alpha
status: implementable
alpha: v1.X
beta: v1.Y
ga: v1.Z

Development work happens and we release v1.X. Given the above fields, I can assume that this KEP is alpha in v1.X. But if, for some reason, the code missed the boat, someone has to go back and edit the KEP to change alpha, beta, and ga. That is prone to failure, and if it is forgotten, then people might take the wrong result.

I think we should instead do soemthing like:

stage: prealpha
status: provisional
alpha: 
beta: 
ga: 

which gets approved to:

stage: prealpha
status: implementable
alpha: v1.X
beta: v1.Y
ga: v1.Z

When we release v1.X, we can update this to:

stage: alpha
status: implementable
alpha: v1.X
beta: v1.Y
ga: v1.Z

This means that someone has to touch the KEP - the same as before, but the failure mode is better. People won't accidentally assume it is ready to use. It would stay in this state until all the code for beta is merged, at which point it becomes:

stage: beta
status: implementable
alpha: v1.X
beta: v1.Y
ga: v1.Z

Obviously, we could model this other ways. E.g. we could have a per-stage status:

status:
  alpha: implemented
  beta: implementable

[thockin]

/sig release

[justaugustus]

cc: @kubernetes/sig-release @kubernetes/enhancements
/area enhancements

[johnbelamaric]

thanks Tim. Here's another one where the CI makes too many assumptions about the various fields being set correctly at the correct times:

#563 (comment)

We had intended to have kepctl manage this metadata in a consistent way, but that has stalled. We need a better solution in the interim.

[thockin]

If we can define the state-machine, we can code it up as CI. "Error: can't transition directly from state FOO to state BAR".

[wojtek-t]

Copying my comment from the slack channel:

I think that what we're missing is basically "two-phase-commit"
so in the ideal world the way I see that is:
- you update the KEP for feature freeze with your intention
- then you do the actual implementation
- once  all the implementation (including metrics, tests, etc.), you mark the KEP as "actual intent"
- and only then you are allow to send a PR changing the value of a feature-gate

that would gives us ways to ensure that the code is actually graduated only when it should be...
and on third step you're allow to update the KEP if that is needed (e.g. something that wasn't predicted was discovered)

Which means that the state machine I'm proposing is (well state-machine seems like a too big word for that, as transitions can go only between adjacent states):

provisional -> intended for Alpha -> code-complete for Alpha [-> graduated Alpha ??] -> intended for Beta -> code-complete for Beta [-> graduated beta ??] -> intended for Stable -> code-complete for Stable -> implemented

In the ideal world how I would see that is:

• we have feature-freeze as it is now [for putting keps into "intended for X" state]
• the there is code-freeze [but PRs touching feature-gates file are not allowed then]
• then it's graduation-freeze [where you mark your KEP as code-complete for X, approvers check if everything that should happen really happened, not just code, but also tests, metrics, ...]
• based on that you're only allowed to bump your feature-gate

I admit that adds more burden on the approvers, but given that code-approvers are not necessary the same people are KEP approvers, in the current maturity state of the project I think it would actually be worth it.

[kikisdeliveryservice]

Q: this is a speculative issue? Because when I was on the release team, the team did go through the KEPs post code freeze to ensure all PRs intended to merge did merge and if not asked the authors to update the KEPs.

While I think there is more granular control we could have on the KEPs, I worry that the reality is that forcing people to update KEPs multiple times within a release post freeze would create a new set of missed updates for authors who previously had no problems creating more issues with up to date KEPs than currently exist.

Thinking of something that is lower friction but provides the verification that @thockin rightly wants about the state: Could we ask sigs to somehow verify the state of the KEPs post code-freeze? I'd be more inclined to do this via the release team for 1 release, then see how it works out, get feedback and then memorialize it in the actual KEP template.

Further, I think a good point is made, that until the release is released, all the KEP statuses are tentative. I think that is something that we should/could more clearly communicate to outside observers, even outside of the KEP itself - perhaps via docs in the enhancements repo and within the KEP website that is being worked on something like: "all KEPs targeted at the current in-progress release: $RELEASENUMBER are only tentative. please wait until the release is cut on $DATE for the final list and status of KEPs in this release" or some such thing...