|
| 1 | +# Sidecar Release Process |
| 2 | + |
| 3 | +This page describes the process for releasing a kubernetes-csi sidecar. |
| 4 | + |
| 5 | +## Prerequisites |
| 6 | + |
| 7 | +The release manager must: |
| 8 | + |
| 9 | +* Be a member of the kubernetes-csi organization. Open an |
| 10 | + [issue](https://github.com/kubernetes/org/issues/new?assignees=&labels=area%2Fgithub-membership&template=membership.md&title=REQUEST%3A+New+membership+for+%3Cyour-GH-handle%3E) in |
| 11 | + kubernetes/org to request membership |
| 12 | +* Be a top level approver for the repository. To become a top level approver, |
| 13 | + the candidate must demonstrate ownership and deep knowledge of the repository |
| 14 | + through active maintenance, responding to and fixing issues, reviewing PRs, |
| 15 | + test triage. |
| 16 | +* Be part of the maintainers or admin group for the repository. admin is a |
| 17 | + superset of maintainers, only maintainers level is required for cutting a |
| 18 | + release. Membership can be requested by submitting a PR to kubernetes/org. |
| 19 | + [Example](https://github.com/kubernetes/org/pull/1467) |
| 20 | + |
| 21 | +## Updating CI Jobs |
| 22 | +Whenever a new Kubernetes minor version is released, our kubernetes-csi CI jobs |
| 23 | +must be updated. |
| 24 | + |
| 25 | +[Our CI jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) have the |
| 26 | +naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`. |
| 27 | + |
| 28 | +1. Jobs should be actively monitored to find and fix failures in sidecars and |
| 29 | + infrastructure changes early in the development cycle. Test failures are sent |
| 30 | + |
| 31 | +1. "-on-master" jobs are the closest reflection to the new Kubernetes version. |
| 32 | +1. Fixes to our prow.sh CI script can be tested in the [CSI hostpath |
| 33 | + repo](https://github.com/kubernetes-csi/csi-driver-host-path) by modifying |
| 34 | + [prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/release-tools/prow.sh) |
| 35 | + along with any overrides in |
| 36 | + [.prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/.prow.sh) |
| 37 | + to mirror the failing environment. Once e2e tests are passing (verify-unit tests |
| 38 | + will fail), then the prow.sh changes can be submitted to [csi-release-tools](https://github.com/kubernetes-csi/csi-release-tools). |
| 39 | +1. Changes can then be updated in all the sidecar repos and hostpath driver repo |
| 40 | + by following the [update |
| 41 | + instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating). |
| 42 | +1. New pull and CI jobs are configured by adding new K8s versions to the top of |
| 43 | + [gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-csi/gen-jobs.sh). |
| 44 | + New pull jobs that have been unverified should be initially made optional by |
| 45 | + setting the new K8s version as |
| 46 | + [experimental](https://github.com/kubernetes/test-infra/blob/a1858f46d6014480b130789df58b230a49203a64/config/jobs/kubernetes-csi/gen-jobs.sh#L40). |
| 47 | +1. Once new pull and CI jobs have been verified, and the new Kubernetes version |
| 48 | + is released, we can make the optional jobs required, and also remove the |
| 49 | + Kubernetes versions that are no longer supported. |
| 50 | + |
| 51 | +## Release Process |
| 52 | +1. Identify all issues and ongoing PRs that should go into the release, and |
| 53 | + drive them to resolution. |
| 54 | +1. Download v2.8+ [K8s release notes |
| 55 | + generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes) |
| 56 | +1. Generate release notes for the release. Replace arguments with the relevant |
| 57 | + information. |
| 58 | + * Clean up old cached information (also needed if you are generating release |
| 59 | + notes for multiple repos) |
| 60 | + ```bash |
| 61 | + rm -rf /tmp/k8s-repo |
| 62 | + ``` |
| 63 | + * For new minor releases on master: |
| 64 | + ```bash |
| 65 | + GITHUB_TOKEN=<token> release-notes --discover=mergebase-to-latest |
| 66 | + --github-org=kubernetes-csi --github-repo=external-provisioner |
| 67 | + --required-author="" --output out.md |
| 68 | + ``` |
| 69 | + * For new patch releases on a release branch: |
| 70 | + ```bash |
| 71 | + GITHUB_TOKEN=<token> release-notes --discover=patch-to-latest --branch=release-1.1 |
| 72 | + --github-org=kubernetes-csi --github-repo=external-provisioner |
| 73 | + --required-author="" --output out.md |
| 74 | + ``` |
| 75 | +1. Compare the generated output to the new commits for the release to check if |
| 76 | + any notable change missed a release note. |
| 77 | +1. Reword release notes as needed. Make sure to check notes for breaking |
| 78 | + changes and deprecations. |
| 79 | +1. If release is a new major/minor version, create a new `CHANGELOG-<major>.<minor>.md` |
| 80 | + file. Otherwise, add the release notes to the top of the existing CHANGELOG |
| 81 | + file for that minor version. |
| 82 | +1. Submit a PR for the CHANGELOG changes. |
| 83 | +1. Submit a PR for README changes, in particular, Compatibility, Feature status, |
| 84 | + and any other sections that may need updating. |
| 85 | +1. Check that all [canary CI |
| 86 | + jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) are passing, |
| 87 | + and that test coverage is adequate for the changes that are going into the release. |
| 88 | +1. Make sure that no new PRs have merged in the meantime, and no PRs are in |
| 89 | + flight and soon to be merged. |
| 90 | +1. Create a new release following a previous release as a template. Be sure to select the correct |
| 91 | + branch. This requires Github release permissions as required by the prerequisites. |
| 92 | + [external-provisioner example](https://github.com/kubernetes-csi/external-provisioner/releases/new) |
| 93 | +1. If release was a new major/minor version, create a new `release-<minor>` |
| 94 | + branch at that commit. |
| 95 | +1. Check [image build status](https://k8s-testgrid.appspot.com/sig-storage-image-build). |
| 96 | +1. Promote images from k8s-staging-sig-storage to k8s.gcr.io/sig-storage. From |
| 97 | + the [k8s image |
| 98 | + repo](https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage), |
| 99 | + run `./generate.sh > images.yaml`, and send a PR with the updated images. |
| 100 | + Once merged, the image promoter will copy the images from staging to prod. |
| 101 | +1. Update [kubernetes-csi/docs](https://github.com/kubernetes-csi/docs) sidecar |
| 102 | + and feature pages with the new released version. |
| 103 | +1. After all the sidecars have been released, update |
| 104 | + CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy) |
| 105 | + and [k/k |
| 106 | + in-tree](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/storage-csi/hostpath/hostpath) |
| 107 | + |
| 108 | +## Adding support for a new Kubernetes release |
| 109 | + |
| 110 | +1. Add the new release to `k8s_versions` in |
| 111 | + https://github.com/kubernetes/test-infra/blob/090dec5dd535d5f61b7ba52e671a810f5fc13dfd/config/jobs/kubernetes-csi/gen-jobs.sh#L25 |
| 112 | + to enable generating a job for it. Set `experimental_k8s_version` |
| 113 | + in |
| 114 | + https://github.com/kubernetes/test-infra/blob/090dec5dd535d5f61b7ba52e671a810f5fc13dfd/config/jobs/kubernetes-csi/gen-jobs.sh#L40 |
| 115 | + to ensure that the new jobs aren't run for PRs unless explicitly |
| 116 | + requested. Generate and submit the new jobs. |
| 117 | +1. Create a test PR to try out the new job in some repo with `/test |
| 118 | + pull-kubernetes-csi-<repo>-<x.y>-on-kubernetes-<x.y>` where x.y |
| 119 | + matches the Kubernetes release. Alternatively, run .prow.sh in that |
| 120 | + repo locally with `CSI_PROW_KUBERNETES_VERSION=x.y.z`. |
| 121 | +1. Optional: update to a [new |
| 122 | + release](https://github.com/kubernetes-sigs/kind/tags) of kind with |
| 123 | + pre-built images for the new Kubernetes release. This is optional |
| 124 | + if the current version of kind is able to build images for the new |
| 125 | + Kubernetes release. However, jobs require less resources when they |
| 126 | + don't need to build those images from the Kubernetes source code. |
| 127 | + This change needs to be tried out in a PR against a component |
| 128 | + first, then get submitted against csi-release-tools. |
| 129 | +1. Optional: propagate the updated csi-release-tools to all components |
| 130 | + with the script from |
| 131 | + https://github.com/kubernetes-csi/csi-release-tools/issues/7#issuecomment-707025402 |
| 132 | +1. Once it is likely to work in all components, unset |
| 133 | + `experimental_k8s_version` and submit the updated jobs. |
| 134 | +1. Once all sidecars for the new Kubernetes release are released, |
| 135 | + either bump the version number of the images in the existing |
| 136 | + [csi-driver-host-path |
| 137 | + deployments](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy) |
| 138 | + and/or create a new deployment, depending on what Kubernetes |
| 139 | + release an updated sidecar is compatible with. If no new deployment |
| 140 | + is needed, then add a symlink to document that there intentionally |
| 141 | + isn't a separate deployment. This symlink is not needed for Prow |
| 142 | + testing because that will use "kubernetes-latest" as fallback. |
| 143 | + Update that link when creating a new deployment. |
| 144 | +1. Create a new csi-driver-host-path release. |
| 145 | +1. Bump `CSI_PROW_DRIVER_VERSION` in prow.sh to that new release and |
| 146 | + (eventually) roll that change out to all repos by updating |
| 147 | + `release-tools` in them. This is used when testing manually. The |
| 148 | + Prow jobs override that value, so also update |
| 149 | + `hostpath_driver_version` in |
| 150 | + https://github.com/kubernetes/test-infra/blob/91b04e6af3a40a9bcff25aa030850a4721e2dd2b/config/jobs/kubernetes-csi/gen-jobs.sh#L46-L47 |
0 commit comments