pohly
diff --git a/‎README.md
Lines changed: 59 additions & 1 deletion b/‎README.md
Lines changed: 59 additions & 1 deletion
diff --git a/‎SIDECAR_RELEASE_PROCESS.md
Lines changed: 106 additions & 0 deletions b/‎SIDECAR_RELEASE_PROCESS.md
Lines changed: 106 additions & 0 deletions
diff --git a/‎build.make
Lines changed: 132 additions & 13 deletions b/‎build.make
Lines changed: 132 additions & 13 deletions
diff --git a/‎cloudbuild.sh
Lines changed: 6 additions & 0 deletions b/‎cloudbuild.sh
Lines changed: 6 additions & 0 deletions
@@ -81,7 +81,7 @@ on what is enabled in Prow, see
 https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-csi
 
 Test results for periodic jobs are visible in
-https://testgrid.k8s.io/sig-storage-csi
+https://testgrid.k8s.io/sig-storage-csi-ci
 
 It is possible to reproduce the Prow testing locally on a suitable machine:
 - Linux host
@@ -106,3 +106,61 @@ Kubernetes releases:
 
     CSI_PROW_KUBERNETES_VERSION=1.13.3 ./.prow.sh
     CSI_PROW_KUBERNETES_VERSION=latest ./.prow.sh
+
+Dependencies and vendoring
+--------------------------
+
+Most projects will (eventually) use `go mod` to manage
+dependencies. `dep` is also still supported by `csi-release-tools`,
+but not documented here because it's not recommended anymore.
+
+The usual instructions for using [go
+modules](https://github.com/golang/go/wiki/Modules) apply. Here's a cheat sheet
+for some of the relevant commands:
+- list available updates: `GO111MODULE=on go list -u -m all`
+- update or add a single dependency: `GO111MODULE=on go get <package>`
+- update all dependencies to their next minor or patch release:
+  `GO111MODULE=on go get ./...` (add `-u=patch` to limit to patch
+  releases)
+- lock onto a specific version: `GO111MODULE=on go get <package>@<version>`
+- clean up `go.mod`: `GO111MODULE=on go mod tidy`
+- update vendor directory: `GO111MODULE=on go mod vendor`
+
+`GO111MODULE=on` can be left out when using Go >= 1.13 or when the
+source is checked out outside of `$GOPATH`.
+
+`go mod tidy` must be used to ensure that the listed dependencies are
+really still needed. Changing import statements or a tentative `go
+get` can result in stale dependencies.
+
+The `test-vendor` verifies that it was used when run locally or in a
+pre-merge CI job. If a `vendor` directory is present, it will also
+verify that it's content is up-to-date.
+
+The `vendor` directory is optional. It is still present in projects
+because it avoids downloading sources during CI builds. If this is no
+longer deemed necessary, then a project can also remove the directory.
+
+Conversion of a repository that uses `dep` to `go mod` can be done with:
+
+    GO111MODULE=on go mod init
+    release-tools/go-get-kubernetes.sh <current Kubernetes version from Gopkg.toml>
+    GO111MODULE=on go mod tidy
+    GO111MODULE=on go mod vendor
+    git rm -f Gopkg.toml Gopkg.lock
+    git add go.mod go.sum vendor
+
+### Updating Kubernetes dependencies
+
+When using packages that are part of the Kubernetes source code, the
+commands above are not enough because the [lack of semantic
+versioning](https://github.com/kubernetes/kubernetes/issues/72638)
+prevents `go mod` from finding newer releases. Importing directly from
+`kubernetes/kubernetes` also needs `replace` statements to override
+the fake `v0.0.0` versions
+(https://github.com/kubernetes/kubernetes/issues/79384). The
+`go-get-kubernetes.sh` script can be used to update all packages in
+lockstep to a different Kubernetes version. Example usage:
+```
+$ ./release-tools/go-get-kubernetes.sh 1.16.4
+```
@@ -0,0 +1,106 @@
+# Sidecar Release Process
+
+This page describes the process for releasing a kubernetes-csi sidecar.
+
+## Prerequisites
+
+The release manager must:
+
+* Be a member of the kubernetes-csi organization. Open an
+  [issue](https://github.com/kubernetes/org/issues/new?assignees=&labels=area%2Fgithub-membership&template=membership.md&title=REQUEST%3A+New+membership+for+%3Cyour-GH-handle%3E) in
+  kubernetes/org to request membership
+* Be a top level approver for the repository. To become a top level approver,
+  the candidate must demonstrate ownership and deep knowledge of the repository
+  through active maintainence, responding to and fixing issues, reviewing PRs,
+  test triage.
+* Be part of the maintainers or admin group for the repository. admin is a
+  superset of maintainers, only maintainers level is required for cutting a
+  release.  Membership can be requested by submitting a PR to kubernetes/org.
+  [Example](https://github.com/kubernetes/org/pull/1467)
+
+## Updating CI Jobs
+Whenever a new Kubernetes minor version is released, our kubernetes-csi CI jobs
+must be updated.
+
+[Our CI jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) have the
+naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
+
+1. Jobs should be actively monitored to find and fix failures in sidecars and
+   infrastructure changes early in the development cycle. Test failures are sent
+   to [email protected].
+1. "-on-master" jobs are the closest reflection to the new Kubernetes version.
+1. Fixes to our prow.sh CI script can be tested in the [CSI hostpath
+   repo](https://github.com/kubernetes-csi/csi-driver-host-path) by modifying
+   [prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/release-tools/prow.sh)
+   along with any overrides in
+   [.prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/.prow.sh)
+   to mirror the failing environment. Once e2e tests are passing (verify-unit tests
+   will fail), then the prow.sh changes can be submitted to [csi-release-tools](https://github.com/kubernetes-csi/csi-release-tools).
+1. Changes can then be updated in all the sidecar repos and hostpath driver repo
+   by following the [update
+   instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating).
+1. New pull and CI jobs are configured by adding new K8s versions to the top of
+   [gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-csi/gen-jobs.sh).
+   New pull jobs that have been unverified should be initially made optional by
+   setting the new K8s version as
+   [experimental](https://github.com/kubernetes/test-infra/blob/a1858f46d6014480b130789df58b230a49203a64/config/jobs/kubernetes-csi/gen-jobs.sh#L40).
+1. Once new pull and CI jobs have been verified, and the new Kubernetes version
+   is released, we can make the optional jobs required, and also remove the
+   Kubernetes versions that are no longer supported.
+
+## Release Process
+1. Identify all issues and ongoing PRs that should go into the release, and
+  drive them to resolution.
+1. Download v2.8+ [K8s release notes
+  generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes)
+1. Generate release notes for the release. Replace arguments with the relevant
+  information.
+    * Clean up old cached information (also needed if you are generating release
+      notes for multiple repos)
+      ```bash
+      rm -rf /tmp/k8s-repo
+      ```
+    * For new minor releases on master:
+        ```bash
+        GITHUB_TOKEN=<token> release-notes --discover=mergebase-to-latest
+        --github-org=kubernetes-csi --github-repo=external-provisioner
+        --required-author="" --output out.md
+        ```
+    * For new patch releases on a release branch:
+        ```bash
+        GITHUB_TOKEN=<token> release-notes --discover=patch-to-latest --branch=release-1.1
+        --github-org=kubernetes-csi --github-repo=external-provisioner
+        --required-author="" --output out.md
+        ```
+1. Compare the generated output to the new commits for the release to check if
+   any notable change missed a release note.
+1. Reword release notes as needed. Make sure to check notes for breaking
+   changes and deprecations.
+1. If release is a new major/minor version, create a new `CHANGELOG-<major>.<minor>.md`
+   file. Otherwise, add the release notes to the top of the existing CHANGELOG
+   file for that minor version.
+1. Submit a PR for the CHANGELOG changes.
+1. Submit a PR for README changes, in particular, Compatibility, Feature status,
+   and any other sections that may need updating.
+1. Check that all [canary CI
+  jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) are passing,
+  and that test coverage is adequate for the changes that are going into the release.
+1. Make sure that no new PRs have merged in the meantime, and no PRs are in
+   flight and soon to be merged.
+1. Create a new release following a previous release as a template. Be sure to select the correct
+   branch. This requires Github release permissions as required by the prerequisites.
+   [external-provisioner example](https://github.com/kubernetes-csi/external-provisioner/releases/new)
+1. If release was a new major/minor version, create a new `release-<minor>`
+   branch at that commit.
+1. Check [image build status](https://k8s-testgrid.appspot.com/sig-storage-image-build).
+1. Promote images from k8s-staging-sig-storage to k8s.gcr.io/sig-storage. From
+   the [k8s image
+   repo](https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage),
+   run `./generate.sh > images.yaml`, and send a PR with the updated images.
+   Once merged, the image promoter will copy the images from staging to prod.
+1. Update [kubernetes-csi/docs](https://github.com/kubernetes-csi/docs) sidecar
+   and feature pages with the new released version.
+1. After all the sidecars have been released, update
+   CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
+   and [k/k
+   in-tree](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/storage-csi/hostpath/hostpath)
@@ -22,6 +22,9 @@
 # including build.make.
 REGISTRY_NAME=quay.io/k8scsi
 
+# Can be set to -mod=vendor to ensure that the "vendor" directory is used.
+GOFLAGS_VENDOR=
+
 # Revision that gets built into each binary via the main.version
 # string. Uses the `git describe` output based on the most recent
 # version tag with a short revision suffix or, if nothing has been
@@ -60,14 +63,27 @@ endif
 # Specific packages can be excluded from each of the tests below by setting the *_FILTER_CMD variables
 # to something like "| grep -v 'github.com/kubernetes-csi/project/pkg/foobar'". See usage below.
 
-build-%:
+# BUILD_PLATFORMS contains a set of <os> <arch> <suffix> triplets,
+# separated by semicolon. An empty variable or empty entry (= just a
+# semicolon) builds for the default platform of the current Go
+# toolchain.
+BUILD_PLATFORMS =
+
+# This builds each command (= the sub-directories of ./cmd) for the target platform(s)
+# defined by BUILD_PLATFORMS.
+$(CMDS:%=build-%): build-%: check-go-version-go
 	mkdir -p bin
-	CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$*
+	echo '$(BUILD_PLATFORMS)' | tr ';' '\n' | while read -r os arch suffix; do \
+		if ! (set -x; CGO_ENABLED=0 GOOS="$$os" GOARCH="$$arch" go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o "./bin/$*$$suffix" ./cmd/$*); then \
+			echo "Building $* for GOOS=$$os GOARCH=$$arch failed, see error(s) above."; \
+			exit 1; \
+		fi; \
+	done
 
-container-%: build-%
+$(CMDS:%=container-%): container-%: build-%
 	docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) .
 
-push-%: container-%
+$(CMDS:%=push-%): push-%: container-%
 	set -ex; \
 	push_image () { \
 		docker tag $*:latest $(IMAGE_NAME):$$tag; \
@@ -89,22 +105,93 @@ build: $(CMDS:%=build-%)
 container: $(CMDS:%=container-%)
 push: $(CMDS:%=push-%)
 
+# Additional parameters are needed when pushing to a local registry,
+# see https://github.com/docker/buildx/issues/94.
+# However, that then runs into https://github.com/docker/cli/issues/2396.
+#
+# What works for local testing is:
+# make push-multiarch PULL_BASE_REF=master REGISTRY_NAME=<your account on dockerhub.io> BUILD_PLATFORMS="linux amd64; windows amd64 .exe; linux ppc64le -ppc64le; linux s390x -s390x"
+DOCKER_BUILDX_CREATE_ARGS ?=
+
+# This target builds a multiarch image for one command using Moby BuildKit builder toolkit.
+# Docker Buildx is included in Docker 19.03.
+#
+# ./cmd/<command>/Dockerfile[.Windows] is used if found, otherwise Dockerfile[.Windows].
+# It is currently optional: if no such file exists, Windows images are not included,
+# even when Windows is listed in BUILD_PLATFORMS. That way, projects can test that
+# Windows binaries can be built before adding a Dockerfile for it.
+#
+# BUILD_PLATFORMS determines which individual images are included in the multiarch image.
+# PULL_BASE_REF must be set to 'master', 'release-x.y', or a tag name, and determines
+# the tag for the resulting multiarch image.
+$(CMDS:%=push-multiarch-%): push-multiarch-%: check-pull-base-ref build-%
+	set -ex; \
+	DOCKER_CLI_EXPERIMENTAL=enabled; \
+	export DOCKER_CLI_EXPERIMENTAL; \
+	docker buildx create $(DOCKER_BUILDX_CREATE_ARGS) --use --name multiarchimage-buildertest; \
+	trap "docker buildx rm multiarchimage-buildertest" EXIT; \
+	dockerfile_linux=$$(if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi); \
+	dockerfile_windows=$$(if [ -e ./cmd/$*/Dockerfile.Windows ]; then echo ./cmd/$*/Dockerfile.Windows; else echo Dockerfile.Windows; fi); \
+	if [ '$(BUILD_PLATFORMS)' ]; then build_platforms='$(BUILD_PLATFORMS)'; else build_platforms="linux amd64"; fi; \
+	if ! [ -f "$$dockerfile_windows" ]; then \
+		build_platforms="$$(echo "$$build_platforms" | sed -e 's/windows *[^ ]* *.exe//g' -e 's/; *;/;/g')"; \
+	fi; \
+	pushMultiArch () { \
+		tag=$$1; \
+		echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix; do \
+			docker buildx build --push \
+				--tag $(IMAGE_NAME):$$arch-$$os-$$tag \
+				--platform=$$os/$$arch \
+				--file $$(eval echo \$${dockerfile_$$os}) \
+				--build-arg binary=./bin/$*$$suffix \
+				--label revision=$(REV) \
+				.; \
+		done; \
+		images=$$(echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix; do echo $(IMAGE_NAME):$$arch-$$os-$$tag; done); \
+		docker manifest create --amend $(IMAGE_NAME):$$tag $$images; \
+		docker manifest push -p $(IMAGE_NAME):$$tag; \
+	}; \
+	if [ $(PULL_BASE_REF) = "master" ]; then \
+			: "creating or overwriting canary image"; \
+			pushMultiArch canary; \
+	elif echo $(PULL_BASE_REF) | grep -q -e 'release-*' ; then \
+			: "creating or overwriting canary image for release branch"; \
+			release_canary_tag=$$(echo $(PULL_BASE_REF) | cut -f2 -d '-')-canary; \
+			pushMultiArch $$release_canary_tag; \
+	elif docker pull $(IMAGE_NAME):$(PULL_BASE_REF) 2>&1 | tee /dev/stderr | grep -q "manifest for $(IMAGE_NAME):$(PULL_BASE_REF) not found"; then \
+			: "creating release image"; \
+			pushMultiArch $(PULL_BASE_REF); \
+	else \
+			: "ERROR: release image $(IMAGE_NAME):$(PULL_BASE_REF) already exists: a new tag is required!"; \
+			exit 1; \
+	fi
+
+.PHONY: check-pull-base-ref
+check-pull-base-ref:
+	if ! [ "$(PULL_BASE_REF)" ]; then \
+		echo >&2 "ERROR: PULL_BASE_REF must be set to 'master', 'release-x.y', or a tag name."; \
+		exit 1; \
+	fi
+
+.PHONY: push-multiarch
+push-multiarch: $(CMDS:%=push-multiarch-%)
+
 clean:
 	-rm -rf bin
 
-test:
+test: check-go-version-go
 
 .PHONY: test-go
 test: test-go
 test-go:
 	@ echo; echo "### $@:"
-	go test `go list ./... | grep -v -e 'vendor' -e '/test/e2e$$' $(TEST_GO_FILTER_CMD)` $(TESTARGS)
+	go test $(GOFLAGS_VENDOR) `go list $(GOFLAGS_VENDOR) ./... | grep -v -e 'vendor' -e '/test/e2e$$' $(TEST_GO_FILTER_CMD)` $(TESTARGS)
 
 .PHONY: test-vet
 test: test-vet
 test-vet:
 	@ echo; echo "### $@:"
-	go vet `go list ./... | grep -v vendor $(TEST_VET_FILTER_CMD)`
+	go vet $(GOFLAGS_VENDOR) `go list $(GOFLAGS_VENDOR) ./... | grep -v vendor $(TEST_VET_FILTER_CMD)`
 
 .PHONY: test-fmt
 test: test-fmt
@@ -118,14 +205,38 @@ test-fmt:
 	fi
 
 # This test only runs when dep >= 0.5 is installed, which is the case for the CI setup.
+# When using 'go mod', we allow the test to be skipped in the Prow CI under some special
+# circumstances, because it depends on accessing all remote repos and thus
+# running it all the time would defeat the purpose of vendoring:
+# - not handling a PR or
+# - the fabricated merge commit leaves go.mod, go.sum and vendor dir unchanged
+# - release-tools also didn't change (changing rules or Go version might lead to
+#   a different result and thus must be tested)
+# - import statements not changed (because if they change, go.mod might have to be updated)
+#
+# "git diff" is intelligent enough to annotate changes inside the "import" block in
+# the start of the diff hunk:
+#
+# diff --git a/rpc/common.go b/rpc/common.go
+# index bb4a5c4..5fa4271 100644
+# --- a/rpc/common.go
+# +++ b/rpc/common.go
+# @@ -21,7 +21,6 @@ import (
+#         "fmt"
+#         "time"
+#
+# -       "google.golang.org/grpc"
+#         "google.golang.org/grpc/codes"
+#         "google.golang.org/grpc/status"
+#
+# We rely on that to find such changes.
+#
+# Vendoring is optional when using go.mod.
 .PHONY: test-vendor
 test: test-vendor
 test-vendor:
 	@ echo; echo "### $@:"
-	@ case "$$(dep version 2>/dev/null | grep 'version *:')" in \
-		*v0.[56789]*) dep check && echo "vendor up-to-date" || false;; \
-		*) echo "skipping check, dep >= 0.5 required";; \
-	esac
+	@ ./release-tools/verify-vendor.sh
 
 .PHONY: test-subtree
 test: test-subtree
@@ -143,11 +254,19 @@ test-shellcheck:
 	@ ret=0; \
 	if ! command -v docker; then \
 		echo "skipped, no Docker"; \
-		return 0; \
+		exit 0; \
         fi; \
 	for dir in $(abspath $(TEST_SHELLCHECK_DIRS)); do \
 		echo; \
 		echo "$$dir:"; \
 		./release-tools/verify-shellcheck.sh "$$dir" || ret=1; \
 	done; \
-	return $$ret
+	exit $$ret
+
+# Targets in the makefile can depend on check-go-version-<path to go binary>
+# to trigger a warning if the x.y version of that binary does not match
+# what the project uses. Make ensures that this is only checked once per
+# invocation.
+.PHONY: check-go-version-%
+check-go-version-%:
+	./release-tools/verify-go-version.sh "$*"
@@ -0,0 +1,6 @@
+#! /bin/bash
+
+# shellcheck disable=SC1091
+. release-tools/prow.sh
+
+gcr_cloud_build