Check devworkspace pod events for "Unrecoverable" events

[amisevsk]

What does this PR do?

Checks events related to the DevWorkspace pod for what we would consider "unrecoverable" events (list stolen from Che 😛)

This requires adding an index to the manager, to allow us to list events with the appropriate fieldSelector.

Also adds get/list/watch Events RBAC to the controller.

What issues does this PR fix or reference?

Closes #465

Is it tested? How?

It's hard to reproduce these events manually; the easiest way I found was to patch commonStorage.go to read

// Line 115
	podAdditions.Volumes = append(podAdditions.Volumes, corev1.Volume{
		Name: commonPVCName,
		VolumeSource: corev1.VolumeSource{
			PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSource{
				ClaimName: commonPVCName+"fake",
			},
		},
	})

and trying to start a non-ephemeral DevWorkspace.

PR Checklist

• E2E tests pass (when PR is ready, comment /test v7-devworkspaces-operator-e2e, v7-devworkspace-happy-path to trigger)
  • v7-devworkspaces-operator-e2e: DevWorkspace e2e test
  • v7-devworkspace-happy-path: DevWorkspace e2e test

[sleshchenko]

On some infrastructure, FailedMount or FailedScheduling can be an usual situation when you just need to be more patient and then on Che we suggest users to drop them from unrecoverable events list.
We should have it configurable in some way or let users annotate particular workspace, maybe get it from namespaced configmap settings if present.
WDYT?

[amisevsk]

I feel like an annotation system might complicate things too much -- we should just do what we think is best (whether that's check or don't check).

One issue I've seen in Che before is that there can be a bit of a thundering herd problem -- if I've got 10 workspaces waiting for the common PVC to be mountable, it can be very confusing because they'll randomly start up and have to be stopped until it gets to the one I want to actually use.

One option that just occurred to me from looking at the Event struct a bit more -- what if we have a threshold for "number of times the event has occurred"? We could fail workspace start if e.g. we've seen the FailedMount even three times.

edit: tested checking the count, and for e.g. a non-existent PVC, it never goes past 1 :/

[amisevsk]

Created #550 to follow up

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amisevsk, JPinkney, sleshchenko

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JPinkney,amisevsk,sleshchenko]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

New changes are detected. LGTM label has been removed.

[amisevsk]

/test v7-devworkspaces-operator-e2e, v7-devworkspace-happy-path

[openshift-ci]

@amisevsk: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/prow/v7-devworkspace-happy-path	05f102f	link	/test v7-devworkspace-happy-path

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[amisevsk]

Happy path tests fail due to #484 (Events not in RBAC used in the test itself).

I'm merging this one now, with the note that we should follow up in #550