B/R e2e add velero restic DC workaround, describe, namespace events, wait for CSI snapshot to be ready, IsDCReady wait for builds, azure-rg #654

kaovilai · 2022-04-29T23:41:09Z

Adding several enhancements to B/R suite test

Restic DC restore workaround
print namespace events to detect install failures
add B/R describe after b/r done
wait for CSI snapshot to be ready after backing up before deleting workload and restoring.
Update IsDCReady to wait for BuildConfig.
Fix label used to get velero pods
Failure logs from backup/restore are now done via downloadrequest
Fix Bug: Azure E2E using -rg resource group #658

Restic DC restore workaround
Allow us to add data verification in the future. Currently we only verify app is running and responsive.
We would need to add after PreBackupVerification which check app is running with PreBackupDataEntry or similar.

Namespace Events
This is useful for debugging pods not starting up or stuck.
Example event that could come up before #650 is merged if secret contains carriage return that would be hard to dig from artifacts.

Event: Error: couldn't find key access_key in Secret openshift-adp/oadp-ts-example-velero-1-aws-registry-secret, Src: kubelet, Reason: Failed

B/R describe
Help diagnose B/R failures when there is nothing in the restore logs. Maybe remove after vmware-tanzu/velero#4743

Wait for CSI snapshot to be ready (Uncovered by Namespace events enhancements)
On restore it is possible that snapshot isn't ready to be used as DataSource for PVC yet as shown at #654 (comment)

Event: failed to provision volume with StorageClass "gp2-csi": error getting handle for DataSource Type VolumeSnapshot by Name velero-mysql-l5sz5: snapshot velero-mysql-l5sz5 is not Ready, Type: Warning, Count: 10, Src: {PersistentVolumeClaim mysql-persistent mysql 72d98c8b-10ff-4a32-a319-88bed69e1227 v1 34225 }, Reason: ProvisioningFailed

We should wait for CSI snapshot to be ready before uninstalling the application.

Example output

2022/05/01 16:19:06 Backup for case mysql-e2e succeeded
2022/05/01 16:19:13 waiting for volume snapshot velero-mysql-6s8sm to be ready
...
2022/05/01 16:20:44 waiting for volume snapshot velero-mysql-6s8sm to be ready
2022/05/01 16:20:44 Uninstalling app for case mysql-e2e

Update IsDCReady to wait for BuildConfig.
We should wait for all builds to be complete before considering DC ready for backup/disaster simulation/restore processes.
Fix #646

Example output

2022/05/01 16:25:56 Installing application for case parks-e2e
2022/05/01 16:25:57 Build is not ready: restify-1
...
2022/05/01 16:26:52 Build is not ready: restify-1
2022/05/01 16:27:02 Running pre-backup function for case parks-e2e

Failure logs from backup/restore are now done via downloadrequest
This prevent unrelated to backup or restore pod errors from showing up.

kaovilai · 2022-04-30T04:45:20Z

Welp. Looks like I opened a new can of worms here
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_oadp-operator/654/pull-ci-openshift-oadp-operator-master-4.9-operator-e2e-aws/1520234660829859840#1:build-log.txt%3A362

Event: Error creating: pods "mysql-6bb6964964-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount, provider "velero-privileged": Forbidden: not usable by user or serviceaccount], Type: Warning, Count: 6, Src: {ReplicaSet mysql-persistent mysql-6bb6964964 39a26745-ee02-4441-be87-7e694f07100b apps/v1 34184 }, Reason: FailedCreate
Event: failed to provision volume with StorageClass "gp2-csi": error getting handle for DataSource Type VolumeSnapshot by Name velero-mysql-l5sz5: snapshot velero-mysql-l5sz5 is not Ready, Type: Warning, Count: 10, Src: {PersistentVolumeClaim mysql-persistent mysql 72d98c8b-10ff-4a32-a319-88bed69e1227 v1 34225 }, Reason: ProvisioningFailed

Fixing error getting handle for DataSource Type VolumeSnapshot by Name velero-mysql-l5sz5: snapshot velero-mysql-l5sz5 is not Ready in #657

kaovilai · 2022-04-30T05:46:42Z

Browsing Azure container for backup shows resource group issue coming from #582

time="2022-04-30T03:54:36Z" level=error msg="Error backing up item" backup=openshift-adp/mysql-e2e-0cd8e63e-c839-11ec-88cf-0a580a81136e error="error getting volume info: rpc error: code = Unknown desc = compute.DisksClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\"ResourceGroupNotFound\" Message=\"Resource group '-rg' could not be found.\"" logSource="pkg/backup/backup.go:417" name=mysql-6bb6964964-kbwxj

looks like

oadp-operator/tests/e2e/scripts/azure_settings.sh

Line 10 in 69ce063

CI_AZURE_RESOURCE_GROUP="${CI_AZURE_RESOURCE_GROUP}-rg"; \

is not getting the value from $AZURE_RESOURCE_FILE

kaovilai · 2022-05-01T05:46:35Z

/retest

kaovilai · 2022-05-01T22:55:52Z

after tests pass except azure will merge #659 into this and close that out to verify. Also want to see if azure errors resource group errors (in velero backup logs) now show up properly. Velero deployment label change caused failure logs to not show up in e2e logs.

kaovilai · 2022-05-01T23:20:49Z

kaovilai · 2022-05-02T01:01:56Z

The resource group error #658 now output in e2e log https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_oadp-operator/654/pull-ci-openshift-oadp-operator-master-4.10-operator-e2e-azure/1520908535485960192#1:build-log.txt%3A400

  Expected
      <[]string | len:2, cap:2>: [
          "time=\"2022-05-02T00:34:54Z\" level=info msg=\"1 errors encountered backup up item\" backup=openshift-adp/mysql-e2e-800070e1-c9af-11ec-9321-0a580a832067 logSource=\"pkg/backup/backup.go:413\" name=mysql-59b8bbcdc8-h9g7d",
          "time=\"2022-05-02T00:34:54Z\" level=error msg=\"Error backing up item\" backup=openshift-adp/mysql-e2e-800070e1-c9af-11ec-9321-0a580a832067 error=\"error getting volume info: rpc error: code = Unknown desc = compute.DisksClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\\\"ResourceGroupNotFound\\\" Message=\\\"Resource group '-rg' could not be found.\\\"\" logSource=\"pkg/backup/backup.go:417\" name=mysql-59b8bbcdc8-h9g7d",
      ]
  to equal
      <[]string | len:0, cap:0>: []
  In [It] at: /go/src/github.com/openshift/oadp-operator/tests/e2e/backup_restore_suite_test.go:141

kaovilai · 2022-05-02T02:35:35Z

@kaovilai: all tests passed!

go.mod

controllers/velero.go

kaovilai

comments

tests/e2e/lib/apps.go

This is useful for debugging pods not starting up or stuck. Example event that would come up before openshift#650 is merged ``` Event: Error: couldn't find key access_key in Secret openshift-adp/oadp-ts-example-velero-1-aws-registry-secret, Src: kubelet, Reason: Failed ```

…store

* fix awk for azure resource group * newline proof awk for resourcegroup

tests/e2e/lib/velero_helpers.go

deepakraj1997

LGTM

openshift-ci · 2022-05-02T18:38:46Z

@kaovilai: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

shubham-pampattiwar

VISACK !

openshift-ci bot requested review from hhpatel14 and shubham-pampattiwar April 29, 2022 23:41

kaovilai mentioned this pull request Apr 30, 2022

E2E Wait for snapshot to be ready for CSI B/R case #657

Closed

kaovilai added the testing label Apr 30, 2022

kaovilai mentioned this pull request Apr 30, 2022

Bug: Azure E2E using -rg resource group #658

Closed

1 task

kaovilai changed the title ~~Print namespace events when installing apps fail~~ B/R Suite Test adds restic DC workaround, namespace events, B/R describe, wait for CSI snapshot to be ready May 1, 2022

kaovilai force-pushed the namespace_events_app_suite_test branch from 60c9620 to 7b8ff54 Compare May 1, 2022 15:38

kaovilai changed the title ~~B/R Suite Test adds restic DC workaround, namespace events, B/R describe, wait for CSI snapshot to be ready~~ B/R Suite test restic DC workaround, namespace events, describe, wait for CSI snapshot to be ready, IsDCReady wait for builds May 1, 2022

kaovilai force-pushed the namespace_events_app_suite_test branch from c74bdc2 to e1cd727 Compare May 1, 2022 22:48

kaovilai mentioned this pull request May 2, 2022

awk for azure resource group can accept unformatted json #659

Closed

kaovilai requested review from dymurray and deepakraj1997 May 2, 2022 01:28

kaovilai commented May 2, 2022

View reviewed changes

go.mod Outdated Show resolved Hide resolved

kaovilai commented May 2, 2022

View reviewed changes

controllers/velero.go Outdated Show resolved Hide resolved

kaovilai commented May 2, 2022

View reviewed changes

deepakraj1997 reviewed May 2, 2022

View reviewed changes

tests/e2e/lib/apps.go Outdated Show resolved Hide resolved

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 2, 2022

kaovilai added 6 commits May 2, 2022 11:51

use metav1 Time

dc47683

format

c27cb13

Get events after time

520c78f

add sort and count to event

a387586

Update lastInstallTime and lastInstallApplicationNamespace for app re…

9aa9081

…store

kaovilai and others added 16 commits May 2, 2022 11:57

Describe backup and restore in backup restore suite test

83c3572

format

43430aa

fix make test, install.AllCRDs param removal

68e727f

log so we know we are waiting on volumesnapshot

bf6142e

Wait for builds before considering app ready

7a9fe01

move B/R describe to ginkgowriter

8ea62d7

add builds ready check to after restic DC restore workaround first step

97346f3

IsVolumeSnapshotsReady ->AreVolumeSnapshotsReady

203c31e

use correct restore name to check for done

900c965

Use GinkgoWriter for describe

d2c0fc2

fix getVeleroPods labels

a3cd1ec

B/R logs request via downloadrequest

f20f536

Check cluster version skip before building dpa

e314324

Azure resource group (#45)

92c32cb

* fix awk for azure resource group * newline proof awk for resourcegroup

remove unused commented codes

8440ffe

break loop after found DC availability condition

641bd1f

kaovilai force-pushed the namespace_events_app_suite_test branch from a6ec56c to 641bd1f Compare May 2, 2022 16:02

openshift-ci bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 2, 2022

kaovilai requested a review from deepakraj1997 May 2, 2022 16:03

kaovilai added 2 commits May 2, 2022 12:06

rebase reverted go.mod velero update

e1467b2

make test go fmt

d370624

kaovilai mentioned this pull request May 2, 2022

[OADP-460] Don't mount cloud provider secret volumes with NoDefaultBackupLocation #662

Merged

1 task

deepakraj1997 suggested changes May 2, 2022

View reviewed changes

tests/e2e/lib/velero_helpers.go Outdated Show resolved Hide resolved

tests/e2e/lib/velero_helpers.go Outdated Show resolved Hide resolved

tests/e2e/lib/velero_helpers.go Outdated Show resolved Hide resolved

tests/e2e/lib/velero_helpers.go Outdated Show resolved Hide resolved

use log.Printf

61a649e

kaovilai requested a review from deepakraj1997 May 2, 2022 17:00

deepakraj1997 approved these changes May 2, 2022

View reviewed changes

shubham-pampattiwar approved these changes May 2, 2022

View reviewed changes

kaovilai merged commit 0068184 into openshift:master May 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

B/R e2e add velero restic DC workaround, describe, namespace events, wait for CSI snapshot to be ready, IsDCReady wait for builds, azure-rg #654

B/R e2e add velero restic DC workaround, describe, namespace events, wait for CSI snapshot to be ready, IsDCReady wait for builds, azure-rg #654

Uh oh!

kaovilai commented Apr 29, 2022 •

edited

Loading

Uh oh!

kaovilai commented Apr 30, 2022 •

edited

Loading

Uh oh!

kaovilai commented Apr 30, 2022 •

edited

Loading

Uh oh!

kaovilai commented May 1, 2022

Uh oh!

kaovilai commented May 1, 2022 •

edited

Loading

Uh oh!

kaovilai commented May 1, 2022

Uh oh!

kaovilai commented May 2, 2022 •

edited

Loading

Uh oh!

kaovilai commented May 2, 2022

Uh oh!

Uh oh!

Uh oh!

kaovilai left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deepakraj1997 left a comment

Uh oh!

openshift-ci bot commented May 2, 2022

Uh oh!

shubham-pampattiwar left a comment

Uh oh!

Uh oh!

B/R e2e add velero restic DC workaround, describe, namespace events, wait for CSI snapshot to be ready, IsDCReady wait for builds, azure-rg #654

B/R e2e add velero restic DC workaround, describe, namespace events, wait for CSI snapshot to be ready, IsDCReady wait for builds, azure-rg #654

Uh oh!

Conversation

kaovilai commented Apr 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaovilai commented Apr 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaovilai commented Apr 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaovilai commented May 1, 2022

Uh oh!

kaovilai commented May 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaovilai commented May 1, 2022

Uh oh!

kaovilai commented May 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaovilai commented May 2, 2022

Uh oh!

Uh oh!

Uh oh!

kaovilai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deepakraj1997 left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented May 2, 2022

Uh oh!

shubham-pampattiwar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kaovilai commented Apr 29, 2022 •

edited

Loading

kaovilai commented Apr 30, 2022 •

edited

Loading

kaovilai commented Apr 30, 2022 •

edited

Loading

kaovilai commented May 1, 2022 •

edited

Loading

kaovilai commented May 2, 2022 •

edited

Loading