Ops user changes should be rolled back when attempt at VCH reconfigure fails

[jzt]

User Statement:

As a user, when I attempt to change ops user permissions during a vic-machine configure, I would expect those changes to be rolled back in the event of a failure.

Details:

Currently, an attempt to use vic-machine configure to grant permissions to an ops user does not work (see issue #7725, and fix #7777). Even with this fix, there exists a scenario in which changes to the ops user during a failing vic-machine configure persist even after a rollback is executed. Logic to undo changes to the ops user permissions should be added to the rollback step.

Acceptance Criteria:

• rollback includes logic to undo any changes to the ops user that were present during a failed attempt at vic-machine configure
• integration/unit tests to exercise this rollback logic

[zjs]

I'm going to mark this as a P1 because it's a regression in behavior since 1.3. I don't think we can reasonably fix this for 1.4 due to the complexity, but I think it's significant enough to call out as something that we should be fixing, but can't.

To restate the impact of this, the effects are twofold:

1. Following the rollback of an unsuccessful change (configure or upgrade) where additional privileges were granted to the operations user, those additional privileges are left as "cruft". This leads to a (slight) violation of the principle of least privilege as we leave the VCH with an operations user which can do more than it needs to.
2. Following the rollback of an unsuccessful change where privileges were revoked from the operations user, the VCH will not start. This would lead to a significant violation of our contract around rollback, in that we do not leave the VCH in a good state. However, currently there may not be any changes which lead to privileges being revoked. (There certainly will in the future though. See, e.g., Update vic-machine configure to support VM-Host Affinity #7561.)

[mdubya66]

Is there a workaround? Adding kind/note.

[zjs]

A high-level workaround would be for the administrator to manually adjust the permissions of the operations user back to what they "should be". That's an annoying sort of workaround though; presumably the whole reason they're using the --ops-grant-perms functionality is that they don't want to be manually fussing with the permissions.

[zjs]

Marking this as blocked by #7818. I think the easiest way to address this would be to use invoke opsuser.GrantOpsUserPerms with the old configuration. This seems simpler and less error-prone than attempting to track the previous state and then restore it.

[zjs]

Gracefully plumbing through the old configuration values seems to be the hardest part of the above idea; the rest of the change looks roughly like:

	if conf.ShouldGrantPerms() {
		err = opsuser.GrantOpsUserPerms(d.op, d.session, conf)
		if err != nil {
			return errors.Errorf("Failed to grant permissions to ops-user, failure: %s", err)
		}
+
+		defer func() {
+			if err != nil {
+				d.op.Info("Attempting to undo changes to operations user due to previous error: %s", err)
+
+				opsuser.GrantOpsUserPerms(d.op, d.session, oldConf)
+			}
+		}()
	}

[stuclem]

Release note:

• Attempts to change operations user permissions with vic-machine configure do not roll back in the event of a failure. #7814
  Changes to the operations user that are made during a failed vic-machine configure operation persist even after a rollback.

@jzt is this OK?

[jzt]

@stuclem Looks good. It may be worth noting that there is a workaround to cleanup: Browse to Administration -> Roles, and delete every role that begins with vch-. WARNING: This workaround should only be performed when there are no other virtual container hosts running in VC under the ops user, as this will remove the roles for any (all) virtual container hosts that are running in VC.