Add support for delete and recreate option to `ExistingResourcePolicy` feature

[shubham-pampattiwar]

Describe the problem/challenge you have

ExistingResourcePolicy feature was implemented in velero 1.9 with only 2 options - none and update We would like to complete the phase 2 of this feature's implementation i.e also provide the delete and recreate option

Design link: #4613

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "The project would be better with this feature added"
• 👎 for "This feature will not enhance the project in a meaningful way"

[sseago]

Current location of the design is here: https://github.com/vmware-tanzu/velero/blob/main/design/Implemented/existing-resource-policy_design.md

Note that there are two aspects of the second implementation phase:

1. add the recreate option (in addition to none and update
2. implement the "approach 3" from the design -- add a second spec field, existingResourcePolicyOverrides to allow per-resource-type overrides to the otherwise-defined existingResourcePolicy

The second one is important here -- since recreate is potentially destructive (it's possible that velero will delete a cluster resource and be unable to recreate it), users may want to limit its use to resource types they know are safe. They may set existingResourcePolicy to update but set a recreate override for pods.

Also, as an implementation comment, I think we should only delete/recreate in cases where update fails. So even if policy is recreate, we first attempt update, and if that fails, we recreate.

[kaovilai]

👋

[reasonerjt]

Per comments in the PR seems there's no consensus on the implementation, we may move this out of v1.12