ESXi diskgroup and host cache creation/configuration + dependencies

[alexbleotu]

What does this PR do?

This PR adds the ability to:

• add/manage VSAN diskgroups on ESXi hosts either via the diskgroup_configured state or via execution functions.
• add capacity disks to diskgroups (done automatically by the diskgroup_configured state)
• create VMFS datastores and assign them host caches on the ESXi hosts; the host caches can have a fixed size or a percentage of capacity of the backing datastores
• remove capacity disks from diskgroups, entire diskgroups, datastores or erase partitions on disks via execution functions only (state functions don't allow that)

Proxy functions:

• Added ability to esxi proxy to connect via the vCenter (not only directly to the ESXi host) - it requres a slightly different proxy config key

Utils functions:

• salt.utils.vsan.get_vsan_disk_management_system
• salt.utils.vsan.get_host_vsan_system
• salt.utils.vsan.create_diskgroup
• salt.utils.vsan.add_capacity_to_diskgroup
• salt.utils.vsan.remove_capacity_from_diskgroup
• salt.utils.vsan.remove_diskgroup
• salt.utils.vmware._get_partition_info
• salt.utils.vmware._get_new_computed_partition_spec
• salt.utils.vmware.create_vmfs_datastore
• salt.utils.vmware.get_host_datastore_system
• salt.utils.vmware.remove_datastore
• Improved logic to filter hosts based on parent in salt.utils.vmware.get_hosts
• salt.utils.vmware._get_scsi_address_to_lun_key_map
• salt.utils.vmware.get_all_luns
• salt.utils.vmware.get_scsi_address_to_lun_map
• salt.utils.vmware.get_disks
• salt.utils.vmware.get_disk_partition_info
• salt.utils.vmware.erase_disk_partitions
• salt.utils.get_diskgroups
• salt.utils._check_disks_in_diskgroup
• salt.utils.vmware.get_host_cache
• salt.utils.vmware.configure_host_cache

Execution functions:

• salt.modules.list_hosts_via_proxy
• salt.modules.vsphere._get_proxy_target
• salt.modules.vsphere.list_disks
• salt.modules.vsphere.erase_disk_partitions
• salt.modules.vsphere.list_disk_partitions
• salt.modules.vsphere.list_diskgroups
• salt.modules.vsphere.create_diskgroup
• salt.modules.vsphere.add_capacity_to_diskgroup
• salt.modules.vsphere.remove_capacity_from_diskgroup
• salt.modules.vsphere.remove_diskgroup
• salt.modules.vsphere.get_host_cache
• salt.modules.vsphere.configure_host_cache
• salt.modules.vsphere.create_vmfs_datastore
• salt.modules.vsphere.remove_datastore

State functions:

• salt.states.esxi additional imports and pyVmomi/python compatibility check
• diskgroups_configured state that configures VSAN diskgroups on ESXi hosts
• host_cache_configured state that configures the host cache on ESXi hosts

Tests written?

No (will add unit tests in the next PRs - I wanted to get the main code merged in)

Please review Salt's Contributing Guide for best practices.

[cachedout]

I'm not sure I'm all that thrilled to have this sort of destructive behavior even available in Salt. Is this a "just because we can" function or is there a strong use case for this? At the very least, I would like to make sure that the defaults here do as little damage as possible.

[alexbleotu]

It is more because I would like everything to be done in salt, making the bulky vsphere client obsolete. This is not included in the state, so I consider it a manual process. Furthermore, one has to invoke salt <host_id> vsphere.erase_disk_partitions <disk_id>. I am not sure how to make it more specific than that - it assumes the user knows exactly the disk_id he/she needs and can't run it on more hosts by mistake because the ids are unique.

This command is very useful if someone has rebuilt the host and has stale partitions on the disks. I agree it is a destructive command, but an advanced user should have it available in salt.

[cachedout]

That is a reasonable point. Thanks, @alexbleotu

[alexbleotu]

Rebased

LE: pylint job needs a rerun

[cachedout]

re-run lint

[alexbleotu]

All tests & pylint pass