VMware cluster states + dependencies #43645

alexbleotu · 2017-09-20T10:43:04Z

What does this PR do?

Adds 3 states and all required dependencies (+ a missing external VSAN library from VMware) to be used with the esxcluster proxy to configure a VMware cluster, rename the cluster's datastore, assign a VSAN license to the cluster

Functions added:

State functions:

cluster_configured state to create/manage ESX clusters
vsan_datastore_configured esxcluster state that renames the vSan datastore on a cluster
licenses_configured state that checks the the VSAN licenses are added to the vCenter and

Execution functions

salt.modules.vspere.list_datastores_via_proxy that returns a list of datastore representations
salt.modules.vspere.rename_datastore that renames a datastore
salt.modules.vsphere.list_licenses that lists all licenses on a vCenter
salt.modules.vsphere.add_license
salt.modules.vsphere._get_entity that describes a cluster and a vcenter
salt.modules.vsphere._validate_entity that validates a cluster and a vcenter description
salt.modules.vsphere.list_assigned_licenses that list the licenses assigned to an entity
salt.modules.vsphere.assign_license that assigns a license to an entity

Utils functions:

salt.utils.vmware.get_storage_system that retrieves an ESXi host's storage system
salt.utils.vmware.get_datastores which returns a list of vim.Datastore objects
salt.utils.vmware.rename_datastore
salt.utils.vmware.get_license_manager
salt.utils.vmware.get_license_assignment_manager
salt.utils.vmware.get_licenses
salt.utils.vmware.add_license
salt.utils.vmware.get_assigned_licenses
salt.utils.vmware.assign_license

Tests written?

Yes (utils functions only)

Please review Salt's Contributing Guide for best practices.

… cluster using the new endpoint

…nter using the new endpoint

…s in a configuration dictionary to a cluster spec

…rding to spec, including configuring VSAN

…rding to spec, including VSAN configuration

…ompares dictionaries

…n datastore on a cluster

…t's storage system

…tastore objects - can retrieve the datastores visible to the root folder, a datacenter, a cluster and an ESXi host - can filter on datastore name and backing disk id (the latter, only if it is a VMFS datastore and the reference is an ESXi host)

…st of datastore representations - can filter on datastore name, backing disk id, backing_disk_scsi_addresses (the last two only apply to VMFS datastore and if the proxy ``esxi`` - pointing to an an ESXi host)

…and a vcenter description

…ses assigned to an entity

…n entity

… added to the vCenter and assigned to the cluster

ghost · 2017-09-20T10:43:09Z

@alexbleotu, thanks for your PR! By analyzing the history of the files in this pull request, we identified @rallytime, @cro and @s0undt3ch to be potential reviewers.

cachedout

Fantastic! Let's start with the pylint fixes and then we'll start reviewing this in-depth. Thanks @alexbleotu!

https://jenkins.saltstack.com/job/PR/job/salt-pr-lint-n/14781/violations/

cachedout

This is one of the best PRs I've seen in a long while. Great job, @alexbleotu. Just a few small comments in-line.

cachedout · 2017-09-20T19:43:09Z

salt/config/schemas/esxcluster.py

+    admission_control_policy = AdmissionControlPolicyItem()
+    default_vm_settings = DefaultVmSettingsItem()
+    hb_ds_candidate_policy = StringItem(
+        title='Hartbeat Datastore Candidate Policy',


Heartbeat?

Good catch!

cachedout · 2017-09-20T20:02:18Z

salt/states/esxcluster.py

+    Python 2.7.9, or newer must be present. This is due to an upstream dependency
+    in pyVmomi 6.0 that is not supported in Python versions 2.7 to 2.7.8. If the
+    version of Python is not in the supported range, you will need to install an
+    earlier version of pyVmomi. See `Issue #29537`_ for more information.


Fair enough but are we detecting this condition and warning the user at all? It's a bit much to ask them to read the docs here to determine what could be a very hard-to-detect condition. I wonder if we could improve this at all.

I am not sure how to do this though. The only thing I can think of is warnings in tests if there is an incompatibility, but there is no guarantee that everyone will run the tests.

Can we detect this in __virtual__() and return False, 'State module did not load: Incompatible versions of Python and pyVmomi present', or something along those lines?

Also, this is minor, but I think the mention of Python 2.6 can be removed to reduce potential confusion. Support for Python 2.6 in salt has been removed as of the 2017.7.0 release.

We should be able to. This needs to happen in all states though and the vsphere execution module. I will add it in this one to esxcluster and I will create a separate PR to fix the rest

Ok, I have a change - not as trivial as expected. Testing it out now. Question is if we should add it in the state files or just in the vsphere execution module where pyVmomi is actually used. @rallytime, @cachedout what do you think?

Ok did both. Let's see if the tests pass. From what I remember they don't actually test the __virtual__ function

cachedout · 2017-09-20T20:04:22Z

salt/states/esxcluster.py

+    WARNING: vsan datastores will not exist until there is at least on host in
+    the cluster; the state assumes that the datastore exists and errors out if i
+    it doesn't; it's up to the user to accept the error or enable the state run
+    when de datastore does exist (grain: vsan_datastore_exists)


de? the perhaps?

There are several typos; will fix ;)

alexbleotu · 2017-09-21T14:56:40Z

#43674 depends on this

cachedout · 2017-09-21T21:41:16Z

@rallytime Re-review please

rallytime

Well done! And thank you for adding tests, too. :)

Alexandru Bleotu added 30 commits September 20, 2017 06:24

Added external VMware VSAN libraries

fccc4af

Added utils.vsan.reconfigure_cluster_vsan that reconfigures VSAN on a…

6dbf784

… cluster using the new endpoint

Added tests for salt.utils.vsan.reconfigure_cluster_vsan

6a86104

Added utils.vsan._wait_for_tasks that waits for VSAN tasks on the vCe…

f796885

…nter using the new endpoint

Added tests for utils.vsan._wait_for_tasks

0db8cba

Added salt.modules.vsphere._apply_cluster_dict that applies the value…

d693820

…s in a configuration dictionary to a cluster spec

Added salt.modules.vsphere.create_cluster that creates a cluster acco…

345b783

…rding to spec, including configuring VSAN

Added salt.modules.vsphere.update_cluster that updates a cluster acco…

01f2410

…rding to spec, including VSAN configuration

Added RecursiveDictDiffer in salt.utils.dictdiffer that recursively c…

ca9235f

…ompares dictionaries

Added tests for RecursiveDictDiffer

81a02f4

Added ListDictDiffer that recursively compares lists of dictionaries

e1822b4

Added tests for list_diff

f042334

Added JSON schema defintions ESX cluster config and all dependencies

3590699

Added cluster_configured state to create/manage ESX clusters

08100d7

Added salt.states.esxcluster._get_vsan_datastore

26207e2

Added vsan_datastore_configured esxcluster state that renames the vSa…

85f328f

…n datastore on a cluster

Added salt.utils.vmware.get_storage_system that retrieves an ESXi hos…

0a5e506

…t's storage system

Added tests for salt.utils.vmware.get_storage_system

457246a

Added tests for salt.utils.vmware.get_datastores

51e881e

Added salt.utils.vmware.rename_datastore

792f4e9

Added tests for salt.utils.vmware.rename_datastore

643f38c

Added salt.modules.vspere.list_datastores_via_proxy that returns a li…

0949e57

…st of datastore representations - can filter on datastore name, backing disk id, backing_disk_scsi_addresses (the last two only apply to VMFS datastore and if the proxy ``esxi`` - pointing to an an ESXi host)

Added salt.modules.vspere.rename_datastore that renames a datastore

b027579

Added tests for salt.modules.vspere.rename_datastore

71b10b8

Added ESXClusterEntitySchema and LicenseSchema JSON schemas

693d234

Added VCenterEntitySchema JSON schema

4394601

Added salt.utils.vmware.get_license_manager

ee47cd0

Added tests for salt.utils.vmware.get_license_manager

9f566fb

Added salt.utils.vmware.get_license_assignment_manager

1d57000

Alexandru Bleotu added 4 commits September 20, 2017 06:29

Added salt.modules.vsphere._validate_entity that validates a cluster …

25ac440

…and a vcenter description

Added salt.modules.vsphere.list_assigned_licenses that list the licen…

4656f26

…ses assigned to an entity

Added salt.modules.vsphere.assign_license that assigns a license to a…

d0a7946

…n entity

Added licenses_configured state that checks the the VSAN licenses are…

7bb5443

… added to the vCenter and assigned to the cluster

Alexandru Bleotu added 4 commits September 20, 2017 09:32

Skip ext vsan libraries pylint checks

9f0d639

Moved ignore_unset_values parameter as a RecursiveDictDiffer attribute

f7f4f34

Fixed tests for the ignore_unset_values change

4f4db43

pylint

8cdaf01

cachedout suggested changes Sep 20, 2017

View reviewed changes

Alexandru Bleotu added 2 commits September 20, 2017 13:40

more pylint

a41b507

Fixed list_differ because of ignore_unset_values change in dictdiffer

aa26f0b

cachedout reviewed Sep 20, 2017

View reviewed changes

Alexandru Bleotu added 6 commits September 20, 2017 17:02

Added InvalidEntityError in salt.exceptions

819e015

Even more pylint

23f58c0

Review changes

491faaa

Imported salt.exceptions in salt.states.esxcluster

1706c24

More pylint

ee0dcd7

Fixed except statement

cbefa4d

alexbleotu mentioned this pull request Sep 21, 2017

VMware vCenter proxy + storage policies states & dependencies #43674

Merged

Alexandru Bleotu added 3 commits September 21, 2017 12:31

Added python/pyvmomi compatibility check to salt.states.esxcluster

5f30477

Added python/pyvmomi compatibility check to salt.modules.vsphere

2f55841

Removed references to Pthon 2.6 as it's no longer supported in Salt

f46afa0

alexbleotu force-pushed the cluster_states-gh branch from 7014b53 to f46afa0 Compare September 21, 2017 16:32

cachedout approved these changes Sep 21, 2017

View reviewed changes

rallytime approved these changes Sep 22, 2017

View reviewed changes

cachedout merged commit 75dd97e into saltstack:develop Sep 22, 2017

alexbleotu mentioned this pull request Sep 22, 2017

DVS states + dependencies #43604

Merged

VMware cluster states + dependencies #43645

VMware cluster states + dependencies #43645

Uh oh!

Conversation

alexbleotu commented Sep 20, 2017

What does this PR do?

Tests written?

Uh oh!

ghost commented Sep 20, 2017

Uh oh!

cachedout left a comment

Choose a reason for hiding this comment

Uh oh!

cachedout left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexbleotu commented Sep 21, 2017

Uh oh!

cachedout commented Sep 21, 2017

Uh oh!

rallytime left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants