vmpooler doesn't seem to recognize ready VMs

[fuero]

Vmpooler doesn't seem to correctly find VMware VMs:

[2017-04-21 20:52:10] [*] [centos-7] starting worker thread
[2017-04-21 20:52:12] [!] [centos-7] 'vmp_r14lejs7ljrsee2' not found in inventory, removed from 'completed' queue
[2017-04-21 20:52:15] [ ] [VMP_CentOS7_3] 'vmp_s306w9dbnqrnil6' is being cloned from 'VMP_CentOS7_3'
[2017-04-21 20:52:39] [+] [VMP_CentOS7_3] 'vmp_s306w9dbnqrnil6' cloned from 'VMP_CentOS7_3' in 23.59 seconds
[2017-04-21 20:52:39] [?] [centos-7] 'vmp_s306w9dbnqrnil6' added to 'discovered' queue
[2017-04-21 20:52:41] [ ] [VMP_CentOS7_3] 'vmp_tm3m2ew577l397j' is being cloned from 'VMP_CentOS7_3'
[2017-04-21 20:53:03] [+] [VMP_CentOS7_3] 'vmp_tm3m2ew577l397j' cloned from 'VMP_CentOS7_3' in 22.87 seconds
[2017-04-21 20:53:04] [?] [centos-7] 'vmp_tm3m2ew577l397j' added to 'discovered' queue
[2017-04-21 20:53:04] [ ] [centos-7] 'vmp_s306w9dbnqrnil6' is being shut down
[2017-04-21 20:53:04] [-] [centos-7] 'vmp_s306w9dbnqrnil6' destroyed in 0.50 seconds

As far as I can tell, it doesn't even attempt to verify if a VM was successfully cloned and is accessible
on the net.

The docs don't go into detail on how the template VM should be set up and what should be provided to make this work.

What I gathered:

• it needs a VM, not a VM template, so that's quite misleading when VMware offers VM templates.
• the VMs need IPs assigned by DHCP.
• VMware Tools or Open VM Tools need to be installed(?).
• vmpooler needs to have SSH access to the VMs.

Here's the (redacted) config I used:

vmpooler.yaml

---
:vsphere:
  server: vcsa.local
  username: vmpooler@vsphere.local
  password: secret
:auth:
  provider: dummy
:redis:
  server: localhost
:config:
  logfile: "/tmp/vmpooler.log"
  prefix: vmp_
  timeout: 120
:pools:
  - name: centos-7
    template:   TEMPLATES/VMP_CentOS7_3
    folder:     "Pooled VMs/CentOS7_3"
    pool:       vmpooler
    datastore:  VMW/A_DATASTORE
    clone_target: A_CLUSTER
    size:       1

[glennsarti]

VMPooler will also need to be able to resolve the hostnames as well.

Also what git SHA is this running on?

[fuero]

SHA1: e3cda9d

VMPooler will also need to be able to resolve the hostnames as well.

As VMPooler generates its hostnames, how do you get this information to the DNS?
Wouldn't it be easier to take the IP address gathered from the VMware tools to connect?

Updated again

Instructions for setting up your template VM, DHCP and DNS:

• Provide a DNS Server & DHCP (E.g. dnsmasq).
• Set up your DHCP that it updates the DNS server with the DHCP client hostname (OOTB behaviour with dnsmasq)
• Set up your vmpooler host to use this DNS server.
• Install Open VM Tools or VMware Tools in your template VM.
• The DHCP should provide option 12 to have the VMs set their hostnames or set the VM's hostname in a different manner.
• Set up the VM to send the hostname passed from guestinfo data from the hypervisor to the DHCP Server. Example for CentOS/RHEL, set up your primary interface's network-script like this:

  ONBOOT=yes
  BOOTPROTO="dhcp"
  DHCP_HOSTNAME="$(/usr/bin/vmtoolsd --cmd 'info-get guestinfo.hostname')"
  

[glennsarti]

Correct. The DHCP server can do the DNS updates on the client's behalf.

Getting the IP address from guest via Tools will also work and I can definitely see that as a use-case. I didn't write the original code so I'm taking a bit of a guess at the reason why this is the way it is:

1. It's simpler to query DNS than querying vSphere for Tools information (DNS client vs RBVMOMI)
2. Additional dependency that all VMs must have functioning Tools install
3. Querying vSphere is expensive and the less number of calls the better
4. It emulates closer to what a user/client would experience i.e. Once a user requests a VM, they will then try to connect to it via DNS name

Point 4 seems to be the strongest argument, in my mind. The bulk of those calls is to ensure the VM is in a ready state, so it makes sense that the VM is ready from the point of view of a user/client.

That said. The VMPooler documentation needs to be updated (or even created!) to state this is a requirement.

[fuero]

I've updated to your PR head (b0b7fef), the wacky VM cloning/trashing behaviour is gone, as expected from a total rewrite. I've moved the config, cloning VMs and connecting with SSH works with the test my setup, as evidenced by a few extra logging statements.

Nonetheless, it doesn't seem that the VM is running, it continuously tries to connect to it despite the connection working.

[fuero]

Additional dependency that all VMs must have functioning Tools install

You need that anyway if I'm not mistaken, as you need to be able to query the hostname vmpooler generated in the VM to update your DNS.

[glennsarti]

Oh yeah...forgot about that hostname check.

Oh I wonder if it's a FQDN issue. I had a problem in my setup where the FQDN was registered in DNS, but the shortname wasn't (or my DNS search suffix list was incomplete)

Pooler only queries DNS for the short name.

[glennsarti]

Oh while my PR does look like a total rewrite, it's not. It's mainly a mass refactor. Most of the vSphere and Pool_Manager code is intact, but segregated into more siloed areas of concern.

[fuero]

Ok, so I got it to work.

What should be checked for by vmpooler:

• config.prefix may not contain non-RFC 952/1123 compliant characters

What to note when setting this up:

• The DHCP should provide option 12 to have the VMs set their hostnames or set the VM's hostname in a different manner.

Example dnsmasq config:

# 192.168.1.1 < your router
# 192.168.1.2 < dnsmasq
# 8.8.8.8, 8.8.4.4 < Google's DNS servers, replace with your own

server=8.8.8.8
server=8.8.4.4
local=/local/
expand-hosts
domain=local
dhcp-range=192.168.1.100,192.168.1.150,12h
dhcp-option=option:router,192.168.1.1
dhcp-option=option:dns-server,192.168.1.2,8.8.8.8,8.8.4.4

[fuero]

Found a small bug making the API choke on hostnames containing a dash:

lib/vmpooler/api/helpers.rb:120 reads:

if domain && hostname =~ /^\w+\.#{domain}$/

but should be:

if domain && hostname =~ /^[a-zA-Z0-9-]+\.#{domain}$/

[mattkirby]

Moved API hostname regex bug to a new issue and will close this one. I'm happy to review a fix for #282 if you'd like to put up a PR.