-
Notifications
You must be signed in to change notification settings - Fork 47
(QENG-5305) Improve vmpooler host selection #242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
04cb1da
4d7e9c7
cf2f9fa
7c71311
2e3e83a
dc8067a
c602421
5a19ef5
f47b804
35817fa
33bdf66
6bf2eb7
0472559
f75d4f6
dd12125
1a01855
b5fd0f7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -224,10 +224,9 @@ def _clone_vm(pool, provider) | |
| $logger.log('s', "[+] [#{pool_name}] '#{new_vmname}' cloned in #{finish} seconds") | ||
|
|
||
| $metrics.timing("clone.#{pool_name}", finish) | ||
| rescue => err | ||
| $logger.log('s', "[!] [#{pool_name}] '#{new_vmname}' clone failed with an error: #{err}") | ||
| rescue => _err | ||
| $redis.srem('vmpooler__pending__' + pool_name, new_vmname) | ||
| raise | ||
| raise _err | ||
| ensure | ||
| $redis.decr('vmpooler__tasks__clone') | ||
| end | ||
|
|
@@ -459,69 +458,17 @@ def _check_snapshot_queue | |
| end | ||
| end | ||
|
|
||
| def migration_limit(migration_limit) | ||
| # Returns migration_limit setting when enabled | ||
| return false if migration_limit == 0 || !migration_limit # rubocop:disable Style/NumericPredicate | ||
| migration_limit if migration_limit >= 1 | ||
| end | ||
|
|
||
| def migrate_vm(vm_name, pool_name, provider) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There is currently no way in Pool Manager to actually add a VM to the migration queue. So this PR will not have any affect yet. Is this by design?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No, it's an oversight. It's fixed now. |
||
| Thread.new do | ||
| begin | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The provider really shouldn't need to contact redis for this. In this case the provider is just removing it from the queue. So the
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I took your suggestion and placed this prior to calling |
||
| _migrate_vm(vm_name, pool_name, provider) | ||
| $redis.srem("vmpooler__migrating__#{pool_name}", vm_name) | ||
| provider.migrate_vm(pool_name, vm_name) | ||
| rescue => err | ||
| $logger.log('s', "[x] [#{pool_name}] '#{vm_name}' migration failed with an error: #{err}") | ||
| remove_vmpooler_migration_vm(pool_name, vm_name) | ||
| end | ||
| end | ||
| end | ||
|
|
||
| def _migrate_vm(vm_name, pool_name, provider) | ||
| $redis.srem('vmpooler__migrating__' + pool_name, vm_name) | ||
|
|
||
| parent_host_name = provider.get_vm_host(pool_name, vm_name) | ||
| raise('Unable to determine which host the VM is running on') if parent_host_name.nil? | ||
| migration_limit = migration_limit $config[:config]['migration_limit'] | ||
| migration_count = $redis.scard('vmpooler__migration') | ||
|
|
||
| if !migration_limit | ||
| $logger.log('s', "[ ] [#{pool_name}] '#{vm_name}' is running on #{parent_host_name}") | ||
| return | ||
| elsif migration_count >= migration_limit | ||
| $logger.log('s', "[ ] [#{pool_name}] '#{vm_name}' is running on #{parent_host_name}. No migration will be evaluated since the migration_limit has been reached") | ||
| return | ||
| else | ||
| $redis.sadd('vmpooler__migration', vm_name) | ||
| host_name = provider.find_least_used_compatible_host(pool_name, vm_name) | ||
| if host_name == parent_host_name | ||
| $logger.log('s', "[ ] [#{pool_name}] No migration required for '#{vm_name}' running on #{parent_host_name}") | ||
| else | ||
| finish = migrate_vm_and_record_timing(vm_name, pool_name, parent_host_name, host_name, provider) | ||
| $logger.log('s', "[>] [#{pool_name}] '#{vm_name}' migrated from #{parent_host_name} to #{host_name} in #{finish} seconds") | ||
| end | ||
| remove_vmpooler_migration_vm(pool_name, vm_name) | ||
| end | ||
| end | ||
|
|
||
| def remove_vmpooler_migration_vm(pool, vm) | ||
| $redis.srem('vmpooler__migration', vm) | ||
| rescue => err | ||
| $logger.log('s', "[x] [#{pool}] '#{vm}' removal from vmpooler__migration failed with an error: #{err}") | ||
| end | ||
|
|
||
| def migrate_vm_and_record_timing(vm_name, pool_name, source_host_name, dest_host_name, provider) | ||
| start = Time.now | ||
| provider.migrate_vm_to_host(pool_name, vm_name, dest_host_name) | ||
| finish = format('%.2f', Time.now - start) | ||
| $metrics.timing("migrate.#{pool_name}", finish) | ||
| $metrics.increment("migrate_from.#{source_host_name}") | ||
| $metrics.increment("migrate_to.#{dest_host_name}") | ||
| checkout_to_migration = format('%.2f', Time.now - Time.parse($redis.hget("vmpooler__vm__#{vm_name}", 'checkout'))) | ||
| $redis.hset("vmpooler__vm__#{vm_name}", 'migration_time', finish) | ||
| $redis.hset("vmpooler__vm__#{vm_name}", 'checkout_to_migration', checkout_to_migration) | ||
| finish | ||
| end | ||
|
|
||
| # Helper method mainly used for unit testing | ||
| def time_passed?(_event, time) | ||
| Time.now > time | ||
|
|
@@ -764,9 +711,6 @@ def _check_pool(pool, provider) | |
| end | ||
|
|
||
| pool_check_response | ||
| rescue => err | ||
| $logger.log('d', "[!] [#{pool['name']}] _check_pool failed with an error: #{err}") | ||
| raise | ||
| end | ||
|
|
||
| # Create a provider object, usually based on the providers/*.rb class, that implements providers/base.rb | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As the migration limit is now a per vSphere provider setting the documentation should be updated too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It probably should be a per vSphere provider setting, but it's still tracking migrations in redis and getting the setting from the global configuration section instead of checking the provider for this setting. I could add this change here. I was considering tackling that and the task_limit concept in a future change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could be as simple as add in the docs
migration limit (Only affects vSphere Provider)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh I see what you're saying, not that it's a per provider setting, but rather that it is only a vsphere provider setting.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah... The docs suggest its global but no longer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added a mention in the example configuration file. I couldn't find another reference to migration_limit, please let me know if I've missed another location this should go.