(POOLER-184) Pool manager retry and exit on failure

[sbeaulie]

Adding a reconnect retry for redis, which by default would retry 10 times,
for a total wait time of ~80 seconds

[sbeaulie]

I tested it locally by pointing to localhost but without a redis running, and it returned the error after 1min20 seconds.

So the retry works, but I cannot reproduce the initial ticket findings that it would keep running after hitting the stack trace.

root@a3389b84408e:/# VMPOOLER_DEBUG=true docker-entrypoint.sh manager
[2020-09-02 16:07:36] starting vmpooler
[2020-09-02 16:08:57] Cannot connect to the redis server: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
warning: thread "Ruby-0-Thread-1: /usr/local/bundle/gems/vmpooler-0.14.5/bin/vmpooler:43" terminated with exception (report_on_exception is true):
Redis::CannotConnectError: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
  establish_connection at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:362
               connect at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:107
        with_reconnect at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:308
               connect at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:106
      ensure_connected at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:381
               process at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:233
               logging at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:320
               process at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:232
                  call at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:126
                   set at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:848
           synchronize at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:69
       mon_synchronize at /opt/jruby/lib/ruby/stdlib/monitor.rb:235
           synchronize at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:69
                   set at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:844
              execute! at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/pool_manager.rb:1567
          with_metrics at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/generic_connection_pool.rb:29
      handle_interrupt at org/jruby/RubyThread.java:759
          with_metrics at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/generic_connection_pool.rb:28
      handle_interrupt at org/jruby/RubyThread.java:759
          with_metrics at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/generic_connection_pool.rb:21
              execute! at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/pool_manager.rb:1565
                <main> at /usr/local/bundle/gems/vmpooler-0.14.5/bin/vmpooler:44
Redis::CannotConnectError: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
  establish_connection at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:362
               connect at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:107
        with_reconnect at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:308
               connect at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:106
      ensure_connected at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:381
               process at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:233
               logging at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:320
               process at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:232
                  call at /usr/local/bundle/gems/redis-4.2.1/lib/redis/client.rb:126
                   set at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:848
           synchronize at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:69
       mon_synchronize at /opt/jruby/lib/ruby/stdlib/monitor.rb:235
           synchronize at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:69
                   set at /usr/local/bundle/gems/redis-4.2.1/lib/redis.rb:844
              execute! at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/pool_manager.rb:1567
          with_metrics at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/generic_connection_pool.rb:29
      handle_interrupt at org/jruby/RubyThread.java:759
          with_metrics at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/generic_connection_pool.rb:28
      handle_interrupt at org/jruby/RubyThread.java:759
          with_metrics at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/generic_connection_pool.rb:21
              execute! at /usr/local/bundle/gems/vmpooler-0.14.5/lib/vmpooler/pool_manager.rb:1565
                <main> at /usr/local/bundle/gems/vmpooler-0.14.5/bin/vmpooler:44
root@a3389b84408e:/# echo $?
1

[mattkirby]

I'm +1, but had a comment about preferring we move the default for 'redis_connection_attempts' to vmpooler.rb.