Submission worker must be able to recover from DB connection failure on startup

[erickpeirson]

Here is a sample start-up log:

arxiv-submission-worker   | /opt/arxiv/agent/config.py:52: UserWarning: Vault integration is disabled
arxiv-submission-worker   |   warnings.warn('Vault integration is disabled')
arxiv-submission-worker   | /opt/arxiv/agent/config.py:180: UserWarning: Certificate verification for Kinesis is disabled; this should not be disabled in production.
arxiv-submission-worker   |   warnings.warn('Certificate verification for Kinesis is disabled; this'
arxiv-submission-worker   | /opt/arxiv/agent/config.py:305: UserWarning: Certificate verification for compiler is disabled; this should not be disabled in production.
arxiv-submission-worker   |   warnings.warn('Certificate verification for compiler is disabled; this'
arxiv-submission-worker   | /opt/arxiv/agent/config.py:382: UserWarning: Certificate verification for plaintext extraction service is disabled; this should not be disabled in production.
arxiv-submission-worker   |   warnings.warn('Certificate verification for plaintext extraction service'
arxiv-submission-worker   | /opt/arxiv/agent/config.py:414: UserWarning: Certificate verification for SMTP is disabled; this should not be disabled in production.
arxiv-submission-worker   |   warnings.warn('Certificate verification for SMTP is disabled; this'
arxiv-submission-worker   | Traceback (most recent call last):
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
arxiv-submission-worker   |     return fn()
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect
arxiv-submission-worker   |     return _ConnectionFairy._checkout(self)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
arxiv-submission-worker   |     fairy = _ConnectionRecord.checkout(pool)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
arxiv-submission-worker   |     rec = pool._do_get()
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
arxiv-submission-worker   |     self._dec_overflow()
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
arxiv-submission-worker   |     compat.reraise(exc_type, exc_value, exc_tb)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 154, in reraise
arxiv-submission-worker   |     raise value
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
arxiv-submission-worker   |     return self._create_connection()
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
arxiv-submission-worker   |     return _ConnectionRecord(self)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
arxiv-submission-worker   |     self.__connect(first_connect_check=True)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
arxiv-submission-worker   |     connection = pool._invoke_creator(self)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
arxiv-submission-worker   |     return dialect.connect(*cargs, **cparams)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 451, in connect
arxiv-submission-worker   |     return self.dbapi.connect(*cargs, **cparams)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect
arxiv-submission-worker   |     return Connection(*args, **kwargs)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/MySQLdb/connections.py", line 204, in __init__
arxiv-submission-worker   |     super(Connection, self).__init__(*args, **kwargs2)
arxiv-submission-worker   | _mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'submission-agent-maria' (111)")
arxiv-submission-worker   | 
arxiv-submission-worker   | The above exception was the direct cause of the following exception:
arxiv-submission-worker   | 
arxiv-submission-worker   | Traceback (most recent call last):
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/bin/celery", line 10, in <module>
arxiv-submission-worker   |     sys.exit(main())
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/celery/__main__.py", line 14, in main
arxiv-submission-worker   |     _main()
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/celery/bin/celery.py", line 326, in main
arxiv-submission-worker   |     cmd.execute_from_commandline(argv)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
arxiv-submission-worker   |     super(CeleryCommand, self).execute_from_commandline(argv)))
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/celery/bin/base.py", line 279, in execute_from_commandline
arxiv-submission-worker   |     argv = self.setup_app_from_commandline(argv)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/celery/bin/base.py", line 481, in setup_app_from_commandline
arxiv-submission-worker   |     self.app = self.find_app(app)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/celery/bin/base.py", line 503, in find_app
arxiv-submission-worker   |     return find_app(app, symbol_by_name=self.symbol_by_name)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/celery/app/utils.py", line 355, in find_app
arxiv-submission-worker   |     sym = symbol_by_name(app, imp=imp)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/celery/bin/base.py", line 506, in symbol_by_name
arxiv-submission-worker   |     return imports.symbol_by_name(name, imp=imp)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
arxiv-submission-worker   |     module = imp(module_name, package=package, **kwargs)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
arxiv-submission-worker   |     return imp(module, package=package)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib64/python3.6/importlib/__init__.py", line 126, in import_module
arxiv-submission-worker   |     return _bootstrap._gcd_import(name[level:], package, level)
arxiv-submission-worker   |   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
arxiv-submission-worker   |   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
arxiv-submission-worker   |   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
arxiv-submission-worker   |   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
arxiv-submission-worker   |   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
arxiv-submission-worker   |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
arxiv-submission-worker   |   File "/opt/arxiv/agent/worker.py", line 25, in <module>
arxiv-submission-worker   |     app = create_app()
arxiv-submission-worker   |   File "/opt/arxiv/agent/factory.py", line 77, in create_app
arxiv-submission-worker   |     database.create_all()
arxiv-submission-worker   |   File "/opt/arxiv/agent/services/database.py", line 79, in create_all
arxiv-submission-worker   |     db.create_all(bind='agent')
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py", line 1033, in create_all
arxiv-submission-worker   |     self._execute_for_all_tables(app, bind, 'create_all')
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py", line 1025, in _execute_for_all_tables
arxiv-submission-worker   |     op(bind=self.get_engine(app, bind), **extra)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4287, in create_all
arxiv-submission-worker   |     ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2032, in _run_visitor
arxiv-submission-worker   |     with self._optional_conn_ctx_manager(connection) as conn:
arxiv-submission-worker   |   File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
arxiv-submission-worker   |     return next(self.gen)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2024, in _optional_conn_ctx_manager
arxiv-submission-worker   |     with self._contextual_connect() as conn:
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
arxiv-submission-worker   |     self._wrap_pool_connect(self.pool.connect, None),
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
arxiv-submission-worker   |     e, dialect, self
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
arxiv-submission-worker   |     util.raise_from_cause(sqlalchemy_exception, exc_info)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 399, in raise_from_cause
arxiv-submission-worker   |     reraise(type(exception), exception, tb=exc_tb, cause=cause)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
arxiv-submission-worker   |     raise value.with_traceback(tb)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
arxiv-submission-worker   |     return fn()
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect
arxiv-submission-worker   |     return _ConnectionFairy._checkout(self)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
arxiv-submission-worker   |     fairy = _ConnectionRecord.checkout(pool)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
arxiv-submission-worker   |     rec = pool._do_get()
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
arxiv-submission-worker   |     self._dec_overflow()
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
arxiv-submission-worker   |     compat.reraise(exc_type, exc_value, exc_tb)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 154, in reraise
arxiv-submission-worker   |     raise value
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
arxiv-submission-worker   |     return self._create_connection()
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
arxiv-submission-worker   |     return _ConnectionRecord(self)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
arxiv-submission-worker   |     self.__connect(first_connect_check=True)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
arxiv-submission-worker   |     connection = pool._invoke_creator(self)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
arxiv-submission-worker   |     return dialect.connect(*cargs, **cparams)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 451, in connect
arxiv-submission-worker   |     return self.dbapi.connect(*cargs, **cparams)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect
arxiv-submission-worker   |     return Connection(*args, **kwargs)
arxiv-submission-worker   |   File "/root/.local/share/virtualenvs/arxiv-AI0WzCV4/lib/python3.6/site-packages/MySQLdb/connections.py", line 204, in __init__
arxiv-submission-worker   |     super(Connection, self).__init__(*args, **kwargs2)
arxiv-submission-worker   | sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'submission-agent-maria' (111)")
arxiv-submission-worker   | (Background on this error at: http://sqlalche.me/e/e3q8)
arxiv-submission-worker exited with code 1