Heartbeat Manager

[dgasmith]

Description

This PR adds manager heartbeats to the project. This allows jobs that have been allocated to managers that do not shut down properly to be recycled and given to other managers.

The heartbeat interval is set by the FractalServer instance and passed to the managers to ensure consistency.

Todos

• Adds heartbeats, closes Manager Heartbeats #95.
• Allows shutdown of CLI invocations via SIGINT and SIGTERM to have a better chance of a clean shutdown, closes Safe kill when run as background thread #90.
• A new FractalClient requests some data from the server to ensure the connection is live. Error messages should be slightly more explicit now.
• When a FractalServer owns a Manager, all manager REST queries are now run in a background thread so that communication is not blocked. This is not exactly an ideal solution as we need to continuously build threads. There are plenty of ways to improve in the future, but the current version works and is relatively robust.

Status

• Ready to go

[QCArchiveBot]

This pull request introduces 3 alerts when merging 961923a into 75c8ccf - view on LGTM.com

new alerts:

• 1 for Module is imported more than once
• 1 for Unused local variable
• 1 for Unused import

---

Comment posted by LGTM.com

[codecov]

Codecov Report

Merging #106 into master will decrease coverage by 12.91%.
The diff coverage is 88.1%.

[codecov]

Codecov Report

Merging #106 into master will decrease coverage by 0.21%.
The diff coverage is 88.29%.

[dgasmith]

PyTest and cov will fail due to:
pytest-dev/pytest-cov#253
conda-forge/pytest-cov-feedstock#14

Waiting another few hours should see it resolved.

[dgasmith]

If anyone has a moment to review this let me know, otherwise merging sometime tonight.