Skip to content

Add extra services checks to kuberhealthy #2978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 20, 2025

Conversation

viniciusdc
Copy link
Contributor

@viniciusdc viniciusdc commented Mar 6, 2025

Reference Issues or PRs

closes #2976

What does this implement/fix?

  • add extra checks for argo, dask-gateway and grafana,

Put a x in the boxes that apply

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds a feature)
  • Breaking change (fix or feature that would cause existing features not to work as expected)
  • Documentation Update
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no API changes)
  • Build related changes
  • Other (please describe):

Testing

  • Did you test the pull request locally?
  • Did you add new tests?

How to test this PR?

Any other comments?

@marcelovilla
Copy link
Member

@viniciusdc I haven't reviewed this PR but we probably want to update the Summary of available healthchecks in our docs.

@dcmcand dcmcand added this to the 2025.4.1 milestone Mar 11, 2025
@viniciusdc viniciusdc added the needs: review 👀 This PR is complete and ready for reviewing label Mar 11, 2025
@marcelovilla marcelovilla self-assigned this Mar 17, 2025
Copy link
Member

@marcelovilla marcelovilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks @viniciusdc 🚀

I'm wondering whether we should still have

def check(
self, stage_outputs: Dict[str, Dict[str, Any]], disable_prompt: bool = False
):
directory = "stages/07-kubernetes-services"
import requests
# suppress insecure warnings
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
def _attempt_connect_url(
url, verify=False, num_attempts=NUM_ATTEMPTS, timeout=TIMEOUT
):
for i in range(num_attempts):
response = requests.get(url, verify=verify, timeout=timeout)
if response.status_code < 400:
print(f"Attempt {i+1} health check succeeded for url={url}")
return True
else:
print(f"Attempt {i+1} health check failed for url={url}")
time.sleep(timeout)
return False
services = stage_outputs[directory]["service_urls"]["value"]
for service_name, service in services.items():
service_url = service["health_url"]
if service_url and not _attempt_connect_url(service_url):
print(
f"ERROR: Service {service_name} DOWN when checking url={service_url}"
)
sys.exit(1)
now that healthchecks are implemented using Kuberhealthy.

@marcelovilla marcelovilla moved this from New 🚦 to In review/QA 👀 in 🪴 Nebari Project Management Mar 19, 2025
@marcelovilla marcelovilla moved this from In review/QA 👀 to Done 💪🏾 in 🪴 Nebari Project Management Mar 19, 2025
@marcelovilla marcelovilla merged commit b25ceea into main Mar 20, 2025
40 of 41 checks passed
@marcelovilla marcelovilla deleted the 2976-enh--favor-kuberhealthy-over-pytest branch March 20, 2025 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs: review 👀 This PR is complete and ready for reviewing
Projects
Status: Done 💪🏾
Development

Successfully merging this pull request may close these issues.

[ENH] - Favor Kuberhealthy over pytest integration tests for service health checks
3 participants