Skip to content

Commit bc8a3b9

Browse files
agullonagullon
committed
manual cherry-pick from openshift#4769 PR into release-4.19 branch
1 parent 0706a12 commit bc8a3b9

File tree

5 files changed

+184
-167
lines changed

5 files changed

+184
-167
lines changed

‎test/bin/ci_phase_boot_and_test.sh

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -51,12 +51,6 @@ cd "${ROOTDIR}/test"
5151
# Set up the hypervisor configuration for the tests and start webserver
5252
bash -x ./bin/manage_hypervisor_config.sh create
5353

54-
# Setup a prometheus server for metrics testing
55-
bash -x ./bin/manage_prometheus.sh start
56-
57-
# Setup a loki server for metrics testing
58-
bash -x ./bin/manage_loki.sh start
59-
6054
# Setup a container registry and mirror images.
6155
bash -x ./bin/mirror_registry.sh
6256

‎test/resources/observability.resource

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
*** Settings ***
2+
Documentation Keywords common to observability test suites
3+
4+
5+
*** Keywords ***
6+
Start Prometheus Server
7+
[Documentation] Start a Prometheus Server on port 9092
8+
Local Command Should Work bash -x ./bin/manage_prometheus.sh start
9+
10+
Stop Prometheus Server
11+
[Documentation] Stop the Prometheus Server
12+
Local Command Should Work bash -x ./bin/manage_prometheus.sh stop
13+
14+
Start Loki Server
15+
[Documentation] Start a Loki Server on port 3100
16+
Local Command Should Work bash -x ./bin/manage_loki.sh start
17+
18+
Stop Loki Server
19+
[Documentation] Stop the Loki Server
20+
Local Command Should Work bash -x ./bin/manage_loki.sh stop

‎test/resources/prometheus.py

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
import libipv6
44

55

6-
def check_prometheus_query(host: str, port: int, query: str) -> None:
6+
def _run_prometheus_query(host: str, port: int, query: str) -> requests.Response:
77
"""Executes the given query against the prometheus instance.
88
Fails if the response is empty."""
99
base_url = f"http://{host}:{port}/api/v1/query"
@@ -16,10 +16,23 @@ def check_prometheus_query(host: str, port: int, query: str) -> None:
1616
raise Exception(f"Prometheus query failed with status code {response.status_code}")
1717
if response.json().get("status") != "success":
1818
raise Exception(f"Prometheus query failed with status: {response.json().get('status')}")
19-
if not response.json().get("data", {}).get("result") or len(response.json().get("data", {}).get("result")) == 0:
19+
return response
20+
21+
22+
def check_prometheus_query(host: str, port: int, query: str) -> None:
23+
response = _run_prometheus_query(host, port, query)
24+
data_result_list: list = response.json().get("data", {}).get("result")
25+
if not data_result_list or len(data_result_list) == 0:
2026
raise Exception("Prometheus query returned no results")
2127

2228

29+
def check_prometheus_query_is_missing(host: str, port: int, query: str) -> None:
30+
response = _run_prometheus_query(host, port, query)
31+
data_result_list: list = response.json().get("data", {}).get("result")
32+
if data_result_list and len(data_result_list) > 0:
33+
raise Exception("Prometheus query returned results")
34+
35+
2336
def check_prometheus_exporter(host: str, port: int, query: str) -> None:
2437
"""Check the metric is available int Prometheus Exporter
2538
Fails if the response is empty."""

‎test/suites/optional/observability.robot

Lines changed: 16 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ Library ../../resources/loki.py
1010
Resource ../../resources/kubeconfig.resource
1111
Resource ../../resources/common.resource
1212
Resource ../../resources/systemd.resource
13+
Resource ../../resources/observability.resource
1314

1415
Suite Setup Setup Suite And Prepare Test Host
1516
Suite Teardown Teardown Suite And Revert Test Host
@@ -31,15 +32,9 @@ Host Metrics Are Exported
3132
... \ [0-9]+, "metrics": [0-9]+, "data points": [0-9]+}
3233
Pattern Should Appear In Log Output ${JOURNAL_CUR} ${pattern} unit="microshift-observability"
3334

34-
Check Prometheus Query
35-
... ${PROMETHEUS_HOST}
36-
... ${PROMETHEUS_PORT}
37-
... system_cpu_time_seconds_total{cpu="cpu0",state="idle"}
38-
39-
Check Prometheus Exporter
40-
... ${USHIFT_HOST}
41-
... ${PROM_EXPORTER_PORT}
42-
... system_cpu_time_seconds_total{cpu="cpu0",state="idle"}
35+
Set Test Variable ${METRIC} system_cpu_time_seconds_total{cpu="cpu0",state="idle"}
36+
Check Prometheus Query ${PROMETHEUS_HOST} ${PROMETHEUS_PORT} ${METRIC}
37+
Check Prometheus Exporter ${USHIFT_HOST} ${PROM_EXPORTER_PORT} ${METRIC}
4338

4439
Kube Metrics Are Exported
4540
[Documentation] The opentelemetry-collector should be able to export kube metrics.
@@ -49,25 +44,13 @@ Kube Metrics Are Exported
4944
... \ [0-9]+, "metrics": [0-9]+, "data points": [0-9]+}
5045
Pattern Should Appear In Log Output ${JOURNAL_CUR} ${pattern} unit="microshift-observability"
5146

52-
Check Prometheus Query
53-
... ${PROMETHEUS_HOST}
54-
... ${PROMETHEUS_PORT}
55-
... container_cpu_time_seconds_total
56-
57-
Check Prometheus Exporter
58-
... ${USHIFT_HOST}
59-
... ${PROM_EXPORTER_PORT}
60-
... container_cpu_time_seconds_total
61-
62-
Check Prometheus Query
63-
... ${PROMETHEUS_HOST}
64-
... ${PROMETHEUS_PORT}
65-
... k8s_pod_cpu_time_seconds_total
47+
Set Test Variable ${METRIC} container_cpu_time_seconds_total
48+
Check Prometheus Query ${PROMETHEUS_HOST} ${PROMETHEUS_PORT} ${METRIC}
49+
Check Prometheus Exporter ${USHIFT_HOST} ${PROM_EXPORTER_PORT} ${METRIC}
6650

67-
Check Prometheus Exporter
68-
... ${USHIFT_HOST}
69-
... ${PROM_EXPORTER_PORT}
70-
... k8s_pod_cpu_time_seconds_total
51+
Set Test Variable ${METRIC} k8s_pod_cpu_time_seconds_total
52+
Check Prometheus Query ${PROMETHEUS_HOST} ${PROMETHEUS_PORT} ${METRIC}
53+
Check Prometheus Exporter ${USHIFT_HOST} ${PROM_EXPORTER_PORT} ${METRIC}
7154

7255
Journald Logs Are Exported
7356
[Documentation] The opentelemetry-collector should be able to export logs to journald.
@@ -78,10 +61,7 @@ Journald Logs Are Exported
7861
... \ "log records": [0-9]+\\}
7962
Pattern Should Appear In Log Output ${JOURNAL_CUR} ${pattern} unit="microshift-observability"
8063

81-
Check Loki Query
82-
... ${LOKI_HOST}
83-
... ${LOKI_PORT}
84-
... {job="journald",exporter="OTLP"}
64+
Check Loki Query ${LOKI_HOST} ${LOKI_PORT} {job="journald",exporter="OTLP"}
8565

8666
Kube Events Logs Are Exported
8767
[Documentation] The opentelemetry-collector should be able to export logs to journald.
@@ -92,10 +72,7 @@ Kube Events Logs Are Exported
9272
... \ "log records": [0-9]+\\}
9373
Pattern Should Appear In Log Output ${JOURNAL_CUR} ${pattern} unit="microshift-observability"
9474

95-
Check Loki Query
96-
... ${LOKI_HOST}
97-
... ${LOKI_PORT}
98-
... {job="kube_events",exporter="OTLP"}
75+
Check Loki Query ${LOKI_HOST} ${LOKI_PORT} {job="kube_events",exporter="OTLP"}
9976

10077
Logs Should Not Contain Receiver Errors
10178
[Documentation] Internal receiver errors are not treated as fatal. Typically these are due to a misconfiguration
@@ -110,6 +87,8 @@ Logs Should Not Contain Receiver Errors
11087
Setup Suite And Prepare Test Host
11188
[Documentation] The service starts after MicroShift starts and thus will start generating pertinent log data
11289
... right away. When the suite is executed, immediately get the cursor for the microshift-observability unit.
90+
Start Prometheus Server
91+
Start Loki Server
11392
Setup Suite
11493
Check Required Observability Variables
11594
Set Test OTEL Configuration
@@ -153,6 +132,8 @@ Teardown Suite And Revert Test Host
153132
[Documentation] Set back original OTEL config and teardown Suite
154133
Set Back Original OTEL Configuration
155134
Teardown Suite
135+
Stop Loki Server
136+
Stop Prometheus Server
156137

157138
Set Back Original OTEL Configuration
158139
[Documentation] Set Back Original OTEL Configuration

0 commit comments

Comments
 (0)