Skip to content

Commit 8301baa

Browse files
tmalovetmalove
committed
add new file microshift-telemetry-service
1 parent 1b4cb64 commit 8301baa

11 files changed

+455
-0
lines changed

‎_topic_maps/_topic_map_ms.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -211,6 +211,8 @@ Topics:
211211
File: microshift-applications
212212
- Name: Deleting or updating Kustomize manifest resources
213213
File: microshift-deleting-resource-manifests
214+
- Name: Configuring Observability
215+
File: microshift-observability-service
214216
- Name: Embedding applications on RHEL for Edge
215217
File: microshift-embedded-apps-on-rhel-edge
216218
- Name: Embedding applications for offline use

‎microshift_install_rpm_opt/microshift-install-optional-rpms.adoc

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,13 @@ include::modules/microshift-install-rpms-olm.adoc[leveloffset=+2]
3434
.Additional resources
3535
* xref:../microshift_running_apps/microshift_operators/microshift-operators-olm.adoc#microshift-operators-olm[Using Operator Lifecycle Manager with {microshift-short}]
3636

37+
include::modules/microshift-otel-install.adoc[leveloffset=+2]
38+
39+
//additional resources for installing Observability
40+
[role="_additional-resources"]
41+
.Additional resources
42+
* xref:../microshift_running_apps/microshift-observability-service.adoc#microshift-observability-service[Using {microshift-short} Observability]
43+
3744
include::modules/microshift-rhoai-install.adoc[leveloffset=+2]
3845

3946
//additional resources for installing RHOAI

‎microshift_running_apps/microshift-observability-service.adoc

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
[id="microshift-observability-service"]
3+
= Using {microshift-short} Observability
4+
include::_attributes/attributes-microshift.adoc[]
5+
:context: microshift-observability-service
6+
7+
toc::[]
8+
9+
{microshift-short} Observability collects and transmits system data for monitoring and analysis. The data includes performance and usage metrics and error reporting.
10+
11+
include::modules/microshift-otel-install.adoc[leveloffset=+1]
12+
[role="_additional-resources"]
13+
.Additional resources
14+
* xref:../microshift_install_rpm_opt/microshift-install-optional-rpms.adoc#microshift-install-optional-rpms[Installing optional RPM packages]
15+
16+
include::modules/microshift-otel-config.adoc[leveloffset=+1]
17+
18+
// include::modules/microshift-otel-start-service.adoc[leveloffset=+1]
19+
20+
include::modules/microshift-otel-config-examples.adoc[leveloffset=+1]
21+
22+
include::modules/microshift-otel-config-small.adoc[leveloffset=+1]
23+
24+
include::modules/microshift-otel-config-medium.adoc[leveloffset=+1]
25+
26+
include::modules/microshift-otel-config-large.adoc[leveloffset=+1]
27+
28+
include::modules/microshift-otel-verify.adoc[leveloffset=+1]
29+
30+
31+

‎modules/microshift-otel-config-examples.adoc

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
// Module included in the following assemblies:
2+
//
3+
// microshift_running_apps/microshift-observability-service.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="microshift-otel-config-examples_{context}"]
7+
= Selecting a {microshift-short} Observability configuration
8+
9+
The amount and complexity of the data depends on predefined configurations. These configurations determine the number of data sources and the amount of collected data that is transmitted. These configurations are defined as small, medium, and large (default).
10+
11+
The `opentelemetry-collector.yaml` file includes specific parameters that are used to collect data for monitoring the system resources. All warnings for cluster events are included in the collected data. {microshift-short} Observability collects and transmits data for the following resources:
12+
13+
* CPU, memory, disk, and network metrics of containers, pods, and node
14+
* Kubernetes events
15+
* Host CPU, memory, disk, and network metrics
16+
* System journals for certain {microshift-short} services, and dependencies
17+
* Metrics exposed by pods that have `prometheus.io/scrape`: `true` annotation
18+
19+
Replace the values of the `exporters.otlp.endpoint` and `services.telemetry.metrics.readers[0].endpoint` fields with the IP address or host name of the remote back end. This IP address resolves to the local node's host name. Any unreachable endpoint is reported in the {microshift-short} observability service logs.

‎modules/microshift-otel-config-large.adoc

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
// Module included in the following assemblies:
2+
//
3+
// microshift_running_apps/microshift-observability-service.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="microshift-otel-config-large_{context}"]
7+
= Selecting a large configuration
8+
9+
. Select a large configuration by adding the following information to the `/etc/microshift/observability/opentelemetry-collector.yaml` file. `Large` is the default configuration.
10+
+
11+
[source,yaml]
12+
----
13+
receivers:
14+
kubeletstats:
15+
auth_type: tls
16+
ca_file: /var/lib/microshift/certs/ca-bundle/client-ca.crt
17+
key_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.key
18+
cert_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.crt
19+
insecure_skip_verify: true
20+
collection_interval: 10s
21+
endpoint: "${env:K8S_NODE_NAME}:10250"
22+
node: ${env:K8S_NODE_NAME}
23+
k8s_api_config:
24+
auth_type: kubeConfig
25+
k8s_events:
26+
auth_type: kubeConfig
27+
hostmetrics:
28+
root_path: /
29+
collection_interval: 10s
30+
scrapers:
31+
cpu:
32+
memory:
33+
network:
34+
disk:
35+
filesystem:
36+
journald:
37+
units:
38+
- microshift
39+
- microshift-observability
40+
- microshift-etcd
41+
- crio
42+
- openvswitch.service
43+
- ovsdb-server.service
44+
- ovs-vswitchd.service
45+
priority: info
46+
prometheus:
47+
config:
48+
scrape_configs:
49+
- job_name: k8s
50+
scrape_interval: 10s
51+
kubernetes_sd_configs:
52+
- kubeconfig_file: /var/lib/microshift/resources/observability-client/kubeconfig
53+
role: pod
54+
relabel_configs:
55+
# Only scrape Pods with annotation "prometheus.io/scrape": "true"
56+
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
57+
action: keep
58+
regex: true
59+
# Use value of "prometheus.io/path" annotation for scraping
60+
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
61+
action: replace
62+
target_label: __metrics_path__
63+
regex: (.+)
64+
# Use value of "prometheus.io/port" annotation for scraping
65+
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
66+
action: replace
67+
regex: ([^:]+)(?::\d+)?;(\d+)
68+
replacement: $1:$2
69+
target_label: __address__
70+
processors:
71+
batch:
72+
resourcedetection/system:
73+
detectors: [ "system" ]
74+
system:
75+
hostname_sources: [ "os" ]
76+
exporters:
77+
otlp:
78+
sending_queue:
79+
storage: file_storage
80+
endpoint: ${env:OTEL_BACKEND}:4317 <1>
81+
tls:
82+
insecure: true
83+
extensions:
84+
file_storage:
85+
directory: /var/lib/microshift-observability
86+
service:
87+
extensions: [ file_storage ]
88+
pipelines:
89+
metrics/kubeletstats:
90+
receivers: [ kubeletstats ]
91+
processors: [ batch ]
92+
exporters: [ otlp ]
93+
metrics/hostmetrics:
94+
receivers: [ hostmetrics ]
95+
processors: [ resourcedetection/system, batch ]
96+
exporters: [ otlp ]
97+
logs/kube_events:
98+
receivers: [ k8s_events ]
99+
processors: [ resourcedetection/system, batch ]
100+
exporters: [ otlp ]
101+
logs/host:
102+
receivers: [ hostmetrics ]
103+
processors: [ resourcedetection/system ]
104+
exporters: [ otlp ]
105+
logs/journald:
106+
receivers: [ journald ]
107+
processors: [ resourcedetection/system ]
108+
exporters: [ otlp ]
109+
metrics/pods:
110+
receivers: [ prometheus ]
111+
processors: [ batch ]
112+
exporters: [ otlp ]
113+
telemetry:
114+
metrics:
115+
readers:
116+
- periodic:
117+
exporter:
118+
otlp:
119+
protocol: http/protobuf
120+
endpoint: http://${env:OTEL_BACKEND}:4318 <2>
121+
----
122+
<1> Replace the variable `${env:OTEL_BACKEND}` with the IP address or host name of the remote back end. This IP address resolves to the local node's host name. Any unreachable endpoint is reported in the `microshift-observability` service logs.
123+
<2> Replace the variable `${env:OTEL_BACKEND}` with the IP address or host name of the remote back end. This IP address resolves to the local node's host name. Any unreachable endpoint is reported in the `microshift-observability` service logs.
124+
125+
. Restart {microshift-short} Observability to complete the configuration selection.

‎modules/microshift-otel-config-medium.adoc

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
// Module included in the following assemblies:
2+
//
3+
// microshift_running_apps/microshift-observability-service.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="microshift-otel-config-medium_{context}"]
7+
= Selecting a medium configuration
8+
9+
. Select a medium configuration by adding the following information to the `/etc/microshift/observability/opentelemetry-collector.yaml` file:
10+
+
11+
[source,yaml]
12+
----
13+
receivers:
14+
kubeletstats:
15+
auth_type: tls
16+
ca_file: /var/lib/microshift/certs/ca-bundle/client-ca.crt
17+
key_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.key
18+
cert_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.crt
19+
insecure_skip_verify: true
20+
collection_interval: 10s
21+
endpoint: "${env:K8S_NODE_NAME}:10250"
22+
node: ${env:K8S_NODE_NAME}
23+
k8s_api_config:
24+
auth_type: kubeConfig
25+
k8s_events:
26+
auth_type: kubeConfig
27+
journald:
28+
units:
29+
- microshift
30+
priority: info
31+
processors:
32+
batch:
33+
resourcedetection/system:
34+
detectors: [ "system" ]
35+
system:
36+
hostname_sources: [ "os" ]
37+
exporters:
38+
otlp:
39+
sending_queue:
40+
storage: file_storage
41+
endpoint: ${env:OTEL_BACKEND}:4317 <1>
42+
tls:
43+
insecure: true
44+
extensions:
45+
file_storage:
46+
directory: /var/lib/microshift-observability
47+
service:
48+
extensions: [ file_storage ]
49+
pipelines:
50+
metrics/kubeletstats:
51+
receivers: [ kubeletstats ]
52+
processors: [ batch ]
53+
exporters: [ otlp ]
54+
logs/kube_events:
55+
receivers: [ k8s_events ]
56+
processors: [ resourcedetection/system, batch ]
57+
exporters: [ otlp ]
58+
logs/journald:
59+
receivers: [ journald ]
60+
processors: [ resourcedetection/system ]
61+
exporters: [ otlp ]
62+
telemetry:
63+
metrics:
64+
readers:
65+
- periodic:
66+
exporter:
67+
otlp:
68+
protocol: http/protobuf
69+
endpoint: http://${env:OTEL_BACKEND}:4318 <2>
70+
----
71+
<1> Replace the variable `${env:OTEL_BACKEND}` with the IP address or host name of the remote back end. This IP address resolves to the local node's host name. Any unreachable endpoint is reported in the `microshift-observability` service logs.
72+
<2> Replace the variable `${env:OTEL_BACKEND}` with the IP address or host name of the remote back end. This IP address resolves to the local node's host name. Any unreachable endpoint is reported in the `microshift-observability` service logs.
73+
74+
. Restart {microshift-short} Observability to complete the configuration selection.

‎modules/microshift-otel-config-small.adoc

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
// Module included in the following assemblies:
2+
//
3+
// microshift_running_apps/microshift-observability-service.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="microshift-otel-config-small_{context}"]
7+
= Selecting a small configuration
8+
9+
. Select a small configuration by adding the following information to the `/etc/microshift/observability/opentelemetry-collector.yaml` file:
10+
+
11+
[source,yaml]
12+
----
13+
receivers:
14+
kubeletstats:
15+
auth_type: tls
16+
ca_file: /var/lib/microshift/certs/ca-bundle/client-ca.crt
17+
key_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.key
18+
cert_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.crt
19+
insecure_skip_verify: true
20+
collection_interval: 10s
21+
endpoint: "${env:K8S_NODE_NAME}:10250"
22+
node: ${env:K8S_NODE_NAME}
23+
k8s_api_config:
24+
auth_type: kubeConfig
25+
k8s_events:
26+
auth_type: kubeConfig
27+
processors:
28+
batch:
29+
resourcedetection/system:
30+
detectors: [ "system" ]
31+
system:
32+
hostname_sources: [ "os" ]
33+
exporters:
34+
otlp:
35+
sending_queue:
36+
storage: file_storage
37+
endpoint: ${env:OTEL_BACKEND}:4317 <1>
38+
tls:
39+
insecure: true
40+
extensions:
41+
file_storage:
42+
directory: /var/lib/microshift-observability
43+
service:
44+
extensions: [ file_storage ]
45+
pipelines:
46+
metrics/kubeletstats:
47+
receivers: [ kubeletstats ]
48+
processors: [ batch ]
49+
exporters: [ otlp ]
50+
logs/kube_events:
51+
receivers: [ k8s_events ]
52+
processors: [ resourcedetection/system, batch ]
53+
exporters: [ otlp ]
54+
telemetry:
55+
metrics:
56+
readers:
57+
- periodic:
58+
exporter:
59+
otlp:
60+
protocol: http/protobuf
61+
endpoint: http://${env:OTEL_BACKEND}:4318 <2>
62+
----
63+
<1> Replace the variable `${env:OTEL_BACKEND}` with the IP address or host name of the remote back end. This IP address resolves to the local node's host name. Any unreachable endpoint is reported in the {microshift-short} service logs.
64+
<2> Replace the variable `${env:OTEL_BACKEND}` with the IP address or host name of the remote back end. This IP address resolves to the local node's host name. Any unreachable endpoint is reported in the {microshift-short} service logs.
65+
66+
. Restart {microshift-short} Observability to complete the configuration selection.

‎modules/microshift-otel-config.adoc

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
// Module included in the following assemblies:
2+
//
3+
// microshift_running_apps/microshift-observability-service.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="microshift-otel-config_{context}"]
7+
= Configuring {microshift-short} Observability
8+
9+
You must configure {microshift-short} Observability after it is installed by specifying a valid endpoint. If an endpoint is not specified, {microshift-short} Observability will not start.
10+
11+
You can specify any OpenTelemetry Protocol (OTLP)-compatible endpoint for each configuration before starting {microshift-short}.
12+
13+
.Procedure
14+
15+
. Update the `/etc/microshift/observability/opentelemetry-collector.yaml` file to specify an OTLP-compatible endpoint with the following information. The endpoint must link to an IP address or host name, and port number of an OTLP service.
16+
+
17+
.OTLP-compatible endpoint configuration
18+
[source,yaml]
19+
----
20+
# ...
21+
exporters:
22+
otlp:
23+
sending_queue:
24+
storage: file_storage
25+
endpoint: ${env:OTEL_BACKEND}:4317 <1>
26+
tls:
27+
insecure: true
28+
# ...
29+
service:
30+
# ...
31+
telemetry:
32+
metrics:
33+
readers:
34+
- periodic:
35+
exporter:
36+
otlp:
37+
protocol: http/protobuf
38+
endpoint: http://${env:OTEL_BACKEND}:4318 <1>
39+
# ...
40+
----
41+
<1> Replace `${env:OTEL_BACKEND}` with the IP address or host name of the remote back end. This IP address resolves to the local node's host name. An unreachable endpoint is reported in the {microshift-short} service logs.
42+
43+
. Each time that you update the `opentelemetry-collector.yaml` file, you must restart {microshift-short} Observability to apply the updates.
44+
+
45+
Restart {microshift-short} Observability by entering the following command:
46+
+
47+
[source,terminal]
48+
----
49+
$ sudo systemctl restart microshift-observability
50+
----
51+

0 commit comments

Comments
 (0)