etcd-shield: fix metrics in staging (#6834)

* etcd-shield: fix metrics in staging

Metrics for etcd-shield were misconfigured in two ways.  First, the
labels selectors were incorrect and not applied to the relevant service.
Applying the correct labels and label selectors allows the
servicemonitor to pick up the correct metrics endpoints.  Second,
because etcd-shield is run not as a user workload but as a core service,
the prometheus-k8s service account in the namespace
`openshift-monitoring` needs to be given appropriate permissions to look
at resources in the `etcd-shield` namespace.  This is fixed by adding a
role and a rolebinding.

Signed-off-by: Andy Sadler <[email protected]>

* remove etcd-shield from dev overlay

etcd-shield shouldn't be a part of the development overlays.  It's an
extra workload not offering anything useful during development
deployments of konflux.

Signed-off-by: Andy Sadler <[email protected]>

---------

Signed-off-by: Andy Sadler <[email protected]>

Author: sadlerap

argo-cd-apps/overlays/development/delete-applications.yaml

@@ -130,3 +130,9 @@ kind: ApplicationSet
 metadata:
   name: pulp-access-controller
 $patch: delete
+---
+apiVersion: argoproj.io/v1alpha1
+kind: ApplicationSet
+metadata:
+  name: etcd-shield
+$patch: delete

components/etcd-shield/base/metrics/metrics-service.yaml

@@ -1,6 +1,8 @@
 apiVersion: v1
 kind: Service
 metadata:
+  labels:
+    app: etcd-shield
   name: etcd-shield-metrics
   namespace: etcd-shield
 spec:
@@ -10,4 +12,5 @@ spec:
   ports:
   - name: metrics
     targetPort: 9100
+    protocol: TCP
     port: 9100

components/etcd-shield/base/metrics/monitor.yaml

@@ -3,7 +3,7 @@ kind: ServiceMonitor
 metadata:
   name: etcd-shield-metrics
   labels:
-    apps: etcd-shield
+    app: etcd-shield
 spec:
   endpoints:
     - interval: 15s
@@ -23,4 +23,4 @@ spec:
         serverName: etcd-shield.etcd-shield.svc
   selector:
     matchLabels:
-      apps: etcd-shield
+      app: etcd-shield

components/etcd-shield/base/metrics/network-policy.yaml

@@ -1,19 +1,19 @@
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
-  name: allow-from-openshift-user-workload-monitoring
+  name: allow-from-openshift-monitoring
   namespace: etcd-shield
 spec:
   podSelector:
     matchLabels:
-      apps: etcd-shield
+      app: etcd-shield
   policyTypes:
   - Ingress
   ingress:
   - from:
     - namespaceSelector:
         matchLabels:
-          kubernetes.io/metadata.name: openshift-user-workload-monitoring
+          kubernetes.io/metadata.name: openshift-monitoring
     ports:
     - protocol: TCP
       port: metrics

components/etcd-shield/base/metrics/rbac.yaml

@@ -21,3 +21,28 @@ subjects:
 - kind: ServiceAccount
   name: metrics-reader
   namespace: etcd-shield
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: monitoring-metrics-reader
+  namespace: etcd-shield
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: etcd-shield-monitor
+subjects:
+- kind: ServiceAccount
+  name: prometheus-k8s
+  namespace: openshift-monitoring
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: etcd-shield-monitor
+  namespace: etcd-shield
+rules:
+- apiGroups: [""]
+  resources: ["pods", "services", "endpoints"]
+  verbs: ["get", "list", "watch"]
+---