Enable reloading of config for aggregator

[sinacek]

Enable reloading of config for aggregator when config file is changed without restart of pod.

Notes:

• https://vector.dev/docs/administration/management/#reloading
• https://github.com/jimmidyson/configmap-reload

[jszwedko]

Hey @sinacek !

Would you be able to provide a bit more detail about what you are trying to accomplish? Is it that you want to use https://github.com/jimmidyson/configmap-reload with Vector to reload when the config map changes? If so, it should work once that tool supports sending SIGHUPs, which seems to be on its roadmap.

If that's not what you want, could you provide a bit more detail about your use-case?

[sinacek]

Hi @jszwedko , that is exactly how you described. Roadmap of which project do you mean?

[jszwedko]

Hi @jszwedko , that is exactly how you described. Roadmap of which project do you mean?

The roadmap of the https://github.com/jimmidyson/configmap-reload project. It looks like they aim to support sending SIGHUPs in the future which means it'll just work with Vector. Assuming that is what you want, I think we can close this issue and just wait for them to add it (jimmidyson/configmap-reload#23).

[sinacek]

That issue is 5 years old, so i'm not sure if he has plan to implement it. Do you have on road map add endpoint to reload as it has prometheus?

[jszwedko]

We don't have plans for that at the moment, but if it'd be useful for you would you want to open a feature request on the Vector repository describing what you would need? https://github.com/vectordotdev/vector/issues/new?assignees=&labels=type%3A+feature&template=feature.yml

[sinacek]

I find that similar issue is already exists, so I send comment there vectordotdev/vector#541

[jusch23]

Hi @jszwedko,
I came across this issue while experimenting with the k8s-sidecar project (https://github.com/kiwigrid/k8s-sidecar) for dynamic reloading of vector configurations. From my tests, it should be possible to set this up with the current vector implementation, but some changes to the helm chart are required. These are the steps that I took:
First, I applied additional RBAC rules to the vector service account to enable listing of configmaps in the vector namespace.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: monitoring-vector
  name: cm-read
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cm-read
  namespace: monitoring-vector
subjects:
- kind: User
  name: system:serviceaccount:monitoring-vector:vector
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: cm-read
  apiGroup: rbac.authorization.k8s.io

Then I installed vector using the current helm chart (0.46.0) and following overrides:

customConfig:
  data_dir: /var/lib/vector
service:
  enabled: true
  ports:
    - name: vector-test
      port: 8080
args:
  - --config-dir
  - "/etc/vector/"
  - -w
  - --allow-empty-config
extraContainers:
  - name: config-loader
    image: kiwigrid/k8s-sidecar:2.0.3
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: config
      mountPath: /etc/vector
    env:
    - name: LABEL
      value: "vector_config"
    - name: LABEL_VALUE
      value: "1"
    - name: FOLDER
      value: "/etc/vector"

Finally, I needed to patch the vector daemonset, so that config volume is an emptyDir and removed readOnly for the volumeMount of the vector config directory, getting the following DaemonSet:

vector-daemonset.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/component: Agent
    app.kubernetes.io/instance: vector
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: vector
    app.kubernetes.io/version: 0.50.0-distroless-libc
    helm.sh/chart: vector-0.46.0
  name: vector
  namespace: monitoring-vector
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: Agent
      app.kubernetes.io/instance: vector
      app.kubernetes.io/name: vector
  template:
    metadata:
      annotations:
        checksum/config: 6938951191aca860ed66ed189cc133b0a2781255ee85250d4dc80c3baef3a651
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: Agent
        app.kubernetes.io/instance: vector
        app.kubernetes.io/name: vector
        vector.dev/exclude: "true"
    spec:
      containers:
      - args:
        - --config-dir
        - /etc/vector/
        - -w
        - --allow-empty-config
        env:
        - name: VECTOR_LOG
          value: info
        - name: VECTOR_SELF_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: VECTOR_SELF_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: VECTOR_SELF_POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: PROCFS_ROOT
          value: /host/proc
        - name: SYSFS_ROOT
          value: /host/sys
        envFrom:
        - secretRef:
            name: loki-tenants-credentials
        image: timberio/vector:0.50.0-distroless-libc
        imagePullPolicy: IfNotPresent
        name: vector
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/vector
          name: data
        - mountPath: /etc/vector/
          name: config
        - mountPath: /var/log/
          name: var-log
          readOnly: true
        - mountPath: /var/lib
          name: var-lib
          readOnly: true
        - mountPath: /host/proc
          name: procfs
          readOnly: true
        - mountPath: /host/sys
          name: sysfs
          readOnly: true
      - env:
        - name: LABEL
          value: vector_config
        - name: LABEL_VALUE
          value: "1"
        - name: FOLDER
          value: /etc/vector
        image: kiwigrid/k8s-sidecar:2.0.3
        imagePullPolicy: IfNotPresent
        name: config-loader
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/vector
          name: config
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: vector
      serviceAccountName: vector
      terminationGracePeriodSeconds: 60
      volumes:
      - name: config
        emptyDir: {}
      - hostPath:
          path: /var/lib/vector
          type: ""
        name: data
      - hostPath:
          path: /var/log/
          type: ""
        name: var-log
      - hostPath:
          path: /var/lib/
          type: ""
        name: var-lib
      - hostPath:
          path: /proc
          type: ""
        name: procfs
      - hostPath:
          path: /sys
          type: ""
        name: sysfs
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate

Do you think this might be a valid approach? If so, I would try adding a PR to make this configurable with the helm chart.

[jszwedko]

Thanks! On first read it seems like this is a reasonable approach but cc/ @vectordotdev/vector for additional thoughts.

[pront]

Hi @jusch23 please create a new issue to track this work. We just need to ensure the defaults are not compromising security (e.g. readOnly should always be the default).