Vector.dev CPU cap in code/internally at 8vcpu?

[pdonorio]

Hello everyone, trying to understand better a behavior observed in k8s with vector daemonset today.

We recently removed CPU limit in resources:

$ kubectl --context ... --namespace vector get pod vector-... -o yaml | yq .spec.containers[0].resources

limits:
  memory: 4Gi
requests:
  cpu: "1"
  memory: 2Gi

We are using some services that send logs to s3 and loki with a very high rate and their are critical so we'd like not to lose any of it.

Since sometimes the CPU was getting too high we wanted to test no CPU limit and let the node autoscaler do its job. But we are seeing some sort of hidden limit around 7.5cpu usage. Today one vector pod got kind of "stuck" and lost some logs, restarting unblocked the situation.

The main question is: is there any other configuration or internal cap for CPUs?

A side question would be: do you have suggestions to increase key pods for vector to perform at best? note: we're using gzip compression and testing snappy as alternative we actually saw an increase in CPU usage.

[pront]

Hi @pdonorio,

No, Vector does not have any built-in CPU limiter. It will use whatever CPU it's given, governed only by Kubernetes CPU requests/limits, quotas and of course, OS resources and hardware capacity. When it comes to Vector, some sinks are not optimized or you could create more batch partitions (if supported by the implementation).

As always I encourage people to share their Vector config and for questions around performance I highly recommend settings up a monitoring dashboard as described here: https://vector.dev/guides/developer/debugging/#visualizing-and-querying-internal-metrics

[pdonorio]

Hi @pront - thank you for a quick reply 🙏

No, Vector does not have any built-in CPU limiter. It will use whatever CPU it's given, governed only by Kubernetes CPU requests/limits, quotas and of course, OS resources and hardware capacity.

Thanks, at least we can rule out variables in our search

When it comes to Vector, some sinks are not optimized or you could create more batch partitions (if supported by the implementation).

Can you explain more about the batch partitions or link references if you have any?

As always I encourage people to share their Vector config and for questions around performance I highly recommend settings up a monitoring dashboard as described here: https://vector.dev/guides/developer/debugging/#visualizing-and-querying-internal-metrics

We did add internal metrics towards datadog, but we're not finding a way to validate this bottleneck because we didn't find a way to add custom tags (e.g. cluster or node name) that would easily allow us to filter and group the values. We just have instances with random names. Do you know if that is possible?

Thank you for this project!

[jlambatl]

@pdonorio, our internal metrics go through a filter transform and then an enrichment, where we add additional tags to the metrics before they go to the sink.

source -> filter -> remap -> sink

Here's the remap (enrichment) transform configuration.

type = "remap"

inputs = [
  "internal_metrics_filtered",
]

# Metric mutation: https://vector.dev/docs/reference/configuration/transforms/remap/#event-data-model
source = '''
  .tags.environment = "${AWS_ENV}"
  .tags.environment_type = "${AWS_ENVTYPE}"
  .tags.instance_type = "${AWS_INSTANCE_TYPE}"
'''

[pdonorio]

thanks this is very helpful