pyroscope.ebpf: Retry after eBPF errors and buffer config updates. #3381
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
simonswine
added
the
area/pyroscope
If the eBPF component encounter an error, it stops the main processing loops, which then caused issues because any config reload for alloy would block the whole agent. Fixes #3332 Also for long running scrapes we were blocking the component evaluation as we are waiting for the config to be updated. I switched this to be a buffered channel, so the config updated are unblocked. In case of multiple updated buffered it will skip the in between updates and go straight to the latest version. Fixes #1371
simonswine
force-pushed
the
20250416_pyroscope.ebpf-error-propagation
branch
from
312558c to
9e1854d
Compare
simonswine
changed the title
simonswine
changed the title
simonswine
commented
Apr 17, 2025
| } | ||
| case <-t.C: | ||
| if !started { | ||
| err := c.session.Start() |
There was a problem hiding this comment.
I imagine main of those errors are permanent (e.g. not enough permissions, we probably should handle those by stopping the ticker. Wdyt @korniltsev
|
@JohanLindvall wonder if this fixes your problem and/or gives you a better error message? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If the eBPF component encounters an error, it stops the main processing loops, which then caused issues because any config reload for alloy would block the whole agent.
Fixes #3332
Also for long running scrapes we were blocking the component evaluation as we are waiting for the config to be updated. I switched this to be a buffered channel, so the config updated are unblocked. In case of multiple updated buffered it will skip the in between updates and go straight to the latest version.
Fixes #1371