Determining where CPU drops/lost events are coming from

[gmarler]

Trying to find out where in a BPF program certain tracelogs are coming from, as I'm not sure how to start.

While watching the output of bpftool prog tracelog, and only while my program that uses libbpf-rs is running, the following appears repeatedly for a subset of my CPUs every few seconds:

...
CPU:47 [LOST 162 EVENTS]
...

The BPF programs are here and here, but I can't even tell which one specifically is generating the events that are being lost.

I presume these CPU:XX [LOST ... EVENTS] messages are coming from the eBPF VM or the kernel.

Not seeing any obvious drops from either the perf event buffers or ring buffers we've set up via bpftool examination of the running programs.

How can I dig deeper?

[danielocfb]

Probably emitted from https://github.com/torvalds/linux/blob/5472d60c129f75282d94ae5ad072ee6dfb7c7246/kernel/trace/trace.c#L4388-L4389

I think the general course of action is to increase buffer size and/or to make sure that event draining can keep up with event creation (and vice versa), as lost events would likely be the result of ring buffer overrun.

[danielocfb]

As for debugging, I think you may be able to register a callback function to be invoked when events are being lost.

[gmarler]

Yes, the location where the messages comes from does check out.

I've tried bumping up the size of every buffer I know how without success. Also, every ring buffer_output() is checked for error like so - unless I'm doing that in a way that won't detect ring overflow.

Going back to review...