Cannot trace syscalls because CONFIG_FTRACE_SYSCALLS is not set

[bukatlib]

Describe the bug

Please consider to enable CONFIG_FTRACE_SYSCALLS kernel option to enable syscall tracing via 'perf trace', currently only raw syscalls can be observed.

$ perf list | grep syscall
raw_syscalls:sys_enter [Tracepoint event]
raw_syscalls:sys_exit [Tracepoint event]

Performance impact should be close to zero if tracing is disabled (nop instructions).

Steps to reproduce the behaviour

perf stat -e 'syscalls:sys_enter_*'
event syntax error: 'syscalls:sys_enter_*'
___ unknown tracepoint

Device (s)

Raspberry Pi 5

System

Raspberry Pi reference 2024-07-04
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 48efb5fc5485fafdc9de8ad481eb5c09e1182656, stage5

Logs

No response

Additional context

No response

[popcornmix]

This seems reasonable to me.

[pelwell]

See #6759. About 40 minutes from now you should be able to install a trial build using sudo rpi-update pulls/6759.

[bukatlib]

Thank you, I tested pulls/6759 and it works:

$ perf trace -e 'syscalls:sys_enter_open*' -a -- sleep 3
     0.000 sleep/2393 syscalls:sys_enter_openat(dfd: CWD, filename: "", flags: RDONLY|CLOEXEC)
     0.053 sleep/2393 syscalls:sys_enter_openat(dfd: CWD, filename: "", flags: RDONLY|CLOEXEC)
     0.340 sleep/2393 syscalls:sys_enter_openat(dfd: CWD, filename: "", flags: RDONLY|CLOEXEC)

$ perf stat -e 'syscalls:sys_enter_open*' -a -- sleep 3

 Performance counter stats for 'system wide':

                 0      syscalls:sys_enter_openat2                                            
                 3      syscalls:sys_enter_openat                                             
                 0      syscalls:sys_enter_open_tree                                          
                 0      syscalls:sys_enter_open_by_handle_at                                      

       3,001874110 seconds time elapsed

$ perf list -v  | grep syscalls | wc -l
620

We can trace more than 300 syscalls with much lower overhead compared to strace and moreover system wide. Size overhead is in my opinion acceptable given the better OS observability.

[pelwell]

@popcornmix's comment on the PR says:

The size increase is more than I expected. Perhaps this needs more thought to see exactly how many users it would benefit.

I did ask a colleague who has been using perfetto for tracing system activity if this would be useful and he says perfetto already supports tracing syscalls using these events:

$ sudo cat /sys/kernel/debug/tracing/available_events | grep syscalls
raw_syscalls:sys_exit
raw_syscalls:sys_enter

[bukatlib]

Thank you for the comment. I suggest to consider adding the tracing support at least to 64 bit kernels, 170kB memory overhead is 0.03 % on 512 MB memory, which is close to negligible.

My view is that tracepoints add minimal overhead, you can control the tracing through a shell in /sys/kernel/tracing directory. Compare it to the above mentioned perfetto which requires dozens of megabytes of the storage.

You are right that you might trace by using raw syscall tracepoints but it is user unfriendly or it requires third-party solution. The RPI kernel without FTRACE_SYSCALLS has from my point of view improper configuration for the official distro package as command perf trace does not work...

[richardweinberger]

Please enable this kernel config option. Just wasted almost half an hour to figure why bpftrace doesn't work as expected.

[pelwell]

I've now merged #6759, but we reserve the right to revert it if we get complaints.

[bukatlib]

Thank you for the config update, good to see that syscalls will be more easily traceable on RPI official kernels.