drgn does not handle non-stripped/with-debug_info symbol files properly

[georges-aureau]

When using -s with a non-stripped/with-debug_info file, drgn does not seem to skip searching for a matching debug_info file, eg:

georges@kumba:~/ktools/drgn$ python -m drgn --version
drgn 0.0.30+27.g4738ddfa (using Python 3.12.3, elfutils 0.190, with libkdumpfile)

georges@kumba:~/ktools/drgn$ file vmcore vmlinux
vmcore:  Kdump compressed dump v6, system Linux, node node3, release 4.18.0-372.9.1.el8.x86_64, version #1 SMP Fri Apr 15 22:12:19 EDT 2022, machine x86_64, domain che-pro
vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=05f54d21df8b3a447231fb623be4d9f5aba57262, with debug_info, not stripped

georges@kumba:~/ktools/drgn$ python -m drgn -c vmcore -s vmlinux --log-level debug
drgn 0.0.30+27.g4738ddfa (using Python 3.12.3, elfutils 0.190, with libkdumpfile)
debug: loading default debugging symbols
debug: provided file vmlinux build ID 05f54d21df8b3a447231fb623be4d9f5aba57262
debug: created main module kernel
debug: couldn't find kernel build ID
debug: loading debugging symbols for 1 modules
debug: kernel (no build ID): trying standard paths in ":.debug:/usr/lib/debug" for debug file
debug: /usr/lib/debug/boot/vmlinux-4.18.0-372.9.1.el8.x86_64: No such file or directory
debug: /usr/lib/debug/lib/modules/4.18.0-372.9.1.el8.x86_64/vmlinux: No such file or directory
debug: /usr/lib/debug/lib/modules/4.18.0-372.9.1.el8.x86_64/vmlinux.debug: No such file or directory
debug: /boot/vmlinux-4.18.0-372.9.1.el8.x86_64: No such file or directory
debug: /lib/modules/4.18.0-372.9.1.el8.x86_64/build/vmlinux: No such file or directory
debug: /lib/modules/4.18.0-372.9.1.el8.x86_64/vmlinux: No such file or directory
debug: debugging symbols loaded for 0/1 modules
warning: can't find loaded modules without kernel debug info
warning: missing debugging symbols for kernel
warning: provided file vmlinux did not match any loaded modules; ignoring
warning: missing some debugging symbols; see https://drgn.readthedocs.io/en/latest/getting_debugging_symbols.html
For help, type help(drgn).
>>> import drgn
>>> from drgn import FaultError, NULL, Object, alignof, cast, container_of, execscript, implicit_convert, offsetof, reinterpret, sizeof, stack_trace
>>> from drgn.helpers.common import *
>>> from drgn.helpers.linux import *

When the symbol file contains a ".debug_info" section, drgn should skip searching for a build-id-matching debug_info file.

[osandov]

Thanks for the report! drgn's main branch is more strict about using build IDs to match files provided with -s, but I didn't handle the fact that old kernel versions didn't save the vmlinux build ID in the core dump. (This was also reported in #462). I'll fix it today. In the meantime, the latest release, 0.0.30, should work.

[georges-aureau]

In this case, the vmlinux has a buildId, it has debug_info, it is not stripped, so there is no need to try finding another file with the same buildId, the file provided has everything, so we don't need to hunt for debug_info. There should be a check for the presence of an ELF ".debug_info" section, and if present, drgn should skip the "want debug_info" file.

[georges-aureau]

And I tested with 0.0.30, with latest commits, see provided output.

[georges-aureau]

Please note that using prog.main_module().try_file("vmlinux") instead of -s vmlinux is going through a code path allowing to load the debug_info:

georges@kumba:~/ktools/drgn$ ll vmlinux
lrwxrwxrwx 1 georges georges 34 Jan 23 22:39 vmlinux -> /data/lxdb/x86_64/rhel/4.18.0-372.9.1.el8.x86_64/vmlinux

georges@kumba:~/ktools/drgn$ cat vmlinux.py
prog.main_module().try_file("vmlinux")

georges@kumba:~/ktools/drgn$ python -m drgn --log-level debug  -c vmcore vmlinux.py
debug: loading default debugging symbols
debug: created main module kernel
debug: couldn't find kernel build ID
debug: loading debugging symbols for 1 modules
debug: kernel (no build ID): trying standard paths in ":.debug:/usr/lib/debug" for debug file
debug: /usr/lib/debug/boot/vmlinux-4.18.0-372.9.1.el8.x86_64: No such file or directory
debug: /usr/lib/debug/lib/modules/4.18.0-372.9.1.el8.x86_64/vmlinux: No such file or directory
debug: /usr/lib/debug/lib/modules/4.18.0-372.9.1.el8.x86_64/vmlinux.debug: No such file or directory
debug: /boot/vmlinux-4.18.0-372.9.1.el8.x86_64: No such file or directory
debug: /lib/modules/4.18.0-372.9.1.el8.x86_64/build/vmlinux: No such file or directory
debug: /lib/modules/4.18.0-372.9.1.el8.x86_64/vmlinux: No such file or directory
debug: debugging symbols loaded for 0/1 modules
warning: can't find loaded modules without kernel debug info
warning: missing debugging symbols for kernel
warning: missing some debugging symbols; see https://drgn.readthedocs.io/en/latest/getting_debugging_symbols.html
debug: kernel (no build ID): trying provided file as debug file
debug: kernel: trying vmlinux
debug: canonical path is /data/lxdb/x86_64/rhel/4.18.0-372.9.1.el8.x86_64/vmlinux
debug: got bias 0x14e00000 from VMCOREINFO
info: kernel: using debug info file /data/lxdb/x86_64/rhel/4.18.0-372.9.1.el8.x86_64/vmlinux
debug: kernel: set build ID 05f54d21df8b3a447231fb623be4d9f5aba57262 from file
debug: kernel: set address range 0xffffffff95e00000-0xffffffff99600000 from file

So there seems to be something wrong with the logics "want debug info file" when using -s vmlinux where vmlinux contains an ELF .debug_info section.

[brenns10]

In this case, the vmlinux has a buildId

The question isn't whether your vmlinux file has a build ID, but rather whether your vmcore file has a build ID. That's what drgn is complaining about here: it cannot find a build ID in your vmcore, so it doesn't have a good way to match your vmlinux against it.

The linux kernel puts the build ID into an easily accessible place starting with commit torvalds/linux@0935288c6e008 ("kdump: append kernel build-id string to VMCOREINFO"), which was merged in 5.9. So your vmcore, because it doesn't have that commit, doesn't contain the build ID for drgn to match against.

Like @osandov said, this isn't a problem if you use v0.0.30 (you can install it from pip, or build it from source after git checkout v0.0.30). And this will get fixed on the main branch soon.

[osandov]

Just pushed a fix. Please re-open if it doesn't work for you.