Skip to content

[lldb][test][win][x86_64] Fix XFAIL and XPASS on LLDB API tests #100477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

[lldb][test][win][x86_64] Fix XFAIL and XPASS on LLDB API tests #100477

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions lldb/test/API/functionalities/multiple-slides/TestMultipleSlides.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
class MultipleSlidesTestCase(TestBase):
NO_DEBUG_INFO_TESTCASE = True

@expectedFailureAll(oslist=["windows"], archs=["x86_64"])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you provide a failure log for this? It passes on our CI (Windows native x86_64).

********************
PASS: lldb-api :: functionalities/multiple-slides/TestMultipleSlides.py (530 of 1191)
Script:
--
C:/Python312/python.exe C:/lldb\test\API\dotest.py -u CXXFLAGS -u CFLAGS --env OBJCOPY=C:/build-lldb-native/./bin/llvm-objcopy.exe --env STRIP=C:/build-lldb-native/./bin/llvm-strip.exe --env LLVM_LIBS_DIR=C:/build-lldb-native/./lib --env LLVM_INCLUDE_DIR=C:/build-lldb-native/include --env LLVM_TOOLS_DIR=C:/build-lldb-native/./bin --arch x86_64 --build-dir C:/build-lldb-native/lldb-test-build.noindex --lldb-module-cache-dir C:/build-lldb-native/lldb-test-build.noindex/module-cache-lldb\lldb-api --clang-module-cache-dir C:/build-lldb-native/lldb-test-build.noindex/module-cache-clang\lldb-api --executable C:/build-lldb-native/./bin/lldb.exe --make c:/gl/b1/3rdpaty/bin/make-wfix.exe --compiler C:/build-lldb-native/./bin/clang.exe --dsymutil C:/build-lldb-native/./bin/dsymutil.exe --llvm-tools-dir C:/build-lldb-native/./bin --lldb-obj-root C:/build-lldb-native/tools/lldb --lldb-libs-dir C:/build-lldb-native/./lib --skip-category=watchpoint C:\lldb\test\API\functionalities\multiple-slides -p TestMultipleSlides.py
--
Exit Code: 0

Command Output (stdout):
--
lldb version 20.0.0git (https://gitlab-ci-token:[email protected]/accesssoftek/lldb-test-scripts.git revision cbd39658b6aed4dbb66a2b775096974c9087e93d)
  clang revision cbd39658b6aed4dbb66a2b775096974c9087e93d
  llvm revision cbd39658b6aed4dbb66a2b775096974c9087e93d
Skipping the following test categories: ['watchpoint', 'libc++', 'libstdcxx', 'dwo', 'dsym', 'gmodules', 'debugserver', 'objc', 'fork', 'pexpect']


--
Command Output (stderr):
--
PASS: LLDB (C:\build-lldb-native\bin\clang.exe-x86_64) :: test_mulitple_slides (TestMultipleSlides.MultipleSlidesTestCase.test_mulitple_slides)

----------------------------------------------------------------------

Ran 1 test in 2.447s



OK


--

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it may be worth cherry-picking this 58d22f1 to have build commands and outputs in the log without tracing hassle.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ty! I'll try that out.

Copy link
Contributor Author

@kendalharland kendalharland Jul 29, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like after my CMake chages this one is passing now. I'll remove it from the PR

Copy link
Contributor Author

@kendalharland kendalharland Jul 29, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, sorry I spoke too soon. On windows I am seeing this fail. The test is built without debug information because this is in its makefile:

C_SOURCES := main.c
MAKE_DSYM := NO

include Makefile.rules

main.o: main.c
	$(CC) $(CFLAGS_NO_DEBUG) -c $< -o $@

@dzhidzhoev I am curious to know how this is passing for you? Without specifying -g or -gdwarf to Clang, lldb cannot resolve the name first and fails for me locally. But adding either of those flags fixes the problem.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For comparison, my objdump output shows no symbol table:

PS C:\workspace\llvm-project> C:\workspace\llvm-project\build\bin\llvm-objdump -t C:\workspace\llvm-project\build\lldb-test-build.noindex\functionalities\multiple-slides\TestMultipleSlides.test_mulitple_slides\a.out

C:\workspace\llvm-project\build\lldb-test-build.noindex\functionalities\multiple-slides\TestMultipleSlides.test_mulitple_slides\a.out:  file format coff-x86-64

SYMBOL TABLE:

Copy link
Member

@dzhidzhoev dzhidzhoev Jul 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

$ F:/Users/vdzhidzhoev/temp/build-lldb-native/bin/clang.exe main.o -gdwarf -O0  -m64 -IF:\Users\vdzhidzhoev\temp\llvm-project\lldb\packages\Python\lldbsuite\test\make/../../../../..//include -IF:/Users/vdzhidzhoev/temp/build-lldb-native/tools/lldb/include -IF:\Users\vdzhidzhoev\temp\llvm-project\lldb\test\API\functionalities\multiple-slides -IF:\Users\vdzhidzhoev\temp\llvm-project\lldb\packages\Python\lldbsuite\test\make -include F:\Users\vdzhidzhoev\temp\llvm-project\lldb\packages\Python\lldbsuite\test\make/test_common.h -fno-limit-debug-info     -fuse-ld=lld  --driver-mode=g++ -o "a.out" -v

clang version 20.0.0git (8d8d23fec72ad6ff77e15e7627ab8cc26ac36d4c)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: F:\Users\vdzhidzhoev\temp\build-lldb-native\bin
Build config: +assertions
 "F:\\Users\\vdzhidzhoev\\temp\\build-lldb-native\\bin\\lld-link" -out:a.out -defaultlib:libcmt -defaultlib:oldnames -nologo -debug main.o

Copy link
Member

@dzhidzhoev dzhidzhoev Jul 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For comparison, my objdump output shows no symbol table:

PS C:\workspace\llvm-project> C:\workspace\llvm-project\build\bin\llvm-objdump -t C:\workspace\llvm-project\build\lldb-test-build.noindex\functionalities\multiple-slides\TestMultipleSlides.test_mulitple_slides\a.out

C:\workspace\llvm-project\build\lldb-test-build.noindex\functionalities\multiple-slides\TestMultipleSlides.test_mulitple_slides\a.out:  file format coff-x86-64

SYMBOL TABLE:

I have the same for a.out. The dump I provided (with "first") was for main.o.
Try launching bin/lldb, "file a.out", "image dump symtab a.out", and there should be "first" symbol. Beware that it will probably show A LOT of entries.

Copy link
Contributor Author

@kendalharland kendalharland Jul 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My output for image dump symtab a.out is empty (just prints the path to a.out). However, if I edit

llvm-project/lldb/test/API/functionalities/multiple-slides/Makefile

Line 11 in b66aa3b

$(CC) $(CFLAGS_NO_DEBUG) -c $< -o $@
to use CFLAGS instead of CFLAGS_NO_DEBUG lldb I can see symfile output for a.out.:

(lldb) image dump symfile
Dumping debug symbols for 1 modules.
SymbolFile dwarf (C:\workspace\llvm-project\build\lldb-test-build.noindex\functionalities\multiple-slides\TestMultipleSlides.test_mulitple_slides\a.out)
Types:

Compile units:

Symtab, file = C:\workspace\llvm-project\build\lldb-test-build.noindex\functionalities\multiple-slides\TestMultipleSlides.test_mulitple_slides\a.out, num_symbols = 0
Manual DWARF index for (x86_64) 'a.out':
Function basenames:
INFO/0000006f "main"

Function fullnames:
INFO/0000006f "main"

Function methods:

Function selectors:

Objective-C class selectors:

Globals and statics:
INFO/0000005a "second"
INFO/0000002a "first"

Types:
INFO/00000053 "__ARRAY_SIZE_TYPE__"
INFO/0000004c "int"

Namespaces:

Do you have a patched or older version of that file, or have settings in your environment that force -gdwarf to be passed to clang? I ask because although I am now able to see debug info, I still am wondering if there's something in my local environment that is causing different behavior. Additionally, LLDB produces no output when I run image lookup -s first, which means the tests still fail locally since they can't resolve the symbols first and second, but lldb can still resolve first and second if I instead run ta v first or ta v second. Something is preventing it from looking up the symbols for first and second even though they are clearly present as globals

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to figure this out. My CMakeCache had -D LLVM_ENABLE_DIA_SDK="OFF" which I must have set in a previous build. I don't recall why I set this, must have been trying to repro someone else's build and forgot. This indeed passes for me now. Thanks so much for your patience and helping me debug!

def test_mulitple_slides(self):
"""Test that a binary can be slid multiple times correctly."""
self.build()
Expand Down
2 changes: 1 addition & 1 deletion .../lang/cpp/class-template-non-type-parameter-pack/TestClassTemplateNonTypeParameterPack.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

class TestCaseClassTemplateNonTypeParameterPack(TestBase):
@expectedFailureAll(
oslist=["windows"], archs=["i[3-6]86", "x86_64"]
oslist=["windows"], archs=["i[3-6]86"]
) # Fails to read memory from target.
@no_debug_info_test
def test(self):
Expand Down
2 changes: 1 addition & 1 deletion ...est/API/lang/cpp/class-template-type-parameter-pack/TestClassTemplateTypeParameterPack.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

class TestCaseClassTemplateTypeParameterPack(TestBase):
@expectedFailureAll(
oslist=["windows"], archs=["i[3-6]86", "x86_64"]
oslist=["windows"], archs=["i[3-6]86"]
) # Fails to read memory from target.
@no_debug_info_test
def test(self):
Expand Down
1 change: 1 addition & 0 deletions lldb/test/API/python_api/thread/TestThreadAPI.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ def test_negative_indexing(self):
self.build()
self.validate_negative_indexing()

@expectedFailureAll(oslist=["windows"], archs=["x86_64"])
Copy link
Contributor Author

@kendalharland kendalharland Jul 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this line, I am still seeing this test fail with:

======================================================================
FAIL: test_StepInstruction_dwarf (TestThreadAPI.ThreadAPITestCase.test_StepInstruction_dwarf)
    Test that StepInstruction preserves the plan stack.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\workspace\llvm-project\lldb\packages\Python\lldbsuite\test\lldbtest.py", line 1756, in test_method
    return attrvalue(self)
           ^^^^^^^^^^^^^^^
  File "C:\workspace\llvm-project\lldb\test\API\python_api\thread\TestThreadAPI.py", line 58, in test_StepInstruction
    self.step_instruction_in_called_function()
  File "C:\workspace\llvm-project\lldb\test\API\python_api\thread\TestThreadAPI.py", line 334, in step_instruction_in_called_function
    self.assertRegex(
AssertionError: Regex didn't match: '.*call_me\\(bool\\)$' not found in 'main' : Stopped in call_me(bool)
Config=x86_64-C:\workspace\llvm-project\build\bin\clang.exe
----------------------------------------------------------------------

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recently updated this for Windows on Arm 3e06392, that's where that regex came from.

We need to know if it's a cosmetic difference or whether it's stopped at a completely different place. Running the test with lldb-dotest and the -t flag will hopefully show you what it was doing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I'll try that and update here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the full output with 3e06392 cherry-picked. https://pastebin.com/LQ9QReBr

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I change this line:

self.runCmd("expr -i 0 -- call_me(true)", check=False)

to this:

self.runCmd("expr -i 0 -- call_me(true)", check=True)

I see this error:

======================================================================
FAIL: test_StepInstruction_dwarf (TestThreadAPI.ThreadAPITestCase.test_StepInstruction_dwarf)
   Test that StepInstruction preserves the plan stack.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\workspace\llvm-project\lldb\packages\Python\lldbsuite\test\lldbtest.py", line 1761, in test_method
    return attrvalue(self)
           ^^^^^^^^^^^^^^^
  File "C:\workspace\llvm-project\lldb\test\API\python_api\thread\TestThreadAPI.py", line 16, in test_StepInstruction
    self.step_instruction_in_called_function()
  File "C:\workspace\llvm-project\lldb\test\API\python_api\thread\TestThreadAPI.py", line 291, in step_instruction_in_called_function
    self.runCmd("expr -i 0 -- call_me(true)", check=True)
  File "C:\workspace\llvm-project\lldb\packages\Python\lldbsuite\test\lldbtest.py", line 1006, in runCmd
    self.assertTrue(self.res.Succeeded(), msg if (msg) else CMD_MSG(cmd))
AssertionError: False is not true : Command 'expr -i 0 -- call_me(true)
Error output:
error: Couldn't look up symbols:
  void call_me(bool)
Hint: The expression tried to call a function that is not present in the target, perhaps because it was optimized out by the compiler.
' did not return successfully
Config=x86_64-C:\workspace\llvm-project\build\bin\clang.exe
----------------------------------------------------------------------

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like my Clang invocation is generating different debug information than expected:

PS C:\workspace\llvm-project> llvm-pdbutil.exe pretty -all C:\workspace\llvm-project\build\lldb-test-build.noindex\python_api\thread\TestThreadAPI.test_StepInstruction_dwarf\a.pdb | Select-String -Pattern call_me

  func [0x00001000+ 0 - 0x00001025-37 | sizeof= 37] (FPO) void __cdecl call_me(bool should_spin)
  func [0x00001000+ 0 - 0x00001025-37 | sizeof= 37] (FPO) void __cdecl call_me(bool should_spin)
  public [0x00001000] ?call_me@@YAX_N@Z

Is the issue here that debug symbols have the additional should_spin parameter name, thus LLDB is not able to look them up since it is search for void call_me(bool)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the same function name/signature.
Btw, I'd recommend adding --env;LLDB_USE_LLDB_SERVER=1 to LLDB_TEST_USER_ARGS cmake argument. Without it, lldb can't even stop on a breakpoint on my machine. It may be helpful if you haven't set it yet.

def test_StepInstruction(self):
"""Test that StepInstruction preserves the plan stack."""
self.build()
Expand Down
Loading