[lldb][test][win][x86_64] XFAIL already failing Shell/Driver tests

[kendalharland]

I'm currently working on getting the LLDB test suites to pass on Windows x86_64 which is not currently included in LLVM CI. These tests are currently failing in this configuration.

See #100474

[llvmbot]

@llvm/pr-subscribers-lldb

Author: Kendal Harland (kendalharland)

Changes

I'm currently working on getting the LLDB test suites to pass on Windows x86_64 which is not currently included in LLVM CI. These tests are currently failing on this configuration.

---

Full diff: https://github.com/llvm/llvm-project/pull/100473.diff

2 Files Affected:

• (modified) lldb/test/Shell/Driver/TestConvenienceVariables.test (+1)
• (modified) lldb/test/Shell/Driver/TestSingleQuote.test (+1)

diff --git a/lldb/test/Shell/Driver/TestConvenienceVariables.test b/lldb/test/Shell/Driver/TestConvenienceVariables.test
index 45dc7673bfc51..36ec0748a5a02 100644
--- a/lldb/test/Shell/Driver/TestConvenienceVariables.test
+++ b/lldb/test/Shell/Driver/TestConvenienceVariables.test
@@ -3,6 +3,7 @@ RUN: mkdir -p %t
 RUN: %build %p/Inputs/hello.cpp -o %t/target.out
 RUN: %lldb %t/target.out -s %p/Inputs/convenience.in -o quit | FileCheck %s
 
+# XFAIL: target=x86_64-{{.*}}-windows{{.*}}
 CHECK: stop reason = breakpoint 1.1
 CHECK: script print(lldb.debugger)
 CHECK-NEXT: Debugger (instance: {{.*}}, id: {{[0-9]+}})
diff --git a/lldb/test/Shell/Driver/TestSingleQuote.test b/lldb/test/Shell/Driver/TestSingleQuote.test
index af321ba04db39..5d721b5a3345c 100644
--- a/lldb/test/Shell/Driver/TestSingleQuote.test
+++ b/lldb/test/Shell/Driver/TestSingleQuote.test
@@ -2,5 +2,6 @@
 # RUN: %clang_host %p/Inputs/hello.c -g -o "%t-'pat"
 # RUN: %lldb -s %s "%t-'pat" | FileCheck %s
 
+# XFAIL: target=x86_64-{{.*}}-windows{{.*}}
 br set -p return
 # CHECK: Breakpoint 1: where = TestSingleQuote.test.tmp-'pat`main

[JDevlieghere]

How are these tests failing? Neither of them seem to be testing something specific to x86_64 and presumably the test are passing on the aarch64 windows buildbot?

[kendalharland]

How are these tests failing? Neither of them seem to be testing something specific to x86_64 and presumably the test are passing on the aarch64 windows buildbot?

I am currently seeing this output. I'm not really knowledgeable enough about LLDB to debug these tests quickly and am trying to get a working Windows x86_64 build ASAP, so apologies if I don't have much insight into why some of these tests are broken. If the output below is meaningful to you and looks like something I could fix pretty quickly - e.g. something I've probably not set in my local environment, I'm happy to try debugging it.

Testing:  0.. 10
FAIL: lldb-shell :: Driver/TestConvenienceVariables.test (48 of 2873)
******************** TEST 'lldb-shell :: Driver/TestConvenienceVariables.test' FAILED ********************
Script:
--
: 'RUN: at line 2';   mkdir -p S:\b\1\tools\lldb\test\Shell\Driver\Output\TestConvenienceVariables.test.tmp
: 'RUN: at line 3';   'C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\python.exe' S:\SourceCache\llvm-project\lldb\test\Shell\helper\build.py --compiler=any --arch=64 --tools-dir=S:/b/1/./bin --libs-dir=S:/b/1/./lib S:\SourceCache\llvm-project\lldb\test\Shell\Driver/Inputs/hello.cpp -o S:\b\1\tools\lldb\test\Shell\Driver\Output\TestConvenienceVariables.test.tmp/target.out
: 'RUN: at line 4';   s:\b\1\bin\lldb.exe --no-lldbinit -S S:/b/1/tools/lldb\test\Shell\lit-lldb-init-quiet S:\b\1\tools\lldb\test\Shell\Driver\Output\TestConvenienceVariables.test.tmp/target.out -s S:\SourceCache\llvm-project\lldb\test\Shell\Driver/Inputs/convenience.in -o quit | s:\b\1\bin\filecheck.exe S:\SourceCache\llvm-project\lldb\test\Shell\Driver\TestConvenienceVariables.test
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 2"
note: command had no output on stdout or stderr
$ "mkdir" "-p" "S:\b\1\tools\lldb\test\Shell\Driver\Output\TestConvenienceVariables.test.tmp"
note: command had no output on stdout or stderr
$ ":" "RUN: at line 3"
note: command had no output on stdout or stderr
$ "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\python.exe" "S:\SourceCache\llvm-project\lldb\test\Shell\helper\build.py" "--compiler=any" "--arch=64" "--tools-dir=S:/b/1/./bin" "--libs-dir=S:/b/1/./lib" "S:\SourceCache\llvm-project\lldb\test\Shell\Driver/Inputs/hello.cpp" "-o" "S:\b\1\tools\lldb\test\Shell\Driver\Output\TestConvenienceVariables.test.tmp/target.out"
# command output:
Cleaning hello.ilk
Cleaning target.out-hello.obj
Cleaning target.pdb
Cleaning target.out



compiling hello.cpp -> target.out-hello.obj
  STDOUT:




linking target.out-hello.obj -> target.out
  STDOUT:


$ ":" "RUN: at line 4"
note: command had no output on stdout or stderr
$ "s:\b\1\bin\lldb.exe" "--no-lldbinit" "-S" "S:/b/1/tools/lldb\test\Shell\lit-lldb-init-quiet" "S:\b\1\tools\lldb\test\Shell\Driver\Output\TestConvenienceVariables.test.tmp/target.out" "-s" "S:\SourceCache\llvm-project\lldb\test\Shell\Driver/Inputs/convenience.in" "-o" "quit"
note: command had no output on stdout or stderr
$ "s:\b\1\bin\filecheck.exe" "S:\SourceCache\llvm-project\lldb\test\Shell\Driver\TestConvenienceVariables.test"
# command stderr:
S:\SourceCache\llvm-project\lldb\test\Shell\Driver\TestConvenienceVariables.test:6:8: error: CHECK: expected string not found in input
CHECK: stop reason = breakpoint 1.1
       ^
<stdin>:1:1: note: scanning from here
(lldb) command source -s 0 'S:/b/1/tools/lldb\test\Shell\lit-lldb-init-quiet'
^
<stdin>:13:14: note: possible intended match here
* thread #1, stop reason = breakpoint 1.2
             ^

Input file: <stdin>
Check file: S:\SourceCache\llvm-project\lldb\test\Shell\Driver\TestConvenienceVariables.test

-dump-input=help explains the following input dump.

Input was:
<<<<<<
           1: (lldb) command source -s 0 'S:/b/1/tools/lldb\test\Shell\lit-lldb-init-quiet'
check:6'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
           2: Executing commands in 'S:\b\1\tools\lldb\test\Shell\lit-lldb-init-quiet'.
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           3: (lldb) command source -C --silent-run true lit-lldb-init
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           4: (lldb) target create "S:\\b\\1\\tools\\lldb\\test\\Shell\\Driver\\Output\\TestConvenienceVariables.test.tmp/target.out"
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           5: Current executable set to 'S:\b\1\tools\lldb\test\Shell\Driver\Output\TestConvenienceVariables.test.tmp\target.out' (x86_64).
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           6: (lldb) command source -s 0 'S:\SourceCache\llvm-project\lldb\test\Shell\Driver/Inputs/convenience.in'
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           7: Executing commands in 'S:\SourceCache\llvm-project\lldb\test\Shell\Driver\Inputs\convenience.in'.
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           8: (lldb) breakpoint set -f hello.cpp -p Hello
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           9: Breakpoint 1: where = target.out`main + 21 at hello.cpp:4, address = 0x0000000140001015
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          10: (lldb) run
check:6'0     ~~~~~~~~~~~
          11: 1 location added to breakpoint 1
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          12: (lldb) Process 51956 stopped
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          13: * thread #1, stop reason = breakpoint 1.2
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
check:6'1                  ?                             possible intended match
          14:  frame #0: 0x00007ff7031b1015 target.out`main(argc=1, argv=0x0000010f1571f7e0) at hello.cpp:4
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          15:  1 #include<stdio.h>
check:6'0     ~~~~~~~~~~~~~~~~~~~~~
          16:  2
check:6'0     ~~~~
          17:  3 int main(int argc, char **argv) {
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          18: -> 4 printf("Hello World\n");
check:6'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           .
           .
           .
>>>>>>

error: command failed with exit status: 1

[DavidSpickett]

S:\SourceCache\llvm-project\lldb\test\Shell\Driver\TestConvenienceVariables.test:6:8: error: CHECK: expected string not found in input
CHECK: stop reason = breakpoint 1.1
       ^
<stdin>:1:1: note: scanning from here
(lldb) command source -s 0 'S:/b/1/tools/lldb\test\Shell\lit-lldb-init-quiet'
^
<stdin>:13:14: note: possible intended match here
* thread #1, stop reason = breakpoint 1.2
             ^

I think it's just that it expected 1.1 and got 1.2. I don't know what the bit after the point means, is it the location? So this did breakpoint set -f hello.cpp -p Hello and somehow 1 location added to breakpoint 1 but yet it hit a second location?

Which doesn't make sense so maybe the .2 means something else.

[DavidSpickett]

Neither of them seem to be testing something specific to x86_64 and presumably the test are passing on the aarch64 windows buildbot?

More likely than not.

Though the test step is not verbose for some reason despite being told to be. I'll see if I can fix that.

[labath]

S:\SourceCache\llvm-project\lldb\test\Shell\Driver\TestConvenienceVariables.test:6:8: error: CHECK: expected string not found in input
CHECK: stop reason = breakpoint 1.1
       ^
<stdin>:1:1: note: scanning from here
(lldb) command source -s 0 'S:/b/1/tools/lldb\test\Shell\lit-lldb-init-quiet'
^
<stdin>:13:14: note: possible intended match here
* thread #1, stop reason = breakpoint 1.2
             ^

I think it's just that it expected 1.1 and got 1.2. I don't know what the bit after the point means, is it the location? So this did breakpoint set -f hello.cpp -p Hello and somehow 1 location added to breakpoint 1 but yet it hit a second location?

Which doesn't make sense so maybe the .2 means something else.

That's exactly what it means. The breakpoint gets one location when its set (the process is not running yet):

      9: Breakpoint 1: where = target.out`main + 21 at hello.cpp:4, address = 0x0000000140001015

and then another when when it runs:

      11: 1 location added to breakpoint 1

Although this test doesn't really care about the number of breakpoint locations (so we could just delete the .1 part), I am wondering if this isn't a symptom of a larger problem. Can you try running these commands manually and print more information about the state of the process (for starters, the output of image list and breakpoint list -v both before and after run command).

Also, the duplicate location reminds me of some similar situations where we (used to?) fail to correctly unique modules in presence of symlinks and stuff. What can you tell us about the path (S:\b\1\tools\lldb\test\Shell\Driver\Output\TestConvenienceVariables.test.tmp\target.out) where this binary is running from? Does it contain any symlinks (junctions), funky drive mappins or anything else that would make the file appear to have multiple paths?

[kendalharland]

S:\SourceCache\llvm-project\lldb\test\Shell\Driver\TestConvenienceVariables.test:6:8: error: CHECK: expected string not found in input
CHECK: stop reason = breakpoint 1.1
       ^
<stdin>:1:1: note: scanning from here
(lldb) command source -s 0 'S:/b/1/tools/lldb\test\Shell\lit-lldb-init-quiet'
^
<stdin>:13:14: note: possible intended match here
* thread #1, stop reason = breakpoint 1.2
             ^

I think it's just that it expected 1.1 and got 1.2. I don't know what the bit after the point means, is it the location? So this did breakpoint set -f hello.cpp -p Hello and somehow 1 location added to breakpoint 1 but yet it hit a second location?
Which doesn't make sense so maybe the .2 means something else.

That's exactly what it means. The breakpoint gets one location when its set (the process is not running yet):

      9: Breakpoint 1: where = target.out`main + 21 at hello.cpp:4, address = 0x0000000140001015

and then another when when it runs:

      11: 1 location added to breakpoint 1

Although this test doesn't really care about the number of breakpoint locations (so we could just delete the .1 part), I am wondering if this isn't a symptom of a larger problem. Can you try running these commands manually and print more information about the state of the process (for starters, the output of image list and breakpoint list -v both before and after run command).

Also, the duplicate location reminds me of some similar situations where we (used to?) fail to correctly unique modules in presence of symlinks and stuff. What can you tell us about the path (S:\b\1\tools\lldb\test\Shell\Driver\Output\TestConvenienceVariables.test.tmp\target.out) where this binary is running from? Does it contain any symlinks (junctions), funky drive mappins or anything else that would make the file appear to have multiple paths?

After #100477 I learned that my CMake configuration is largely to blame for most of the failures I was observing. TestConvenienceVariables.test is now passing for me and I am investigating why TestSingleQuote.test is still failing. I will fold this change into #100476 and work with @dzhidzhoev to figure out why my build results do match those in his Windows Native CI and those in lldb-aarch-windows first.