go.testFlags doesn't pass items before -args to debug binaries

[mgabeler-lee-6rs]

What version of Go, VS Code & VS Code Go extension are you using?

• Run go version to get version of Go from the VS Code integrated terminal.
  • go version go1.16.6 linux/amd64
• Run gopls -v version to get version of Gopls from the VS Code integrated terminal.
  • golang.org/x/tools/gopls v0.7.0
• Run code -v or code-insiders -v to get version of VS Code or VS Code Insiders.
  • 1.58.2 c3f126316369cd610563c75b1b1725e0679adfb3 x64
• Check your installed extensions to get the version of the VS Code Go extension
  • 0.26.0
• Run Ctrl+Shift+P (Cmd+Shift+P on Mac OS) > Go: Locate Configured Go Tools command.

Checking configured tools....

GOBIN: undefined
toolsGopath: 
gopath: /home/mgl/src/go
GOROOT: /usr/lib/go-1.16
PATH: /home/mgl/.nodenv/shims:/usr/lib/ccache:/home/mgl/src/mgl-incident-tools/bin:/home/mgl/bin:/home/mgl/.local/bin:/home/mgl/.nodenv/bin:/home/mgl/src/go/bin:/sbin:/usr/local/sbin:/usr/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/mgl/.npm/bin

   gopkgs: /home/mgl/src/go/bin/gopkgs installed
   go-outline: /home/mgl/src/go/bin/go-outline installed
   gotests: /home/mgl/src/go/bin/gotests installed
   gomodifytags: gomodifytags not installed
   impl: /home/mgl/src/go/bin/impl installed
   goplay: /home/mgl/src/go/bin/goplay installed
   dlv: /home/mgl/src/go/bin/dlv installed
   dlv-dap: dlv-dap not installed
   staticcheck: /home/mgl/src/go/bin/staticcheck installed
   gopls: /home/mgl/src/go/bin/gopls installed

Share the Go related settings you have added/edited

Run Preferences: Open Settings (JSON) command to open your settings.json file.
Share all the settings with the go. or ["go"] or gopls prefixes.

Details

  "go.useLanguageServer": true,
  "go.gotoSymbol.includeImports": true,
  "go.buildOnSave": "workspace",
  "go.lintOnSave": "workspace",
  "go.vetOnSave": "workspace",
  "go.testOnSave": true,
  "go.testTimeout": "10s",
  "go.delveConfig": {
    "dlvLoadConfig": {
      "followPointers": true,
      "maxVariableRecurse": 1,
      "maxStringLen": 1024,
      "maxArrayValues": 1024,
      "maxStructFields": -1
    },
    "apiVersion": 2,
    "showGlobalVariables": false
  },
  "go.testFlags": [
    "-v"
  ],
  "go.toolsManagement.autoUpdate": true,

Describe the bug

This is basically a re-hash / follow-up to old issue microsoft/vscode-go#2115

If you set go.testFlags to ["-v"], then you get verbose test output when running tests but not debugging them. If you set it to ["-args","-test.v"] then you get verbose output when debugging but not when running. You have to combine the two via ["-v", "-args", "-test.v"] to get both behaviors.

This is quite confusing. It seems that testFlags elements before -args ought to be passed to the debug binary with the test. prefix added, the same way go test does when non-debug running tests.

Steps to reproduce the behavior:

1. Set go.testFlags to ["-v"]
2. Select a test
3. Set a breakpoint within it
4. When the breakpoint hits, run ps auxww | grep test or such to see the running processes, observe that the test is run as /path/to/something.test -run ^TestName$, instead of as /path/to/something.test -run ^TestName$ -test.v

[findleyr]

I agree that this is confusing, though I'm not sure we can change the behavior at this point. We will discuss this in our triage meeting.

For reference, would you find a separate configuration option for test arguments to be equally confusing?

[mgabeler-lee-6rs]

From my perspective, there's "flags for building the test binary", and "flags for running the test binary". I would like to be able to configure the latter once, and have them apply equally whether I'm running or debugging that binary. If having go.testBuildFlags and go.testRunFlags was the "easy" way to do that, it'd be fine with me. My only concern with that model would be the confusion of folks expecting -v to "work", and not realizing that it has to be 'spelled' -test.v.

Also, poking around, I'm seeing some extra fun behavior where go test ./package/ -args test.v ... doesn't print the verbose stuff, and I instead have to write it as ... -test.v=true. But when running a compiled test binary, -test.v works fine. After looking at what go test -v does for exec calls under strace, I ... think that go test is itself interpreting the -v arg to control output suppression, in addition to passing it to the test binary it builds & runs to control what output is generated.

Which makes me think that making the use case of "I want vscode-go to always run my tests with verbose enabled" work without arcane invocations may simply require a special case no matter what (absent any changes to go test itself, anyways).

go test ./package/ -args test.v behaving differently than go test ./package/ -args test.v=true ... just feels like a 🐛 though 🙂

[findleyr]

We discussed this today, and consensus was that we can't change this behavior at this point: we can't break users who have configurations containing -args.

Also, poking around, I'm seeing some extra fun behavior where go test ./package/ -args test.v ... doesn't print the verbose stuff, and I instead have to write it as ... -test.v=true

I'm not seeing this behavior. I see -test.v and -test.v=true behaving identically. It would be very surprising if they behaved differently, given that this is a bool flag.

[mgabeler-lee-6rs]

I'm not seeing this behavior

Hmm ... yeah, I think that was an artifact of caching, or more likely typos. Doing a more careful, isolated test (below), I'm finding that no variation on -test.v after -args produces verbose output from go test ./packagename, as it seems that, even if the test app emits the output, go test suppresses it.

So it seems that, to reliably get verbose test output with the extension, the only way to do it right now is to double-request it by using -v -args -test.v.

I guess the question is, should the extension replicate how go test converts "before the -args" flags to "after the -args" flags? I'm not sure that making it do so would be a breaking change? Certainly users would no longer be able to replicate the "inconsistent" behavior of only getting verbose output when they don't debug the tests, but I don't personally see the utility in that behavior 😉

To me, having the extension act like go test is desirable, as is having "Run test" and "Debug test" run the test with the same args. Having those two behave differently is, to me, the crux of the "bug" here.

---

More fiddly bits:

Setup I'm working with:

1. go mod init example
2. cat demo_test.go:

package example
import "testing"
func TestDemo(t *testing.T) {
    t.Log("verbose output")
}

• go test -args -test.v -- produces verbose output
• go test ./ -args -test.v -- does not produce verbose output (strace shows the test binary is producing it, but the test runner is suppressing it)

I think the difference here is, subtly, captured by the go help test output:

The first, called local directory mode, occurs when go test is invoked with no package arguments ...
In this mode, go test compiles the package sources and tests found in the current directory and then runs the resulting test binary ...

The second, called package list mode, occurs when go test is invoked with explicit package arguments ...
In this mode, go test compiles and tests each of the packages listed on the command line.
If a package test passes, go test prints only the final 'ok' summary line.
(emphasis mine)

Confusing, but documented 🤷‍♂️