debug: hard to debug a binary built with -trimpath

[hyangah]

go version: go1.18beta1 (but the problem applies to older versions of go too)
dlv version: 2f13672

When a binary is built with -trimpath, part of file paths are removed from the debug info.

It is recommended to configure substitutePath to handle this.
But, it is challenging to come up with good substitutePath rules since -trimpath produces file paths in the form of:

• standard libs: /
  e.g.: runtime/map.go
• 3rd party dependencies in module cache: module@version/package/path/file.go
  e.g.: golang.org/x/mod@v0.5.1/semver/semver.go (this can be interpreted as a path relative to MODCACHE)
• files in main module (workspace): module/package/path/file.go
  e.g: work/main.go (when the main module's name is work)

More complication:

• vendored 3rd party dependencies or replaced 3rd party dependencies
  -> original module@version/package/path/file.go is used instead of file path in vendor or replace directory

<autogenerated>
golang.org/x/mod@v0.5.1/semver/semver.go
...
runtime/map.go
sort/sort.go
...
work/main.go

Repro:

~/ww % cat go.mod
module work

go 1.17

require golang.org/x/mod v0.5.1

~/ww % cat main.go
package main

import (
        "golang.org/x/mod/semver"
)

func main() {
        if semver.IsValid("1.0.0") {
                println("valid")
        }
}

Launch configuration

        {
            "name": "Launch Package",
            "type": "go",
            "request": "launch",
            "mode": "debug",
            "program": "${fileDirname}",
            "buildFlags": "-trimpath",
            "stopOnEntry": true,
            "showLog": true,
            "logOutput": "dap"
        }

Delve DAP log

2021-12-31T17:18:43-05:00 debug layer=dap [<- from client]{"seq":15,"type":"request","command":"disconnect","arguments":{"restart":true,"terminateDebuggee":true}}
2021-12-31T17:18:43-05:00 debug layer=dap halting
2021-12-31T17:18:43-05:00 debug layer=dap process not running
2021-12-31T17:18:43-05:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"output","body":{"category":"console","output":"Detaching and terminating target process\n","source":{}}}
Detaching and terminating target process
2021-12-31T17:18:43-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":15,"success":true,"command":"disconnect"}
2021-12-31T17:18:43-05:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"terminated","body":{}}
2021-12-31T17:18:43-05:00 debug layer=dap DAP server stopping...
2021-12-31T17:18:43-05:00 debug layer=dap DAP server stopped
dlv dap (81795) exited with code: 0
Starting: /Users/hakim/go/bin/dlv-dap dap --check-go-version=false --listen=127.0.0.1:50368 --log=true --log-output=dap --log-dest=3 from /Users/hakim/ww
DAP server listening at: 127.0.0.1:50368
2021-12-31T17:18:43-05:00 debug layer=dap DAP server pid = 81923
2021-12-31T17:18:43-05:00 debug layer=dap DAP connection 1 started
2021-12-31T17:18:43-05:00 debug layer=dap [<- from client]{"seq":1,"type":"request","command":"initialize","arguments":{"clientID":"vscode","clientName":"Visual Studio Code","adapterID":"go","locale":"en-us","linesStartAt1":true,"columnsStartAt1":true,"pathFormat":"path","supportsVariableType":true,"supportsVariablePaging":true,"supportsRunInTerminalRequest":true,"supportsMemoryReferences":true,"supportsProgressReporting":true,"supportsInvalidatedEvent":true}}
2021-12-31T17:18:43-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":1,"success":true,"command":"initialize","body":{"supportsConfigurationDoneRequest":true,"supportsFunctionBreakpoints":true,"supportsConditionalBreakpoints":true,"supportsEvaluateForHovers":true,"supportsSetVariable":true,"supportsExceptionInfoRequest":true,"supportTerminateDebuggee":true,"supportsDelayedStackTraceLoading":true,"supportsLogPoints":true,"supportsDisassembleRequest":true,"supportsClipboardContext":true,"supportsSteppingGranularity":true,"supportsInstructionBreakpoints":true}}
2021-12-31T17:18:43-05:00 debug layer=dap [<- from client]{"seq":2,"type":"request","command":"launch","arguments":{"name":"Launch Package","type":"go","request":"launch","mode":"debug","program":".","buildFlags":"-trimpath","stopOnEntry":true,"showLog":true,"logOutput":"dap","__configurationTarget":5,"packagePathToGoModPathMap":{"/Users/hakim/ww":"/Users/hakim/ww"},"debugAdapter":"dlv-dap","dlvToolPath":"/Users/hakim/go/bin/dlv-dap","env":{},"__buildDir":"/Users/hakim/ww","__sessionId":"23c200ff-23a4-4336-9b50-c03565f1a05c"}}
2021-12-31T17:18:43-05:00 debug layer=dap parsed launch config: {
	"mode": "debug",
	"program": ".",
	"buildFlags": "-trimpath",
	"stopOnEntry": true,
	"backend": "default",
	"stackTraceDepth": 50
}
2021-12-31T17:18:43-05:00 debug layer=dap building from "/Users/hakim/ww": [go build -o /var/folders/bw/6r6k9d113sv1_vvzk_1kfxbm001py5/T/__debug_bin2675042589 -gcflags all=-N -l -trimpath .]
2021-12-31T17:18:43-05:00 debug layer=dap launching binary '/var/folders/bw/6r6k9d113sv1_vvzk_1kfxbm001py5/T/__debug_bin2675042589' with config: {
	"mode": "debug",
	"program": "/Users/hakim/ww",
	"cwd": "/Users/hakim/ww",
	"buildFlags": "-trimpath",
	"output": "/var/folders/bw/6r6k9d113sv1_vvzk_1kfxbm001py5/T/__debug_bin2675042589",
	"dlvCwd": "/Users/hakim/ww",
	"stopOnEntry": true,
	"backend": "default",
	"stackTraceDepth": 50
}
WARNING: undefined behavior - version of Delve is too old for Go version 1.18.-1 (maximum supported version 1.17)
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"initialized"}
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":2,"success":true,"command":"launch"}
2021-12-31T17:18:44-05:00 debug layer=dap [<- from client]{"seq":3,"type":"request","command":"setBreakpoints","arguments":{"source":{"name":"main.go","path":"/Users/hakim/ww/main.go"},"breakpoints":[{"line":8}],"lines":[8]}}
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":3,"success":true,"command":"setBreakpoints","body":{"breakpoints":[{"verified":false,"message":"could not find file /Users/hakim/ww/main.go","source":{}}]}}
2021-12-31T17:18:44-05:00 debug layer=dap [<- from client]{"seq":4,"type":"request","command":"setFunctionBreakpoints","arguments":{"breakpoints":[]}}
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":4,"success":true,"command":"setFunctionBreakpoints","body":{"breakpoints":[]}}
2021-12-31T17:18:44-05:00 debug layer=dap [<- from client]{"seq":5,"type":"request","command":"setInstructionBreakpoints","arguments":{"breakpoints":[]}}
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":5,"success":true,"command":"setInstructionBreakpoints","body":{"breakpoints":[]}}
2021-12-31T17:18:44-05:00 debug layer=dap [<- from client]{"seq":6,"type":"request","command":"configurationDone","arguments":{}}
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"stopped","body":{"reason":"entry","threadId":1,"allThreadsStopped":true}}
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"output","body":{"category":"console","output":"Type 'dlv help' for list of commands.\n","source":{}}}
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":6,"success":true,"command":"configurationDone"}
Type 'dlv help' for list of commands.
2021-12-31T17:18:44-05:00 debug layer=dap [<- from client]{"seq":7,"type":"request","command":"threads"}
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":7,"success":true,"command":"threads","body":{"threads":[{"id":1,"name":"Dummy"}]}}
2021-12-31T17:18:44-05:00 debug layer=dap [<- from client]{"seq":8,"type":"request","command":"threads"}
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":8,"success":true,"command":"threads","body":{"threads":[{"id":1,"name":"Dummy"}]}}
2021-12-31T17:18:44-05:00 debug layer=dap [<- from client]{"seq":9,"type":"request","command":"stackTrace","arguments":{"threadId":1,"levels":1,"format":{}}}
2021-12-31T17:18:44-05:00 debug layer=dap Unable to produce stack trace: unknown goroutine 1
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":9,"success":false,"command":"stackTrace","message":"Unable to produce stack trace","body":{"error":{"id":2004,"format":"Unable to produce stack trace: unknown goroutine 1","showUser":false}}}
2021-12-31T17:18:44-05:00 debug layer=dap [<- from client]{"seq":10,"type":"request","command":"stackTrace","arguments":{"threadId":1,"levels":20,"format":{}}}
2021-12-31T17:18:44-05:00 debug layer=dap Unable to produce stack trace: unknown goroutine 1
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":10,"success":false,"command":"stackTrace","message":"Unable to produce stack trace","body":{"error":{"id":2004,"format":"Unable to produce stack trace: unknown goroutine 1","showUser":false}}}
2021-12-31T17:18:44-05:00 debug layer=dap [<- from client]{"seq":11,"type":"request","command":"stackTrace","arguments":{"threadId":1,"levels":19,"format":{}}}
2021-12-31T17:18:44-05:00 debug layer=dap Unable to produce stack trace: unknown goroutine 1
2021-12-31T17:18:44-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":11,"success":false,"command":"stackTrace","message":"Unable to produce stack trace","body":{"error":{"id":2004,"format":"Unable to produce stack trace: unknown goroutine 1","showUser":false}}}
dlv sources main.go
2021-12-31T17:18:51-05:00 debug layer=dap [<- from client]{"seq":12,"type":"request","command":"evaluate","arguments":{"expression":"dlv sources main.go","context":"repl","format":{}}}
2021-12-31T17:18:51-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":12,"success":true,"command":"evaluate","body":{"result":"work/main.go","presentationHint":{},"variablesReference":0}}
work/main.go
dlv sources semver.go
2021-12-31T17:18:56-05:00 debug layer=dap [<- from client]{"seq":13,"type":"request","command":"evaluate","arguments":{"expression":"dlv sources semver.go","context":"repl","format":{}}}
2021-12-31T17:18:56-05:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":13,"success":true,"command":"evaluate","body":{"result":"golang.org/x/mod@v0.5.1/semver/semver.go","presentationHint":{},"variablesReference":0}}
golang.org/x/mod@v0.5.1/semver/semver.go

[hyangah]

@suzmue @polinasok This may affect blaze users and google internal binary debugging.

[polinasok]

Related discussion in go-delve/delve#2754

[firelizzard18]

I think this is a better place for it, so I'm copying my comment from #2609.

Capitalization is another potential pitfall. The project I am working on has a dependency with a capitalized name in the path. With -trimpath the binary has paths such as github.com/AccumulateNetwork/jsonrpc2/v15@v15.0.0-20220517212445-953ad957e040/handler.go which needs to be resolved to /home/ME/go/pkg/mod/github.com/!accumulate!network/jsonrpc2/v15@v15.0.0-20220517212445-953ad957e040/handler.go. I have { "to": "", "from": "${env:HOME}/go/pkg/mod/" } as my last substitution rule, but that doesn't cover the capitalization case. Ultimately the conflict is that the go module system uses the !a substitution rule for filesystem paths, but does not apply that rule when synthesizing paths for -trimpath.

[dlipovetsky]

In fa820d4, we document a strategy for constructing substitution paths, namely:

"substitutePath": [
  // Main module.
  {
   "from": "${workspaceFolder}",
   "to": "moduleName",
  },
  // Module cache paths.
  {
   "from": "${env:HOME}/go/pkg/mod/github.com",
   "to": "github.com",
  },
  {
   "from": "${env:HOME}/go/pkg/mod/golang.org",
   "to": "golang.org",
  },
  ...
  // Standard library paths.
  // This rule should come last since the empty "to" will match every path.
  { "from": "/path/to/local/goroot/pkg" , "to": ""}
 ],

I use a different strategy. My main module depends on many modules in the cache, and this set of modules changes over time. The standard library packages, on the other hand, are relatively stable. I wanted to avoid enumerating the modules in the cache, so instead I enumerated the standard library packages.

Because the standard library packages have no prefix (e.g. just fmt/errors.go, rather than go/fmt/errors.go), I had to create a substitution for every package. I also used the executable's go version to create the substitutions. (I use different go versions at the same time, so I needed to use the correct one for this executable.)

"substitutePath": [
    // Main module, built with -trimpath
    {
        "from": "${workspaceFolder}",
        "to": "moduleName",
    },
    // Packages in GOROOT (standard library)
    {
        "from": "/home/myuser/.local/go/1.20.1/src/archive",
        "to": "archive"
    },
    // ... Other entries omitted to save space ...
    {
        "from": "/home/myuser/.local/go/1.20.1/src/vendor",
        "to": "vendor"
    },
    // Modules in GOMODCACHE
    {
        "from": "/home/myuser/.local/go/pkg/mod",
        "to": "",
    },
]

While I find my strategy easier to use manually, I think would be easier to automatically generate substitutions for modules in the module cache. We would still need to generate one substitution for the standard library packages, and for that we would need (a) the go version of the executable (available as metadata in the executable), and (b) the path to the GOROOT of that go version (must be provided by the user).