cmd/link: compress debug info on windows #25927

dajoo75 · 2018-06-17T09:52:49Z

What did you do?

Compiled a basic hello world program on 64-bit Windows 10 with go tip (+b459e00), go 1.8.1 and go 1.10.3 and compared the binary sizes.

               hello_go1.8.1.exe  |  hello_go1.10.3.exe  |  hello_+b459e00.exe
binary size    1 633 280 bytes    |  2 057 728 bytes     |  2 848 768 bytes
                                  |                      |
.rdata                     N/A    |             N/A      |    752 128 bytes
.text          1 068 032 bytes    |  1 181 696 bytes     |    575 488 bytes
.debug_info      260 608 bytes    |    396 800 bytes     |    457 216 bytes
.debug_loc                 N/A    |             N/A      |    404 480 bytes
.symtab           98 304 bytes    |    141 312 bytes     |    142 336 bytes
.debug_frame      70 144 bytes    |     75 264 bytes     |     76 288 bytes
.debug_line       68 519 bytes    |     72 192 bytes     |    139 264 bytes
.debug_pubtypes   30 720 bytes    |     42 496 bytes     |     44 544 bytes
.debug_pubnames   17 408 bytes    |     40 960 bytes     |     41 472 bytes
.data             15 360 bytes    |     77 824 bytes     |     78 336 bytes
.debug_ranges              N/A    |     25 600 bytes     |    133 632 bytes
.idata             1 536 bytes    |      1 024 bytes     |      1 024 bytes
.debug_aranges       512 bytes    |              N/A     |              N/A
.debug_abbrev        512 bytes    |        512 bytes     |        512 bytes
.debug_gdb_scripts         N/A    |        512 bytes     |        512 bytes

What did you expect to see?

The binary size to be about the same at tip (+b459e00) as the previous release (go1.10.3 at the time of writing this). And if possible, the binary size to have gone down.

Other related issues: #6853, #11799

What did you see instead?

The binary size at tip (+b459e00) has increased by 38 percent since the previous release (go1.10.3 at the time of writing this).

Does this issue reproduce with the latest release (go1.10.3)?

Yes

System details

go version devel +b459e00 Sat Jun 16 13:35:45 2018 +0000 windows/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="C:\Users\loki\AppData\Local\go-build"
GOEXE=".exe"
GOHOSTARCH="amd64"
GOHOSTOS="windows"
GOOS="windows"
GOPATH="C:\home\projects\go"
GOPROXY=""
GORACE=""
GOROOT="C:\home\installs\go\go.tip"
GOTMPDIR=""
GOTOOLDIR="C:\home\installs\go\go.tip\pkg\tool\windows_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\loki\AppData\Local\Temp\go-build103655438=/tmp/go-build -gno-record-gcc-switches"
VGOMODROOT=""
GOROOT/bin/go version: go version devel +b459e00 Sat Jun 16 13:35:45 2018 +0000 windows/amd64
GOROOT/bin/go tool compile -V: compile version devel +b459e00 Sat Jun 16 13:35:45 2018 +0000
gdb --version: GNU gdb (GDB) 8.0

The text was updated successfully, but these errors were encountered:

ALTree · 2018-06-17T11:33:54Z

As you word it, this would technically be a dup of #6853.

What I think it's worth noting is that CL 118276 (cmd/link: compress DWARF sections in ELF binaries) only helped with ELF binaries. So I think it would be better if you renamed this issue to point out this specific issue (i.e., make this issue a TODO reminder to compress debug info on windows, too).

In theory we had #11799 (cmd/link: compress debug info) for the general problem, but it got closed after the ELF work.

dajoo75 · 2018-06-17T12:07:38Z

@ALTree I've renamed the issue to cmd/link: compress debug info on windows as you suggested.

Others more in the know about Windows, the PE file format and the debug story around Windows (gdb, delve etc.) have to chime in on the correct way forward.

dajoo75 · 2018-06-17T20:49:30Z

I tried to get this working on Windows (made some small changes to cmd/link and, debug/pe) and hello world.exe went from 2.8MB to 1.9MB. I almost got delve to start but didn't have time to finish it.

There doesn't seem to be an insurmountable amount of work left to do to get compression working on Windows as well (since most of the work seems to be done already in #11799) but I may be wrong (the devil is in the details and all that).

I would feel better if the original authors of #11799 looked at this as well (together with Alex) since they know what they are doing :).

cc @heschik, @alexbrainman, @ianlancetaylor, @aclements

aarzilli · 2018-06-18T06:16:06Z

@dajoo75 what approach are you taking? Elf supports compression but PE doesn't (based on a ctrl-f search in the relevant MSDN page), so .zdebug?

dajoo75 · 2018-06-18T06:42:31Z

Yes .zdebug. That seems to be the way the Go team have solved this. Dwarf sections in a PE file. Package debug/pe has a DWARF-function that delve uses. I’m decompressing the sections inside debug/pe just like the debug/elf solution. The consumer, e.g. delve, only sees uncompressed data, at least that was the idea... but I didn’t have time to finish it. /Daniel

…

18 juni 2018 kl. 08:17 skrev Alessandro Arzilli ***@***.***>: @dajoo75 what approach are you taking? Elf supports compression but PE doesn't (based on a ctrl-f search in the relevant MSDN page), so .zdebug? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

aarzilli · 2018-06-18T08:52:59Z

Ah, I didn't notice the switch to zdebug. In that case: https://github.com/derekparker/delve/pull/1237

alexbrainman · 2018-06-18T10:08:56Z

Thank you @dajoo75 for creating the issue.

If I understand correctly, CL 118276 added compression for ELF DWARF sections. I would try and add some code to make .zdebug_... sections appear in Windows executables, and see where it gets us. I doubt I would have time before weekend. So others are welcome to this.

I also wonder if debuggers used by our users will support this. Lucky we have @aarzilli here, so he can fix Delve. Not so easy to fix gdb. We will see.

Alex

aarzilli · 2018-06-18T10:13:46Z

Not so easy to fix gdb. We will see.

I think gdb was the original user of the zdebug convention, I imagine that should be fine. The problem is lldb.

alexbrainman · 2018-06-18T10:29:52Z

The problem is lldb.

Does lldb work on Windows?

Alex

aarzilli · 2018-06-18T10:33:10Z

Does lldb work on Windows?

Partially: https://lldb.llvm.org/status.html

heschi · 2018-06-18T19:24:59Z

@dajoo75 I'm happy to review a CL and help you over some bumps; I'd have to set up a Windows machine to even start on this and it sounds like you're well into it. We are pretty deep into the freeze so it'd be nice to get it done soon.

Can someone confirm that on Windows we're basically ignoring the platform convention for debug info, which I guess would either be COFF or PDB? As long as that's the case and none of the standard tools work anyway, I think it makes sense to go ahead and compress it. I doubt lldb is very popular there, even if it does work.

ianlancetaylor · 2018-06-18T21:59:34Z

@heschik That is correct: we only generate DWARF. COFF debug info is entirely inadequate. As far as I know PDB is not publicly documented.

dajoo75 · 2018-06-18T22:27:49Z

@heschik Thank you for helping out. I don’t feel I have enough of a grip on the solution right now to send a CL I believe in so I’ll leave this one to Alex (or someone else) more experienced in the PE and DWARF format for now. /Daniel

…

18 juni 2018 kl. 21:25 skrev Heschi Kreinick ***@***.***>: @dajoo75 I'm happy to review a CL and help you over some bumps; I'd have to set up a Windows machine to even start on this and it sounds like you're well into it. We are pretty deep into the freeze so it'd be nice to get it done soon. Can someone confirm that on Windows we're basically ignoring the platform convention for debug info, which I guess would either be COFF or PDB? As long as that's the case and none of the standard tools work anyway, I think it makes sense to go ahead and compress it. I doubt lldb is very popular there, even if it does work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

heschi · 2018-06-18T23:20:37Z

Okay. @alexbrainman, if you don't claim this by tomorrow I'll take a crack at it.

alexbrainman · 2018-06-19T02:44:53Z

Okay. @alexbrainman, if you don't claim this by tomorrow I'll take a crack at it

Please, do. I will try it myself, but I do not know when.

Alex

gopherbot · 2018-06-19T21:34:25Z

Change https://golang.org/cl/119816 mentions this issue: cmd/link: enable DWARF compression on Windows

gopherbot · 2018-06-19T21:34:26Z

Change https://golang.org/cl/119815 mentions this issue: debug/elf,macho,pe: support compressed DWARF

Since we're going to start compressing DWARF on Windows and maybe Darwin, copy the ELF support for .zdebug sections to macho and pe. The code is almost completely the same across the three. While I was here I added support for compressed .debug_type sections, which I presume were overlooked before. Tests will come in a later CL once we can actually generate compressed PE/Mach-O binaries, since there's no other good way to get test data. Updates #25927, #11799 Change-Id: Ie920b6a16e9270bc3df214ce601a263837810376 Reviewed-on: https://go-review.googlesource.com/119815 Run-TryBot: Heschi Kreinick <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Brad Fitzpatrick <[email protected]> Reviewed-by: Austin Clements <[email protected]>

be5invis · 2018-10-19T09:16:34Z

@ianlancetaylor
PDB's definition: https://github.com/Microsoft/microsoft-pdb

ianlancetaylor · 2018-10-19T13:50:11Z

@be5invis Thanks.

dajoo75 changed the title ~~Windows binaries too big and growing~~ cmd/link: compress debug info on windows Jun 17, 2018

ALTree added this to the Go1.12 milestone Jun 17, 2018

ALTree added NeedsFix The path to resolution is known, but the work has not been done. Debugging OS-Windows labels Jun 17, 2018

agnivade mentioned this issue Jun 18, 2018

cmd/link: compress debug info #11799

Closed

gopherbot closed this as completed in 83515df Jun 19, 2018

golang locked and limited conversation to collaborators Oct 19, 2019

gopherbot added the FrozenDueToAge label Oct 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/link: compress debug info on windows #25927

cmd/link: compress debug info on windows #25927

dajoo75 commented Jun 17, 2018 •

edited

Loading

ALTree commented Jun 17, 2018

dajoo75 commented Jun 17, 2018 •

edited

Loading

dajoo75 commented Jun 17, 2018

aarzilli commented Jun 18, 2018

dajoo75 commented Jun 18, 2018 via email

aarzilli commented Jun 18, 2018

alexbrainman commented Jun 18, 2018

aarzilli commented Jun 18, 2018

alexbrainman commented Jun 18, 2018

aarzilli commented Jun 18, 2018

heschi commented Jun 18, 2018

ianlancetaylor commented Jun 18, 2018

dajoo75 commented Jun 18, 2018 via email

heschi commented Jun 18, 2018

alexbrainman commented Jun 19, 2018

gopherbot commented Jun 19, 2018

gopherbot commented Jun 19, 2018

be5invis commented Oct 19, 2018

ianlancetaylor commented Oct 19, 2018

cmd/link: compress debug info on windows #25927

cmd/link: compress debug info on windows #25927

Comments

dajoo75 commented Jun 17, 2018 • edited Loading

What did you do?

What did you expect to see?

What did you see instead?

Does this issue reproduce with the latest release (go1.10.3)?

System details

ALTree commented Jun 17, 2018

dajoo75 commented Jun 17, 2018 • edited Loading

dajoo75 commented Jun 17, 2018

aarzilli commented Jun 18, 2018

dajoo75 commented Jun 18, 2018 via email

aarzilli commented Jun 18, 2018

alexbrainman commented Jun 18, 2018

aarzilli commented Jun 18, 2018

alexbrainman commented Jun 18, 2018

aarzilli commented Jun 18, 2018

heschi commented Jun 18, 2018

ianlancetaylor commented Jun 18, 2018

dajoo75 commented Jun 18, 2018 via email

heschi commented Jun 18, 2018

alexbrainman commented Jun 19, 2018

gopherbot commented Jun 19, 2018

gopherbot commented Jun 19, 2018

be5invis commented Oct 19, 2018

ianlancetaylor commented Oct 19, 2018

dajoo75 commented Jun 17, 2018 •

edited

Loading

dajoo75 commented Jun 17, 2018 •

edited

Loading