cmd/link: compress debug info #11799

josharian · 2015-07-20T18:25:53Z

Compressing our debug info might offer a significant, cheap file size win.

Things to do:

Figure out which targets to compress debug info for by default.
Provide a linker flag to control compression on/off.
Teach the linker to compress debug info.
Teach the Go packages how to read compressed debug info.

Compressed debug info is supported by gdb and the GNU binutils.
Yes, we should do it in our toolchain for relevant targets.

Related: #11773

mwhudson · 2015-07-20T22:20:52Z

It's only tangentially related, but it's been at the back of my mind to try to figure out what would be required to make stripping Go binaries (and so being able to use detached debugging symbols) more of a supported thing.

josharian · 2016-03-03T19:35:23Z

cc @crawshaw @derekparker

I think that this could reduce the size of cmd/go by about 15%.

How I got that number (OS X commands):

$ dwarfdump -R `which go`
[snip]
Segments
Segment Name     vmaddr           vmsize           fileoff          filesize         maxprot  initprot nsects   flags
---------------- ---------------- ---------------- ---------------- ---------------- -------- -------- -------- --------
__PAGEZERO       0000000000000000 0000000000001000 0000000000000000 0000000000000000 00000000 00000000 00000000 00000000
__TEXT           0000000000001000 00000000007bf000 0000000000000000 00000000007bf000 00000007 00000005 00000006 00000000
__DATA           00000000007c0000 0000000000045060 00000000007bf000 0000000000020ac0 00000003 00000003 00000005 00000000
__LINKEDIT       0000000000806000 0000000000083c4c 0000000000ac6000 0000000000083c4c 00000007 00000003 00000000 00000000
__DWARF          0000000000805060 0000000000000000 00000000007e0000 00000000002e5968 00000000 00000000 00000008 00000000

Sections
Section Name     Segment Name     addr             size             offset   align    reloff   nreloc   flags    reserv1  reserv2  reserv3  size     size %
---------------- ---------------- ---------------- ---------------- -------- -------- -------- -------- -------- -------- -------- -------- ======== ======
__text           __TEXT           0000000000002000 00000000003a6f38 00001000 00000004 00000000 00000000 00000400 00000000 00000000 00000000    3.65M 33.49%
__rodata         __TEXT           00000000003a8f40 0000000000283f0c 003a7f40 00000005 00000000 00000000 00000000 00000000 00000000 00000000    2.52M 23.06%
__typelink       __TEXT           000000000062ce50 000000000000e7f0 0062be50 00000003 00000000 00000000 00000000 00000000 00000000 00000000   57.98K  0.52%
__gosymtab       __TEXT           000000000063b640 0000000000000000 0063a640 00000000 00000000 00000000 00000000 00000000 00000000 00000000    0      0.00%
__gopclntab      __TEXT           000000000063b640 000000000018415b 0063a640 00000005 00000000 00000000 00000000 00000000 00000000 00000000    1.52M 13.90%
__symbol_stub1   __TEXT           00000000007bf7a0 00000000000000d8 007be7a0 00000005 00000000 00000000 80000408 00000000 00000006 00000000  216      0.00%
__nl_symbol_ptr  __DATA           00000000007c0000 0000000000000138 007bf000 00000002 00000000 00000000 00000006 00000024 00000000 00000000  312      0.00%
__noptrdata      __DATA           00000000007c0140 0000000000014008 007bf140 00000005 00000000 00000000 00000000 00000000 00000000 00000000   80.01K  0.72%
__data           __DATA           00000000007d4160 000000000000c960 007d3160 00000005 00000000 00000000 00000000 00000000 00000000 00000000   50.34K  0.45%
__bss            __DATA           00000000007e0ac0 000000000001e138 00000000 00000005 00000000 00000000 00000001 00000000 00000000 00000000  120.30K  1.08%
__noptrbss       __DATA           00000000007fec00 0000000000006460 00000000 00000005 00000000 00000000 00000001 00000000 00000000 00000000   25.09K  0.22%
__debug_abbrev   __DWARF          0000000000805060 00000000000000ff 007e0000 00000000 00000000 00000000 02000000 00000000 00000000 00000000  255      0.00%
__debug_line     __DWARF          000000000080515f 00000000000ac904 007e00ff 00000000 00000000 00000000 02000000 00000000 00000000 00000000  690.25K  6.18%
__debug_frame    __DWARF          00000000008b1a63 000000000005c574 0088ca03 00000000 00000000 00000000 02000000 00000000 00000000 00000000  369.36K  3.31%
__debug_info     __DWARF          000000000090dfd7 0000000000154b49 008e8f77 00000000 00000000 00000000 02000000 00000000 00000000 00000000    1.33M 12.20%
__debug_pubnames __DWARF          0000000000a62b20 000000000005f4c1 00a3dac0 00000000 00000000 00000000 02000000 00000000 00000000 00000000  381.19K  3.41%
__debug_pubtypes __DWARF          0000000000ac1fe1 000000000002897f 00a9cf81 00000000 00000000 00000000 02000000 00000000 00000000 00000000  162.37K  1.45%
__debug_aranges  __DWARF          0000000000aea960 0000000000000030 00ac5900 00000000 00000000 00000000 02000000 00000000 00000000 00000000   48      0.00%
__debug_gdb_scri __DWARF          0000000000aea990 0000000000000035 00ac5930 00000000 00000000 00000000 02000000 00000000 00000000 00000000   53      0.00%
$ # manually convert fileoff and filesize from hex to dec, calculate rough compressed dwarf size
$ head -c 8257536 `which go` | tail -c 3037544 | gzip | wc -c
 1048123
$ # find current binary size
$ wc -c `which go`
 11836492
$ # calculate savings as a percent of current size
$ python -c "print(100*(3037544-1048123)/11836492.)"
16.807522026

I also double-checked that I'm interpreted fileoff and filesize correctly by using otool and xxd to dump just the __debug_info section and compressed it; the compression ratio for __debug_info is even better than for the entirety of the dwarf segment.

Note that this probably includes updating the dwarf stdlib to handle compression.

See also #5158.

minux · 2016-03-03T20:12:34Z

I think debug/elf already supports compressed sections. Could we compress other sections as well (and do decompression at runtime)? The only segment we can't decompress at runtime is the code segment.

josharian · 2017-07-12T00:41:25Z

I took yet another stab at this, but I am having a hard time making friends with the linker, so I'm going to leave this for @aarzilli or @heschik.

Here are some very useful implementation details from @ianlancetaylor, copied from #20463:

DWARF compression is done at the file container level. There are two ways to do it. The older way is to prefix the DWARF section name with a 'z', as in '.zdebug_info'. The newer way, which applies to sections of all types, not just DWARF sections, is to set the SHF_COMPRESSED flag in the ELF section flags.

Clearly Mach-O doesn't support the SHF_COMPRESSED flag, but I don't see any reason that it wouldn't support .zdebug_*. Looking at the gdb source code, it seems that it ought to work. I suppose the only way to find out is to try it and see what happens.

A .zdebug-* section starts with the four characters "ZLIB" followed by a uint64 of the uncompressed size in big-endian order followed by the compressed data.

SHF_COMPRESSED works differently. The section data starts with a compression header: a 32-bit word with the compression type (1 for ZLIB), on 64-bit systems a 32-bit padding word, a 32-bit or 64-bit word with the uncompressed section size, a 32-bit or 64-bit word with the uncompressed alignment.

heschi · 2017-07-14T15:42:01Z

I've spent a little time digging into this and I'll leave some notes for myself or whoever comes next.

The linker doesn't have the .debug_info section as a whole unit in memory at any point. dwarfp contains each top-level DIE as an individual symbol, and they're written to the output file individually by Dwarfblk in data.go. So that's the only place that we can insert compression of the whole section, perhaps by checking a new flag in the Section type.

This won't work for external linking, because the .debug_info section contains relocations to text symbols that will be placed by the external linker. New versions of GNU ld (I'm not sure when it was added exactly) support --compress-debug-sections to compress debug sections passed to. (gcc has another flag -gz, but presumably that's just for the individual CUs it's creating, not the whole section. That doesn't help with the Go CUs.)

ianlancetaylor · 2017-12-15T02:56:40Z

I don't think we should compress the debug info when doing external linking. The object file that we create is only going to live briefly, probably in /tmp, likely on a RAM disk. It will be deleted after the external linker runs. Compressing the debug info in that case will buy us nothing; we should leave it to the external linker.

heschi · 2017-12-15T19:04:15Z

I agree, it's both impossible and undesirable :)

I might work on this for 1.11 as part of getting DWARF location lists enabled by default, to offset the increase in binary size.

ghost · 2017-12-30T17:51:55Z

What about providing the option to completely strip debug info? The -w and -s flags don't remove this info.

ianlancetaylor · 2017-12-30T20:07:27Z

@r04r That is a completely different problem that should be discussed on a different issue, not here. Thanks.

ghost · 2017-12-30T20:14:17Z

@ianlancetaylor This issue seems to be about acquiring a "significant, cheap file size win.". Instead of compressing, just removing this information seems more significant, and cheaper. Is it viable? Should I make another issue?

ianlancetaylor · 2017-12-30T20:45:46Z

@r04r This issue is about a cheap file size win without losing any information. Removing the debug info produces a binary that can not be used with a debugger, so that is a different matter--not a win at all for people who want to use a debugger.

Yes, if the linker's -s and -w flags do not work as documented, please open a different issue.

josharian · 2018-03-16T02:17:40Z

Marking as release-blocker for 1.11, since https://go-review.googlesource.com/c/go/+/100738 just went in and executable sizes are up >15%.

heschi · 2018-03-20T18:54:39Z

Spent a little more time. More notes, adding to #11799 (comment).

Revisiting it, I'm not totally sure where to put the compression any more. The linker calculates the output's layout in (*Link).address(), then uses it in (*Link).reloc() to compute relocation targets and rewrite symbols as necessary. The DWARF cannot have been compressed before that, since it contains relocations. But compression could alter the layout, invalidating the previous results. In practice, the only things that come after the .debug sections on my Linux/AMD64 machine are .note.go.buildid, .symtab, and .strtab. I don't think any of those ought to be relocation targets. As long as we don't mind making that a guarantee, we can add a compression step beween reloc and Asmb, using the .zdebug approach for both Mach-O and ELF based on Ian's commentary above. I don't know about PE.

aarzilli · 2018-03-20T19:05:11Z

Other options:

Move all .debugs all the way to the end of the file
Do the relocations for .debugs only, compress, do the relocations for everything else.

aclements · 2018-03-22T20:42:38Z

To expand on @heschik's comment, perhaps the root of the problem here is that (*Link).address() does two things: it assigns virtual addresses, which has to be done before reloc; and it assigns file offsets. These are logically independent. If we could disentangle them, we could assign virtual addresses, then apply relocations, then compress the DWARF section, then do file layout. I suspect the reason they're entangled is simply that there's no simple list of segments anywhere. The list is all unrolled into code and it would be awful to duplicate that code, so it's all in address.

bradfitz · 2018-05-04T17:31:03Z

I'd like to stress that this really should happen for Go 1.11, as our binaries have gotten 20% bigger since Go 1.10 and that's not something users will be super happy about.

aclements · 2018-05-04T17:41:38Z

I'll see if I can put together a fix for this, but it's definitely not an easy change. We'll see how it turns out.

ianlancetaylor · 2018-05-04T18:02:59Z

@aclements I'm also happy to look at this if you prefer.

gopherbot · 2018-06-12T17:40:07Z

Change https://golang.org/cl/118276 mentions this issue: cmd/link: compress DWARF sections in ELF binaries

gopherbot · 2018-06-12T17:41:26Z

Change https://golang.org/cl/118277 mentions this issue: cmd/link: compress debug sections in external linking mode

Currently these two forms of layout are done in a single pass. This makes it difficult to compress DWARF sections because that must be done after relocations are applied, which must happen after virtual address layout, but we can't layout the file until we've compressed the DWARF sections. Fix this by separating the two layout steps. In the process, we can also unify the copy-pasted code in Link.address to compute file offsets. Currently, each instance of this is slightly different, but there's no reason for it to be. For example, we don't perform PEFILEALIGN alignment on Segrodata or Selreltodata even when HeadType == Hwindows, but it turns out it doesn't matter whether you do or don't because these segments simply don't exist on Windows. Hence, in the unified code path, we do this alignment for all segments. Likewise, there are two ways of computing Fileoff: seg.Vaddr - prev.Vaddr + prev.Fileoff and prev.Fileoff + uint64(Rnd(int64(prev.Filelen), int64(*FlagRound))) At the moment, these always have the same value, but the latter will continue to work after we start compressing sections on disk. Tested by comparing test binaries for all packages in std before and after this change for GOOS={linux,windows,darwin,plan9}. All binaries are identical. For #11799. Change-Id: If09f28771bb4d78dd392fd58b8d7c9d5f22b0b9f Reviewed-on: https://go-review.googlesource.com/111682 Run-TryBot: Austin Clements <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]>

jcajka · 2018-06-13T11:36:25Z

bd83774 seems to be cause of issue #25863

gopherbot · 2018-06-13T21:36:12Z

Change https://golang.org/cl/118716 mentions this issue: cmd/link: separate virtual address layout from file layout

Currently these two forms of layout are done in a single pass. This makes it difficult to compress DWARF sections because that must be done after relocations are applied, which must happen after virtual address layout, but we can't layout the file until we've compressed the DWARF sections. Fix this by separating the two layout steps. In the process, we can also unify the copy-pasted code in Link.address to compute file offsets. Currently, each instance of this is slightly different, but there's no reason for it to be. For example, we don't perform PEFILEALIGN alignment on Segrodata or Selreltodata even when HeadType == Hwindows, but it turns out it doesn't matter whether you do or don't because these segments simply don't exist on Windows. Hence, in the unified code path, we do this alignment for all segments. Likewise, there are two ways of computing Fileoff: seg.Vaddr - prev.Vaddr + prev.Fileoff and prev.Fileoff + uint64(Rnd(int64(prev.Filelen), int64(*FlagRound))) At the moment, these always have the same value, but the latter will continue to work after we start compressing sections on disk. Tested by comparing test binaries for all packages in std before and after this change for GOOS={linux,windows,darwin,plan9}. All binaries are identical. For golang#11799. Change-Id: If09f28771bb4d78dd392fd58b8d7c9d5f22b0b9f Reviewed-on: https://go-review.googlesource.com/111682 Run-TryBot: Austin Clements <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]>

Currently these two forms of layout are done in a single pass. This makes it difficult to compress DWARF sections because that must be done after relocations are applied, which must happen after virtual address layout, but we can't layout the file until we've compressed the DWARF sections. Fix this by separating the two layout steps. In the process, we can also unify the copy-pasted code in Link.address to compute file offsets, which currently has some unnecessary variation. Unlike the current file offset computation, which depends on virtual addresses, the new computation only uses file offsets and sizes. This will let us compress the file representation of a segment and create the file layout based on its on-disk size rather than its original in-memory size. Tested by comparing the test binary for the "strings" package on all supported GOOS/GOARCH combinations. All binaries are identical (except, of course, their build IDs). This is a second attempt at CL 111682. For #11799. Fixes #25863. Change-Id: If09f28771bb4d78dd392fd58b8d7c9d5f22b0b9e Reviewed-on: https://go-review.googlesource.com/118716 Run-TryBot: Austin Clements <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]>

Forked from CL 111895. For #11799. Change-Id: Ie1346ac2c9122de494823b9058df3a0971e9dfe1 Reviewed-on: https://go-review.googlesource.com/118277 Run-TryBot: Heschi Kreinick <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]> Reviewed-by: Austin Clements <[email protected]>

robpike · 2018-06-18T00:56:58Z

This should be left open until all platforms, not just ELF, do this.

agnivade · 2018-06-18T03:38:38Z

Then #25927 should be closed I believe.

aclements · 2018-06-18T13:54:36Z

I think it's fine to leave #25927 open for the Windows-specific aspects of this. There's already been a decent amount of discussion there.

For macOS, I'm concerned because my understanding is that lldb doesn't support compressed DWARF sections in Mach-O binaries (in fact, it didn't support them in ELF binaries until last month). And I believe dsymutil doesn't support them either (@heschik?)

heschi · 2018-06-18T18:23:19Z

Well, dsymutil doesn't matter so much since we only use it during external linking, and we would need the host linker to do the compression in that case. But there's no --compress-debug-sections on macOS. I suppose we could do the compression in machoCombineDwarf.

Then there's the question of whether we want to. dwarfdump and lldb won't support it, but Delve could. So I suppose it's technically feasible as long as we're willing to leave lldb behind. I would want a flag so that I could use dwarfdump when necessary though.

gopherbot · 2018-06-19T21:34:25Z

Change https://golang.org/cl/119816 mentions this issue: cmd/link: enable DWARF compression on Windows

gopherbot · 2018-06-19T21:34:26Z

Change https://golang.org/cl/119815 mentions this issue: debug/elf,macho,pe: support compressed DWARF

Simple follow-on to CL 118276. Everything worked except that the compressed sections need to be aligned at PEFILEALIGN. Fixes #25927 Updates #11799 Change-Id: Iec871defe30e3e66055d64a5ae77d5a7aca355f5 Reviewed-on: https://go-review.googlesource.com/119816 Run-TryBot: Heschi Kreinick <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Brad Fitzpatrick <[email protected]> Reviewed-by: Austin Clements <[email protected]>

Since we're going to start compressing DWARF on Windows and maybe Darwin, copy the ELF support for .zdebug sections to macho and pe. The code is almost completely the same across the three. While I was here I added support for compressed .debug_type sections, which I presume were overlooked before. Tests will come in a later CL once we can actually generate compressed PE/Mach-O binaries, since there's no other good way to get test data. Updates #25927, #11799 Change-Id: Ie920b6a16e9270bc3df214ce601a263837810376 Reviewed-on: https://go-review.googlesource.com/119815 Run-TryBot: Heschi Kreinick <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Brad Fitzpatrick <[email protected]> Reviewed-by: Austin Clements <[email protected]>

gopherbot · 2018-06-20T21:12:08Z

Change https://golang.org/cl/120155 mentions this issue: cmd/link: support DWARF compression on Darwin

We want to compress DWARF even on macOS, but the native toolchain isn't going to understand it. Add a flag that can be used to disable compression, then add Darwin to the whitelist used during internal linking. Unlike GNU ld, the Darwin linker doesn't have a handy linker flag to do compression. But since we're already doing surgery to put the DWARF in the output executable in the first place, compressing it at the same time isn't unduly difficult. This does have the slightly odd effect of compressing some Apple proprietary debug sections, which absolutely nothing will understand. Leaving them uncompressed didn't make much sense, though, since I doubt they're useful without (say) __debug_info. Updates #11799 Change-Id: Ie00b0215c630a798c59d009a641e2d13f0e7ea01 Reviewed-on: https://go-review.googlesource.com/120155 Run-TryBot: Heschi Kreinick <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Austin Clements <[email protected]>

heschi · 2018-06-22T19:55:09Z

Here's the state of things as of that last commit. We now enable DWARF compression by default on almost all platforms. I didn't do anything for plan9, wasm, and nacl, but I'm not actually sure if they get debug information anyway.

To debug a binary with DWARF compression, you'll need a modern version of GDB or a version of Delve after derekparker/delve@440b440 compiled with tip Go. Tip versions of lldb might work, but I haven't tested them. Compression can be disabled with -ldflags=-compressdwarf=false if there's any problems, or if you need to use inspection tools like macOS dwarfdump that don't understand the compressed DWARF.

I'm not aware of any significant remaining work to do here, so closing. Please comment if I missed something.

gopherbot · 2018-09-27T21:54:17Z

Change https://golang.org/cl/138182 mentions this issue: doc: mention -compressdwarf=false on gdb page

Update #11799 Change-Id: I2646a52bfb8aecb67a664a7c6fba25511a1aa49f Reviewed-on: https://go-review.googlesource.com/138182 Reviewed-by: Heschi Kreinick <[email protected]> Reviewed-by: David Chase <[email protected]>

gopherbot · 2018-09-27T22:04:56Z

Change https://golang.org/cl/138184 mentions this issue: [release-branch.go1.11] doc: mention -compressdwarf=false on gdb page

josharian added this to the Unplanned milestone Jul 20, 2015

minux mentioned this issue Aug 15, 2016

cmd/go: add test to ensure upx can compress our binaries #16706

Closed

bradfitz modified the milestones: Go1.10, Unplanned May 7, 2017

josharian mentioned this issue May 23, 2017

cmd/compile: use common DWARF prefixes? #20463

Closed

heschi added the Debugging label Jul 17, 2017

rsc modified the milestones: Go1.10, Go1.11 Nov 22, 2017

ianlancetaylor mentioned this issue Dec 15, 2017

cmd/link: DWARF data too large #5158

Closed

mvdan mentioned this issue Feb 19, 2018

all: unstripped binary size growth between Go 1.9 and 1.10 #23934

Closed

FiloSottile mentioned this issue Feb 20, 2018

all: binaries too big and growing #6853

Open

josharian added the release-blocker label Mar 16, 2018

dgryski mentioned this issue May 8, 2018

Understand compressed debug info go-delve/delve#1202

Closed

gopherbot closed this as completed in 594eae5 Jun 15, 2018

siebenmann mentioned this issue Jun 16, 2018

cmd/link: test failure in TestLldbPython from commit 594eae5 onward #25925

Closed

dajoo75 mentioned this issue Jun 17, 2018

cmd/link: compress debug info on windows #25927

Closed

robpike reopened this Jun 18, 2018

heschi closed this as completed Jun 22, 2018

golang locked and limited conversation to collaborators Sep 27, 2019

gopherbot added the FrozenDueToAge label Sep 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/link: compress debug info #11799

cmd/link: compress debug info #11799

josharian commented Jul 20, 2015

mwhudson commented Jul 20, 2015

josharian commented Mar 3, 2016

minux commented Mar 3, 2016 via email

josharian commented Jul 12, 2017

heschi commented Jul 14, 2017

ianlancetaylor commented Dec 15, 2017

heschi commented Dec 15, 2017

ghost commented Dec 30, 2017

ianlancetaylor commented Dec 30, 2017

ghost commented Dec 30, 2017

ianlancetaylor commented Dec 30, 2017

josharian commented Mar 16, 2018

heschi commented Mar 20, 2018

aarzilli commented Mar 20, 2018

aclements commented Mar 22, 2018

bradfitz commented May 4, 2018

aclements commented May 4, 2018

ianlancetaylor commented May 4, 2018

gopherbot commented Jun 12, 2018

gopherbot commented Jun 12, 2018

jcajka commented Jun 13, 2018

gopherbot commented Jun 13, 2018

robpike commented Jun 18, 2018

agnivade commented Jun 18, 2018

aclements commented Jun 18, 2018

heschi commented Jun 18, 2018

gopherbot commented Jun 19, 2018

gopherbot commented Jun 19, 2018

gopherbot commented Jun 20, 2018

heschi commented Jun 22, 2018 •

edited

Loading

gopherbot commented Sep 27, 2018

gopherbot commented Sep 27, 2018

cmd/link: compress debug info #11799

cmd/link: compress debug info #11799

Comments

josharian commented Jul 20, 2015

mwhudson commented Jul 20, 2015

josharian commented Mar 3, 2016

minux commented Mar 3, 2016 via email

josharian commented Jul 12, 2017

heschi commented Jul 14, 2017

ianlancetaylor commented Dec 15, 2017

heschi commented Dec 15, 2017

ghost commented Dec 30, 2017

ianlancetaylor commented Dec 30, 2017

ghost commented Dec 30, 2017

ianlancetaylor commented Dec 30, 2017

josharian commented Mar 16, 2018

heschi commented Mar 20, 2018

aarzilli commented Mar 20, 2018

aclements commented Mar 22, 2018

bradfitz commented May 4, 2018

aclements commented May 4, 2018

ianlancetaylor commented May 4, 2018

gopherbot commented Jun 12, 2018

gopherbot commented Jun 12, 2018

jcajka commented Jun 13, 2018

gopherbot commented Jun 13, 2018

robpike commented Jun 18, 2018

agnivade commented Jun 18, 2018

aclements commented Jun 18, 2018

heschi commented Jun 18, 2018

gopherbot commented Jun 19, 2018

gopherbot commented Jun 19, 2018

gopherbot commented Jun 20, 2018

heschi commented Jun 22, 2018 • edited Loading

gopherbot commented Sep 27, 2018

gopherbot commented Sep 27, 2018

heschi commented Jun 22, 2018 •

edited

Loading