cmd/compile: statictmps can prevent heap objects from being collected

[randall77]

package main

import (
	"fmt"
	"runtime"
)

var x = [][]int{[]int{1}}

func main() {
	s := make([]int, 1e8)
	runtime.SetFinalizer(&s[0], func(p *int) { fmt.Println("finalized") })
	x[0] = s
	x = nil
	runtime.GC()
	runtime.GC()
	runtime.GC()
}

When I run this program, the finalizer never executes. The big slice will live in the heap forever.

The original state of the program has x pointing to a compiler-allocated global variable statictmp1. That variable has one []int slot in it, which in turn points to a second statictmp2 variable holding a single 1.

When we do x[0]=s, we set statictmp1 to point to the heap instead. Then when we do x = nil, statictmp1 is now unreachable. But statictmp1 now points to an object in the heap, and we still scan statictmp1 at every garbage collection, because it is a global.

If instead we allocated statictmp1 on the heap, this problem would go away. There's a tradeoff here which I'm not sure how to resolve. The current situation prioritizes fast startup and preferring global data over heap data, but it can result in imprecise retention of objects.

A better fix would be to include a global "object" in the GC marking phase for each global variable. (This is similar to how stack objects work.) Named globals would be in the root set, but unnamed ones like the statictmps would only be live if a heap object or other live global pointed to it.

I'm not sure it is worth fixing this problem. I sort of stumbled on it while working on #29013 but I haven't seen any instances in the wild. Thought it would be worth documenting it here in case someone had a better idea or someone found an actual real-world instance.

Note that a similar situation also applies to statictmp2. Afterx[0]=s, statictmp2 is dead. We can't collect statictmp2 because it is allocated in the globals area. We could collect it if it was allocated on the heap. But because it doesn't hold a pointer to heap objects, it isn't a big deal either way. (This also has a parallel to stack objects, which we can't collect directly - we can only ignore their contents.)

@aclements @RLH

[aclements]

@cherrymui and I were just discussing this and came up with some interesting ideas:

First, and somewhat obviously, we'd split up the "true" globals that are always roots from the coincidental globals that might become unreachable. True globals we could continue to scan exactly as we do today.

For coincidental globals, the GC would need to be able to find object boundaries and pointer/scalar information, and keep mark bits to avoid infinite scan loops (though we wouldn't need to use the mark bits for sweeping).

@cherrymui suggested that we could structure this global space literally the same way we do heap spans. The linker already sorts globals by size; it would just have to round them up to size classes and arrange them in pages appropriately. Then we can find object boundaries in the same way we do for regular heap spans. We could also lay out pointer/scalar information exactly as the heap does, either embedding it at the end of these spans, or as allocation headers. Finally, the linker could write out some minimal description of these spans, which the runtime could realize into literal mspan objects at startup. This would let us reuse a lot of code paths around object finding, pointer/scalar bitmaps, and mark bits. It would also naturally help with a lot of weird corner cases, like what happens if the program attaches a finalizer to one of these objects.