runtime: scavenger unable to release idle heap on ARM64 with 64KB OS pages

[arnav-chakraborty]

Summary

On ARM64 Linux with 64KB OS pages, the Go scavenger retains nearly all idle heap memory instead of releasing it back to the OS. The same program on x86 Linux with 4KB pages releases 100% of idle memory. This causes Go programs on ARM64 to consume significantly more RSS than their live heap requires.

Environment

• Go version: go1.21 (also reproduced with go1.22)
• OS: Linux (kernel 5.15+)
• ARM64: 64KB OS pages (getconf PAGESIZE → 65536)
• x86: 4KB OS pages (getconf PAGESIZE → 4096)

64KB pages are the default kernel configuration on major cloud ARM platforms (AWS Graviton, Ampere Altra) for TLB efficiency. This is not a niche or misconfigured setup — it is the standard for server ARM Linux.

Transparent Huge Pages (THP) are not a factor: on our ARM hosts, Hugepagesize is 512 MB (effectively never used; AnonHugePages: 0 kB). On x86, THP uses 2 MB pages (AnonHugePages: ~100 MB), but the scavenger operates at base page granularity regardless of THP.

Reproducer

Single-file, zero dependencies, CGO_ENABLED=0:

package main

import (
	"fmt"
	"math/rand"
	"os"
	"runtime"
	"runtime/debug"
	"time"
)

const MB = 1 << 20

func main() {
	n := 20_000_000
	size := 200
	freeFrac := 0.95

	fmt.Fprintf(os.Stderr, "arch=%s page_size=%d series=%d size=%d free_fraction=%.2f\n",
		runtime.GOARCH, os.Getpagesize(), n, size, freeFrac)

	// Allocate N byte slices.
	objects := make([][]byte, n)
	for i := range objects {
		b := make([]byte, size)
		b[0] = 1
		objects[i] = b
	}
	runtime.GC()
	printStats("after alloc")

	// Free a random fraction: shuffle, nil the tail, reslice.
	keep := int(float64(n) * (1 - freeFrac))
	rand.Shuffle(len(objects), func(i, j int) {
		objects[i], objects[j] = objects[j], objects[i]
	})
	for i := keep; i < len(objects); i++ {
		objects[i] = nil
	}
	objects = objects[:keep]

	runtime.GC()
	debug.FreeOSMemory()
	time.Sleep(2 * time.Second)
	printStats("after free+scavenge")

	runtime.KeepAlive(objects)
}

func printStats(phase string) {
	var ms runtime.MemStats
	runtime.ReadMemStats(&ms)
	trapped := ms.HeapIdle - ms.HeapReleased
	fmt.Printf("%-25s heap_alloc=%-6d heap_inuse=%-6d heap_idle=%-6d heap_released=%-6d trapped=%-6d sys=%-6d [MB]\n",
		phase,
		ms.HeapAlloc/MB, ms.HeapInuse/MB, ms.HeapIdle/MB, ms.HeapReleased/MB, trapped/MB, ms.Sys/MB)
}

Build and run:

GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -o repro_arm64 .
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o repro_amd64 .

Results

Tested on two Linux hosts — identical binary, identical workload, identical Go version:

ARM64 (page_size=65536):
after alloc               heap_alloc=4425   heap_inuse=4465   heap_idle=5      heap_released=5      trapped=0      sys=4587   [MB]
after free+scavenge       heap_alloc=656    heap_inuse=3921   heap_idle=548    heap_released=5      trapped=543    sys=4587   [MB]

x86 (page_size=4096):
after alloc               heap_alloc=4425   heap_inuse=4465   heap_idle=1      heap_released=1      trapped=0      sys=4582   [MB]
after free+scavenge       heap_alloc=656    heap_inuse=3922   heap_idle=548    heap_released=548    trapped=0      sys=4587   [MB]

Metric (MB)	ARM64	x86
heap_alloc (live data)	656	656
heap_inuse (spans with live objects)	3,921	3,922
heap_idle (fully empty spans)	548	548
heap_released (returned to OS)	5	548
trapped (idle but unreleased)	543	0

Both architectures have identical live heap (656 MB) and identical idle memory (548 MB). x86 releases all 548 MB back to the OS. ARM64 releases only 5 MB.

Note on HeapInuse vs HeapAlloc gap

We also observe that HeapInuse remains significantly higher than HeapAlloc after freeing (~3.9 GB vs 656 MB). This indicates a large number of spans still contain at least one live object and have not transitioned to HeapIdle. This is consistent with fragmentation — after randomly freeing 95% of objects, most spans retain at least one survivor (200-byte objects in size class 224 give 36 objects per 8KB span; P(span fully empty) = 0.95^36 ≈ 16%). The HeapIdle of 548 MB represents only the spans that happened to have all 36 objects freed. The trapped memory problem compounds on top of this: even those 548 MB of fully-empty spans cannot be released on ARM64 because they share 64KB pages with spans that still have live objects.

Hypothesis

The scavenger releases memory via madvise(MADV_DONTNEED) at OS page granularity. It cannot partially release an OS page — all spans within the page must be empty before the page can be returned to the OS.

On ARM64 with 64KB pages, each OS page covers approximately 8 contiguous Go spans (8KB each). A page can only be released when all 8 spans on it are empty. On x86, each 8KB span covers exactly two 4KB OS pages — when a span becomes empty, both its pages are immediately releasable (assuming no adjacent span shares the page boundary).

With 200-byte objects (Go size class 224, 36 objects per 8KB span), after randomly freeing 95% of objects, ~16% of spans become fully empty (~548 MB). On ARM64, these empty spans are scattered across 64KB pages that also contain spans with surviving objects, preventing release. On x86, the finer 4KB page granularity allows most empty spans' pages to be released independently.

Other possible contributing factors

We have not ruled out:

• Scavenger pacing or heuristics that may behave differently on large-page systems
• Differences in span reuse patterns between architectures
• Interaction between physical page size and the runtime's page allocator

Scale of the problem

We encountered this running a time series database (M3DB) in production on both ARM64 (64KB pages) and x86 (4KB pages) with identical Go versions and comparable workloads. With ~17M active objects and 31-33 GB heap on ARM64, the RSS overhead reaches 15-20 GB compared to x86 nodes with similar heap sizes.

Workload shape

In production, the heap grows monotonically during bootstrap (loading millions of series from disk/peer node) and then stabilizes at steady state. Ongoing churn comes from encoder rollovers and new series ingestion, but the overall heap size remains relatively constant. The trapped memory accumulates during the bootstrap phase and persists at steady state — it is not a transient spike that resolves over time.

Additional experiments

We ran further experiments with a more comprehensive reproducer to understand what factors influence the amount of trapped memory:

Single size class vs multiple size classes

The reproducer above uses a single size class (200-byte []byte). In a separate experiment modeling our production workload with 6 allocations per logical object across 6+ different size classes (~4.8 KB total per object), trapped memory on ARM64 increased from 543 MB to 14.9 GB for a 24 GB heap. Consolidating all per-object data into a single allocation (one size class) reduced trapped memory by 46%.

This suggests that cross-size-class interference — where spans from different size classes share the same 64KB page — amplifies the problem significantly.

Spatial locality experiment

Using mmap + madvise(MADV_DONTNEED) to manage memory directly (bypassing Go's heap), we tested two freeing patterns:

• Random free (same pattern as the above reproducer): ARM64 released 47% of pages, x86 released 91%
• Contiguous free (objects that die together were allocated together): Both ARM64 and x86 released 95% of pages, producing identical RSS (~1.1 GB)

This confirms that the problem is not inherent to 64KB pages themselves, but arises from the interaction between page size and the spatial distribution of surviving objects across pages.

What helps / what doesn't

From our experiments:

• Fewer size classes per logical object: Consolidating 6 allocations into 1 per object reduced trapped memory by 46% on ARM64.
• Spatial locality: When objects with correlated lifetimes are allocated contiguously, ARM64 and x86 produce identical RSS.
• debug.FreeOSMemory(): In this reproducer, calling debug.FreeOSMemory() did not significantly increase HeapReleased on ARM64.
• GOMEMLIMIT: Setting a memory limit forces more aggressive GC but did not change the page-granularity release constraint in our testing.

Impact

ARM64 servers with 64KB pages (common in cloud ARM instances — e.g., AWS Graviton, Ampere Altra) running Go programs with significant heap churn will retain substantially more RSS than equivalent x86 deployments. This affects:

• Memory-based autoscaling decisions
• Container memory limits and OOM risk
• Cost efficiency of ARM64 migration

Possible directions

We're not certain what the appropriate runtime-level response would be, but potential areas that might be relevant include:

• Page-aware span placement: Heuristics to prefer placing spans from the same size class on the same physical page, reducing cross-class interference on large-page systems.
• Page-size-matched spans: On systems where os.Getpagesize() exceeds the default span size (e.g., 64KB pages), the runtime could use spans that match the OS page size (64KB spans on 64KB-page systems). This would ensure each span aligns to exactly one OS page, making every freed span immediately releasable without depending on neighboring spans.
• Large-page-aware scavenger heuristics: On systems with pages larger than the span size, the scavenger could account for the reduced probability of freeing entire pages.
• Exposing the tradeoff to users: Documentation or tooling to help users understand and mitigate the RSS overhead on 64KB-page systems.

Any guidance on workarounds or planned runtime improvements for large-page systems would be highly appreciated.

[gabyhelp]

Related Issues

• runtime: scavenging doesn't reduce reported RSS on darwin, may lead to OOMs on iOS #29844 (closed)
• runtime: make the scavenger more prompt #16930 (closed)
• runtime: the scavenger doesn't call `MADV_NOHUGEPAGE` like it used to #55328 (closed)
• runtime: smarter scavenging #30333 (closed)
• runtime: RSS keeps on increasing, suspected scavenger issue #36398 (closed)
• runtime: excessive memory use between 1.21.0 -> 1.21.1 due to hugepages and the linux/amd64 `max_ptes_none` default of 512 #64332 (closed)
• runtime: scavenger not freeing all possible memory on darwin #47656
• runtime: RSS creeps over 1GB even though heap is 4MB #13552 (closed)
• runtime: GC: heap idle is not released to linux #33376 (closed)
• Runtime should reclaim off-heap memory #43402 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

cc @golang/runtime

[prattmic]

On ARM64 with 64KB pages, each OS page covers approximately 8 contiguous Go spans (8KB each). A page can only be released when all 8 spans on it are empty. On x86, each 8KB span covers exactly two 4KB OS pages — when a span becomes empty, both its pages are immediately releasable (assuming no adjacent span shares the page boundary).

With 200-byte objects (Go size class 224, 36 objects per 8KB span), after randomly freeing 95% of objects, ~16% of spans become fully empty (~548 MB). On ARM64, these empty spans are scattered across 64KB pages that also contain spans with surviving objects, preventing release. On x86, the finer 4KB page granularity allows most empty spans' pages to be released independently.

It doesn't sound like this has much to do with span sizes. The core problem here is the size of the page itself.

A page can only be released once it is completely free. A 4KB page can hold ~20 200-byte objects (ignoring size class and metadata overhead). A 64KB page can hold ~327 200-byte objects. If you randomly free 95% of objects, the probability of making a 64KB page completely empty is much lower than a 4KB page.

Perhaps that span size/layout has some impact. You later mention seeing worse results with mixed size classes that I fully thought through, but it still seems to me that the core problem here is just that fragmentation problems scale up with page size.

[arnav-chakraborty]

Hi @prattmic ,

Thanks for the thoughtful response! You're perhaps absolutely right that the core driver is page size — larger pages inherently make it harder to fully free a page under random freeing patterns.

I may be wrong about the span details (I am a not runtime expert), but one thing I noticed is that the simple "objects packed directly into pages" model doesn't quite match what we observed. Under that model, a 4KB page holding ~20 objects would have P(all freed) = 0.95^20 ≈ 36%, meaning x86 should retain ~64% of pages. But we actually see x86 releasing 100% of idle memory (all 548 MB).

My understanding (which could be incomplete) is that there's a span layer in between — Go allocates objects into ~8KB spans, and the scavenger tracks whether spans are idle, not individual objects. An OS page becomes releasable when all spans overlapping it are idle. On x86, an 8KB span maps neatly onto 2 × 4KB pages, so when a span goes idle its pages are immediately releasable. On ARM64, 8 spans share a single 64KB page, so all 8 need to be idle before anything can be released.

But you may well be right that this is a more straightforward page-size problem than I'm making it, and the span details are secondary. Definitely appreciate correction from anyone more familiar with the runtime internals!