Skip to content

Commit 3298185

Browse files
leftwoAlan Hanson
leftwo
and
Alan Hanson
authored
Bump up the reservoir percentage (#10791)
Increase the reservoir percentage from 86.3 to 91.0 A rough breakdown of where memory goes: * Memory reserved for kernel page_t * Control plane earmark * Reservoir (reserved for VMs) * Crucible, and anything else. The Formula for reservoir is: Reservoir = (physical_ram_bytes - max_page_t_bytes - control_plane_earmark_bytes) * .863 The formula for page_t is: max_page_t_bytes = physical_ram_bytes / 4096 * 120 This change takes the reservoir percentage from 86.3 to 91.0. Current sled-agent logs on a cosmo showing the breakdown (before this change): ``` 00:19:36.348Z INFO SledAgent (MemoryReservations): Calculated eligible VMM reservoir size control_plane_earmark_bytes = 47244640256 max_page_t_bytes = 44289997440 physical_ram_bytes = 1511765245952 vmm_eligible = 1420230608256 00:19:36.348Z INFO SledAgent: 86.3% of 1420230608256 VMM eligible ram = 1225659058266 bytes) (Some correction happens after this, some 2M boundary stuff) 00:19:36.349Z INFO SledAgent: Setting reservoir size to 1168878 MiB bytes ``` Current Cosmo memstat output (before this change): ``` 2F8JEXDK # mdb -ke ::memstat Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 16243364 63450 4% Boot pages 11 0 0% ZFS File Data 4764391 18610 1% VMM Memory 299232768 1168878 81% Anon 2083177 8137 1% Exec and libs 157547 615 0% Page cache 810780 3167 0% Free (cachelist) 66052 258 0% Free (freelist) 45723173 178606 12% Total 369081263 1441723 Physical 369081261 1441723 ``` (This example is from a fresh install that was not very busy, I don't have a memstat output from a busy cosmo before making the reservoir change, so the VMM % is correct, but the other values might not be a perfect example) Current sled agent logs on a gimlet (before this change): ``` 22:11:17.419Z INFO SledAgent (MemoryReservations): Calculated eligible VMM reservoir size control_plane_earmark_bytes = 47244640256 max_page_t_bytes = 31834245120 physical_ram_bytes = 1086608900096 vmm_eligible = 1007530014720 22:11:17.419Z INFO SledAgent: 86.3% of 1007530014720 VMM eligible ram = 869498433450 bytes) ``` Current memstat on gimlet (before this change): ``` Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 15288787 59721 6% Boot pages 11 0 0% ZFS File Data 13899117 54293 5% VMM Memory 212279808 829218 80% Anon 5488274 21438 2% Exec and libs 133384 521 0% Page cache 7178 28 0% Free (cachelist) 226922 886 0% Free (freelist) 17959846 70155 7% Total 265283327 1036262 Physical 265283325 1036262 ``` New sled-agent logs on a cosmo showing the breakdown (with this change): ``` 00:31:18.563Z INFO SledAgent (MemoryReservations): Calculated eligible VMM reservoir size control_plane_earmark_bytes = 47244640256 file = sled-hardware/src/lib.rs:139 max_page_t_bytes = 48316529280 physical_ram_bytes = 1649204199424 vmm_eligible = 1553643029888 00:31:18.564Z INFO SledAgent: 91% of 1553643029888 VMM eligible ram = 1413815157198 bytes) 00:31:18.564Z INFO SledAgent: Setting reservoir size to 1348318 MiB bytes ``` New memstat on Cosmo (with this change): ``` 2CN2M459 # mdb -ke ::memstat Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 19892915 77706 5% Boot pages 11 0 0% ZFS File Data 20732947 80988 5% VMM Memory 345169408 1348318 86% Anon 5371715 20983 1% Exec and libs 134535 525 0% Page cache 6692 26 0% Free (cachelist) 125564 490 0% Free (freelist) 11201908 43757 3% Total 402635695 1572795 Physical 402635693 1572795 ``` New sled-agent logs on a cosmo showing the breakdown (with this change): ``` 01:14:39.886Z INFO SledAgent (MemoryReservations): Calculated eligible VMM reservoir size control_plane_earmark_bytes = 47244640256 file = sled-hardware/src/lib.rs:139 max_page_t_bytes = 31834245120 physical_ram_bytes = 1086608900096 vmm_eligible = 1007530014720 01:14:39.887Z INFO SledAgent: 91% of 1007530014720 VMM eligible ram = 916852313395 bytes) 01:14:39.888Z INFO SledAgent: Setting reservoir size to 874378 MiB bytes ``` New memstat on Gimlet (with this change): ``` BRM42220036 # mdb -ke ::memstat Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 13871627 54186 5% Boot pages 11 0 0% ZFS File Data 16122310 62977 6% VMM Memory 223840768 874378 84% Anon 4072931 15909 2% Exec and libs 201520 787 0% Page cache 17281 67 0% Free (cachelist) 139376 544 0% Free (freelist) 7017503 27412 3% Total 265283327 1036262 Physical 265283325 1036262 ``` I ran this change on London for about a week to see if I could get any sign of memory pressure and I was not able to. I manually added to the reservoir on dogfood and it's been that way for two days now, though there was an update and I had to go back and re-add to the reservoir. There are a bunch of logs of memory stats gathered from dogfood at: /staff/core/omicron-7911/ Co-authored-by: Alan Hanson <alan@oxide.computer>
1 parent 57769f6 commit 3298185

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

‎smf/sled-agent/gimlet/config.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ control_plane_memory_earmark_mb = 45056
4747
#
4848
# We expect this should be higher, but currently a lack of long-uptime
4949
# varied-usage measurements leads us to be cautious.
50-
vmm_reservoir_percentage = 86.3
50+
vmm_reservoir_percentage = 91.0
5151

5252
# Swap device size for the system. The device is a sparsely allocated zvol on
5353
# the internal zpool of the M.2 that we booted from.

0 commit comments

Comments
 (0)