Switch zone space exhausted from logs

[rcgoodfellow]

We found a switch zone that looked like this

# zfs list rpool/zone/oxz_switch
NAME                    USED  AVAIL     REFER  MOUNTPOINT
rpool/zone/oxz_switch  2.97G  4.82M     2.97G  /zone/oxz_switch

due to this

-rw-r--r--   1 root     root     1703936 May  8 07:30 oxide-dendrite:default.log.0
drwxr-xr-x   2 root     sys          132 May  8 07:30 .
-rw-r--r--   1 root     root      393216 May  8 07:15 oxide-dendrite:default.log.1
-rw-r--r--   1 root     root     1441792 May  8 07:00 oxide-dendrite:default.log.2
-rw-r--r--   1 root     root     1383575 May  8 06:47 oxide-mg-ddm:default.log
-rw-r--r--   1 root     root     14286848 May  8 06:45 oxide-dendrite:default.log.3
-rw-r--r--   1 root     root     29766746 May  8 06:45 oxide-wicketd:default.log
-rw-r--r--   1 root     root      289898 May  8 06:45 oxide-pumpkind:default.log
-rw-r--r--   1 root     root     3612606464 May  8 06:31 oxide-dendrite:default.log.4
-rw-r--r--   1 root     root     4185253382 May  8 06:31 oxide-dendrite:default.log
-rw-r--r--   1 root     root     80839571 May  8 06:31 oxide-mgd:default.log
-rw-r--r--   1 root     root     61343803 May  8 06:31 oxide-mgs:default.log
-rw-r--r--   1 root     root     40323877 May  8 06:31 oxide-tfport:default.log
-rw-r--r--   1 root     root     2709568 May  8 06:26 oxide-lldpd:default.log
-rw-r--r--   1 root     root     4195630867 May  8 06:15 oxide-dendrite:default.log.5
-rw-r--r--   1 root     root     4203580961 May  8 06:00 oxide-dendrite:default.log.6
-rw-r--r--   1 root     root     4194655483 May  8 05:45 oxide-dendrite:default.log.7
-rw-r--r--   1 root     root     4196811310 May  8 05:30 oxide-dendrite:default.log.8
-rw-r--r--   1 root     root     4177313624 May  8 05:15 oxide-dendrite:default.log.9
-rw-r--r--   1 root     root     115733711 May  8 05:15 oxide-mgs:default.log.0
-rw-r--r--   1 root     root     107470637 May  8 05:00 oxide-mgd:default.log.0
-rw-r--r--   1 root     root     4191302754 May  8 05:00 oxide-dendrite:default.log.10
-rw-r--r--   1 root     root     4188813289 May  8 04:45 oxide-dendrite:default.log.11
-rw-r--r--   1 root     root     4222101031 May  8 04:30 oxide-dendrite:default.log.12

A sample log file can be found at /staff/core/dendrite-226. Due to production urgency we needed to move quickly with this one so that is all I have at the moment.

[rcgoodfellow]

We noticed that the oxControlService0/omicron6 address was missing from this switch zone, which may be the cause of the rage. Not sure why that address was missing yet.

[rcgoodfellow]

This happened after a pfexec reboot in the switch zone. The ox* addresses do not have persistent bits set in the OS, so we lose them on reboot. So that explains why the address is gone. We may be able to reproduce the log rampage by knocking out the ox* addrs.

[Nieuwejaar]

The log sample is only 28k, which makes it hard to glean anything.

Is this on a local system or at a customer site?

What version of dendrite was this running? If it was before we disabled the interrupt logging, that could easily explain the massive log files. I don't see any of that in the log sample, but it only includes about 100 entries.