Serial device rate limiter

CHANGELOG.md

@@ -14,6 +14,10 @@ and this project adheres to
   support for Vsock Unix domain socket path overriding on snapshot restore. More
   information can be found in the
   [docs](docs/vsock.md/#unix-domain-socket-renaming).
+- [#5824](https://github.com/firecracker-microvm/firecracker/pull/5824): Add
+  optional rate limiting to serial console output, configurable via the
+  `rate_limiter` field on `PUT /serial`. A new metric is exposed under `uart`:
+  `rate_limiter_dropped_bytes`.
 
 ### Changed
 

docs/device-api.md

@@ -32,6 +32,7 @@ BadRequest - HTTP response.
 | `vsock`                   |    O     |       O        |      O       |        O         |     O      |      O       |     O      |      O      |     O      |
 | `entropy`                 |    O     |       O        |      O       |        O         |     O      |      O       |   **R**    |      O      |     O      |
 | `pmem/{id}`               |    O     |       O        |      O       |        O         |     O      |      O       |     O      |    **R**    |     O      |
+| `serial`                  |    O     |     **R**      |      O       |        O         |     O      |      O       |     O      |      O      |     O      |
 
 ## Input Schema
 
@@ -106,6 +107,8 @@ specification:
 |                           | path_on_host       |    O     |       O        |      O       |        O         |     O      |      O       |     O      |    **R**    |     O      |
 |                           | root_device        |    O     |       O        |      O       |        O         |     O      |      O       |     O      |    **R**    |     O      |
 |                           | read_only          |    O     |       O        |      O       |        O         |     O      |      O       |     O      |    **R**    |     O      |
+| `SerialConfig`            | serial_out_path    |    O     |       O        |      O       |        O         |     O      |      O       |     O      |      O      |     O      |
+|                           | rate_limiter       |    O     |       O        |      O       |        O         |     O      |      O       |     O      |      O      |     O      |
 | `MemoryHotplugConfig`     | total_size_mib     |    O     |       O        |      O       |        O         |     O      |      O       |     O      |      O      |   **R**    |
 |                           | slot_size_mib      |    O     |       O        |      O       |        O         |     O      |      O       |     O      |      O      |   **R**    |
 |                           | block_size_mi      |    O     |       O        |      O       |        O         |     O      |      O       |     O      |      O      |   **R**    |
@@ -115,7 +118,7 @@ specification:
 either virtio-block or vhost-user-block devices.
 
 \*\* The `TokenBucket` can be configured with any combination of virtio-net,
-virtio-block and virtio-rng devices.
+virtio-block, virtio-rng and serial devices.
 
 ## Output Schema
 

docs/prod-host-setup.md

@@ -27,17 +27,17 @@ recommended.
 
 Firecracker implements the 8250 serial device, which is visible from the guest
 side and is tied to the Firecracker/non-daemonized jailer process stdout.
-Without proper handling, because the guest has access to the serial device, this
-can lead to unbound memory or storage usage on the host side. Firecracker does
-not offer users the option to limit serial data transfer, nor does it impose any
-restrictions on stdout handling. Users are responsible for handling the memory
-and storage usage of the Firecracker process stdout. We suggest using any
-upper-bounded forms of storage, such as fixed-size or ring buffers, using
-programs like `journald` or `logrotate`, or redirecting to `/dev/null` or a
-named pipe. Furthermore, we do not recommend that users enable the serial device
-in production. To disable it in the guest kernel, use the `8250.nr_uarts=0` boot
-argument when configuring the boot source. Please be aware that the device can
-be reactivated from within the guest even if it was disabled at boot.
+Without proper handling, because the guest has access to the serial device. This
+can lead to unbound memory or storage usage on the host side. Users are
+responsible for handling the memory and storage usage of the Firecracker process
+stdout. We recommend using the rate limiter for the serial data transfer that
+Firecracker offers or any upper-bounded forms of storage, such as fixed-size or
+ring buffers, using programs like `journald` or `logrotate`, or redirecting to
+`/dev/null` or a named pipe. Furthermore, we do not recommend that users enable
+the serial device in production. To disable it in the guest kernel, use the
+`8250.nr_uarts=0` boot argument when configuring the boot source. Please be
+aware that the device can be reactivated from within the guest even if it was
+disabled at boot.
 
 If Firecracker's `stdout` buffer is non-blocking and full (assuming it has a
 bounded size), any subsequent writes will fail, resulting in data loss, until

src/firecracker/src/api_server/request/serial.rs

@@ -21,6 +21,8 @@ pub(crate) fn parse_put_serial(body: &Body) -> Result<ParsedRequest, RequestErro
 mod tests {
     use std::path::PathBuf;
 
+    use vmm::vmm_config::TokenBucketConfig;
+
     use super::*;
     use crate::api_server::parsed_request::tests::vmm_action_from_request;
 
@@ -30,6 +32,28 @@ mod tests {
 
         let expected_config = SerialConfig {
             serial_out_path: Some(PathBuf::from("serial")),
+            rate_limiter: None,
+        };
+        assert_eq!(
+            vmm_action_from_request(parse_put_serial(&Body::new(body)).unwrap()),
+            VmmAction::ConfigureSerial(expected_config)
+        );
+    }
+
+    #[test]
+    fn test_parse_put_serial_with_rate_limiter() {
+        let body = r#"{
+            "serial_out_path": "serial",
+            "rate_limiter": {"size": 1024, "one_time_burst": 65536, "refill_time": 1000}
+        }"#;
+
+        let expected_config = SerialConfig {
+            serial_out_path: Some(PathBuf::from("serial")),
+            rate_limiter: Some(TokenBucketConfig {
+                size: 1024,
+                one_time_burst: Some(65536),
+                refill_time: 1000,
+            }),
         };
         assert_eq!(
             vmm_action_from_request(parse_put_serial(&Body::new(body)).unwrap()),

src/firecracker/swagger/firecracker.yaml

@@ -1720,6 +1720,14 @@ definitions:
       serial_out_path:
         type: string
         description: Path to a file or named pipe on the host to which serial output should be written.
+      rate_limiter:
+        $ref: "#/definitions/TokenBucket"
+        description:
+          Optional token bucket for rate limiting serial console output
+          bandwidth. When present, guest writes to the serial port that
+          exceed the configured rate are silently dropped and counted in
+          the `uart.rate_limiter_dropped_bytes` metric. When absent
+          (the default), serial output is not rate-limited.
 
   MemoryHotplugConfig:
     type: object

src/vmm/src/arch/aarch64/fdt.rs

@@ -583,7 +583,7 @@ mod tests {
         cmdline.insert("console", "/dev/tty0").unwrap();
 
         device_manager
-            .attach_legacy_devices_aarch64(&vm, &mut event_manager, &mut cmdline, None)
+            .attach_legacy_devices_aarch64(&vm, &mut event_manager, &mut cmdline, None, None)
             .unwrap();
         let dummy = Arc::new(Mutex::new(DummyDevice::new()));
         device_manager

src/vmm/src/builder.rs

@@ -195,6 +195,7 @@ pub fn build_microvm_for_boot(
         &vcpus_exit_evt,
         &vm,
         vm_resources.serial_out_path.as_ref(),
+        vm_resources.serial_rate_limiter(),
     )?;
 
     let vm = Arc::new(vm);
@@ -285,6 +286,7 @@ pub fn build_microvm_for_boot(
         event_manager,
         &mut boot_cmdline,
         vm_resources.serial_out_path.as_ref(),
+        vm_resources.serial_rate_limiter(),
     )?;
 
     device_manager.attach_vmgenid_device(&vm)?;

src/vmm/src/device_manager/legacy.rs

@@ -149,7 +149,7 @@ mod tests {
     use vmm_sys_util::eventfd::EventFd;
 
     use super::*;
-    use crate::devices::legacy::serial::SerialOut;
+    use crate::devices::legacy::serial::{SerialOut, SerialOutInner};
     use crate::devices::legacy::{EventFdTrigger, SerialEventsWrapper};
     use crate::vstate::vm::tests::setup_vm_with_memory;
 
@@ -164,7 +164,7 @@ mod tests {
                     SerialEventsWrapper {
                         buffer_ready_event_fd: None,
                     },
-                    SerialOut::Sink,
+                    SerialOut::new(SerialOutInner::Sink, None),
                 ),
                 input: None,
             })),

src/vmm/src/device_manager/mod.rs

@@ -30,7 +30,7 @@ use crate::devices::legacy::I8042Device;
 #[cfg(target_arch = "aarch64")]
 use crate::devices::legacy::RTCDevice;
 use crate::devices::legacy::SerialDevice;
-use crate::devices::legacy::serial::SerialOut;
+use crate::devices::legacy::serial::{SerialOut, SerialOutInner};
 use crate::devices::pseudo::BootTimer;
 use crate::devices::virtio::ActivateError;
 use crate::devices::virtio::balloon::BalloonError;
@@ -42,6 +42,7 @@ use crate::devices::virtio::pmem::persist::PmemPersistError;
 use crate::devices::virtio::rng::persist::EntropyPersistError;
 use crate::devices::virtio::transport::mmio::{IrqTrigger, MmioTransport};
 use crate::devices::virtio::vsock::{VsockError, VsockUnixBackendError};
+use crate::rate_limiter::TokenBucket;
 use crate::resources::VmResources;
 use crate::snapshot::Persist;
 use crate::utils::open_file_nonblock;
@@ -134,13 +135,23 @@ impl DeviceManager {
         event_manager: &mut EventManager,
         output: Option<&PathBuf>,
         state: Option<&serial::SerialState>,
+        rate_limiter: Option<TokenBucket>,
     ) -> Result<Arc<Mutex<SerialDevice>>, std::io::Error> {
         let (serial_in, serial_out) = match output {
-            Some(path) => (None, open_file_nonblock(path).map(SerialOut::File)?),
+            Some(path) => (
+                None,
+                SerialOut::new(
+                    SerialOutInner::File(open_file_nonblock(path)?),
+                    rate_limiter,
+                ),
+            ),
             None => {
                 Self::set_stdout_nonblocking();
 
-                (Some(std::io::stdin()), SerialOut::Stdout(std::io::stdout()))
+                (
+                    Some(std::io::stdin()),
+                    SerialOut::new(SerialOutInner::Stdout(std::io::stdout()), rate_limiter),
+                )
             }
         };
 
@@ -179,9 +190,15 @@ impl DeviceManager {
         vm: &Vm,
         serial_output: Option<&PathBuf>,
         serial_state: Option<&serial::SerialState>,
+        serial_rate_limiter: Option<TokenBucket>,
     ) -> Result<PortIODeviceManager, DeviceManagerCreateError> {
         // Create serial device
-        let serial = Self::setup_serial_device(event_manager, serial_output, serial_state)?;
+        let serial = Self::setup_serial_device(
+            event_manager,
+            serial_output,
+            serial_state,
+            serial_rate_limiter,
+        )?;
         let reset_evt = vcpus_exit_evt
             .try_clone()
             .map_err(DeviceManagerCreateError::EventFd)?;
@@ -203,10 +220,17 @@ impl DeviceManager {
         vcpus_exit_evt: &EventFd,
         vm: &Vm,
         serial_output: Option<&PathBuf>,
+        serial_rate_limiter: Option<TokenBucket>,
     ) -> Result<Self, DeviceManagerCreateError> {
         #[cfg(target_arch = "x86_64")]
-        let legacy_devices =
-            Self::create_legacy_devices(event_manager, vcpus_exit_evt, vm, serial_output, None)?;
+        let legacy_devices = Self::create_legacy_devices(
+            event_manager,
+            vcpus_exit_evt,
+            vm,
+            serial_output,
+            None,
+            serial_rate_limiter,
+        )?;
 
         Ok(DeviceManager {
             mmio_devices: MMIODeviceManager::new(),
@@ -292,6 +316,7 @@ impl DeviceManager {
         event_manager: &mut EventManager,
         cmdline: &mut Cmdline,
         serial_out_path: Option<&PathBuf>,
+        serial_rate_limiter: Option<TokenBucket>,
     ) -> Result<(), AttachDeviceError> {
         // Serial device setup.
         let cmdline_contains_console = cmdline
@@ -302,7 +327,12 @@ impl DeviceManager {
             .contains("console=");
 
         if cmdline_contains_console {
-            let serial = Self::setup_serial_device(event_manager, serial_out_path, None)?;
+            let serial = Self::setup_serial_device(
+                event_manager,
+                serial_out_path,
+                None,
+                serial_rate_limiter,
+            )?;
             self.mmio_devices.register_mmio_serial(vm, serial, None)?;
             self.mmio_devices.add_mmio_serial_to_cmdline(cmdline)?;
         }
@@ -549,6 +579,7 @@ impl<'a> Persist<'a> for DeviceManager {
             constructor_args.vm,
             constructor_args.vm_resources.serial_out_path.as_ref(),
             serial_state.as_ref(),
+            constructor_args.vm_resources.serial_rate_limiter(),
         )?;
 
         // Restore MMIO devices
@@ -608,7 +639,7 @@ pub(crate) mod tests {
         #[cfg(target_arch = "x86_64")]
         let legacy_devices = PortIODeviceManager {
             stdio_serial: Arc::new(Mutex::new(
-                SerialDevice::new(None, SerialOut::Sink, None).unwrap(),
+                SerialDevice::new(None, SerialOut::new(SerialOutInner::Sink, None), None).unwrap(),
             )),
             i8042: Arc::new(Mutex::new(
                 I8042Device::new(EventFd::new(libc::EFD_NONBLOCK).unwrap()).unwrap(),
@@ -634,15 +665,15 @@ pub(crate) mod tests {
         let mut cmdline = Cmdline::new(4096).unwrap();
         let mut event_manager = EventManager::new().unwrap();
         vmm.device_manager
-            .attach_legacy_devices_aarch64(&vmm.vm, &mut event_manager, &mut cmdline, None)
+            .attach_legacy_devices_aarch64(&vmm.vm, &mut event_manager, &mut cmdline, None, None)
             .unwrap();
         assert!(vmm.device_manager.mmio_devices.rtc.is_some());
         assert!(vmm.device_manager.mmio_devices.serial.is_none());
 
         let mut vmm = default_vmm();
         cmdline.insert("console", "/dev/blah").unwrap();
         vmm.device_manager
-            .attach_legacy_devices_aarch64(&vmm.vm, &mut event_manager, &mut cmdline, None)
+            .attach_legacy_devices_aarch64(&vmm.vm, &mut event_manager, &mut cmdline, None, None)
             .unwrap();
         assert!(vmm.device_manager.mmio_devices.rtc.is_some());
         assert!(vmm.device_manager.mmio_devices.serial.is_some());

src/vmm/src/device_manager/persist.rs

@@ -373,6 +373,7 @@ impl<'a> Persist<'a> for MMIODeviceManager {
                         constructor_args.event_manager,
                         constructor_args.vm_resources.serial_out_path.as_ref(),
                         serial_state.as_ref(),
+                        constructor_args.vm_resources.serial_rate_limiter(),
                     )?;
 
                     dev_manager.register_mmio_serial(vm, serial, Some(state.device_info))?;