Write auto-expander for the disk

[yuvipanda]

Optionally, when the disk gets to about 99% full, we could have a sidecar running that:

1. Automatically talks to the cloud API
2. Resizes the disk up
3. Resizes the xfs volume up

This would need specific support for cloud providers + ways to setup permissions for doing so.

We should support EBS and GCP to start with. This should be a python program running as a sidecar that can be configured with different cloud providers.

What we want to learn: can we automate here, and what kind of time does it save us?

What we need to do

• Create a Python script in jupyterhub-home-nfs/auto-expander.py
• Have the script use any needed secret config from a k8s secret
• In the script, using something like psutil.disk_usage, check if the home directories disk has less than 10% free space
• When the above condition is met, run a disk resize function for either aws or gcp, followed by an xfs resize
• Write the AWS resize function
  • setup a boto3.Client with the appropriate creds, maybe available as secret env vars
  • use https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2/client/modify_volume.html to resize the volume
  • bash script example https://docs.aws.amazon.com/cloud9/latest/user-guide/move-environment-resize.html using REST API
• Write the GCP resize function
  • use https://docs.cloud.google.com/python/docs/reference/compute/latest/google.cloud.compute_v1.services.disks.DisksClient?hl=en to resize the disk

Definition of Done

• such a script exists
• a sidecar container that runs this script exists
• it has been verified that it works