Research spike: investigate solutions around pulling images that exceed VCH tmpfs size

[jzt]

When pulling an image that exceeds the available tmpfs space on the VCH, a "no space left on device" error occurs as the /tmp partition becomes full.

Currently, we have two solutions in mind:

1. Implement a shared buffer on the portlayer side that will read up to a certain watermark, block downloads, drain to disk, repeat.

2. Implement a large ephemeral disk to use as temporary space instead of using tmpfs on the VCH. This disk should be created at VCH provision time, must be large enough (10s of GB) to support multiple concurrent large image pulls, and must be able to resize (shrink) itself on VCH restart.

We need to research the complexity and correctness of both solutions, with a preference for correctness over less complex.

bug2157676

[hickeng]

Bumping to high - the fact that we can shutdown the endpointVM simply by pulling a large image from a fast repo is going to be compounded with Harbor deployments and large images are more likely to occur in enterprise than elsewhere.

[mdubya66]

doing the priority dance, by definition a research spike is not high