Feature request: Even distribution of IP addresses, when multiple subnets per AZ are available

[peanball]

Is your feature request related to a problem? Please describe.

See background in "additional context" below

When a BOSH network contains multiple subnets for the same AZ, it will create VMs distributed into the specified AZs with even distribution. It will however take addresses from the first subnet, until that subnet is exhausted / filled. Only then will it move on to the next subnet, etc.

It is understood that this is an additional feature and should not change default behavior.

Describe the solution you'd like
It would be nice if BOSH could evenly distribute VMs into all available subnets evenly, as it does for AZs.

Describe alternatives you've considered

We are currently assigning static IP addresses manually from known pools, based on the subnets provided by the BOSH cloud config, and assign them manually.

Additional context

When deploying CF using BOSH on hyperscalers, very large CF foundations require NAT gateways in order to manage the apps' egress requests.

NAT gateways are finite resources and can be exhausted with connections. Using further NAT gateways requires separate subnets, at least on AWS and some other hyperscalers. In order to divide egress traffic onto these multiple NAT gateways, subnets are created, each of which is tied to a NAT gateway.

This is also why the even distribution of VMs to subnets becomes important, as it in turn distributes the egress from those VMs to the different NAT gateways.

[rkoster]

Was discussed in FI working group meeting, seems like a good idea. It would be good to wait for the IPv6 director changes that are currently in-flight, because there is a bigger refactor being done as part of that work which probably touches code that would also be related to this feature #2611.

[peanball]

Yes. This came exactly out of the implementation efforts for IPv6.

As for IPv6, we don't need NAT gateways, this whole logic wasn't needed. But you can't just mix and match subnets with BOSH easily, at least not in our setup. Subnets are also dual-stack and what we do for IPv4 then equally applies to IPv6. So we are (in the interim) doing static IPv6 address assignment in our manifest generator, outside of BOSH. And that just feels wrong ;-)

Another item might be to figure out a good way to attach a Diego VM to different subnets, where we may need this split-logic for IPv4, but not for IPv6. This is something that needs further checking though, and the features described in this issue stands on its own.

I wrote this now so we wouldn't forget about it and tackle it sooner than later.