Skip to content

Commit 4d45cdf

Browse files
gran-vmvgran-vmv
authored
[flexible-ipam] Update doc for Antrea 1.5 (#3164)
Signed-off-by: gran <gran@vmware.com>
1 parent c2f3670 commit 4d45cdf

2 files changed

Lines changed: 163 additions & 50 deletions

File tree

‎docs/antrea-ipam.md‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,3 +46,160 @@ cluster. Valid range is 16 to 30. Default is 24.
4646

4747
- `nodeCIDRMaskSizeIPv6`: Mask size for IPv6 Node CIDR in IPv6 or dual-stack
4848
cluster. Valid range is 64 to 126. Default is 64.
49+
50+
## Antrea Flexible-IPAM
51+
52+
Antrea supports flexible control over Pod IP addressing since version 1.4. This can be
53+
achieved by configuring `IPPool` CRD with a desired set of IP ranges. The `IPPool` can be
54+
annotated to Namespace, Pod and PodTemplate of StatefulSet/Deployment. Antrea will manage
55+
IP address assignment for corresponding Pods according to `IPPool` spec.
56+
57+
Note that the IP pool annotation cannot be updated or deleted without recreating the
58+
resource. An `IPPool` can be extended, but cannot be shrunk if already assigned to a
59+
resource. The IP ranges of IPPools must not overlap, otherwise it would lead to undefined
60+
behavior.
61+
62+
Regular `Subnet per Node` IPAM will continue to be used for resources without the IP pool
63+
annotation, or when `AntreaIPAM` feature is disabled.
64+
65+
### Usage
66+
67+
#### Enable AntreaIPAM feature gate
68+
69+
Users need to enable `AntreaIPAM` from the featureGates map defined in antrea.yml for
70+
both Controller and Agent:
71+
72+
```yaml
73+
antrea-controller.conf: |
74+
...
75+
featureGates:
76+
# Enable flexible IPAM mode for Antrea. This mode allows to assign IP Ranges to Namespaces,
77+
# Deployments and StatefulSets via IP Pool annotation.
78+
AntreaIPAM: true
79+
...
80+
antrea-agent.conf: |
81+
...
82+
featureGates:
83+
# Enable flexible IPAM mode for Antrea. This mode allows to assign IP Ranges to Namespaces,
84+
# Deployments and StatefulSets via IP Pool annotation.
85+
AntreaIPAM: true
86+
...
87+
```
88+
89+
#### Create IPPool CR
90+
91+
The following example YAML manifests create an IPPool CR.
92+
93+
```yaml
94+
apiVersion: "crd.antrea.io/v1alpha2"
95+
kind: IPPool
96+
metadata:
97+
name: pool1
98+
spec:
99+
ipVersion: 4
100+
ipRanges:
101+
- start: "10.2.0.12"
102+
end: "10.2.0.20"
103+
gateway: "10.2.0.1"
104+
prefixLength: 24
105+
```
106+
107+
#### IPPool Annotations on Namespace
108+
109+
The following example YAML manifests create a Namespace to allocate Pod IPs from the IP pool.
110+
111+
```yaml
112+
kind: Namespace
113+
metadata:
114+
annotations:
115+
ipam.antrea.io/ippools: 'pool1'
116+
...
117+
```
118+
119+
#### IPPool Annotations on Pod (available since Antrea 1.5)
120+
121+
Since Antrea 1.5, Pod IPPool annotation is supported and has a higher priority than the
122+
Namespace IPPool annotation. This annotation can be added to `PodTemplate` of a
123+
controller resource such as StatefulSet and Deployment.
124+
125+
Pod IP annotation is supported for a single Pod to specify a fixed IP for the Pod.
126+
127+
Examples of annotations on a Pod or PodTemplate:
128+
129+
```yaml
130+
kind: StatefulSet
131+
spec:
132+
replicas: 1 # Do not increase replicas if there is pod-ips annotation in PodTemplate
133+
template:
134+
metadata:
135+
annotations:
136+
ipam.antrea.io/ippools: 'sts-ip-pool1' # This annotation will be set automatically on all Pods managed by this resource
137+
ipam.antrea.io/pod-ips: '<ip-in-sts-ip-pool1>'
138+
...
139+
```
140+
141+
```yaml
142+
kind: StatefulSet
143+
spec:
144+
replicas: 4
145+
template:
146+
metadata:
147+
annotations:
148+
ipam.antrea.io/ippools: 'sts-ip-pool1' # This annotation will be set automatically on all Pods managed by this resource
149+
# Do not add pod-ips annotation to PodTemplate if there is more than 1 replica
150+
...
151+
```
152+
153+
```yaml
154+
kind: Pod
155+
metadata:
156+
annotations:
157+
ipam.antrea.io/ippools: 'pod-ip-pool1'
158+
...
159+
```
160+
161+
```yaml
162+
kind: Pod
163+
metadata:
164+
annotations:
165+
ipam.antrea.io/ippools: 'pod-ip-pool1'
166+
ipam.antrea.io/pod-ips: '<ip-in-pod-ip-pool1>'
167+
...
168+
```
169+
170+
```yaml
171+
kind: Pod
172+
metadata:
173+
annotations:
174+
ipam.antrea.io/pod-ips: '<ip-in-namespace-pool>'
175+
...
176+
```
177+
178+
#### Persistent IP for StatefulSet Pod (available since Antrea 1.5)
179+
180+
A StatefulSet Pod's IP will be kept after Pod restarts, when the IP is allocated from the
181+
annotated IPPool.
182+
183+
### Data path behaviors
184+
185+
When `AntreaIPAM` is enabled, `antrea-agent` will connect the Node's network interface
186+
to the OVS bridge at startup, and it will detach the interface from the OVS bridge and
187+
restore its configurations at exit. Node may lose network connection when `antrea-agent`
188+
or OVS daemons are stopped unexpectedly, which can be recovered by rebooting the Node.
189+
`AntreaIPAM` Pods' traffic will not be routed by local Node's network stack.
190+
191+
All traffic to a local Pod will be sent to the Pod's OVS port directly, after the
192+
destination MAC is rewritten to the Pod's MAC address. This includes `AntreaIPAM` Pods
193+
and regular `Subnet per Node` IPAM Pods, even they are not in the same subnets.
194+
Inter-Node traffic will be sent to the Node network from the source Node, and forwarded
195+
to the destination Node by the Node network.
196+
197+
### Requirements for this Feature
198+
199+
As of now, this feature is supported on Linux Nodes, with IPv4, `system` OVS datapath
200+
type, and `noEncap`, `noSNAT` traffic mode.
201+
202+
The IPs in the `IPPools` must be in the same "underlay" subnet as the Node IP, because
203+
inter-Node traffic of AntreaIPAM Pods is forwarded by the Node network. Only a single IP
204+
pool can be included in the Namespace annotation. In the future, annotation of up to two
205+
pools for IPv4 and IPv6 respectively will be supported.

‎docs/feature-gates.md‎

Lines changed: 6 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -209,59 +209,15 @@ there is a risk of conflicts in CIDR allocation between the two.
209209

210210
### AntreaIPAM
211211

212-
`AntreaIPAM` is a feature that allows flexible control over Pod IP Addressing. This can
213-
be achieved by configuring custom resource `IPPool` with desired set of IP ranges.
214-
The pool can be annotated to Namespace, and Antrea will manage IP address assignment for
215-
corresponding Pods according to pool spec. In the future, support will be extended to
216-
Deployments and StatefulSets, as well as requesting specific IP via Pod annotation.
217-
218-
Note that IP pool annotation can not be updated, but rather re-created. IP Pool can be
219-
extended, but cannot be shrunk if already assigned to a resource. The IP ranges of IP
220-
Pools must not overlap, otherwise it would lead to undefined behavior.
221-
222-
Regular `Subnet per Node` IPAM will continue to be used for resources without IP pool
223-
annotation, or when `AntreaIPAM` feature is disabled.
224-
225-
Usage example:
226-
227-
```yaml
228-
apiVersion: "crd.antrea.io/v1alpha2"
229-
kind: IPPool
230-
metadata:
231-
name: pool1
232-
spec:
233-
ipVersion: 4
234-
ipRanges:
235-
- start: "10.2.0.12"
236-
end: "10.2.0.20"
237-
gateway: "10.2.0.1"
238-
prefixLength: 24
239-
```
240-
241-
```yaml
242-
kind: Namespace
243-
metadata:
244-
annotations:
245-
ipam.antrea.io/ippools: 'pool1'
246-
```
247-
248-
#### Data path change for this feature
249-
250-
When `AntreaIPAM` is enabled, `antrea-agent` will connect the Node's network interface
251-
to the OVS bridge at startup, and it will detach the interface from the OVS bridge and
252-
restore its configurations at exit. Node may lose network connection when `antrea-agent`
253-
or OVS daemons are stopped unexpectedly, which can be recovered by rebooting the Node.
254-
`AntreaIPAM` Pods' traffic will not be routed by local Node's network stack.
255-
256-
All traffic to a local Pod will be sent to the Pod's OVS port directly, after the
257-
destination MAC is rewritten to the Pod's MAC address. This includes `AntreaIPAM` Pods
258-
and regular `Subnet per Node` IPAM Pods, even they are not in the same subnets.
259-
Inter-Node traffic will be sent to the Node network from the source Node, and forwarded
260-
to the destination Node by the Node network.
212+
`AntreaIPAM` feature allows flexible control over Pod IP addressing. This can be
213+
achieved by configuring `IPPool` CRD with a desired set of IP ranges. The `IPPool` can be
214+
annotated to Namespace, Pod and PodTemplate of StatefulSet/Deployment. Antrea will manage
215+
IP address assignment for corresponding Pods according to `IPPool` spec.
216+
Refer to this [document](antrea-ipam.md) for more information.
261217

262218
#### Requirements for this Feature
263219

264-
In Antrea 1.4, this feature is supported on Linux Nodes, with IPv4, `system` OVS datapath
220+
As of now, this feature is supported on Linux Nodes, with IPv4, `system` OVS datapath
265221
type, and `noEncap`, `noSNAT` traffic mode.
266222

267223
The IPs in the `IPPools` must be in the same "underlay" subnet as the Node IP, because

0 commit comments

Comments
 (0)