Add docs for TrafficControl feature by tnqn · Pull Request #3800 · antrea-io/antrea

tnqn · 2022-05-17T05:42:21Z

Add a document to introduce the feature
Add a cookbook to describe how to use Antrea with Suricata to provide
IDS to Pods

Signed-off-by: Quan Tian qtian@vmware.com

antoninbas

great documentation

antoninbas · 2022-05-17T21:26:59Z

+following specification creates a new TrafficControl object named
+"mirror-web-app", which mirrors all traffic from or to any Pod with the
+`app=web` label and send them to a collector running on "10.0.10.2" encapsulated
+within VXLAN tunnel:


within a VXLAN tunnel

antoninbas · 2022-05-17T21:28:59Z

+`podSelector`. If set with a `namespaceSelector`, all Pods from Namespaces
+selected by the `namespaceSelector` will be selected. Specific Pods from
+specific Namespaces can be selected by providing both a `podSelector` and a
+`namespaceSelector`. Empty `appliedTo` selects nothing. The field is mandatory.


IIRC, empty podSelector and empty namespaceSelector can be used to select all Pods in the cluster?

that was also my first thought but I found we made it select nothing from the beginning for NetworkPolicy and Egress so have to keep consistency:

antrea/pkg/controller/grouping/group_entity_index.go

Line 739 in 017f8be

// The group selects nothing when all selectors are missing.

https://github.com/antrea-io/antrea/blob/main/docs/egress.md#appliedto

ok, but jut to make sure, the following (which is what I meant by having both selectors be empty) will select all Pods in the cluster:

appliedTo: namespaceSelector: podSelector:

while the following (both selectors unset) will select nothing:

appliedTo:

the above two examples mean the same, selecting nothing, because namespaceSelector and podSelector are "omitempty".
To select all Pods, we need to use either:

appliedTo: namespaceSelector: {}

or

appliedTo: podSelector: {}

antoninbas · 2022-05-17T21:29:38Z

+### Action
+
+The `action` field specifies which action should be taken for the traffic. It
+can be `Mirror` or `Redirect`. For `Mirror` action, `targetPort` must be


the Mirror action

antoninbas · 2022-05-17T21:29:53Z

+
+The `action` field specifies which action should be taken for the traffic. It
+can be `Mirror` or `Redirect`. For `Mirror` action, `targetPort` must be
+specified to the port to which the traffic will be mirrored to. For `Redirect`


targetPort must be set to the port to which the traffic will be mirrored.

antoninbas · 2022-05-17T21:30:30Z

+
+The `action` field specifies which action should be taken for the traffic. It
+can be `Mirror` or `Redirect`. For `Mirror` action, `targetPort` must be
+specified to the port to which the traffic will be mirrored to. For `Redirect`


For the Redirect action

antoninbas · 2022-05-17T21:32:11Z

+**device**: This specifies a network device on all Nodes. A Pod's traffic will
+be redirected or mirrored to the network device on the same Node that hosts the
+Pod. The network device must exist on all Nodes and Antrea will attach it to the
+OVS bridge if not already attached. To use a network device, `name` of the


antoninbas · 2022-05-17T21:33:00Z

+  name: eno2
+```
+
+**geneve**: This specifies a remote destination based on GENEVE tunnel. All


nit: I would say "a remote destination for a GENEVE tunnel"

same below

antoninbas · 2022-05-17T21:38:14Z

+To replicate Pod traffic to threat detection engines for analysis, create a
+TrafficControl with the `Mirror` action, and set the `targetPort` to an OVS
+internal port that threat detection engines will capture traffic from. This
+cookbook uses `tap0` as the port name and perform intrusion detection for Pods


s/perform/performs

antoninbas · 2022-05-17T21:39:14Z

+
+You can then generate malicious requests to trigger alerts. For ingress, you can
+fake a web application attack against the Pod with the following command
+(assumed that the Pod IP is 10.10.2.3):


s/assumed/assuming

tnqn

@antoninbas thanks for the review.

tnqn · 2022-05-26T15:11:46Z

+To replicate Pod traffic to threat detection engines for analysis, create a
+TrafficControl with the `Mirror` action, and set the `targetPort` to an OVS
+internal port that threat detection engines will capture traffic from. This
+cookbook uses `tap0` as the port name and perform intrusion detection for Pods


tnqn · 2022-05-26T15:12:14Z

+
+You can then generate malicious requests to trigger alerts. For ingress, you can
+fake a web application attack against the Pod with the following command
+(assumed that the Pod IP is 10.10.2.3):


tnqn · 2022-05-26T15:12:39Z

+following specification creates a new TrafficControl object named
+"mirror-web-app", which mirrors all traffic from or to any Pod with the
+`app=web` label and send them to a collector running on "10.0.10.2" encapsulated
+within VXLAN tunnel:


tnqn · 2022-05-26T15:17:33Z

+`podSelector`. If set with a `namespaceSelector`, all Pods from Namespaces
+selected by the `namespaceSelector` will be selected. Specific Pods from
+specific Namespaces can be selected by providing both a `podSelector` and a
+`namespaceSelector`. Empty `appliedTo` selects nothing. The field is mandatory.


that was also my first thought but I found we made it select nothing from the beginning for NetworkPolicy and Egress so have to keep consistency:

antrea/pkg/controller/grouping/group_entity_index.go

Line 739 in 017f8be

// The group selects nothing when all selectors are missing.

https://github.com/antrea-io/antrea/blob/main/docs/egress.md#appliedto

tnqn · 2022-05-26T15:19:25Z

+### Action
+
+The `action` field specifies which action should be taken for the traffic. It
+can be `Mirror` or `Redirect`. For `Mirror` action, `targetPort` must be


tnqn · 2022-05-26T15:19:37Z

+can be `Mirror` or `Redirect`. For `Mirror` action, `targetPort` must be
+specified to the port to which the traffic will be mirrored to. For `Redirect`
+action, both `targetPort` and `returnPort` need to be specified, the latter of
+which represents a port from which the traffic will be sent back to OVS and be


tnqn · 2022-05-26T15:20:21Z

+**ovsInternal**: This specifies an OVS internal port on all Nodes. A Pod's
+traffic will be redirected or mirrored to the OVS internal port on the same Node
+that hosts the Pod. The port doesn't need to exist in advance, Antrea will
+create the port if it doesn't exist. To use an OVS internal port, `name` of the 


tnqn · 2022-05-26T15:20:25Z

+**device**: This specifies a network device on all Nodes. A Pod's traffic will
+be redirected or mirrored to the network device on the same Node that hosts the
+Pod. The network device must exist on all Nodes and Antrea will attach it to the
+OVS bridge if not already attached. To use a network device, `name` of the


tnqn · 2022-05-26T15:21:34Z

+  name: eno2
+```
+
+**geneve**: This specifies a remote destination based on GENEVE tunnel. All


jianjuns

Not finished yet. Will continue.

jianjuns · 2022-05-26T17:12:02Z

+
+### TrafficControl
+
+`TrafficControl` enables a CRD API for Antrea that manages and manipulates the


How about "controls" to be consistent with TrafficControl.

jianjuns · 2022-05-26T17:12:34Z

+### TrafficControl
+
+`TrafficControl` enables a CRD API for Antrea that manages and manipulates the
+transmission of Pod traffic. It allows users to mirror or redirect specific


As we do not support matching specific traffic, we can remove this "specific"?

jianjuns · 2022-05-26T17:15:22Z

+`TrafficControl` enables a CRD API for Antrea that manages and manipulates the
+transmission of Pod traffic. It allows users to mirror or redirect specific
+traffic originating from specific Pods or destined to specific Pods to a local
+network device or a remote destination via various tunnels. It provides full


How about changing "provides" to "enables", or "enables a monitoring solution to get full visibility..."

jianjuns · 2022-05-26T17:17:05Z

+  purposes such as troubleshooting, intrusion detection, and so on.
+
+- You want to redirect network traffic passing in or out of a set of Pods to
+  applications that analyze traffic, enforce policies, and reject traffic to


Remove "analyze traffic" as that should fall into the mirroring bucket?

jianjuns · 2022-05-26T17:20:33Z

+      remoteIP: 10.0.10.2
+```
+
+### Redirecting specific traffic to local analyzer


Should we just use "local receiver" to be generic? Typically, redirection is not for an analyzer that does not change/drop the packets.

jianjuns · 2022-05-26T17:23:34Z

+  prevent intrusion.
+
+This guide demonstrates how to configure `TrafficControl` to achieve the above
+result.


results

goals?

jianjuns · 2022-05-26T17:24:37Z

+
+TrafficControl was introduced in v1.7 as an alpha feature. A feature gate,
+`TrafficControl` must be enabled on the antrea-agent in the `antrea-config`
+ConfigMap like the following options for the feature to work:


like the following options -> like the following?

Maybe it to the end of the sentence.

sorry, what does your second suggestion mean?

Typo. I meant:

must be enabled on the antrea-agent in the antrea-config ConfigMap for the feature to work, like the following:

jianjuns · 2022-05-26T17:28:41Z

+For example, suppose you have a set of Pods which contain a label `app=web`, the
+following specification creates a new TrafficControl object named
+"mirror-web-app", which mirrors all traffic from or to any Pod with the
+`app=web` label and send them to a collector running on "10.0.10.2" encapsulated


"receiver" to be generic

jianjuns · 2022-05-26T17:39:58Z

+
+To replicate Pod traffic to threat detection engines for analysis, create a
+TrafficControl with the `Mirror` action, and set the `targetPort` to an OVS
+internal port that threat detection engines will capture traffic from. This


Just say "Suricata"?

jianjuns · 2022-05-26T17:40:26Z

+
+### Step 3: Deploy Suricata as a DaemonSet
+
+Suricata supports many possible configuration options, we will focus on the


but we will

focus on -> just take

we will just take the basics in the cookbook? is it correct?

jianjuns · 2022-05-26T17:43:04Z

+### Step 3: Deploy Suricata as a DaemonSet
+
+Suricata supports many possible configuration options, we will focus on the
+basics in the cookbook. The YAML file for Suricata DaemonSet is included in the


We should mention are using the image from https://github.com/jasonish/docker-suricata?

jianjuns · 2022-05-26T17:44:34Z

+tail -f /var/log/suricata/fast.log
+```
+
+You can then generate malicious requests to trigger alerts. For ingress, you can


For ingress traffic?

jianjuns · 2022-06-13T16:12:39Z

+`TrafficControl` enables a CRD API for Antrea that controls and manipulates the
+transmission of Pod traffic. It allows users to mirror or redirect traffic
+originating from specific Pods or destined to specific Pods to a local network
+device or a remote destination via various tunnels. It enables a monitoring


"via a tunnel of various types"?

jianjuns · 2022-06-13T16:15:33Z

+
+TrafficControl was introduced in v1.7 as an alpha feature. A feature gate,
+`TrafficControl` must be enabled on the antrea-agent in the `antrea-config`
+ConfigMap like the following options for the feature to work:


Typo. I meant:

must be enabled on the antrea-agent in the antrea-config ConfigMap for the feature to work, like the following:

jianjuns · 2022-06-13T16:16:12Z

+
+A TrafficControl in Kubernetes is a REST object. Like all the REST objects, you
+can POST a TrafficControl definition to the API server to create a new instance.
+For example, suppose you have a set of Pods which contain a label `app=web`, the


suppose -> supposing

jianjuns · 2022-06-13T16:17:49Z

+set to the port to which the traffic will be mirrored. For the `Redirect`
+action, both `targetPort` and `returnPort` need to be specified, the latter of
+which represents the port from which the traffic will be sent back to OVS and be
+forwarded to its original destination.


This might not be true if the packet has been modified?

what kind of modification you mean? I thought the packet can either be forwarded or dropped, other kind of modification may not work as we do not make the returned packet go through the whole pipeline again.
I changed "will" to "could"

Ok. Should document the packets should not be modified?

Added "Once redirected, a packet should be either dropped or sent back to OVS without modification, otherwise it would lead to undefined behavior."

jianjuns · 2022-06-13T16:20:08Z

+selected Pods' traffic will be mirrored to the destination via an ERSPAN tunnel.
+The `remoteIP` field must be provided to specify the IP address of the
+destination. If ERSPAN session ID is desired, the `sessionID` field can be
+specified to an integer in the range 0-1,023. The `version` field must be


We do not have a default value for version?

@jianjuns We don't have a default value for version since it is not a pointer type in API, and it is int type.

jianjuns · 2022-06-13T16:22:11Z

+destination. Therefore, if an intrusion prevention system or a network firewall
+is configured to capture and forward traffic between `tap0` and `tap1`, it can
+actively scan forwarded network traffic for malicious activities and known
+attack patterns, and drop traffic determined to be malicious.


the traffic

jianjuns · 2022-06-13T16:23:46Z

+## What's next
+
+With the `TrafficControl` capability, Antrea can be used with threat detection
+engines to provide network-based IDS/IPS to Pods, and provides a reference


". We provide a reference cookbook..."?

jianjuns · 2022-06-13T16:24:44Z

+
+This guide will describe how to use Project Antrea with threat detection
+engines, in order to provide network-based intrusion detection service to your
+Pods. In this scenario, Antrea is used for the default network. For the sake of


default Pod network?

jianjuns · 2022-06-13T16:25:48Z

+* [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
+
+The [TrafficControl](../../traffic-control.md) capability was added in Antrea
+version 1.7. Therefore, an Antrea version >= 1.7.0 should be used to configure


We typically use "v1.7.0"

jianjuns · 2022-06-13T16:27:40Z

+
+### Step 2: Configure TrafficControl resource
+
+To replicate Pod traffic to threat detection engines for analysis, create a


Change "threat detection engines" to "Suricata" too?

hongliangl · 2022-06-13T23:44:27Z

+  key: 1
+```
+
+**erspan**: This specifies a remote destination for an ERSPAN tunnel. All


Should we mention the requirement of ERSPAN?

It can be used with 4.19-rc6+ kernel and iproute2-ss180813?

I don't know the exact version in which ERSPAN is added to kernel, the version you provided is a rc version, then I suppose 4.19 but I didn't find it was mentioned in release note, do you have a link for reference?
why iproute2 matters here?

Page 19 in http://vger.kernel.org/lpc_net2018_talks/erspan-linux-presentation.pdf

It's an "Linux Native Mode tunnel example" with 4.19-rc6+ kernel and iproute2-ss180813, not saying the supported is added in that version and iproute2 is required for OVS ERSPAN.

Got it, for OVS ERSPAN, iproute2 is not related, but according to this http://vger.kernel.org/lpc_net2018_talks/erspan-linux.pdf, ERSPAN was added to Linux kernel in 4.14 for IPv4 and 4.16 for IPv6., we should mention this?

but I remember you said 4.15 didn't work in your testbed? Unless we have accurate information, I wouldn't mention its version requirement given it's in alpha phase, to avoid misleading users. We can add the information anytime once we have verified it.

Yes, it didn't work in 4.15 in my testbed actually. Maybe we can add the information in the future.

hongliangl · 2022-06-13T23:51:55Z

+
+`TrafficControl` is a CRD API that manages and manipulates the transmission of
+Pod traffic. It allows users to mirror or redirect specific traffic originating
+from specific Pods or destined to specific Pods to a local network device or a


Suggested change

from specific Pods or destined to specific Pods to a local network device or a

from specific Pods or destined for specific Pods to a local network device or a

codecov-commenter · 2022-06-14T03:08:15Z

Codecov Report

Merging #3800 (e0aa3bb) into main (7817152) will decrease coverage by 27.07%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             main    #3800       +/-   ##
===========================================
- Coverage   64.48%   37.41%   -27.08%     
===========================================
  Files         291      116      -175     
  Lines       42795    16089    -26706     
===========================================
- Hits        27598     6020    -21578     
+ Misses      12974     9574     -3400     
+ Partials     2223      495     -1728

Flag	Coverage Δ
integration-tests	`37.41% <ø> (?)`
kind-e2e-tests	`?`
unit-tests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...lers/multicluster/commonarea/remote_common_area.go
...er/registry/networkpolicy/groupassociation/rest.go
...agent/flowexporter/connections/deny_connections.go
pkg/agent/util/net_linux.go
...nt/apiserver/handlers/serviceexternalip/handler.go
pkg/agent/querier/querier.go
pkg/agent/config/node_config.go
pkg/controller/networkpolicy/convert.go
pkg/agent/multicluster/mc_route_controller.go
pkg/antctl/command_message.go
... and 397 more

* Add a document to introduce the feature * Add a cookbook to describe how to use Antrea with Suricata to provide IDS to Pods Signed-off-by: Quan Tian <qtian@vmware.com>

tnqn · 2022-06-14T04:35:34Z

/skip-all

tnqn added action/release-note Indicates a PR that should be included in release notes. kind/documentation Categorizes issue or PR as related to a documentation. labels May 17, 2022

tnqn force-pushed the tc-doc branch 2 times, most recently from cf5ec58 to eb34894 Compare May 17, 2022 05:58

tnqn requested review from antoninbas, hongliangl, jianjuns, wenqiq and yfauser May 17, 2022 05:59

tnqn commented May 17, 2022

View reviewed changes

Comment thread docs/traffic-control.md

tnqn force-pushed the tc-doc branch from eb34894 to fb10fa3 Compare May 17, 2022 13:48

antoninbas reviewed May 17, 2022

View reviewed changes

tnqn added this to the Antrea v1.7 release milestone May 18, 2022

tnqn mentioned this pull request May 26, 2022

Support Pod traffic control in Antrea #3324

Closed

5 tasks

tnqn commented May 26, 2022

View reviewed changes

tnqn force-pushed the tc-doc branch from fb10fa3 to 0fedb4b Compare May 26, 2022 15:30

tnqn mentioned this pull request May 26, 2022

Consider supporting IPsec tunnel for TrafficControl #3834

Closed

jianjuns reviewed May 26, 2022

View reviewed changes

tnqn force-pushed the tc-doc branch 2 times, most recently from 5e2f72f to 613b9be Compare June 13, 2022 16:22

jianjuns reviewed Jun 13, 2022

View reviewed changes

hongliangl reviewed Jun 13, 2022

View reviewed changes

tnqn force-pushed the tc-doc branch from 613b9be to 12fce62 Compare June 14, 2022 03:00

jianjuns previously approved these changes Jun 14, 2022

View reviewed changes

Add docs for TrafficControl feature

e0aa3bb

* Add a document to introduce the feature * Add a cookbook to describe how to use Antrea with Suricata to provide IDS to Pods Signed-off-by: Quan Tian <qtian@vmware.com>

tnqn dismissed jianjuns’s stale review via e0aa3bb June 14, 2022 04:33

tnqn force-pushed the tc-doc branch from 12fce62 to e0aa3bb Compare June 14, 2022 04:33

hongliangl approved these changes Jun 14, 2022

View reviewed changes

tnqn merged commit b6fbf14 into antrea-io:main Jun 14, 2022

tnqn deleted the tc-doc branch June 14, 2022 07:45


		### TrafficControl

		`TrafficControl` enables a CRD API for Antrea that manages and manipulates the


		### Step 3: Deploy Suricata as a DaemonSet

		Suricata supports many possible configuration options, we will focus on the


		### Step 2: Configure TrafficControl resource

		To replicate Pod traffic to threat detection engines for analysis, create a

	from specific Pods or destined to specific Pods to a local network device or a
	from specific Pods or destined for specific Pods to a local network device or a

Conversation

tnqn commented May 17, 2022

Uh oh!

Uh oh!

antoninbas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tnqn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jianjuns left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment