recalculate network policies for established connections

[aojea]

Fixes: #246

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [aojea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[danwinship]

So I was imagining something more like: "when policies/namespaces/pods change, figure out what IPs are no longer allowed, and then drop conntrack entries for those IPs". But I guess k-n-p doesn't work like that at all so that wouldn't make sense.

I don't remember all the details of how k-n-p currently works, but one thing to note is that some events that require updating NetworkPolicy enforcement are guaranteed not to require deleting conntrack entries. In particular, adding or deleting a Pod or Namespace does not require deleting conntrack entries.

[aojea]

it does not work flushing the conntrack entry, the connection is not sent through the queue, subsequent packets are treated as established

[aojea]

we need to do something like this , is the only way I find out we can "break" existing connections https://lore.kernel.org/netfilter-devel/20250313231341.3040002-1-aojea@google.com/T/#m44c410cd56bdc62cc717798a0e28ee4dcaaab9ad

[aojea]

Ok, I have a better understanding now of the problem, assume TCP and UDP only by now

For UDP just clean up the conntrack entry, next one will be sent to userspace with the nfqueu
For TCP, things get complicated here as there area a lot of ifs and deleting the conntrack entry directly does not work

• if nf_conntrack_tcp_loose is enabled (default), conntrack picks up already established connections, so current logic to only evaluate the first packet we'll not work if we just flush the conntrack entry because it will be considered ESTABLISHED. Conntrack stores the tcp sequence numbers and windows to understand if a packet belongs to the connection
• Good old TCP RST attack as in https://man.archlinux.org/man/extra/dsniff/tcpkill.8.en, it will require for us to have the sequence number and window to forge a packet, storing all the information is a no go, but we can try to get it from the conntrack , no idea if is already exposed or there is some ebpf magic that can return it, but injecting packets seems very intrusive.
• Add nftables reject rules per tuple, but this does not look like scales well and also does not handle connections that are idle, so we do have to monitor the conntrack entry to understand for how long we should leave the nftables rule to reject that connection
• sock_destroy , https://lwn.net/Articles/666220/ , this seems to be exactly what we are looking for, ss -K already implements it., the challenge will be to find the socket, and it is also exposed via netlink,

https://lwn.net/Articles/1037388/
https://lpc.events/event/16/contributions/1358/attachments/1015/2101/socket-termination-lpc2022.pdf
https://docs.ebpf.io/linux/kfuncs/bpf_sock_destroy/

It seems cilium folks went through this already and have pretty good description of the problem and are adding the necessary tooling to the stack cilium/cilium#25169

[danwinship]

• sock_destroy , https://lwn.net/Articles/666220/ , this seems to be exactly what we are looking for

There's not always a socket though (eg, NodePort connection proxied to a pod on another node).

EDIT: oh, wait, that doesn't matter for NetworkPolicy purposes; you can postpone evaluation until the destination node

[aojea]

EDIT: oh, wait, that doesn't matter for NetworkPolicy purposes; you can postpone evaluation until the destination node

yeah, since we want to break existing connections, and it seems we need to be "invasive" we need to make local decisions only ... current code can optimize making decisions for the flow in both directions in the originating node ...

By the way, talked with one team mate working on the ebpf socket destroy problem internally, and he told me that there is no better solution than iterating through namespaces to find the sockets ... I need to check if we can get the information from the conntrack entry or we can store the relation Pod/IP to network namespace and use it, then the good news is that we can just use netlink, is what ss -K uses

[aojea]

@danwinship it took me more than I expected, found also some bugs on the libraries, but now is working as expected, as we can see in the logs of the e2e test added

2025-10-06T10:28:49.002179755Z stderr F I1006 10:28:49.002083       1 networkpolicy.go:302] "Pod has limited all ingress traffic" pod="dev/client" policy="prod/default-deny-ingress"
2025-10-06T10:28:49.002191927Z stderr F I1006 10:28:49.002116       1 engine.go:67] "Packet denied by ingress policy"
2025-10-06T10:28:49.002197247Z stderr F I1006 10:28:49.002129       1 controller.go:407] Connection no longer allowed by network policiespacket[0] 10.244.1.14:47600 10.244.1.13:8080 TCP
2025-10-06T10:28:49.002216743Z stderr F I1006 10:28:49.002139       1 logging.go:67] "Evaluating packet" direction="Egress" srcPod="prod/webserver" dstPod="dev/client" packet=<
2025-10-06T10:28:49.002219518Z stderr F 	[0] 10.244.1.13:8080 10.244.1.14:47600 TCP
2025-10-06T10:28:49.002221853Z stderr F  >
2025-10-06T10:28:49.002236761Z stderr F I1006 10:28:49.002153       1 logging.go:67] "Evaluating packet" direction="Ingress" srcPod="prod/webserver" dstPod="dev/client" packet=<
2025-10-06T10:28:49.002240167Z stderr F 	[0] 10.244.1.13:8080 10.244.1.14:47600 TCP
2025-10-06T10:28:49.002242962Z stderr F  >
2025-10-06T10:28:49.002245156Z stderr F I1006 10:28:49.002170       1 networkpolicy.go:190] "Ingress NetworkPolicies does not apply"
2025-10-06T10:28:49.002247581Z stderr F I1006 10:28:49.002175       1 engine.go:71] "Packet accepted by policy"
2025-10-06T10:28:49.002253492Z stderr F I1006 10:28:49.002184       1 controller.go:383] Skipping conntrack entry not involving managed IPsflowtcp	6 src=172.18.0.3 dst=172.18.0.2 sport=46466 dport=6443 packets=0 bytes=0	src=172.18.0.2 dst=172.18.0.3 sport=6443 dport=46466 packets=0 bytes=0 mark=0x0 start=1970-01-01 00:00:00 +0000 UTC stop=1970-01-01 00:00:00 +0000 UTC timeout=86397(sec)
2025-10-06T10:28:49.002256277Z stderr F I1006 10:28:49.002204       1 controller.go:443] "Destroying TCP connections" logger="firewall-enforcer" connections=1
2025-10-06T10:28:49.002298074Z stderr F I1006 10:28:49.002218       1 socketterminator.go:62] Finding route for IP: 10.244.1.14
2025-10-06T10:28:49.002407918Z stderr F I1006 10:28:49.002325       1 socketterminator.go:72] Found routes for IP 10.244.1.14: [{Ifindex: 15 Dst: 10.244.1.14/32 Src: 10.244.1.1 Gw: <nil> Flags: [] Table: 254 Realm: 0}]
2025-10-06T10:28:49.002561144Z stderr F I1006 10:28:49.002490       1 socketterminator.go:91] Found veth interface veth598fa711 with NetNsID 2
2025-10-06T10:28:49.002755444Z stderr F I1006 10:28:49.002693       1 socketterminator.go:124] Executing callback in namespace ID 2
2025-10-06T10:28:49.002760464Z stderr F I1006 10:28:49.002705       1 socketterminator.go:130] Attempting to destroy socket for packet [0] 10.244.1.14:47600 10.244.1.13:8080 TCP
2025-10-06T10:28:49.002763569Z stderr F I1006 10:28:49.002712       1 socketterminator.go:139] Attempting to destroy socket: 10.244.1.14:47600 -> 10.244.1.13:8080
2025-10-06T10:28:49.061181812Z stderr F I1006 10:28:49.061004       1 socketterminator.go:149] Successfully destroyed socket: 10.244.1.14:47600 -> 10.244.1.13:8080
2025-10-06T10:28:49.061205656Z stderr F I1006 10:28:49.061055       1 controller.go:342] "Enforcing firewall policies on existing connections" logger="firewall-enforcer" elapsed="68.161332ms"

please let me know if you want to review

[aojea]

talked offline, explained current algorithm and implementation and we are good on merging and iterating