Use websockets instead of SPDY for streaming

[Kidswiss]

Summary

This PR switches the streaming protocol for application aware backups from SPDY to WebSockets. This is how upstream kubectl exec does it and should fix truncated backups (#1109). Websocket exec is supported by default since Kubernetes 1.31.

For clusters older than 1.31 a fallback option exists --insecure-allow-podexec-spdy-fallback=true. The fallback is disabled by default because of the risk of silent data corruption.

Checklist

For Code changes

• Categorize the PR by setting a good title and adding one of the labels:
  bug, enhancement, documentation, change, breaking, dependency
  as they show up in the changelog
• PR contains the label area:operator
• Commits are signed off
• Link this PR to related issues
• I have not made any changes in the charts/ directory.

[bastjan]

LGTM 🚀

[bastjan]

I was able to trigger the issue by forcing the old SPDY streaming in kubectl:

❯ for i in 1 2 3 4 5; do KUBECTL_REMOTE_COMMAND_WEBSOCKETS=false kubectl exec "${POD:?}" -- sh -c 'seq 1 200000000' | tail -n1; done 
200000000
19
200000000
200000000
19%                                                                                                                                                                             

Which leads me to believe we should add an option to disable the fallback and not risk any silent fails.

Edit:

I propose something like an environment variable K8UP_INSECURE_ALLOW_PODEXEC_SPDY_FALLBACK with a description that it can lead to incomplete backups.

[Kidswiss]

I like the idea. I'm going to add such an env var.

[Kidswiss]

@bastjan good call with this! Because I now found a RBAC issue that would otherwise cause a fallback.

Websocket exec calls are a "GET" while SPY are a "PUT".

EDIT: I'm also going against the checkbox here and also change the chart, because without the change the e2e tests will fail.

[bastjan]

@bastjan good call with this! Because I now found a RBAC issue that would otherwise cause a fallback.

RBAC my eternal "follow up fix for ..." enemy.

[Kidswiss]

@bastjan can you have a final quick look? Had to kick the tests a bit due to the k8s version bump.

[bastjan]

We should probably release this at least as a minor and add a doc page we can link to in the release.

[bastjan]

Something like

diff --git a/docs/modules/ROOT/pages/explanations/system-requirements.adoc b/docs/modules/ROOT/pages/explanations/system-requirements.adoc
index f5022cfa..8097cf06 100644
--- a/docs/modules/ROOT/pages/explanations/system-requirements.adoc
+++ b/docs/modules/ROOT/pages/explanations/system-requirements.adoc
@@ -2,7 +2,9 @@
 
 == Supported Kubernetes Versions
 
-K8up (v2 or later) officially only supports recent stable Kubernetes versions.
+K8up (v2 or later) officially only supports recent stable Kubernetes versions with support for WebSocket connections in the API server (`1.31` or later).
+
+Older Kubernetes versions may work by setting `--insecure-allow-podexec-spdy-fallback=true` in the operator, but are not officially supported and support may be removed in future releases.
 
 K8up v1 (not maintained anymore) supports legacy Kubernetes clusters such as OpenShift `3.11` (Kubernetes 1.11).
 

[Kidswiss]

Thanks for the review!

I'll do a minor k8up release after the tests are done.