Early Header Mutation

[ryanrolds]

Description

This PR add Early Header Manipulation to the HTTP Connection Manager settings. It allows request headers to be manipulated before they are acted on by tracing and other logic. The goal of the work is to allow customers to override Zipkin/B3 tracing headers.

API changes

• Added headers.options.gloo.solo.io.EarlyHeaderManipulation to HttpConnectionManagerSettings.

Code changes

• Updated ENVOY_GLOO_IMAGE a version with the EHM extension enabled in Envoy
• During tests, the wapper was using a 1.18 image tag. Passing VERSION as ENVOY_IMAGE_TAG in the test Make target to get it using the locally built wrapper
• Added go-httpbin and and a new upstream generator to make testing headers easier
• Added ProcessHcmNetworkFilter to the headers plugin, handled the new setting

Docs changes

• Added details about the new settings

Context

• https://soloio.slab.com/posts/early-header-mutation-design-doc-u5bw3c1m
• [Migrated] Expose "early_header_mutation" filter in the Gloo control plane solo-io/gloo#9604
• https://github.com/solo-io/envoy-gloo-ee/issues/734

Testing steps

I've test locally with Kind by setting and exercising the new logic with curl, including configuring a real Zipkin tracer.

1. Stand-up an empty Kind cluster with this PR's Gloo running
2. Apply this manifest:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: zipkin
---
apiVersion: v1
kind: Service
metadata:
  name: zipkin
  labels:
    app: zipkin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: zipkin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zipkin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zipkin
      version: v1
  template:
    metadata:
      labels:
        app: zipkin
        version: v1
    spec:
      serviceAccountName: zipkin
      containers:
      - image: openzipkin/zipkin
        imagePullPolicy: IfNotPresent
        name: zipkin
        ports:
        - containerPort: 9411
---
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: zipkin
  namespace: default
spec:
  discoveryMetadata: {}
  kube:
    serviceName: zipkin
    serviceNamespace: default
    servicePort: 9411
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80
---
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: default-httpbin-8000
  namespace: default
spec:
  discoveryMetadata: {}
  kube:
    serviceName: httpbin
    serviceNamespace: default
    servicePort: 8000
---
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: httpbin
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
      - prefix: /
      routeAction:
        single:
          upstream:
            name: default-httpbin-8000
            namespace: default
---
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  labels:
    app: gloo
  name: gateway-proxy
  namespace: gloo-system
spec:
  bindAddress: '::'
  bindPort: 8080
  httpGateway:
    options:
      httpConnectionManagerSettings:
        tracing:
          verbose: true
          requestHeadersForTags:
            - path
            - origin
          zipkinConfig:
            collectorEndpoint: /api/v2/spans
            collectorEndpointVersion: HTTP_JSON
            collectorUpstreamRef:
              name: zipkin
              namespace: default
        earlyHeaderManipulation: 
          headersToAdd:
          - header:
              key: X-B3-TRACEID
              value: "%REQ(x-override-traceid)%"
          - header:
              key: X-B3-SPANID
              value: "%REQ(x-override-spanid)%"
  proxyNames:
  - gateway-proxy
  ssl: false
  useProxyProto: false
---
apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: gloo
    meta.helm.sh/release-namespace: gloo-system
  labels:
    app: gloo
    app.kubernetes.io/managed-by: Helm
    gateway-proxy-id: gateway-proxy
    gloo: gateway-proxy
  name: gateway-proxy-envoy-config
  namespace: gloo-system
data:
  envoy.yaml: |
    layered_runtime:
      layers:
      - name: static_layer
        static_layer:
          overload:
            global_downstream_max_connections: 250000
          upstream:
            healthy_panic_threshold:
              value: 0
      - name: admin_layer
        admin_layer: {}
    node:
      cluster: gateway
      id: "{{.PodName}}.{{.PodNamespace}}"
      metadata:
        # Specifies the proxy's in-memory xds cache key (see projects/gloo/pkg/xds/envoy.go)
        # This value needs to match discoveryNamespace (or "writeNamespace") in the settings template
        role: gloo-system~gateway-proxy
    static_resources:
      listeners:
        - name: prometheus_listener
          address:
            socket_address:
              address: 0.0.0.0
              port_value: 8081
          filter_chains:
            - filters:
                - name: envoy.filters.network.http_connection_manager
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                    codec_type: AUTO
                    stat_prefix: prometheus
                    route_config:
                      name: prometheus_route
                      virtual_hosts:
                        - name: prometheus_host
                          domains:
                            - "*"
                          routes:
                            - match:
                                path: "/ready"
                                headers:
                                - name: ":method"
                                  exact_match: GET
                              route:
                                cluster: admin_port_cluster
                            - match:
                                prefix: "/metrics"
                                headers:
                                - name: ":method"
                                  exact_match: GET
                              route:
                                prefix_rewrite: /stats/prometheus
                                cluster: admin_port_cluster
                    http_filters:
                      - name: envoy.filters.http.router
                        typed_config:
                          "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
      
      clusters:
      - name: gloo.gloo-system.svc.cluster.local:9977
        alt_stat_name: xds_cluster
        connect_timeout: 5.000s
        load_assignment:
          cluster_name: gloo.gloo-system.svc.cluster.local:9977
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: gloo.gloo-system.svc.cluster.local
                    port_value: 9977
        http2_protocol_options: {}
        upstream_connection_options:
          tcp_keepalive:
            keepalive_time: 60
        type: STRICT_DNS
        respect_dns_ttl: true
      - name: rest_xds_cluster
        alt_stat_name: rest_xds_cluster
        connect_timeout: 5.000s
        load_assignment:
          cluster_name: rest_xds_cluster
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: gloo.gloo-system.svc.cluster.local
                    port_value: 9976
        upstream_connection_options:
          tcp_keepalive:
            keepalive_time: 60
        type: STRICT_DNS
        respect_dns_ttl: true
      - name: wasm-cache
        connect_timeout: 5.000s
        load_assignment:
          cluster_name: wasm-cache
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: gloo.gloo-system.svc.cluster.local
                    port_value: 9979
        upstream_connection_options:
          tcp_keepalive:
            keepalive_time: 60
        type: STRICT_DNS
        respect_dns_ttl: true
      - name: admin_port_cluster
        connect_timeout: 5.000s
        type: STATIC
        lb_policy: ROUND_ROBIN
        load_assignment:
          cluster_name: admin_port_cluster
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: 127.0.0.1
                    port_value: 19000
      - name: zipkin
        connect_timeout: 1s
        type: STRICT_DNS
        load_assignment:
          cluster_name: zipkin
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: zipkin
                    port_value: 9411
    dynamic_resources:
      ads_config:
        transport_api_version: V3
        api_type: GRPC
        rate_limit_settings: {}
        grpc_services:
        - envoy_grpc: {cluster_name: gloo.gloo-system.svc.cluster.local:9977}
      cds_config:
        resource_api_version: V3
        ads: {}
      lds_config:
        resource_api_version: V3
        ads: {}
    admin:
      access_log_path: /dev/null
      address:
        socket_address:
          address: 127.0.0.1
          port_value: 19000

3. Port forward to the proxy
   a. kubectl -n gloo-system port-forward services/gateway-proxy 8080:80
4. Confirm behavior with curl.
   a. curl -H "x-override-traceid: asdfasdf" -H "x-override-spanid: zxcvzxcv" http://localhost:8080/get
   b. curl http://localhost:8080/get

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

BOT NOTES:
resolves solo-io#9604

[solo-changelog-bot]

Issues linked to changelog:
solo-io#9604

[nfuden]

/kick for build bot showing

[github-actions]

Visit the preview URL for this PR (updated for commit 29e2475):

https://gloo-edge--pr10262-rolds-early-header-m-ajfjhd9z.web.app

_{(expires Thu, 14 Nov 2024 20:51:21 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}

[davidjumani]

Can we add another test that verifies the behaviour of appendAction ?

[davidjumani]

Can we add a comment that mentions the need for this specific test and how it solves an actual use case

[nfuden]

🚢

[davidjumani]

/kick buildbot stuck