Set `Forwarded` / `X-Forwarded-For` when proxying HTTP traffic

[paultiplady]

Feature Request

When the linkerd-proxy forwards an HTTP request, append a Forwarded or X-Forwarded-For header entry so that upstreams can tell where the forwarded request came from.

What problem are you trying to solve?

It's common for HTTP services to log or consider the source-IP of incoming requests. For example, for rate-limiting or auditing. Usually external Load Balancers will set the X-Forwarded-For header, and Linkerd doesn't interfere with that. However for intra-cluster traffic (east/west), there is currently no good way for meshed services to learn the real source-IP of HTTP traffic, since the Linkerd proxy rewrites the source IP to 127.0.0.1.

How should the problem be solved?

When the linkerd-proxy forwards an HTTP request, inject a Forwarded or X-Forwarded-For header.

This typically involves appending the IP to the list, if it already exists, like:

X-Forwarded-For: 1.1.1.1, 2.2.2.2

Any alternatives you've considered?

It might be desirable to pick one of those headers, or to support both. XFF is universally supported. Forwarded is not (e.g. it's not supported in Django: https://code.djangoproject.com/ticket/30729).

How would users interact with this feature?

Presumably this would be injected annotation config, something like

annotations:
  config.linkerd.io/http-proxy-header: "x-forwarded-for" | "forwarded"

[paultiplady]

Tangentially related: #3403

[grampelberg]

This has enough moving parts, we'll want to get an RFC going for it before implementation. I'm particularly interested in the security implications, as anyone can add a XFF header themselves at any point.

[joewilliams]

This would be a really nice feature and is fairly standard feature in most proxy set ups. We are bumping into related x-forwarded-for issues over in this discussion as well #5683

[joewilliams]

Thinking about this a bit more over the weekend, I am wondering if it makes sense to have a couple configuration options. One for the behavior ignore, append vs overwrite and two for the name of the header. I could see a couple scenarios where this could come in handy. For instance, lets say I want x-forwarded-for to be untouched but want l5d-forwarded-for to be written instead. This would ignore x-forwarded-for and overwrite (or append) l5d-forwarded-for with the value linkerd would have used for x-forwarded-for.

annotations:
  config.linkerd.io/http-proxy-header: "l5d-forwarded-for"
  config.linkerd.io/http-proxy-header-behavior: "overwrite"

For the OP example:

annotations:
  config.linkerd.io/http-proxy-header: "x-forwarded-for"
  config.linkerd.io/http-proxy-header-behavior: "append"

Never touch x-forwarded-for:

annotations:
  config.linkerd.io/http-proxy-header-behavior: "ignore"

I think this would be some what equivalent to how Envoy deals with use_remote_address but a bit less confusing on how the configuration combinations actually work. 😅

[kleimkuhler]

@joewilliams Sorry there has been no response on this. We'll make sure to discuss and consider options to solve this issue and your linked discussion.

For your l5d-forwarded-for example, what use case do you have in mind for when both that and x-forwarded-for are set? Do you see a scenario when they would be different? It seems worth at least adding the x-forwarded-for header without configuration, but your suggestion is definitely a lot more to consider given the possible headers and valid values.

[jsoref]

x-forwarded-for would tell me about the external proxy, and l5d-forwarded-for would tell me where in the cluster the request came from.

I have an nginx backend pod that offers debugging information for clients, and it'd be nice for me to not mention the linkerd internal details.

[sgrzemski]

Any progress on that?

[linuxeryt]

How's it going