Activator retries its own requests.

[markusthoemmes]

Expected Behavior

The activator should retry a request for a limited amount of time until the revision is really ready.

Actual Behavior

The activator sends a retry request, which will arrive at itself if the revision is not ready. This results also in a retry loop, but a very unexpected one. For instance, the retry headers sent to the client look like this:

< x-activator-num-retries: 1
< x-activator-num-retries: 1
< x-activator-num-retries: 1
< x-activator-num-retries: 1
< x-activator-num-retries: 1
< x-activator-num-retries: 1
< x-activator-num-retries: 1
< x-activator-num-retries: 1
< x-activator-num-retries: 1
< x-activator-num-retries: 1
< x-activator-num-retries: 1
< x-activator-num-retries: 1
< x-activator-num-retries: 1

rather than:

< x-activator-num-retries: 10

Steps to Reproduce the Problem

1. Deploy the sample app
2. Let it scale to 0
3. Try to scale from 0 with curl -v

[dprotaso]

This is most likely due to in cluster traffic being routed through the istio ingress gateway - the PR that made the change #1789

Istio is routing it back to the activator

Timely discussion that happened yesterday that provided me this context: https://knative.slack.com/archives/CA9RHBGJX/p1534783270000100

/cc @tcnghia

[markusthoemmes]

@dprotaso thanks for for the context. My current implementation proposal would be to add the retry header in the activator even on the retrying requests and then filter them out, returning a 503 from the activator itself. Does that make sense?

[tcnghia]

That is strange, I believe the activator sends request to the Revision's Service, and there is no routing rule that routes such to the activator again.

i.e. the two ways a request can hit activator are:
(1) it arrives through an external request talking to the knative-ingressgateway ELB
(2) it arrives through an internal request talking to route.ns.svc.cluster.local

I believe activator sends requests to {revision}.ns.svc.cluster.local, which doesn't go through the activator again.

Let me double check code to see if this has changed recently.

[tcnghia]

https://github.com/knative/serving/blob/master/pkg/activator/revision.go#L128

There is still a possibility that the sidecar proxy does something "clever" based on the Host header and causing us to be re-routed to activator again. In that case may be we should be clearing the Host header to avoid the loop.

[markusthoemmes]

@tcnghia ahh, yeah that makes sense. Clearing the Host header was the only thing that popped to my mind when looking through recent changes. I'll try to see if that clears things up.

[markusthoemmes]

Actually, maybe we should indeed route things through istio again? #1846 proposes something similar and maybe that proofs to be more resilient in terms of network programming? If we bypass istio, the activator can not actually be certain that network programming has completed 100%. It can though, if we go the full-route, even on the retried paths.