Test Max Retries

[allenwyma]

I made an HTTP adapter:

session = requests.Session()
session.verify = False

# Try max of 3 times before erroring out
retry_adapter = requests.adapters.HTTPAdapter(max_retries=3)
session.mount('https://', retry_adapter)

I wanted to make sure it will automatically retry, but it seems it's not getting triggered right:

exception = ConnectionError('Bad connection')
with responses.RequestsMock(assert_all_requests_are_fired=False) as rsp:
   rsp.add(responses.POST, url, body=exception)
   rsp.add(responses.POST, url, body=exception)
   rsp.add(responses.POST, url, body='{}')
   session.post(url=url, json={})

It just keeps raising the exception.

[tuukkamustonen]

Looks like responses mocks the adapter.send() call in Session.send() and re-routes it into RequestsMock, that just doesn't support retrying...

[lsh-0]

just encountered this myself

[cjw296]

me too, doesn't look like anyone from Sentry is monitoring this issue tracker though...

[kathawala]

I have a max-retries branch on my fork (https://github.com/kathawala/responses/tree/max-retries) which allows this to run without raising an exception:

import responses
import requests

exception = ConnectionError('Bad connection')
url = 'https://www.example.com'
with responses.RequestsMock(assert_all_requests_are_fired=False, max_retries=3) as rsp:
   rsp.add(responses.POST, url, body=exception)
   rsp.add(responses.POST, url, body=exception)
   rsp.add(responses.POST, url, body=exception)
   rsp.add(responses.POST, url, body='{}')
   requests.post(url=url, json={})

But what's the use case for this? Just want to know before I submit a PR. It's not clear to me how this could be helpful.

[cjw296]

@kathawala - I configured a max_retries as Retry() instance and I wanted to check it would work as I expected. Would your branch help with that?

[kathawala]

@cjw296 Hmm, I'm not exactly sure what you mean. What do you mean you "configured a max_retries as Retry() instance"? Code examples would help me understand :)

[cjw296]

here's the code I'd like to test:

from requests.adapters import HTTPAdapter
from requests.packages.urllib3 import Retry
...
        retry = Retry(
            total=5,
            status=3,
            status_forcelist=[404],
            backoff_factor=0.5,
        )
        adapter = HTTPAdapter(max_retries=retry)
        self.session.mount(self.base_url+ASYNC_CHECK_BASE_URL, adapter)

[dcramer]

I'm still confused as to what this is testing. To me it seems like this is testing "does requests retry correctly", which is not what this library should care about. If that's the case, and im not misunderstanding, I'd prefer we not add the extra functionality. If there is indeed some world where it makes sense to have adapters that fail/retry within a test suite, we can consider landing it, but right now this smells like a bad approach.

[kathawala]

I have the same gut feeling as well that it seems unnecessary and likely to cause problems later on. I would also like to know what this is useful for (i.e. what is the use case under which this is desired functionality?). Can you explain the use case?

Anyways, to answer your question, the properties of retry other than total will be ignored on my branch, at the moment.

[cjw296]

@kathawala - the use case here is an async api that returns 404 until a result is ready. I don't particularly like the api, but I have no control over it.

@dcramer - max_retries, in particular when using a Retry object, has quite a lot of complicated configuration and so I'd like to have high level tests that verify I have the configuration correct.
In this case, I'd like to put in a set of mock responses for a url, returning combinations of connection errors, 404s and 200s. I'd also mock out whatever Retry is using as it's source of sleep. That way, I can check that the code retries-on-404 a max of 3 times with an incrementing delay between each request.

Responses has exactly the api that I'm looking for, it's just that when using it, all Retry is currently, understandably, ignored.

Put differently: how would you test that the Retry configuration above is behaving as required and expected?

[dcramer]

@cjw296 you could test that you are calling the Retry API correctly, and lock it in (by proving it actually works against a real HTTP request). That could be as easy as mocking the Retry object that's called and asserting the calling params. I think its important to understand that the goal of responses isn't to implement a full webserver, but to make simple HTTP tests easy (vs mocking out the responses on your own).

I'm ok if we implement retries, but it needs to be implemented per the full requests spec and pass through correctly. Given that it may end up being far too complex for this one-off request.

[cjw296]

@dcramer - I don't think calling the Retry API correctly is very useful, you end up with symmetric testing that doesn't check that the retrying you've configured behaves as you expect.

I'm glad the goal of responses isn't to to implement a full webserver, I looked at pytest-httpbin for that, and the dependencies were pretty terrifying! Not to mention that backoff delays and the like would take their actual time, rather than being mocked out.

I agree that it would ideally be per the full requests spec, and may well end up being too complex for the demand so far. Would be great to keep this issue open for others who also want this to chime in...

[foxx]

Just ran into this issue myself, it's somewhat surprising that using a mocked response results in unexpected behaviour. (e.g. expected requests functionality is effectively being removed when using a mocked response)

[k4nar]

Same here. And I think I have a valid use case:

I want to retry a query in some cases, and behave differently otherwise. For example I want to retry if I have a 503 and abort everything if the max number of retries was exceeded, or avoid the retry and fail gracefully if I have a 404.

In order to detect if I was failing due to too many retries, I was catching MaxRetryError. But requests actually encapsulate this error, so this code was never called.
I couldn't see this behavior in my tests because I wasn't able to simulate a situation where a requests was raising a MaxRetryError.