Better handling with inactive sources

[szyb]

Short description
Sometimes sources not working for a while and then comeback alive. During the downtime, when our cache is expired, each request from WWW and DSM are sent directly to the source server, so it can flood the source server with our requests. This can lead to block website's IP(s) as it acts as potential DDOS attack.

Motivation

• Avoid request flood the source server when it is down
• Decrease size of logs

[szyb]

I'm thinking of auto disable feature. It could disable source for a while and check periodically whether enable it or not.

[szyb]

Now I'm thinking about counting number of failures and decide when to query server one more time.
i.e, (just a sample):
when failureCount < 5 then query the server (no wait)
when >=5 and < 10 then schedule next query to be executed in an hour
when >=10 and < 20 then schedule next query to be executed in 4 hours
when >= 20 and < 30 then schedule next query to be executed in 12 hours
when >= 30 then schedule next query to be executed in 24 hours
In this sample after ~7 days we will send request only once per 24 hours.

Need to also clear number of failures when valid response will be received. FailureCount should be stored per source (no matter what version or model is used) and maybe kept in memory (so the restart will clear the state)