POLLING_FREQUENCY not agreeing with nb_jobs in 2.0.50

[mattxtaz]

In 2.0.49 feeds are polled once every 60 minutes as per the default (60/round_robin). But in 2.0.50 feeds are only polled once every 120 minutes. Is there a race condition between the polling scheduler and the next check time? Next check counts down until it's at 0 minutes but then the scheduler finds 0 or sometimes 1 feed to update.

You can clearly see where I upgraded to the latest code after the user agent bug was fixed:.

EDIT: Actually, now I look at this, was there something wrong before then as well? This should be logged once per hour surely, but even on the older version that wasn't running once an hour.

Nov 17 14:39:51 tao miniflux[90960]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 17 16:39:51 tao miniflux[90960]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 17 21:39:51 tao miniflux[90960]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 18 00:39:51 tao miniflux[90960]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 18 01:39:51 tao miniflux[90960]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 18 03:39:51 tao miniflux[90960]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 18 07:39:51 tao miniflux[90960]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 18 08:39:51 tao miniflux[90960]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 18 09:56:04 tao miniflux[53902]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 18 18:56:04 tao miniflux[53902]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 19 00:56:04 tao miniflux[53902]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 19 09:56:04 tao miniflux[53902]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 19 18:36:57 tao miniflux[68870]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 19 19:40:09 tao miniflux[72015]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 19 20:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 19 21:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=0
Nov 19 22:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 19 23:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=0
Nov 20 00:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 20 01:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=1
Nov 20 02:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=65
Nov 20 03:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=1
Nov 20 04:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 20 05:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=0
Nov 20 06:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 20 07:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=0
Nov 20 08:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=66
Nov 20 09:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=0
Nov 20 10:45:55 tao miniflux[74958]: level=INFO msg="Created a batch of feeds" nb_jobs=66

[jaydeethree]

I think I'm also seeing this - since updating to 2.0.50 the POLLING_FREQUENCY is no longer being used consistently. It looks like feeds sometimes refresh at the POLLING_FREQUENCY, and sometimes they only refresh half as often as the POLLING_FREQUENCY.

[fguillot]

See these pull-requests for more details:

• Take RSS TTL field into consideration to schedule next check date #2147
• Refactor Batch Builder and prevent accidental and excessive refreshes from the web ui #2146
• Avoid excessive manual polling with default scheduler #2133

These changes have been made to address concerns from website publishers. More adjustments and teaks might happen in future releases.

[mattxtaz]

OK. Those PRs make sense. But the feed on my own website does not have any TTL field at all and I am not manually refreshing them either. So I would expect it to be read once an hour as the POLLING_FREQUENCY is set to 60 minutes but it's only being read once every two hours. I've worked around this by changing my POLLING_FREQUENCY to 30 minutes which restores my expected behavior of refreshing feeds hourly. Which completely negates your changes.

[jaydeethree]

My feeds also do not set a TTL, but they are being refreshed less often than the POLLING_FREQUENCY.

[shizunge]

https://github.com/miniflux/v2/pull/2133/files#diff-8ba709255a37f47b6c483ce944cec9a7b7118268365ad8cf11b7916724902ff8L125 looks like the suspect.

The old way to calculate the next check at:

f.NextCheckAt = time.Now()

The new calculation:

f.NextCheckAt = time.Now().Add(time.Minute * time.Duration(config.Opts.PollingFrequency()))

Let us look at how it works now (new calculation):
Assume there are 10 feeds. BATCH_SIZE=10 and POLLING_FREQUENCY=10, which means polling 10 feeds every 10 minutes.

1. At time 0 minute, the next check at of Feed A to Feed J are all set to 0.
2. At time 10 minute, we build the batch (see https://github.com/miniflux/v2/blob/main/internal/storage/batch.go)
3. Due to some processing latency, at time 10 minute 10 second, we set f.NextCheckAt to now + PollingFrequency which is 20 minute and 10 second.
4. At 20 minute, we build the batch again, but this time since the next check at (20 minute 10 second) is after Now (20 minute), the feed won't be added to the batch.
5. At 30 minute, the feeds are added to the batch.

The old calculation at step 3 will set the f.NextCheckAt to 10 minute 10 second, thus at step 4, 20 minute, the feeds will be added to the batch.

"NextCheckAt" should be interpreted as "Do not check before". When the scheduler is round robin, it does not really care what is the exact value of NextCheckAt, it only cares the relative order of the feeds.

That is why I don't think showing last check at and next check at to user is really helpful. (It would be helpful to developer but not user) If you agree with me, please consider to up vote #785.

Is there a work around?
You may play with BATCH_SIZE and POLLING_FREQUENCY. No matter what is POLLING_SCHEDULER, the total number feeds it can poll, is subject to these two values.
In the above example, "polling 10 feeds every 10 minutes" means "in an hour, polling 60 feeds". You can set BATCH_SIZE=1 and POLLING_FREQUENCY=1 to achieve the same thing. When the NextCheckAt is 20 minute 10 second, the feed won't be added to batch at 20 minute, but it would be added to the batch at 21 minute instead of 30 minute, slightly better.

Another improvement that can help this case is to separate feed's POLLING_FREQUENCY and when to run the scheduler.
Either

1. Run feed scheduler every minute https://github.com/miniflux/v2/blob/main/internal/cli/scheduler.go. This will increase the CPU usage, hopefully not too much.
2. Add a new setting SCHEDULER_ROUND_ROBIN_INTERVAL_MIN

[mattxtaz]

Thank you for your detailed analysis. That all makes perfect sense. Yes, it seems quite wrong that in the feeds section you see something like "Last Check: now / Next check: 59m59s" which counts down to zero minutes and then...... nothing happens, for a whole hour, until the scheduler runs again and finally adds a batch job an hour after the counter expired. I've worked around the problem by setting my polling frequency to 30 minutes so I still always get my feeds updated once an hour as expected but I don't like this. It feels wrong. I'm hoping there are going to be tweaks and changes to this by the next version, but if this is how it's going to be from now on then I may have to build my own version which always reverts this code to how it used to be in 2.0.49.

I understand the need to stop people doing stupid things like fetching the feed once a minute, but if a website operator can't deal with the feed being fetched once an hour then there's something very wrong.

[Qeynos]

On the surface, the default setting is 60 minutes to refresh, but in fact most of them refresh at 120 minutes, and 60 minutes refresh randomly occurs (such as github release.atom), which is a bit counter-intuitive.