[FR]: Sync speed: BazQux, Reedah, and rssgurard

[umbellipher]

Brief description of the feature request.

Windows 10, fast internet connection.

I emailed the dev of BazQux to ask why syncing between it and rssguard is slower than syncing between Reedah and rssguard. Here is his response:

...from what I see in logs RSS Guard uses horribly inefficient syncing algorithm. It just fetches all content of all feeds every time (and makes login call before every feed fetch).

Maybe Reedah does not return many items in per feed fetch and works faster due to this (it's faster to download and process 20350 items than 500350 items).

Try to limit number of fetched items per feed to 100 or maybe 50. It won't help much but syncing should become a bit faster.

And it worth to ask RSS Guard developer to implement better syncing algorithm:
https://github.com/bazqux/bazqux-api#the-right-way-to-sync

I don't want to limit fetched items as he suggests. I wondered if @martinrotter has any comments, or whether this is going to be a matter for version 4, when it arrives?

[martinrotter]

Hello @umbellipher. There were some synchronization speed problems reported in #382 and I fixed them.

Please, now download latest development build and test if the problems still exists on that build. Then report back here with your findings.

[martinrotter]

@umbellipher any updates?

[umbellipher]

Yes, thanks. I was using the latest dev build anyway. I have stripped accounts out of rssguard and subscribed, separately, opml files from BazQux [B] and Reedah [R] in turn, so that rssguard was only subscribed ever to one of them.

B is still appreciably slower that R.

I really don't want to limit numbers, for fear of missing items. I would much prefer to use B to R, but the slowness with B in rssguard is noticeable.

Why should there be a speed difference between the two? Can anything be done, while we wait eagerly for v4?

[martinrotter]

I will do comparative test, but the only probably/possible cause is that Bazqux server is slower and/or network connection to Bazqux server is slower than Reedah. In other words. Bazqux could have higher PING.

[umbellipher]

I fear that may be so. I'd be interested to see the results of your test.

[martinrotter]

@umbellipher OK, I imported the same OPML into both services and added both into latest dev. version of RSS Guard.

1. Feed list with icons is fetched much faster from Reedah than from Bazqux.
2. I tested both services with 100 messages-per-feed limit. Bazqux downloaded 3700 messages in 100 seconds. Reedah 960 messages in 20 seconds. You may usk why each service returns different count of messages. It is because some feeds were correctly recognized and unrecognized by some of those services. Anyway it took rougly 0.02 seconds for each service to process single message, fairly similar number for both of services. So it seems that they are similarly fast.
3. One thing that is hugely different is that Bazqux somehow caches/remembers literally almost all articles EVER produced by the feed. So for example it returns 500 messages (if not limited by rssguard) for the feed while the live feed file in fact only contains 25 messages. Bazqux simply remembers messages for the feed many weeks/months from the past. Reedah, on the other hand, does not remember/store old messages from the feed., because I added the same OPML and Reedah does not even display any messages.

So it seems both services provide similar speed for me, but Bazqux remembers much much more messages than Reedah and also returns much more messages, that's why it is slower because the amount of transferred data is much bigger and Bazqux server must process/prepare more data for API request.

There is really no possible fix for this, except for #339. I can employ algorithm they propose. It would speed up drastically all subsequent RSS Guard message fetching (except for very first fetching after you add it to RSS Guard). However, the algorithm has some serious flaws:

1. It only downloads IDs of UNREAD and STARRED items. It only downloads messages newly assigned to some tags.
2. It recommends to remove READ, UNTAGGED, NON-important messages from RSS Guard's database, WTF?
3. It downloads contents only for those articles which are not present in RSS Guard's DB, which means that if content of the message is changed remotely (for example if Bazqux downloads newer version of the message), then the message is not re-downloaded.

In other words, the algorithm tries to minimize the amount of data transferred over the internet. I could write similar advanced algorithm which would try to inteligently download only really really needed message contents. The behavior would have to be opt-in as it would fit only some users clearly.

There are quite big options on how the algorithm should behave. It could download list of unread messages, list of starred messages and list of message/tag assignment and automatically mark all other local messages as read/unstarred, untagged. Also, it could deploy ot parameter to download only messages newer than certain date/time (and remember the value; initial fetching would download all messages regardless of ot). There are literally like 10 options on how to fine tune the potential algorithm.

[martinrotter]

@umbellipher OK, I made very basic test with "algorithm" using ot= option. Initial fetching of the feeds from huge OPML file took 600 seconds. All subsequent fetches took 40 seconds. Certainly interesting. So if you feel like #339 would make sense to really really implement, let me know and I will try to make it working for Greader plugin as PoC.

[umbellipher]

That's really helpful, thanks. Yes, #339 for Greader plugin would be wonderful, and I am most grateful.

[vshabanov]

However, the algorithm has some serious flaws:

1. It only downloads IDs of UNREAD and STARRED items. It only downloads messages newly assigned to some tags.

You actually can download READ IDs too. You can get all IDs (without &xt=user/-/state/com.google/read) and then UNREAD. So you could sync all items and mark read/unread based on UNREAD IDs.

2. It recommends to remove READ, UNTAGGED, NON-important messages from RSS Guard's database, WTF?

BazQux does not keep more than 500 items per feed. Plus it has filtering which can hide any article at any time. So it worth to remove articles that are no longer on the server. But you could leave READ articles if you want (but still remove those that are no longer on the server)

3. It downloads contents only for those articles which are not present in RSS Guard's DB, which means that if content of the message is changed remotely (for example if Bazqux downloads newer version of the message), then the message is not re-downloaded.

There is currently no flag that article was updated and no option in API to fetch IDs of updated articles. So you need to redownload all articles each sync and it will slowdown things a lot.

There are quite big options on how the algorithm should behave. It could download list of unread messages, list of starred messages and list of message/tag assignment and automatically mark all other local messages as read/unstarred, untagged. Also, it could deploy ot parameter to download only messages newer than certain date/time (and remember the value; initial fetching would download all messages regardless of ot). There are literally like 10 options on how to fine tune the potential algorithm.

There are many options, but please, don't use ot. There are lots of issues with it.

[umbellipher]

I've informed Vladimir, the BazQux dev, that you're going to do this, and pointed him to this page.

NB, he commented:

I've added a comment to clarify some things.

presumably to https://github.com/bazqux/bazqux-api#the-right-way-to-sync

[umbellipher]

Ah, no, I see that the comments are to this page!

[martinrotter]

@vshabanov Thanks for some input, I will address the issue for the 4.0.0 milestone definitely. :)