Doc: Add section that compares the various MessageStore Implementations (pro+cons) [INT-2609]

[spring-operator]

Magne Rasmussen opened INT-2609 and commented

We need to expand the Message Store chapter in the reference documentation. While we have sections providing details regarding the various available MessageStore implementation, we should better describe the various pros-and-cons between the various available MessageStore implementation.

Original Description:

As described on the forum, a JDBC message store effectively serializes messages. It also blocks the sender. The attached project (same as on forum) is a trivial demonstration of this effect.

---

Affects: 2.2 M2

Reference URL: http://forum.springsource.org/showthread.php?126666-JDBC-message-store-serializes-messages

Attachments:

• spring-int-test.zip (10.19 kB)

Issue Links:

• Refactor JdbcMessageStore - Provide separate implementations for MessageStore and MessageGroupStore [INT-2672] #6647 Refactor JdbcMessageStore - Provide separate implementations for MessageStore and MessageGroupStore

1 votes, 4 watchers

[spring-operator]

Gunnar Hillert commented

Moving to 2.2 M3 Sprint 4 as the scope changed.

[spring-operator]

Magne Rasmussen commented

Does this mean that it won't be fixed, then? The way it works now, I can hardly see any use at all for the JDBC message store!

[spring-operator]

Gunnar Hillert commented

Hi Magne,

As far as I see it, your problem stems from the fact that your Transactions are open (uncommitted) for such a long time that any concurrent transactions are locked out and eventually time-out.

In your example, you are polling the queue, which is backed by the JDBC MessageStore every 100ms, polling for exactly one 1 message at a time. This will also mark the begin of your transaction.

You will then send the message to the "outputChannel", which is a DirectChannel and thus operates on the same Thread. Therefore, if your ServiceActivator is executed, the Transaction is still open. Furthermore, in your ServiceActivator you are doing a Thread.sleep(2000L). Only thereafter is your transaction committed.

With that constellation, your transaction timeout of 5sec is a bit tight and your concurrent TX (slightly delayed) will eventually timeout. Maybe, if you have very long running processes, you may want to consider a compensating transaction model.

Ultimately, I don't necessarily see an issue with our JDBC Message Store implementation. I hope this is explains my view point sufficiently. I am certainly open for any other suggestions.

Cheers,

Gunnar

[spring-operator]

Magne Rasmussen commented

Hi Gunnar,

The example is an exaggeration to demonstrate the point that access to the JDBC message store is serialized, so that tasks put into the store can only be processed one by one. Even if the transactions are short-lived, or you pull several messages off the store each time, it doesn't change the fact that only one thread at a time is allowed access to the store. What is the point of connecting a thread pool polling the queue if only one thread can actually do anything worthwhile at any given moment? (Ignoring the fact that several threads may actually be doing the same work. This is in fact quite likely, depending on your transaction isolation level.)

I hope you understand that I'm not trying to bash anyone here. I really like how easy it is to use SI, but it was my hope that we were able to avoid installing, configuring, and monitoring a whole JMS stack just to reliably (persisted, only-once) do some asynchronous tasks in parallel. If you think that I can achieve this in any other way with SI, please point me in the right direction.

Regards,

Magne

[spring-operator]

Gunnar Hillert commented

Hi Magne,

I finally found the time to investigate the issue further. I have created a separate Jira story for that #6647 as in wraps several sub-issues. The main issue seems to be, though, that we update the message group table as part of the Message retrieval operation. Therefore, multiple threads try to access the same record which causes time-outs.

For channel-backing JDBC message stores, we may not need the group table or at least we could potentially work around it. This would be a more involved task as we also would need to separate MessageStore and GroupMessageStore.

But I think this should be possible but probably won't be able to do this for the Spring Integration 2.2 release.

Would you be able to possible relax your transactional requirements (E.g. use rather compensational transactions in case erros occur in components further downstream)?

Cheers,

Gunnar

[spring-operator]

Gunnar Hillert commented

For the documentation task, this blog post which is referenced widely, may provide some information documenting the various message stores:

http://www.engineyard.com/blog/2011/5-subtle-ways-youre-using-mysql-as-a-queue-and-why-itll-bite-you/

[spring-operator]

Artem Bilan commented

We already mention in the doc that JDBC for queue is an anti-pattern: https://docs.spring.io/spring-integration/docs/current/reference/html/jdbc.html#custom-message-insertion.

Not sure what else we can say about other database types, but I find that as out scope of this project to deep dive into specific DB details.