Hawtio
Replies: 7 comments 13 replies
|
Version Information 
|
|
@mlalam What does "disable the jolokia updateRate configuration" mean? If Jolokia doesn't send an update request, it will never catch any changes in your application at the server side. You can try changing the updateRate to The more important question is, what would you like to achieve by doing so? Also, the latest Hawtio version is 5.2.1. Try the latest. |
|
The architecture is quite simple actually. |
|
@tadayosi / @grgrzybek , We have hundreds of queues and time it takes to load / refresh hawtio. The jmx UI takes several minutes in hawtio and within few seconds after successful load (due to default 5000 (5 seconds)), hawtio UI triggers another refresh and it keeps happening for each user's browser session sending lots of requests to the server making the server's CPU spike and toasting the server altogether. We have hundreds of users in our enterprise and the magnitude of requests going to the server is huge as well. That's the reason why we want to disable the auto-refresh and let the user decide when to refresh the browser data. The default value for the updateRate in user preference (Jolokia) is 5 seconds. How can I disable or increase this value globally for all users? |
|
@mlalam with the utmost respect for your scenario and the operational constraints you described, I’d like to add a perspective that often gets overlooked when discussing JMX/Jolokia/Hawtio. The architecture of any message broker (ActiveMQ, RabbitMQ, Artemis or others) guarantees exactly two things: 2) if the consumer ACKs a message, the broker considers it delivered. Everything that happens before and after these two points is the responsibility of the application components (producers and consumers), not the broker. Speaking from direct experience as a sysadmin: when we first deployed our system, we received reports of “missing messages”. In reality, consumers were receiving them but not returning the expected response, or silently failing during processing. The broker had done exactly what it was supposed to do: receive the message and hand it off. And from the developer side, the fix was straightforward: I corrected the software. The broker wasn’t losing anything. The application simply wasn’t handling errors, retries or temporary connectivity issues properly. It is perfectly normal for a consumer to fail, crash, or be unable to process a message in those cases you re‑queue or re‑send the missing messages. Let’s just say: unexpected things happen when you operate without safeguards. Coming back to JMX: when dozens or hundreds of users open the console at the same time, each browser becomes an independent Jolokia client polling the broker. With many queues and many MBeans, this creates a load that no broker can sustain for long not because it “doesn’t scale”, but because JMX was never designed as a multi‑user monitoring layer. In practice, the broker is not slowing down “on its own”: the usage pattern is generating a volume of JMX requests that directly impacts the JVM. And I want to say this very clearly and respectfully: you are doing the right thing by providing a proper administrative console to the people who actually need to troubleshoot the system. Hawtio is the right tool for that. The problem is that many of the people using it do not seem to understand that this usage pattern is actually making the situation worse. When something appears slow or stuck, the correct place to look is almost always the application logs of the producers and consumers not the broker. The broker cannot show real‑time flows, business logic, retries, exceptions, or workflow failures. It simply doesn’t have that information. A JMX console is an administrative tool, not an observability platform. When it is used as a shared dashboard by many teams, the inevitable result is overload on the JMX/Jolokia layer. Increasing the global refresh interval may help reduce the baseline load, but it won’t address the underlying issue: the broker is not a persistence layer, not a coherence layer, and not the right place to perform distributed debugging. And if I may add this with complete respect: the current usage pattern looks very similar to a situation where the broker is not trusted to do what it is designed to do. If performance issues are appearing today, it is very likely that they are being self‑induced by the number of concurrent accesses to the JMX console, not by the broker itself. And let me be very specific: there is an architectural aspect of JMX that is often misunderstood. And just like in photography, when you take a snapshot, the subjects must “freeze” for the picture to be accurate. The JVM pauses the relevant subsystems long enough to collect the data requested by the JMX client. If you take one picture occasionally, this is perfectly fine. If you take hundreds of pictures per second, from hundreds of clients, the system spends more time “freezing for the camera” than actually doing its work. In other words: if you take too many snapshots, nothing can move anymore. (GC is similar sometimes) This is exactly what happens when many users open Hawtio at the same time. Each browser becomes a JMX photographer asking the JVM to “say cheese” every few seconds. The result is not a performance issue in the broker it is simply the JVM doing what JMX was designed to do. |
|
What is your recommendation to get observability on Broker performance at each queue/producer/consumer/Enqueue/Dequeue/in-flight messages level if we cannot use Hawtio in multi-user interface mode? Also, if each team wants to manage (aka administrator) their queues on their own, how can we support such model? |
|
Thanks for different perspectives! For the record - the biggest and most obvious bottleneck we had was the https://github.com/jolokia/jolokia-integration was created to "optimize" these responses to cache similar MBean metadata. And it helped a lot when we had such number of Artemis Queues/Addresses MBeans. But we don't have (yet) such optimization for classic ActiveMQ. And a scenario where thousands of users hit Jolokia agent very often is a different thing than optimization of the |
|
@mlalam glad to help if this point us in the right direction. I think there is an important distinction to clarify here, because “observability”, “administration”, and “troubleshooting” are three very different concerns, and Hawtio/JMX only covers one of them. Observability means understanding how the component is performing over time: throughput, enqueue/dequeue rates, in‑flight messages, consumer lag, JVM health, etc. This is normally achieved through a collector that periodically gathers metrics and exposes them as time‑series for dashboards. For example, Prometheus, Micrometer, OpenTelemetry, or even OpenNMS (which has excellent JMX support) feeding into Grafana. With this approach, every team can see the broker’s performance without ever touching the broker itself. They get dashboards, trends, alerts, and historical data and that's exactly what observability is supposed to provide. Hawtio, on the other hand, is not an observability tool. It is an administrative console built on top of JMX. JMX provides synchronous snapshots of the JVM state, and it was never designed for multi‑user, real‑time monitoring. When many users open Hawtio at the same time, each browser becomes a JMX client polling the broker every few seconds. This inevitably overloads the JVM. For this reason, access to Hawtio should be limited to the small number of people who actually need to perform administrative tasks, and only when necessary. For example, creating and deleting persistent queues is an administrative action that should not be performed casually. If a queue is meant to be temporary, a non‑persistent queue is more appropriate, because it disappears automatically when empty. Regarding “each team managing its own queues”: this is unusual in a production environment. Queue administration is typically centralized, just like Active Directory administration. In production, there is normally a broker administrator who handles requests (tickets) for queue creation, deletion, or configuration changes. The number of such tickets is actually a very useful metric: if it grows, it usually indicates issues in the design or reliability of the producers/consumers, not in the broker. In staging or test environments, giving Hawtio access to everyone is fine, performance is not critical, and teams can experiment freely. In fact, I would even recommend giving each team its own isolated broker instance in those environments. In practice, teams often create and delete queues that interfere with each other, and managing strict separation through RBAC in Artemis or ActiveMQ is extremely complex and error‑prone. RBAC‑driven multi‑tenancy is far beyond the intended scope of a test environment, and isolating brokers is usually simpler, safer, and more productive for everyone. But in production, the broker is shared infrastructure. Teams should not “administer” it; they should administer their own software. And for troubleshooting message flow, the correct place to look is always the logs of the producers and consumers. The broker cannot show real‑time flows or business‑level failures because it simply does not have that information. So the recommended model is: This separation keeps the broker healthy, avoids JMX overload, and gives every team exactly the visibility they need without turning Hawtio into something it was never designed to be. N.B.: I’m not here to recommend or promote any specific product beyond the scope of this issue. However, for the sake of transparency, I can say that in my own environments I use Grafana, Prometheus and OpenNMS for observability, and they work extremely well together. |
|
@gzurowski this is exactly the kind of optimization you do on a solid product when you want to push it harder: focused, targeted and surgical. |
|
OK, this gets interesting ;) If you use the classic ActiveMQ, we may have to work on this. The thing is that with current optimizations provided with Hawtio and https://github.com/jolokia/jolokia-integration project, we had JMX UI with tens of thousands of queues loading within seconds. Please provide these details, as we'd like to make the experience better for all users. For the refresh rate, I'm afraid for now users have to change it (the config is stored in browser's local storage)... |
|
We are using classic ActiveMQ. Is there a way to load that value from hawtio init param via web.xml or something for global setting experience instead of defaulting it to 5 seconds? |
|
For ActiveMQ classic, you can use the last version of Hawtio 4.x: Then, you can customise the update rate at the Preferences > Connect. |
|
Thanks @tadayosi . We are looking for a way to set the updateRate value globally for all users to higher value instead of asking each user to set it up individually via Preferences. |
|
Currently there is no way to customise the default values globally other than assembling your own custom console and invoking the preference services at the hawtio ts script's bootstrap to override them. It's not a fully-tested approach though. Providing a way to override the default config parameters in the application itself is a valid feature request. We can work on it for a next minor release. But be aware that we no longer release Hawtio 4.x. You'll have to wait for the community ActiveMQ classic plugin to support Hawtio 5.x. |
|
I've created: to track potential solutions. |
Uh oh!
There was an error while loading. Please reload this page.
Is there a way to disable the jolokia updateRate configuration from server side, not through UI (Preferences->Jolokia) ?
All reactions