Spurious aten events

[fnchooft]

Describe the bug

Hello.

We are experiencing something with aten, what we are calling "spurious events", and we would like to know if you guys could give us some insigths.

We have a cluster of 8 cpus (seperate boards) and all of them have aten started. We noticed that some boards receive up/down events, while others do not.

For instance, there are 3 boards, board1, board3 and board8. Sometimes,board8 receives board1 down, but board3 does not. And board1 was never down.

So, we start debugging and find out that it wasn´t a network issue.
Then we moved to cpu usage, and figure that we had a sub-system which was taking a lot of cpu.

After disabling these sub-systems, we still had "spurious events". Less, but still. So we kept debugging, and notice some ntp issues on our system.

We are at this stage right now. We tried to generate those events moving time forward. The events happens but not in a deterministic way.

Could NTP be a source of the problem?

We are using time correction with multi_time_warp.

The vm.args parameters look like this:

## https://www.erlang.org/doc/apps/erts/time_correction#multi-time-warp-mode
+C multi_time_warp
+c true

In a first attempt to fix the issue, we changed aten_detect:ts() to use monotonic_time:

Link: https://github.com/rabbitmq/aten/blob/main/src/aten_detect.erl#L73

-erlang:system_time(microsecond).
+erlang:monotonic_time(microsecond).

instead of system_time, are we safe from time warping of the monotonic clock?
Should we play with aten parameters instead?
What is the best approach here?

Kind regards,

Reproduction steps

1. Enable time-warp as described in time_correction-docs from OTP/Erlang
2. Setup cluster with NTP-clients
3. Test the setup while adapting/distorting time on the boards/nodes

Expected behavior

Aten should not generate "spurious event" while nodes are reachable.

Additional context

No response

[kjnilsson]

Are each of the erlang nodes single cpu systems? Then there could be an issue with parallelism as other work may stop the aten sink process from running in a timely manner, combined with warping times could well result in false positives.

Aten will never guarantee that there won't be false positives btw, you need to supplement the aten notification with additional probes/checks before you take any form of drastic action.

Did it help at all to change to monotonic time? tbh aten probably should use monotonic time instead of system time.

[fnchooft]

Hi - thanks for promoting this to a discussion.

The CPU is a 4-core intel atom - cpu_topology looks like this:

erlang:system_info(cpu_topology).
[
  {processor,[
    {core,{logical,0}},
    {core,{logical,1}},
    {core,{logical,2}},
    {core,{logical,3}}
  ]}
]

The first thing we did was as you indicated. We added an additional check to make sure it is not a false-positive.

• The test is super simple: net_adm:ping(Node) - if pong we discard the event.

The second thing we did was the code-change - and the system seems "calmer" right now.

We have the system running under tests for about 3 days now - without any "spurious"-events.
However as soon as we finish the tests - we can provide feedback here.

Kind regards,

[kjnilsson]

by code change do you mean you switched to monotonic time?

There are a few configuration parameters you could still try to tweak how the detector works:

{env,[{poll_interval,1000},
          {scaling_factor,1.5},
          {heartbeat_interval,100},
          {detection_threshold,0.99}]}]}.

The poll_interval sets how often the detector will evaluate the heartbeat data - you can change this to check more often.

scaling_factor can be increased to reduce false positives.

detection_threshold is already set very high so probably can't be increased much to reduce false positives.

[fnchooft]

Hello, just a quick feedback. The system has not shown any instabilities.

We did do the code-change as indicated on Line 73 in the aten_detect-module.

We are considering to patch locally - but the aten-team were to consider a fix - that would be fine as well.

Thanks again.

Kind regards,

Fabian

[kjnilsson]

which version of erlang OTP are you using?

[fnchooft]

Erlang/OTP 24 [erts-12.3.2.12]

[michaelklishin]

This belongs to a discussion, which this repository apparently did not have enabled.