force_torque_sensor_broadcaster tests fail on Galactic Source Job in GitHub Workflows

[roncapat]

I cloned the repository and spawned the "Test ros2_controllers".

  2: [ RUN      ] ForceTorqueSensorBroadcasterTest.SensorName_Publish_Success
  2: 1636098896.664650 [0] test_force: using network interface eth0 (udp/10.1.1.107) selected arbitrarily from: eth0, docker0
  2: /home/runner/work/ros2_controllers/ros2_controllers/ros_ws/src/cav7w0ygslo/ros2_controllers/force_torque_sensor_broadcaster/test/test_force_torque_sensor_broadcaster.cpp:96: Failure
  2: Expected equality of these values:
  2:   wait_for(subscription)
  2:     Which is: 1
  2:   rclcpp::WaitResultKind::Ready
  2:     Which is: 0
  2: /home/runner/work/ros2_controllers/ros2_controllers/ros_ws/src/cav7w0ygslo/ros2_controllers/force_torque_sensor_broadcaster/test/test_force_torque_sensor_broadcaster.cpp:260: Failure
  2: Expected equality of these values:
  2:   wrench_msg.header.frame_id
  2:     Which is: ""
  2:   frame_id_
  2:     Which is: "fts_sensor_frame"
  2: [  FAILED  ] ForceTorqueSensorBroadcasterTest.SensorName_Publish_Success (20168 ms)
  2: [ RUN      ] ForceTorqueSensorBroadcasterTest.InterfaceNames_Publish_Success
  2: 1636098916.833366 [0] test_force: using network interface eth0 (udp/10.1.1.107) selected arbitrarily from: eth0, docker0
  2: /home/runner/work/ros2_controllers/ros2_controllers/ros_ws/src/cav7w0ygslo/ros2_controllers/force_torque_sensor_broadcaster/test/test_force_torque_sensor_broadcaster.cpp:96: Failure
  2: Expected equality of these values:
  2:   wait_for(subscription)
  2:     Which is: 1
  2:   rclcpp::WaitResultKind::Ready
  2:     Which is: 0
  2: /home/runner/work/ros2_controllers/ros2_controllers/ros_ws/src/cav7w0ygslo/ros2_controllers/force_torque_sensor_broadcaster/test/test_force_torque_sensor_broadcaster.cpp:284: Failure
  2: Expected equality of these values:
  2:   wrench_msg.header.frame_id
  2:     Which is: ""
  2:   frame_id_
  2:     Which is: "fts_sensor_frame"
  2: [  FAILED  ] ForceTorqueSensorBroadcasterTest.InterfaceNames_Publish_Success (20211 ms)

It happens only in the "Source Job" and never in the "Binary Jobs".
If I run the test locally with colcon after cloning the repo on my PC, everything works as expected.

Noticed while working on PR #260.

[destogl]

Yeah this is an annoying issue with our flaky tests. I had it failing on rolling running locally few days ago, but don't have time to debug this right now. This is related to #132

Do you have time or interest to tackle this? I can support you doing that. We can make a call before starting because I tested already quite some options but wasn't able to get the tests stable (running 100 iterations on my computer). [This offer is also for anybody else who would like to tackle this].

[roncapat]

This may also be related: #27
If publisher would allow introspection/access from the test class, we could move the wait logic on the publisher and away from the subscriber.

Otherwise, let me check the PR you referenced when I have a bit more time and I can tell you if in the following weeks I have enought spare time to contribute timely :)

[destogl]

@roncapat did you start to work on this? If not @jaron-l would start looking at this.

[roncapat]

Unfortunately, no time available yet. Happy if someone is going to handle this one :)

[jaron-l]

I was able to figure out that the test fails because sometimes this line evaluates to false which causes the update function to never publish in InterfaceNames_Publish_Success. I ran the test 500 times and every failure of this test corresponded to that line evaluating to false. Some unanswered questions I have are why this would only affect this test as that code seems to be setup and tested the exact same way on the other tests in that fixture and then the obvious question of why that line would ever evaluate to false when the tests are run serially and no other test should hold the lock.

[jaron-l]

I found the problem. When the controller is constructed, it initializes a real time publisher which starts a thread for the publish loop. That loop sets the turn_ variable which dictates if the loop is real time or not. The ForceTorqueBroadcaster initializes and finishes the test before the publishing loop has time to set that turn_ variable for the first time, thus it skips allowing the publisher to publish. I will have a sample fix up in a few minutes.