Get ci.ros2.org to pass with rmw_zenoh

[Yadunund]

After a huge effort in #493, we have all tests for all packages in the ros2.repos file passing with rmw_zenoh in local workspaces 🎉

However this required us to start the zenohd router locally which does not happen automatically when running a CI job on ci.ros2.org. The ros-infrastructure team is working on middleware agnostic mechanisms to start a pre-defined process but this may not be ready before the RMW freeze for Kilted on April 7.

A workaround in the meanwhile (to demonstrate that rmw_zenoh is indeed capable of passing CI) is to have ci.ros2.org run the usual tests with multicast scouting enabled for discovery. This would be akin to DDS-based middlewares. Once the infra tooling is in place, we can revert these changes.

The good news is that while we already have all tests for all packages passing locally, when a job runs on ci.ros2.org, tests for most packages run with the default middleware and only a select few in the ros2.repos file generate tests for every middleware found in the workspace. These packages rely on the get_available_rmw_implementations macro to retrieve all middlewares and generate tests for these middlewares. Therefore the goal is to update the tests in these repos to set environment variables to run rmw_zenoh tests with multicast discovery enabled. ie, set

export ZENOH_ROUTER_CHECK_ATTEMPTS=-1
export ZENOH_CONFIG_OVERRIDE='scouting/multicast/enabled=true'
export RUST_LOG=z=error

The packages that require changes are

• demos: Set envars to run tests with rmw_zenoh_cpp with multicast discovery demos#711
  • composition
  • image_tools
  • intra_process_test
  • demo_nodes_cpp
  • pendulum_control
• rcl: Set envars to run tests with rmw_zenoh_cpp with multicast discovery rcl#1218
  • rcl
  • rcl_action
• rclcpp: Set envars to run tests with rmw_zenoh_cpp with multicast discovery rclcpp#2776
• ros2cli: Allow zenoh tests to run with multicast ros2cli#992
  • ros2param
  • ros2service
  • ros2topic
  • ros2doctor
  • ros2node
  • ros2lifecycle
  • ros2action
• system_tests: Set envars to run tests with rmw_zenoh_cpp with multicast discovery system_tests#567
  • test_quality_of_service
  • test_communication
  • test_security
  • test_rclcpp
• sros2
• rosbag2: Set envars to run tests with rmw_zenoh_cpp with multicast discovery rosbag2#1946
  • rosbag2_cpp
  • rosbag2_transport

For detailed failures see

• Linux
• Linux-aarch64
• Linux-rhel
• Windows

[YuanYuYuan]

Hi @Yadunund! Since the previous issue has been closed, let me answer you here. We can set

export ZENOH_CONFIG_OVERRIDE='scouting/multicast/enabled=true;scouting/multicast/address="224.0.0.224:0"'

for port auto selection, but I found that more tests fail compared to generating the ports in a script.

[YuanYuYuan]

Regarding the failures reported by CI, I suspect the rmw_zenohd router doesn't work well. See the log here.

[Yadunund]

Regarding the failures reported by CI, I suspect the rmw_zenohd router doesn't work well. See the log here.

@YuanYuYuan to clarify, all these tests are failing because the Zenoh router has not been started before running the tests. Hence we need to add changes like ros2/rcl#1218 to these packages.

[YuanYuYuan]

@YuanYuYuan to clarify, all these tests are failing because the Zenoh router has not been started before running the tests. Hence we need to add changes like ros2/rcl#1218 to these packages.

I see. Then let's focus on the multicast version first. Here are the tests that haven't steadily passed on a local Ubuntu.

╭───┬────────────────────╮
│ 0 │ ros2param          │
│ 1 │ lttngpy            │
│ 2 │ rosbag2_tests      │
│ 3 │ launch_testing_ros │
│ 4 │ ros2doctor         │
│ 5 │ ros2lifecycle      │
│ 6 │ ros2topic          │
╰───┴────────────────────╯

[Yadunund]

We are down to 1 test failure on Linux/RHEL and 75 on Windows.

• Linux
• Linux-aarch64
• Linux-rhel
• Windows

[YuanYuYuan]

Remaining test failures: https://ci.ros2.org/job/ci_windows/23718/#showFailuresLink

• rosbag2/test_record_qos_profiles
  • Windows-only. It also occurs at other RMWs.
  • [Test Failure] ros2bag.test.test_record_qos_profiles fails on Windows 10 rosbag2#1947
• sros2/test_generate_policy
  • Flaky
• test_launch_ros/pytest.missing_result
  • Windows-only. Flaky

[Yadunund]

CI started with: https://raw.githubusercontent.com/ros2/ros2/3f873e9c5a79ce3b47f70199708b21c07007cd7d/ros2.repos

• Linux
• Linux-aarch64 -> flaky sros2
  • Linux-aarch64
• Linux-rhel
• Windows -> flaky ros2bag.test.test_record_qos_profiles.test_record_qos_profiles
  • Windows