EventsCBGExecutor keeps queued events of removed nodes, component container segfaults on shutdown #3312

Generated by Generative AI

Claude Code (claude-opus-5-5): root-cause analysis, reproducer and fix were developed with it; I reviewed and verified the results.

Operating System:

Ubuntu 24.04.5 LTS, x86_64

ROS version or commit hash:

Jazzy, ros-jazzy-rclcpp 28.1.22; also present on rolling (ebf57e3)

RMW implementation (if applicable):

rmw_zenoh_cpp and rmw_fastrtps_cpp (both crash 10/10 with the reproducer)

RMW Configuration (if applicable):

No response

Client library (if applicable):

rclcpp

'ros2 doctor --report' output

No response

Steps to reproduce issue

A component container with the EventsCBGExecutor crashes on Ctrl-C if a loaded component still has events queued in the executor when it shuts down.

Full reproducer (ament package, build and run instructions):
https://gist.github.com/spurnvoj/368641954e673feea50e89dd2e007074

Two components in separate libraries, loaded with intra-process comms into the stock component_container_events_cbg:

// libpub.so: publishes nav_msgs/Odometry every 2 ms
timer_ = create_wall_timer(2ms, [this] {
    pub_->publish(std::make_unique<nav_msgs::msg::Odometry>());
  });

// libsub.so: 20 subscriptions with a slow callback, so events stay queued
for (int i = 0; i < 20; i++) {
  subs_.push_back(create_subscription<nav_msgs::msg::Odometry>(
      "odom", 10, [](nav_msgs::msg::Odometry::ConstSharedPtr) {
        std::this_thread::sleep_for(2ms);
      }));
}

1. Build both libraries with -fno-gnu-unique. GCC otherwise marks some symbols STB_GNU_UNIQUE and glibc never really unloads the library, which hides the crash. Packages from the ROS build farm have no such symbols, so they unload.
2. Load both with use_intra_process_comms: true into component_container_events_cbg.
3. After a few seconds, send SIGINT (Ctrl-C) to the launch process group.

Result: 10 of 10 runs crash. Without the slow callback (nothing left in the queue) it crashes only occasionally (1 of 20).

Expected behavior

The container shuts down cleanly.

Actual behavior

The container exits with SIGSEGV (exit code -11) in the executor destructor:

#0  0x...839a in ?? () from /opt/ros/jazzy/lib/librclcpp.so
#1  0x...ac53 in ?? () from /opt/ros/jazzy/lib/librclcpp.so
#2  0x...ae46 in ?? () from /opt/ros/jazzy/lib/librclcpp.so
#3  rclcpp::executors::EventsCBGExecutor::~EventsCBGExecutor() () from /opt/ros/jazzy/lib/librclcpp.so
#4  0x... in ?? ()
#5  main ()

The faulting instruction is a virtual call (call *0x18(%rax), the control block's _M_destroy) through a vtable at an unmapped address. libsub.so and libpub.so are no longer mapped in the core dump.

Additional information

Root cause: the scheduler never forgets callback groups. CBGScheduler::remove_callback_group() (scheduler.hpp) is never called (see // FIXME inform scheduler about remove cbgs in EventsCBGExecutor::sync_callback_groups()). So every CallbackGroupHandle stays alive together with its ready_entities queue until the executor is destroyed. Queued but not executed events are ReadyEntitys holding weak_ptrs to the node's entities.

In component_container_events_cbg the executor is created before the ComponentManager and therefore destroyed after it. ~ComponentManager destroys the components and unloads their libraries. ~EventsCBGExecutor then releases the last weak_ptrs, and freeing the control block calls code that lived in the unloaded library.

Swapping the declaration order in the container's main() (executor destroyed first) avoids it: 0 of 5 crashes with the reproducer (CONTAINER=workaround in the gist).

Not fixed by #3190 (28.1.22 includes it). Related: #2567 (same symptom, different cause).

In our system (MRS UAV system, several component containers) this crashed the core container in 4 of 15 Ctrl-C runs.

I have a fix that clears the queued events when the callback group's entity cache is destroyed, with a regression test. PR to follow.