CPP function's execution duration follows bimodal distribution

Question

Why does my C++ function's execution times follow a bimodal distribution?

In my C++ code, I load an external library, a data processor (either a cpp class compiled to a shared object which is dynamically loaded, or a python class), then call its method data_received N times and measure how much time it each execution of data_received took. So I instantiate the class whose method I call only once at the beginning before measuring anything. However, in both versions, the execution durations follow a bimodal distribution (Figure 1. cpp, Figure 2. python).

One thing to note here is that I'm running these functions as a callback whenever a ROS node receives a message. Publisher publishes a message with a frequency of 5000, so every 0.2 milliseconds. So these functions are called roughly once every 0.2 milliseconds.

In addition, I tried calling these functions simply 1000000 times normally, so without the callback. In this case, the execution duration distributions also follow this kind of bimodal distribution, but less strongly (Figure 3. and Figure 4.).

So there is a clear difference between whether I run the code normally sequentially or as a callback periodically. However, I don't understand why this should matter. Moreover, these durations seem to be periodic - so they come as groups. This is illustrated in Figure 5. This applies to both cpp and python and also when running the code normally, not as callbacks (but the effect is less clear).

I would guess that this is somehow related to CPU scheduling? However, I'm running on a Linux PREEMPT_RT kernel 5.15.55-rt48, have set the process priority to -98 and scheduling policy to SCHED_RR. I'm using Release build. I'm building the project with colcon colcon build --packages-select <ros package> --cmake-args -DCMAKE_BUILD_TYPE=Release and the following CMakeLists.txt. My compiler is GNU 9.4.0.

cmake_minimum_required(VERSION 3.8)
project(data_processor)

message(STATUS "compiler: ${CMAKE_CXX_COMPILER_ID} ${CMAKE_CXX_COMPILER_VERSION}")

if (CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
    add_compile_options(-Wall -Wextra -Wpedantic)
endif ()

find_package(ament_cmake REQUIRED)
find_package(rclcpp REQUIRED)
find_package(std_msgs REQUIRED)
find_package(mtms_interfaces REQUIRED)
find_package(fpga_interfaces REQUIRED)

set(MATLAB_FIND_DEBUG true)
# MATLAB
find_package(Matlab)
if (Matlab_FOUND)
    # Following 5 lines taken from https://stackoverflow.com/questions/8880802/cmake-linking-shared-library
    # Fixes runtime error "error while loading shared libraries: libMatlabDataArray.so: cannot open shared object file: No such file or directory"
    SET(CMAKE_SKIP_BUILD_RPATH FALSE)
    SET(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE)
    SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib64")
    SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
    SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib64")

    set(LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:${Matlab_ROOT_DIR}/extern/bin/glnxa64:${Matlab_ROOT_DIR}/sys/os/glnxa64)
    include_directories(${Matlab_ROOT_DIR}/extern/include/)
    link_directories(${Matlab_ROOT_DIR}/extern/bin/glnxa64)
else ()
    message(STATUS "MATLAB NOT FOUND")
endif ()

add_executable(
        data_processor
        src/data_processor.cpp
        src/processor.cpp
        src/headers/processor.h
        src/python_processor.cpp
        src/matlab_processor.cpp
        src/compiled_matlab_processor.cpp
        src/matlab_processor_interface.cpp
        src/headers/matlab_processor.h
        src/headers/python_processor.h
        src/headers/compiled_matlab_processor.h
        src/headers/scheduling_utils.h
        src/headers/scheduling_utils.cpp
        src/headers/matlab_processor_interface.h
        src/headers/fpga_event.h
        src/headers/data_processor.h
        src/headers/matlab_helpers.h
        src/matlab_helpers.cpp)

target_include_directories(data_processor
        PUBLIC
        $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/lib>
        $<INSTALL_INTERFACE:lib>)

ament_target_dependencies(data_processor rclcpp std_msgs mtms_interfaces fpga_interfaces)

if (Matlab_FOUND)
    # MATLAB, Linker to libMatlabEngine in link_directories
    target_link_libraries(data_processor MatlabDataArray)
    target_link_libraries(data_processor MatlabEngine)
endif ()

# Python
find_package(PythonLibs)
if (PYTHONLIBS_FOUND)
    message(STATUS "Python found")
    include_directories(${PYTHON_INCLUDE_DIRS})
    target_link_libraries(data_processor ${PYTHON_LIBRARIES})
else ()
    message(STATUS "Python not found")
endif ()

install(TARGETS
        data_processor
        DESTINATION lib/${PROJECT_NAME}
        )

install(
        DIRECTORY launch
        DESTINATION share/${PROJECT_NAME}
)

if (BUILD_TESTING)
    find_package(ament_lint_auto REQUIRED)
    ament_lint_auto_find_test_dependencies()
endif ()

ament_package()

Figure 1. Callback durations in microseconds, cpp

Figure 2. Callback durations in microseconds, python

Figure 3. Normal durations in microseconds, cpp

Figure 4. Normal durations in microseconds, python

Figure 5. Periodic durations in microseconds, python

Answer 1

Compiling the dynamically loaded C++ library as a Release build fixed issues with C++. However, it still remains a question what causes the bimodality on the python version.