What is the output format of CARLA rgb camera sensor

Question

I see many python example for the RGB camera. But can not find any C++ example for the rgb camera output. How is the data stored in the callback data of CARLA RGB Camera Sensor? It is possible to get the data output by reading code- but thought someone might have already decoded the camera output using something like boost view.

Answer 1

DISCLAIMER: This answer is valid as per 0.9.8, bear in mind that CARLA's C++ API is still under development and changes often, in case of doubt check out the source code or their C++ Reference .

The C++ API passes a boost::shared_ptr<carla::sensor::SensorData> to the sensor callback, that's the base class for all the sensor data types (images, point clouds, etc.). Unlike Python, the C++ API does not automatically downcast to the specific type, that's an extra step you have to do. For cameras, you should downcast to carla::sensor::data::Image (alias for ImageTmpl<Color> )

camera->Listen([](auto data) {
    auto image = boost::dynamic_pointer_cast<carla::sensor::data::Image>(data);
    assert(image != nullptr);
    parseImage(image);
});

This image can be used as an array of pixels

for (auto& pixel : *image)
{
    std::cout << pixel.r << ", " << pixel.g << ", " << pixel.b << "\n";
}

Alternatively, if you prefer to access the buffer directly use the image->data() , its layout in memory is BGRA.

Conversions to Boost GIL view are also provided in carla::image::ImageView , together with some convenient color converters in case you want to create a color-converted view, for instance to colorize semantic segmentation images. This is very useful to display images in more human friendly colors without modifying the original image or making copies.

using namespace carla::image;

// Boost GIL image view.
auto view = ImageView::MakeView(*image);

// View semantic segmentation as CityScapes color palette.
auto colorized = ImageView::MakeColorConvertedView(view, ColorConverter::CityScapesPalette{});

Finally, that view can be saved to disk with ImageIO , however if use this you need to link against boost_filesystem, libpng, libjpeg, and libtiff. So I would avoid including this unless you really need it.

ImageIO::WriteView("image.png", colorized);