Save Data in C++, Load from Python - Recommended Data Formats

Question

I have a ROS/CPP simulator that saves large amounts of data to a rosbag (around 90 MB). I want to read this data frequently from Python and since reading rosbags is slow and cumbersome, I currently have another python script that reads the rosbag and saves the relevant contents to a HDF5 file.

It would be nice though to be able to just save the data from the simulator directly (in C++) and then read it from my scripts (in Python). So I was wondering which data format I should use. It should be:

1. Fast to load from Python
2. Be compact (so ideally a binary of some sort)
3. Be easy to use

You might be wondering why I don't just save to HDF5 from my C++ simulator, but it just doesn't seem to be easy. There is basically nothing on forums such as Stackoverflow and the HDF5 Group website is opaque, seems to have some complicated licensing and very poor examples. I just want something quick and dirty that I can get running this afternoon.

Answer 1

You may want to have a look at HDFql as it is a high-level language (similar to SQL) to manage HDF5 files. Amongst others, HDFql supports C++ and Python. There are some examples that illustrates how to use HDFql in these languages here .

Answer 2

I see two solutions that can be useful for your problem :

• LV : Length Value that you can store directly in binary into a file.
• JSON : This does not add many data more than you need, and there are many libraries in Python or C++ that can simplify you the work

Answer 3

协议缓冲区是C ++和Python中语言绑定的一个选项，尽管它可能比今天下午运行快速/肮脏的时间花费更多。