[英]Parsing binary file too slow in C++ using memory-mapped files
我正在尝试以整数方式解析二进制文件，以检查 integer 值是否满足特定条件，但循环非常慢。
此外，我发现memory-mapped file是快速将文件读入 memory 的最快速度，因此我使用以下基于Boost的代码：
unsigned long long int get_file_size(const char *file_path) {
const filesystem::path file{file_path};
const auto generic_path = file.generic_path();
return filesystem::file_size(generic_path);
}
boost::iostreams::mapped_file_source read_bytes(const char *file_path,
const unsigned long long int offset,
const unsigned long long int length) {
boost::iostreams::mapped_file_params parameters;
parameters.path = file_path;
parameters.length = static_cast<size_t>(length);
parameters.flags = boost::iostreams::mapped_file::mapmode::readonly;
parameters.offset = static_cast<boost::iostreams::stream_offset>(offset);
boost::iostreams::mapped_file_source file;
file.open(parameters);
return file;
}
boost::iostreams::mapped_file_source read_bytes(const char *file_path) {
const auto file_size = get_file_size(file_path);
const auto mapped_file_source = read_bytes(file_path, 0, file_size);
return mapped_file_source;
}
我的测试用例大致如下：
inline auto test_parsing_binary_file_performance() {
const auto start_time = get_time();
const std::filesystem::path input_file_path = "...";
const auto mapped_file_source = read_bytes(input_file_path.string().c_str());
const auto file_buffer = mapped_file_source.data();
const auto file_buffer_size = mapped_file_source.size();
LOG_S(INFO) << "File buffer size: " << file_buffer_size;
auto printed_lap = (long) (file_buffer_size / (double) 1000);
printed_lap = round_to_nearest_multiple(printed_lap, sizeof(int));
LOG_S(INFO) << "Printed lap: " << printed_lap;
std::vector<int> values;
values.reserve(file_buffer_size / sizeof(int)); // Pre-allocate a large enough vector
// Iterate over every integer
for (auto file_buffer_index = 0; file_buffer_index < file_buffer_size; file_buffer_index += sizeof(int)) {
const auto value = *(int *) &file_buffer[file_buffer_index];
if (value >= 0x30000000 && value < 0x49000000 - sizeof(int) + 1) {
values.push_back(value);
}
if (file_buffer_index % printed_lap == 0) {
LOG_S(INFO) << std::setprecision(4) << file_buffer_index / (double) file_buffer_size * 100 << "%";
}
}
LOG_S(INFO) << "Values found count: " << values.size();
print_time_taken(start_time, false, "Parsing binary file");
}
memory-mapped file读取几乎按预期立即完成，但在我的机器上以整数方式迭代它太慢了，尽管硬件非常好（ SSD等）：
2020-12-20 13:04:35.124 ( 0.019s) [main thread ]Tests.hpp:387 INFO| File buffer size: 419430400
2020-12-20 13:04:35.124 ( 0.019s) [main thread ]Tests.hpp:390 INFO| Printed lap: 419432
2020-12-20 13:04:35.135 ( 0.029s) [main thread ]Tests.hpp:405 INFO| 0%
2020-12-20 13:04:35.171 ( 0.065s) [main thread ]Tests.hpp:405 INFO| 0.1%
2020-12-20 13:04:35.196 ( 0.091s) [main thread ]Tests.hpp:405 INFO| 0.2%
2020-12-20 13:04:35.216 ( 0.111s) [main thread ]Tests.hpp:405 INFO| 0.3%
2020-12-20 13:04:35.241 ( 0.136s) [main thread ]Tests.hpp:405 INFO| 0.4%
2020-12-20 13:04:35.272 ( 0.167s) [main thread ]Tests.hpp:405 INFO| 0.5%
2020-12-20 13:04:35.293 ( 0.188s) [main thread ]Tests.hpp:405 INFO| 0.6%
2020-12-20 13:04:35.314 ( 0.209s) [main thread ]Tests.hpp:405 INFO| 0.7%
2020-12-20 13:04:35.343 ( 0.237s) [main thread ]Tests.hpp:405 INFO| 0.8%
2020-12-20 13:04:35.366 ( 0.261s) [main thread ]Tests.hpp:405 INFO| 0.9%
2020-12-20 13:04:35.399 ( 0.293s) [main thread ]Tests.hpp:405 INFO| 1%
2020-12-20 13:04:35.421 ( 0.315s) [main thread ]Tests.hpp:405 INFO| 1.1%
2020-12-20 13:04:35.447 ( 0.341s) [main thread ]Tests.hpp:405 INFO| 1.2%
2020-12-20 13:04:35.468 ( 0.362s) [main thread ]Tests.hpp:405 INFO| 1.3%
2020-12-20 13:04:35.487 ( 0.382s) [main thread ]Tests.hpp:405 INFO| 1.4%
2020-12-20 13:04:35.520 ( 0.414s) [main thread ]Tests.hpp:405 INFO| 1.5%
2020-12-20 13:04:35.540 ( 0.435s) [main thread ]Tests.hpp:405 INFO| 1.6%
2020-12-20 13:04:35.564 ( 0.458s) [main thread ]Tests.hpp:405 INFO| 1.7%
2020-12-20 13:04:35.586 ( 0.480s) [main thread ]Tests.hpp:405 INFO| 1.8%
2020-12-20 13:04:35.608 ( 0.503s) [main thread ]Tests.hpp:405 INFO| 1.9%
2020-12-20 13:04:35.636 ( 0.531s) [main thread ]Tests.hpp:405 INFO| 2%
2020-12-20 13:04:35.658 ( 0.552s) [main thread ]Tests.hpp:405 INFO| 2.1%
2020-12-20 13:04:35.679 ( 0.574s) [main thread ]Tests.hpp:405 INFO| 2.2%
2020-12-20 13:04:35.702 ( 0.597s) [main thread ]Tests.hpp:405 INFO| 2.3%
2020-12-20 13:04:35.727 ( 0.622s) [main thread ]Tests.hpp:405 INFO| 2.4%
2020-12-20 13:04:35.769 ( 0.664s) [main thread ]Tests.hpp:405 INFO| 2.5%
2020-12-20 13:04:35.802 ( 0.697s) [main thread ]Tests.hpp:405 INFO| 2.6%
2020-12-20 13:04:35.831 ( 0.726s) [main thread ]Tests.hpp:405 INFO| 2.7%
2020-12-20 13:04:35.860 ( 0.754s) [main thread ]Tests.hpp:405 INFO| 2.8%
2020-12-20 13:04:35.887 ( 0.781s) [main thread ]Tests.hpp:405 INFO| 2.9%
2020-12-20 13:04:35.924 ( 0.818s) [main thread ]Tests.hpp:405 INFO| 3%
2020-12-20 13:04:35.956 ( 0.850s) [main thread ]Tests.hpp:405 INFO| 3.1%
2020-12-20 13:04:35.998 ( 0.893s) [main thread ]Tests.hpp:405 INFO| 3.2%
2020-12-20 13:04:36.033 ( 0.928s) [main thread ]Tests.hpp:405 INFO| 3.3%
2020-12-20 13:04:36.060 ( 0.955s) [main thread ]Tests.hpp:405 INFO| 3.4%
2020-12-20 13:04:36.102 ( 0.997s) [main thread ]Tests.hpp:405 INFO| 3.5%
2020-12-20 13:04:36.132 ( 1.026s) [main thread ]Tests.hpp:405 INFO| 3.6%
...
2020-12-20 13:05:03.456 ( 28.351s) [main thread ]Tests.hpp:410 INFO| Values found count: 10650389
2020-12-20 13:05:03.456 ( 28.351s) [main thread ] benchmark.cpp:31 INFO| Parsing binary file took 28.341 second(s)
解析那些419 MB总是需要大约 28 - 70 秒。 即使在Release模式下编译也无济于事。 有什么办法可以缩短这个时间吗？ 我正在执行的操作似乎不应该那么低效。
请注意，我正在使用GCC 10为Linux 64-bit进行编译。
编辑：
正如评论中所建议的，使用带有advise()的memory-mapped file也无助于提高性能：
boost::interprocess::file_mapping file_mapping(input_file_path.string().data(), boost::interprocess::read_only);
boost::interprocess::mapped_region mapped_region(file_mapping, boost::interprocess::read_only);
mapped_region.advise(boost::interprocess::mapped_region::advice_sequential);
const auto file_buffer = (char *) mapped_region.get_address();
const auto file_buffer_size = mapped_region.get_size();
...
考虑到评论/答案，到目前为止吸取的教训：
advise(boost::interprocess::mapped_region::advice_sequential)没有帮助reserve()或以完全正确的大小调用它可以使性能翻倍int *上迭代比在char *上迭代要慢一些std::set比std::vector收集结果要慢一些正如xanatos memory-mapped file所暗示的那样，它们在性能上具有欺骗性，因为它们不会立即将整个文件读入 memory。 在处理过程中，页面未命中会导致多次磁盘访问，从而严重降低性能。
在这种情况下，首先将整个文件读入 memory 然后遍历 memory 会更有效：
inline std::vector<std::byte> load_file_into_memory(const std::filesystem::path &file_path) {
std::ifstream input_stream(file_path, std::ios::binary | std::ios::ate);
if (input_stream.fail()) {
const auto error_message = "Opening " + file_path.string() + " failed";
throw std::runtime_error(error_message);
}
auto current_read_position = input_stream.tellg();
input_stream.seekg(0, std::ios::beg);
auto file_size = std::size_t(current_read_position - input_stream.tellg());
if (file_size == 0) {
return {};
}
std::vector<std::byte> buffer(file_size);
if (!input_stream.read((char *) buffer.data(), buffer.size())) {
const auto error_message = "Reading from " + file_path.string() + " failed";
throw std::runtime_error(error_message);
}
return buffer;
}
现在性能更容易接受，总共大约3 - 15 seconds 。
这让我想起了大约 40 年前我第一次遇到缓慢。 由衡量进度的百分比条引起。 注释掉该部分并再次测量。 还要测量容量储备，并检查所需的实际容量——如果是 1%，那么你就是在浪费空间，从而浪费时间。
unsigned long long可能代价高昂。 unsignedlong还不够吗？所以：
const auto pct_factor = file_buffer_size == 0 ? 0.0 : 100 / (double)file_buffer_size;
values.reserve(file_buffer_size / sizeof(int));
for (auto file_buffer_index = 0, long pct_countdown = 0; file_buffer_index < file_buffer_size; file_buffer_index += sizeof(int)) {
const auto value = *(int *) &file_buffer[file_buffer_index];
if (value >= 0x30000000 && value < 0x49000000 - sizeof(int) + 1) {
values.push_back(value);
}
if (pct_countdown-- < 0) {
pct_countdown = printed_lap;
const auto pct = file_buffer_index * pct_factor;
LOG_S(INFO) << std::setprecision(4) << pct << "%";
}
}
values - 是否需要它。 一套可能就足够了。 我承认我对*(int *)有疑问。 使用int*指针并增加它似乎也更直接。
如何使用Boost的`mapped_file_sink`类刷新内存映射文件？
[英]How to flush memory-mapped files using Boost's `mapped_file_sink` class?
如何在没有NMI的情况下通过内存映射文件在C ++和JAVA之间进行通信？
[英]How to communicate between C++ and JAVA via Memory-Mapped file without NMI?
sqlite C\C++ 库是否将 DB 文件保留为内存映射文件？ 是否有可能做到这一点？
[英]Does sqlite C\C++ library keep DB file as Memory-mapped file? and is it possible to make it do so?
读取txt文件并使用c ++和boost内存映射文件快速解析它们
[英]Reading txt files and parsing them fast using c++ and boost memory mapped files
只读内存映射寄存器在C中用`volatile const`定义，但在C ++中只用`volatile`定义
[英]Read-only memory-mapped registers defined with `volatile const` in C but only `volatile` in C++
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.