![](/img/trans.png)
[英]How to flush memory-mapped files using Boost's `mapped_file_sink` class?
[英]Parsing binary file too slow in C++ using memory-mapped files
我正在嘗試以整數方式解析二進制文件,以檢查 integer 值是否滿足特定條件,但循環非常慢。
此外,我發現memory-mapped file
是快速將文件讀入 memory 的最快速度,因此我使用以下基於Boost
的代碼:
unsigned long long int get_file_size(const char *file_path) {
const filesystem::path file{file_path};
const auto generic_path = file.generic_path();
return filesystem::file_size(generic_path);
}
boost::iostreams::mapped_file_source read_bytes(const char *file_path,
const unsigned long long int offset,
const unsigned long long int length) {
boost::iostreams::mapped_file_params parameters;
parameters.path = file_path;
parameters.length = static_cast<size_t>(length);
parameters.flags = boost::iostreams::mapped_file::mapmode::readonly;
parameters.offset = static_cast<boost::iostreams::stream_offset>(offset);
boost::iostreams::mapped_file_source file;
file.open(parameters);
return file;
}
boost::iostreams::mapped_file_source read_bytes(const char *file_path) {
const auto file_size = get_file_size(file_path);
const auto mapped_file_source = read_bytes(file_path, 0, file_size);
return mapped_file_source;
}
我的測試用例大致如下:
inline auto test_parsing_binary_file_performance() {
const auto start_time = get_time();
const std::filesystem::path input_file_path = "...";
const auto mapped_file_source = read_bytes(input_file_path.string().c_str());
const auto file_buffer = mapped_file_source.data();
const auto file_buffer_size = mapped_file_source.size();
LOG_S(INFO) << "File buffer size: " << file_buffer_size;
auto printed_lap = (long) (file_buffer_size / (double) 1000);
printed_lap = round_to_nearest_multiple(printed_lap, sizeof(int));
LOG_S(INFO) << "Printed lap: " << printed_lap;
std::vector<int> values;
values.reserve(file_buffer_size / sizeof(int)); // Pre-allocate a large enough vector
// Iterate over every integer
for (auto file_buffer_index = 0; file_buffer_index < file_buffer_size; file_buffer_index += sizeof(int)) {
const auto value = *(int *) &file_buffer[file_buffer_index];
if (value >= 0x30000000 && value < 0x49000000 - sizeof(int) + 1) {
values.push_back(value);
}
if (file_buffer_index % printed_lap == 0) {
LOG_S(INFO) << std::setprecision(4) << file_buffer_index / (double) file_buffer_size * 100 << "%";
}
}
LOG_S(INFO) << "Values found count: " << values.size();
print_time_taken(start_time, false, "Parsing binary file");
}
memory-mapped file
讀取幾乎按預期立即完成,但在我的機器上以整數方式迭代它太慢了,盡管硬件非常好( SSD
等):
2020-12-20 13:04:35.124 ( 0.019s) [main thread ]Tests.hpp:387 INFO| File buffer size: 419430400
2020-12-20 13:04:35.124 ( 0.019s) [main thread ]Tests.hpp:390 INFO| Printed lap: 419432
2020-12-20 13:04:35.135 ( 0.029s) [main thread ]Tests.hpp:405 INFO| 0%
2020-12-20 13:04:35.171 ( 0.065s) [main thread ]Tests.hpp:405 INFO| 0.1%
2020-12-20 13:04:35.196 ( 0.091s) [main thread ]Tests.hpp:405 INFO| 0.2%
2020-12-20 13:04:35.216 ( 0.111s) [main thread ]Tests.hpp:405 INFO| 0.3%
2020-12-20 13:04:35.241 ( 0.136s) [main thread ]Tests.hpp:405 INFO| 0.4%
2020-12-20 13:04:35.272 ( 0.167s) [main thread ]Tests.hpp:405 INFO| 0.5%
2020-12-20 13:04:35.293 ( 0.188s) [main thread ]Tests.hpp:405 INFO| 0.6%
2020-12-20 13:04:35.314 ( 0.209s) [main thread ]Tests.hpp:405 INFO| 0.7%
2020-12-20 13:04:35.343 ( 0.237s) [main thread ]Tests.hpp:405 INFO| 0.8%
2020-12-20 13:04:35.366 ( 0.261s) [main thread ]Tests.hpp:405 INFO| 0.9%
2020-12-20 13:04:35.399 ( 0.293s) [main thread ]Tests.hpp:405 INFO| 1%
2020-12-20 13:04:35.421 ( 0.315s) [main thread ]Tests.hpp:405 INFO| 1.1%
2020-12-20 13:04:35.447 ( 0.341s) [main thread ]Tests.hpp:405 INFO| 1.2%
2020-12-20 13:04:35.468 ( 0.362s) [main thread ]Tests.hpp:405 INFO| 1.3%
2020-12-20 13:04:35.487 ( 0.382s) [main thread ]Tests.hpp:405 INFO| 1.4%
2020-12-20 13:04:35.520 ( 0.414s) [main thread ]Tests.hpp:405 INFO| 1.5%
2020-12-20 13:04:35.540 ( 0.435s) [main thread ]Tests.hpp:405 INFO| 1.6%
2020-12-20 13:04:35.564 ( 0.458s) [main thread ]Tests.hpp:405 INFO| 1.7%
2020-12-20 13:04:35.586 ( 0.480s) [main thread ]Tests.hpp:405 INFO| 1.8%
2020-12-20 13:04:35.608 ( 0.503s) [main thread ]Tests.hpp:405 INFO| 1.9%
2020-12-20 13:04:35.636 ( 0.531s) [main thread ]Tests.hpp:405 INFO| 2%
2020-12-20 13:04:35.658 ( 0.552s) [main thread ]Tests.hpp:405 INFO| 2.1%
2020-12-20 13:04:35.679 ( 0.574s) [main thread ]Tests.hpp:405 INFO| 2.2%
2020-12-20 13:04:35.702 ( 0.597s) [main thread ]Tests.hpp:405 INFO| 2.3%
2020-12-20 13:04:35.727 ( 0.622s) [main thread ]Tests.hpp:405 INFO| 2.4%
2020-12-20 13:04:35.769 ( 0.664s) [main thread ]Tests.hpp:405 INFO| 2.5%
2020-12-20 13:04:35.802 ( 0.697s) [main thread ]Tests.hpp:405 INFO| 2.6%
2020-12-20 13:04:35.831 ( 0.726s) [main thread ]Tests.hpp:405 INFO| 2.7%
2020-12-20 13:04:35.860 ( 0.754s) [main thread ]Tests.hpp:405 INFO| 2.8%
2020-12-20 13:04:35.887 ( 0.781s) [main thread ]Tests.hpp:405 INFO| 2.9%
2020-12-20 13:04:35.924 ( 0.818s) [main thread ]Tests.hpp:405 INFO| 3%
2020-12-20 13:04:35.956 ( 0.850s) [main thread ]Tests.hpp:405 INFO| 3.1%
2020-12-20 13:04:35.998 ( 0.893s) [main thread ]Tests.hpp:405 INFO| 3.2%
2020-12-20 13:04:36.033 ( 0.928s) [main thread ]Tests.hpp:405 INFO| 3.3%
2020-12-20 13:04:36.060 ( 0.955s) [main thread ]Tests.hpp:405 INFO| 3.4%
2020-12-20 13:04:36.102 ( 0.997s) [main thread ]Tests.hpp:405 INFO| 3.5%
2020-12-20 13:04:36.132 ( 1.026s) [main thread ]Tests.hpp:405 INFO| 3.6%
...
2020-12-20 13:05:03.456 ( 28.351s) [main thread ]Tests.hpp:410 INFO| Values found count: 10650389
2020-12-20 13:05:03.456 ( 28.351s) [main thread ] benchmark.cpp:31 INFO| Parsing binary file took 28.341 second(s)
解析那些419 MB
總是需要大約 28 - 70 秒。 即使在Release
模式下編譯也無濟於事。 有什么辦法可以縮短這個時間嗎? 我正在執行的操作似乎不應該那么低效。
請注意,我正在使用GCC 10
為Linux 64-bit
進行編譯。
編輯:
正如評論中所建議的,使用帶有advise()
的memory-mapped file
也無助於提高性能:
boost::interprocess::file_mapping file_mapping(input_file_path.string().data(), boost::interprocess::read_only);
boost::interprocess::mapped_region mapped_region(file_mapping, boost::interprocess::read_only);
mapped_region.advise(boost::interprocess::mapped_region::advice_sequential);
const auto file_buffer = (char *) mapped_region.get_address();
const auto file_buffer_size = mapped_region.get_size();
...
考慮到評論/答案,到目前為止吸取的教訓:
advise(boost::interprocess::mapped_region::advice_sequential)
沒有幫助reserve()
或以完全正確的大小調用它可以使性能翻倍int *
上迭代比在char *
上迭代要慢一些std::set
比std::vector
收集結果要慢一些正如xanatos
memory-mapped file
所暗示的那樣,它們在性能上具有欺騙性,因為它們不會立即將整個文件讀入 memory。 在處理過程中,頁面未命中會導致多次磁盤訪問,從而嚴重降低性能。
在這種情況下,首先將整個文件讀入 memory 然后遍歷 memory 會更有效:
inline std::vector<std::byte> load_file_into_memory(const std::filesystem::path &file_path) {
std::ifstream input_stream(file_path, std::ios::binary | std::ios::ate);
if (input_stream.fail()) {
const auto error_message = "Opening " + file_path.string() + " failed";
throw std::runtime_error(error_message);
}
auto current_read_position = input_stream.tellg();
input_stream.seekg(0, std::ios::beg);
auto file_size = std::size_t(current_read_position - input_stream.tellg());
if (file_size == 0) {
return {};
}
std::vector<std::byte> buffer(file_size);
if (!input_stream.read((char *) buffer.data(), buffer.size())) {
const auto error_message = "Reading from " + file_path.string() + " failed";
throw std::runtime_error(error_message);
}
return buffer;
}
現在性能更容易接受,總共大約3 - 15 seconds
。
這讓我想起了大約 40 年前我第一次遇到緩慢。 由衡量進度的百分比條引起。 注釋掉該部分並再次測量。 還要測量容量儲備,並檢查所需的實際容量——如果是 1%,那么你就是在浪費空間,從而浪費時間。
unsigned long long
可能代價高昂。 unsignedlong
還不夠嗎?所以:
const auto pct_factor = file_buffer_size == 0 ? 0.0 : 100 / (double)file_buffer_size;
values.reserve(file_buffer_size / sizeof(int));
for (auto file_buffer_index = 0, long pct_countdown = 0; file_buffer_index < file_buffer_size; file_buffer_index += sizeof(int)) {
const auto value = *(int *) &file_buffer[file_buffer_index];
if (value >= 0x30000000 && value < 0x49000000 - sizeof(int) + 1) {
values.push_back(value);
}
if (pct_countdown-- < 0) {
pct_countdown = printed_lap;
const auto pct = file_buffer_index * pct_factor;
LOG_S(INFO) << std::setprecision(4) << pct << "%";
}
}
values
- 是否需要它。 一套可能就足夠了。 我承認我對*(int *)
有疑問。 使用int*
指針並增加它似乎也更直接。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.