C ++搜索算法-处理大量数据

Question

I have a code that searches in files for a string, File(s) can be 1mg of size or 1gig or bigger.

I get the file data with ReadFile() WinAPI and Convert into Hex, Then search for a string (Which is hexed before) in the Converted Data.

I used this code for search (string search):

std::string searchStr = "48656C6C6FA"
std::string fileData = ToHex(inputString);

if(fileData.find(searchStr, 0) != std::string::npos)
{
    std::cout << FileName;
}

It takes almost 11 sec to search for string in 2900 files.

Is there any other search algorithm or function to be faster? This way (above) sometimes missed the string and not perfectly works.

Answer 1

If you have a smaller file (like a few megabytes, or even a couple of hundred megabytes, depending on the amount of memory your system have) then read it all into memory, otherwise I recommend using memory mapped files . If the file is to big to be mapped you can use a sliding window or double-buffering algorithm to read blocks of the data from the file into memory.

Then to search for a specific sequence of bytes, you do a linear search through the contents of the file, looking for the first byte of the sequence you search for (in the case of 0x48656C6C6FA that's 0xFA ) . If found then you attempt to match the second byte in the sequence (in the example that's 0xC6 ) to the next byte from the file, and so on until you have matched the whole sequence.

If the second (or continuing) byte doesn't match, you continue your search for the first byte.

This has O(n) complexity, where n is the number of bytes in the file. Unless you know beforehand that the data you search for is in a specific part of the file, that's the best you're going to get.

---

If the files exists on an SSD you can use threads to search, one thread per file. But not all 2900 files at once, that will swamp the processor. Instead have 4-8 threads doing the search (depending on the number of cores of your system), and as soon as one thread is finished with a file, then it takes the next.

Can't be used on a spinning-disk drive, as it will thrash the disk while the heads are seeking back and forth as the threads are trying to read.

Answer 2

Speed: use a memory mapped file

Accuracy: use std::search using binary values.

eg

#include <algorithm>
#include <cstdint>
#include <tuple>
#include <vector>

// some function to return a pointer to the first byte in the file and the length 
extern std::tuple<const std::uint8_t*, std::size_t> get_file_bounds();

int main()
{
    auto [begin, size] = get_file_bounds();
    auto search_string = std::vector<std::uint8_t> {
        0x48,
        0x65,
        0x6C,
        0x6C,
        0x6F
    };

    auto iter = std::search(begin, begin + size, 
                            search_string.begin(), search_string.end());

    if (iter != begin + size)
    {
        // found the sequence 
    }
    else 
    {
        // didn't find it
    }

}

Answer 3

For search strings as short as the ones you have (5 1/2 bytes apparently), the bottleneck will often be the disk I/O. I suspect that those 2900 files may be on a harddisk. That would translate to roughly 4 ms per file, which is quite decent.

Sure, the conversion to hex may be a bit clumsy, but given the 5 1/2 bytes (11 hex digits) it might not be entirely unreasonable. Ie you might not get a major speed improvement if the HDD is the real bottleneck.

So to check, measure how much time you spend if you don't search in the 2900 fies, and just read them in. Don't even convert them to hex. No matter how smart the search algorithm, the time you'll need for disk I/O is a lower bound. If this isn't good enough, get a fast SSD.

Answer 4

For a faster string search algorithm, take a look at the Boyer Moore search algorithm. Boost (and c++17) has such an implementation.

Also, avoid converting the file into hex (std::strings can contain '\\0' characters).

And if you file IO is limiting, memory mapped files might be the way forward..

Answer 5

While this is probably a storage bottleneck problem there are string search algorithms that can be significantly faster than linear, for example Boyer Moore (described at https://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string_search_algorithm ), they do require processing the search pattern and ave some memory overhead compared to a linear search.

The basic idea is to know how many characters can be skipped based on what you find at a given index. (iE start at fileData[patternLLen-1] and if the character isn't even in the search pattern you can next look at fileData[patternLen+patternLen-1] and so on.

The longer your pattern the more likely such an algorithm is to be an improvement over a straight linear search. The boost library already has implementations of several such improved string search algorithms (found in boost/algorithm\\searching/).