搜索非常大的彩虹表文件

Question

I am looking for the best way to search through a very large rainbow table file (13GB file). It is a CSV-style file, looking something like this:

1f129c42de5e4f043cbd88ff6360486f; somestring
78f640ec8bf82c0f9264c277eb714bcf; anotherstring
4ed312643e945ec4a5a1a18a7ccd6a70; yetanotherstring

... you get the idea - there are about ~900 Million lines, always with a hash, semicolon, clear text string.

So basically, the program should look if a specific hash is lited in this file.

Whats the fastest way to do this? Obviously, I can't read the entire file into memory and then put a strstr() on it.

So whats the most efficent way to do this?

1. read file line by line, always to a strstr() ;
2. read larger chunk of the file (eg 10.000 lines), do a strstr()

Or would it be more efficient import all this data into an MySQL database and then search for the hash via SQL querys?

Any help is appreciated

Answer 1

The best way to do it would be to sort it and then use a binary search -like algorithm on it. After sorting it, it will take around O(log n) time to find a particular entry where n is the number of entries you have. Your algorithm might look like this:

1. Keep a start offset and end offset. Initialize the start offset to zero and end offset to the file size.
2. If start = end, there is no match.
3. Read some data from the offset (start + end) / 2.
4. Skip forward until you see a newline. (You may need to read more, but if you pick an appropriate size (bigger than most of your records) to read in step 3, you probably won't have to read any more.)
5. • If the hash you're on is the hash you're looking for, go on to step 6.
   • Otherwise, if the hash you're on is less than the hash you're looking for, set start to the current position and go to step 2.
   • If the hash you're on is greater than the hash you're looking for, set end to the current position and go to step 2.
6. Skip to the semicolon and trailing space. The unhashed data will be from the current position to the next newline.

This can be easily converted into a while loop with breaks.

Importing it into MySQL with appropriate indices and such would use a similarly (or more, since it's probably packed nicely) efficient algorithm.

Answer 2

Your last solution might be the easiest one to implement as you move the whole performance optimizing to the database (and usually they are optimized for that).

strstr is not useful here as it searches a string, but you know a specific format and can jump and compare more goal oriented. Thing about strncmp , and strchr .

The overhead for reading a single line would be really high (as it is often the case for file IO). So I'd recommend reading a larger chunk and perform your search on that chunk. I'd even think about parallelizing the search by reading the next chunk in another thread and do comparison there aswell.

You can also think about using memory mapped IO instead of the standard C file API. Using this you can leave the whole contents loading to the operating system and don't have to care about caching yourself.

Of course restructuring the data for faster access would help you too. For example insert padding bytes so all datasets are equally long. This will provide you "random" access to your data stream as you can easily calculate the position of the nth entry.

Answer 3

I'd start by splitting the single large file into 65536 smaller files, so that if the hash begins with 0000 it's in the file 00/00data.txt , if the hash begins with 0001 it's in the file 00/01data.txt , etc. If the full file was 12 GiB then each of the smaller files would be (on average) 208 KiB.

Next, separate the hash from the string; such that you've got 65536 "hash files" and 65536 "string files". Each hash file would contain the remainder of the hash (the last 12 digits only, because the first 4 digits aren't needed anymore) and the offset of the string in the corresponding string file. This would mean that (instead of 65536 files at an average of 208 KiB each) you'd have 65536 hash files at maybe 120 KiB each and 65536 string files at maybe 100 KiB each.

Next, the hash files should be in a binary format. 12 hexadecimal digits costs 48 bits (not 12*8=96-bits). This alone would halve the size of the hash files. If the strings are aligned on a 4 byte boundary in the strings file then a 16-bit "offset of the string / 4" would be fine (as long as the string file is less than 256 KiB). Entries in the hash file should be sorted in order, and the corresponding strings file should be in the same order.

After all these changes; you'd use the highest 16-bits of the hash to find the right hash file, load the hash file and do a binary search. Then (if found) you'd get the offset for the start of the string (in the strings file) from entry in the hash file, plus get the offset for the next string from next entry in the hash file. Then you'd load data from the strings file, starting at the start of the correct string and ending at the start of the next string.

Finally, you'd implement a "hash file cache" in memory. If your application can allocate 1.5 GiB of RAM, then that'd be enough to cache half of the hash files. In this case (half the hash files cached) you'd expect that half the time the only thing you'd need to load from disk is the string itself (eg probably less than 20 bytes) and the other half the time you'd need to load the hash file into the cache first (eg 60 KiB); so on average for each lookup you'd be loading about 30 KiB from disk. Of course more memory is better (and less is worse); and if you can allocate more than about 3 GiB of RAM you can cache all of the hash files and start thinking about caching some of the strings.

A faster way would be to have a reversible encoding, so that you can convert a string into an integer and then convert the integer back into the original string without doing any sort of lookup at all. For an example; if all your strings use lower case ASCII letters and are a max. of 13 characters long, then they could all be converted into a 64-bit integer and back (as 26^13 < 2^63). This could lead to a different approach - eg use a reversible encoding (with bit 64 of the integer/hash clear) where possible; and only use some sort of lookup (with bit 64 of the integer/hash set) for strings that can't be encoded in a reversible way. With a little knowledge (eg carefully selecting the best reversible encoding for your strings) this could slash the size of your 13 GiB file down to "small enough to fit in RAM easily" and be many orders of magnitude faster.