超快速的搜索整数子串的方法

Question

I am in a situation, where I need to find an integer substring into a range of extremely large strings. I thought of using vector of vectors to store the range of integer strings and similarly I store the integer string to be searched in a vector. Example below:

//vector of 5 vectors
std::vector<std::vector<int>> vec(5);
// elements= {10,5,8,23,15,32,12,34,56,55,43,12,33,4}

and the substring into a vector

//vector with integer substring
std::vector<int> vec1;
//elements = {5,8,23}

and I use std::search to perform the search operation over the vector of vectors to find the vector , something like this

for( int i = 0; i < vec.size(); i++) // searching read into 
   {
     auto pos = std::search(vec[i].begin(), vec[i].end(), vec1.begin(), vec1.end());
// some more code
}

On testing it took about 1m to search for 1000 strings from the range of 10 vectors each of length 500000 .

There are some data structures that are ultra fast such as unordered_map , but i doubt to use the data structure for my data. I would appreciate any suggestion or links to any container or data structure that are efficient in terms of both time and space.

Note:

1) There is no possibility to sort the data, as I loose the data representation by sorting.

2) I am not searching for individual items, indeed for substrings of integers.

Edit

The original length of string may be 100000000 in each vector and the length of substrings 100 , 1million in number.

Answer 1

Here's my attempt at a fast solution -- on my 2.7GHz Mac mini it is able to find the locations of the 1000 "substrings" in 1357 milliseconds. It does this by first building up an index of all the locations where each integer appears in the big vectors, so that for each of the substrings it doesn't have to search everywhere, but instead only in locations where that substring might actually start. One caveat is that the index takes up quite a bit of extra RAM, and takes some time to build; so this may or may not be a practical solution, depending on your use case. (but note that it only has to be built once, unless/until you move on to searching a different set of big vectors)

#include <algorithm>
#include <vector>
#include <cmath>
#include <cstdint>
#include <chrono>
#include <iostream>
#include <unordered_map>

using namespace std;

// Store a vector index and an offset into the vector efficiently
// Supports up to 256 vectors and offsets up to 16777216
static inline uint32_t GetVectorLocationKey(uint8_t whichVector, uint32_t offsetIntoVector)
{
   return ((((uint32_t)whichVector)<<24)|offsetIntoVector);
}

static inline void GetVectorLocationFromKey(uint32_t key, uint8_t & retWhichVector, uint32_t & retOffsetIntoVector)
{
   retWhichVector = (key >> 24) & 0xFF;
   retOffsetIntoVector = (key & 0xFFFFFF);
}

static inline bool SubstringExistsAtOffset(const int * bigVector, const vector<int> & substring)
{
   const int * smallVector = &substring[0];
   const size_t subLen = substring.size();
   for (size_t i=0; i<subLen; i++) if (bigVector[i] != smallVector[i]) return false;
   return true;
}

int main(int, char **)
{
   // Create some large vectors to search in
   vector<vector<int> > big_vectors;
   const size_t num_big_vectors = 5;
   const size_t big_vector_size = 500000;
   for (size_t i=0; i<num_big_vectors; i++)
   {
      big_vectors.push_back(vector<int>());
      vector<int> & v = big_vectors.back();
      for (size_t j=0; j<big_vector_size; j++) v.push_back(rand()%100);
   }

   // Pick out some small "substring" vectors to search for within the large vectors
   vector<vector<int> > substrings;
   const size_t num_substrings = 1000;
   const size_t substring_size = 14;
   for (size_t i=0; i<num_substrings; i++)
   {
      substrings.push_back(vector<int>());
      size_t whichBigVector = rand()%num_big_vectors;
      size_t offsetIntoVector = rand()%(big_vector_size-substring_size);
      vector<int> & v = substrings.back();
      const vector<int> & bigVector = big_vectors[whichBigVector];
      for (size_t j=0; j<substring_size; j++) v.push_back(bigVector[offsetIntoVector+j]);
   }

   // Now we'll build up a map so that for any given integer we'll
   // have immediate access to a list of the locations it is at.
   // That way we can jump immediately to those locations rather than
   // having to scan through the entire set of big_vectors
   unordered_map<int, vector<uint32_t> > index;
   for (size_t i=0; i<big_vectors.size(); i++)
   {
      const vector<int> & bigVector = big_vectors[i];
      for (size_t j=0; j<bigVector.size()-substring_size; j++)
      {
         int val = bigVector[j];
         index[val].push_back(GetVectorLocationKey(i, j));
      }
   }

   // Now for the time-critical part:  Let's see how fast we
   // can find our substrings within the larger vectors!
   std::chrono::steady_clock::time_point begin = std::chrono::steady_clock::now();
   vector<vector<uint32_t> > results;
   for (size_t i=0; i<substrings.size(); i++)
   {
      results.push_back(vector<uint32_t>());
      vector<uint32_t> & resultVec = results.back();

      const vector<int> & substring = substrings[i];
      const int firstVal = substring[0];
      const vector<uint32_t> & lookup = index[firstVal];
      for (size_t j=0; j<lookup.size(); j++)
      {
         const uint32_t key = lookup[j];
         uint8_t whichVector;
         uint32_t offsetIntoVector;
         GetVectorLocationFromKey(key, whichVector, offsetIntoVector);

         const vector<int> & bigVector = big_vectors[whichVector];
         if (SubstringExistsAtOffset(&bigVector[offsetIntoVector], substring)) resultVec.push_back(key);
      }
   }
   std::chrono::steady_clock::time_point end = std::chrono::steady_clock::now();

   cout << " Total time spent finding " << substrings.size() << " substrings was " << std::chrono::duration_cast<std::chrono::milliseconds>(end-begin).count() << " milliseconds." << std::endl;

   cout << endl << endl << "RESULTS:" << endl;
   for(size_t i=0; i<results.size(); i++)
   {
      const vector<uint32_t> & result = results[i];
      for (size_t j=0; j<result.size(); j++)
      {
         const uint32_t key = result[j];
         uint8_t whichVector;
         uint32_t offsetIntoVector;
         GetVectorLocationFromKey(key, whichVector, offsetIntoVector);

         cout << "An instance of substring #" << i << " was found in bigVector #" << (int)whichVector << " at offset " << offsetIntoVector << endl;

         // Let's just double-check that the substring actually exists where I said it did
         // It would be embarrassing to find out I'm not actually finding them correctly :P
         const vector<int> & bigVector = big_vectors[whichVector];
         const vector<int> & substring = substrings[i];
         for (size_t k=0; k<substring.size(); k++)
         {
            if (bigVector[offsetIntoVector+k] != substring[k]) cout << "ERROR BAD RESULT in substring #" << i << " at offset " << k << endl;
         }
      }
   }
}