在干草堆中搜索几个等长的针（Python）

Question

I am looking for a way to search a large string for a large number of equal length substrings.

My current method is basically this:

offset = 0
found = []

while offset < len(haystack):
  current_chunk = haystack[offset*8:offset*8+8]
  if current_chunk in needles:
     found.append(current_chunk)
  offset += 1

This is painfully slow. Is there a better python way of doing this?

Answer 1

More Pythonic, much faster:

for needle in needles:
    if needle in haystack:
        found.append(needle)

Edit: With some limited testing here are test results

This algorithm: 0.000135183334351

Your algorithm: 0.984048128128

Much faster.

Answer 2

I think that you can break it up on a multicore and parallelize your search. Something along the lines of:

from multiprocessing import Pool

text = "Your very long string"

"""
A generator function for chopping up a given list into chunks of
length n.
"""
def chunks(l, n):
  for i in xrange(0, len(l), n):
    yield l[i:i+n]

def searchHaystack(haystack, needles):
    offset = 0
    found = []

    while offset < len(haystack):
      current_chunk = haystack[offset*8:offset*8+8]
      if current_chunk in needles:
      found.append(current_chunk)
      offset += 1
    return(needles)

# Build a pool of 8 processes
pool = Pool(processes=8,)

# Fragment the string data into 8 chunks
partitioned_text = list(chunks(text, len(text) / 8))

# Generate all the needles found
all_the_needles = pool.map(searchHaystack, partitioned_text, needles)