Fastest check if line starts with value in list?

Question

I have thousands of values (as list but might convert to dictionary or so if that helps) and want to compare to files with millions of lines . What I want to do is to filter lines in files to only the ones starting with values in the list .

What is the fastest way to do it?

My slow code:

  for line in source_file:
    # Go through all IDs
    for id in my_ids:
      if line.startswith(str(id) + "|"):
        #replace comas with semicolons and pipes with comas
        target_file.write(line.replace(",",";").replace("|",","))

Answer 1

If you sure the line starts with id + "|", and "|" will not present in id, I think you could play some trick with "|". For example:

my_id_strs = map(str, my_ids)
for line in source_file:
    first_part = line.split("|")[0]
    if first_part in my_id_strs:
        target_file.write(line.replace(",",";").replace("|",","))

Hope this will help :)

Answer 2

Use string.translate to do replace. Also you can do a break after you match the id.

from string import maketrans

trantab = maketrans(",|", ";,")

ids = ['%d|' % id for id in my_ids]

for line in source_file:
    # Go through all IDs
    for id in ids:
      if line.startswith(id):
        #replace comas with semicolons and pipes with comas
        target_file.write(line.translate(trantab))
        break

or

from string import maketrans

#replace comas with semicolons and pipes with comas
trantab = maketrans(",|", ";,")
idset = set(my_ids)

for line in source_file:
    try:
        if line[:line.index('|')] in idset:            
            target_file.write(line.translate(trantab))
    except ValueError as ve:
        pass

Answer 3

Use a regular expression. Here is an implementation:

import re

def filterlines(prefixes, lines):
    pattern = "|".join([re.escape(p) for p in prefixes])
    regex = re.compile(pattern)
    for line in lines:
        if regex.match(line):
            yield line

We build and compile a regular expression first (expensive, but once only), but then the matching is very, very fast.

Test code for the above:

with open("/usr/share/dict/words") as words:
    prefixes = [line.strip() for line in words]

lines = [
    "zoo this should match",
    "000 this shouldn't match",
]

print(list(filterlines(prefixes, lines)))