如何从巨大的txt文件中提取模式

Question

I have huge text file which consists of random letters from AZ and I want to extract some characters out. The tricky part is that given the following input:

AFVAJFLDVAJPQDVAJDSNJKVAJGHD

and the pattern VAJ , I want to extract each match until the end of the string. I want the following output:

[ "VAJFLDVAJPQDVAJDSNJKVAJGHD", "VAJPQDVAJDSNJKVAJGHD", "VAJDSNJKVAJGHD", "VAJGHD" ]

Answer 1

You can use str.find() to find the index, where your pattern occurs. You can then slice the string accordingly. An implementaion could look like this:

def find(inp, what):
  matches = []
  while what in inp:
    idx = inp.find(what)
    matches.append(inp[idx:])
    # remove the previous pattern from the string
    inp = inp[idx+len(what):]

  return matches

You can use it with find("AFVAJFLDVAJPQDVAJDSNJKVAJGHD", "VAJ") .

Answer 2

This calls for regular expressions with sub-group matching. ( https://docs.python.org/3.5/library/re.html#match-objects )

My test file data.txt :

QWEEEFVAJFLDVAJPQDVAJDSNJKVAJGHD
AFVAJFLDVAJPQDVAJDSNJKHFGHERQWFS
ONLY_TWO_VAJsOOVAJ123VAQQWERTY
START_VAJs_with_more_VAJ123VAJ_space_between
AAPVAJRCGVAJJKYVAJJJJJJJJVAJOOOO
AAPVAJRCGVAJJKYVAJJJJJJJJQQQOOOOO

Python code:

import re

pattern = "VAJ"

re_str = pattern + "..." + "(" + pattern + "..." +"(" +  pattern + "(.*)))"
regex = re.compile(re_str)

regex_extra = re.compile(pattern + ".*")

for line in open("data.txt"):
    line = line.strip()
    match = regex.search(line)
    if match:
        result = list()
        result.append(match.group(0))   # entire regex match
        result.append(match.group(1))   # outer regex parenthesis'ed group
        result.append(match.group(2))   # middle regex parenthesis'ed group

        # Most inner regex parenthesis'ed group contains rest of the line.
        # Use this to find extra pattern.
        #
        the_rest = match.group(3)
        match_extra = regex_extra.search(the_rest)
        if match_extra:   # If one more <pattern> in the rest of the line
            result.append(match_extra.group(0))   # add it to the result list

        # Output         
        print(result)

Ouput:

['VAJFLDVAJPQDVAJDSNJKVAJGHD', 'VAJPQDVAJDSNJKVAJGHD', 'VAJDSNJKVAJGHD', 'VAJGHD']
['VAJFLDVAJPQDVAJDSNJKHFGHERQWFS', 'VAJPQDVAJDSNJKHFGHERQWFS', 'VAJDSNJKHFGHERQWFS']
['VAJRCGVAJJKYVAJJJJJJJJVAJOOOO', 'VAJJKYVAJJJJJJJJVAJOOOO', 'VAJJJJJJJJVAJOOOO', 'VAJOOOO']
['VAJRCGVAJJKYVAJJJJJJJJQQQOOOOO', 'VAJJKYVAJJJJJJJJQQQOOOOO', 'VAJJJJJJJJQQQOOOOO']

Hugeness of the file is not a problem with this code, as long as the longest line fits a few times in memory it should be OK.