在python中的大型文本文件（> 1GB）上查找特定字符串的最有效方法是什么？

Question

I am developing a string filter for huge process log file in distributed system.

These log files are >1GB and contains millions of lines.These logs contains special type of message blocks which are starting from "SMsg{" and end from "}". My program is reading the whole file line by line and put the line numbers which the line contains "SMsg{" to an list.Here is my python method to do that.

 def FindNMsgStart(self,logfile):

        self.logfile = logfile

        lf = LogFilter()

        infile = lf.OpenFile(logfile, 'Input')
        NMsgBlockStart = list()


        for num, line in enumerate(infile.readlines()):
            if re.search('SMsg{', line):                
                NMsgBlockStart.append(num)


        return NMsgBlockStart

This is my lookup function to search any kind of word in the text file.

def Lookup(self,infile,regex,start,end):

        self.infile = infile
        self.regex = regex
        self.start = start
        self.end = end
        result = 0


        for num, line in enumerate(itertools.islice(infile,start,end)):
            if re.search(regex, line):
                result = num + start
                break




        return result        

Then I will get that list and find the end for each starting block through the whole file. Following is my code for find the end.

def FindNmlMsgEnd(self,logfile,NMsgBlockStart):

        self.logfile = logfile
        self.NMsgBlockStart = NMsgBlockStart

        NMsgBlockEnd = list()

        lf = LogFilter() 

        length = len(NMsgBlockStart)


        if length > 0:
            for i in range (0,length):
                start=NMsgBlockStart[i]                
                infile = lf.OpenFile(logfile, 'Input')
                lines = lf.LineCount(logfile, 'Input')
                end = lf.Lookup(infile, '}', start, lines+1)               
                NMsgBlockEnd.append(end)


            return NMsgBlockEnd
        else:
            print("There is no Normal Message blocks.") 

But those method are never efficient enough to handle huge files. The program is running long time without a result.

1. Is there efficient way to do this?
2. If yes, How could I do this?

I am doing another filters too , But first I need to find a solution for this basic problem.I am really new to python. Please help me.

Answer 1

I see a couple of issues that are slowing your code down.

The first seems to be a pretty basic error. You're calling readlines on your file in the FindNMsgStart method, which is going to read the whole file into memory and return a list of its lines.

You should just iterate over the lines directly by using enumerate(infile) . You do this properly in the other functions that read the file, so I suspect this is a typo or just a simple oversight.

The second issue is a bit more complicated. It involves the general architecture of your search.

You're first scanning the file for message start lines, then searching for the end line after each start. Each end-line search requires re-reading much of the file, since you need to skip all the lines that occur before the start line. It would be a lot more efficient if you could combine both searches into a single pass over the data file.

Here's a really crude generator function that does that:

def find_message_bounds(filename):
    with open(filename) as f:
        iterator = enumerate(f)
        for start_line_no, start_line in iterator:
            if 'SMsg{' in start_line:
                for end_line_no, end_line in iterator:
                    if '}' in end_line:
                        yield start_line_no, end_line_no
                        break

This function yields start, end line number tuples, and only makes a single pass over the file.

I think you can actually implement a one-pass search using your Lookup method, if you're careful with the boundary variables you pass in to it.

def FindNmlMsgEnd(self,logfile,NMsgBlockStart):

    self.logfile = logfile
    self.NMsgBlockStart = NMsgBlockStart

    NMsgBlockEnd = list()

    lf = LogFilter() 
    infile = lf.OpenFile(logfile, 'Input')
    total_lines = lf.LineCount(logfile, 'Input')

    start = NMsgBlockStart[0]
    prev_end = -1
    for next_start in NMsgBlockStart[:1]:
        end = lf.Lookup(infile, '}', start-prev_end-1, next_start-prev_end-1) + prev_end + 1
        NMsgBlockEnd.append(end)

        start = next_start
        prev_end = end

    last_end = lf.Lookup(infile, '}', start-prev_end-1, total_lines-prev_end-1) + prev_end + 1
    NMsgBlockEnd.append(last_end)

    return NMsgBlockEnd

It's possible I have an off-by-one error in there somewhere, the design of the Lookup function makes it difficult to call repeatedly.