在 Python3 中不使用 FOR 的模式匹配后返回一个巨大文件的确切行

Question

I am new to Python. My problem here is that: I want to match a pattern against a large file and return matching lines(not just the matched string) from it. I DO NOT want a FOR loop for this as my file is huge. I am using mmap for reading the file.

in the above file, if I search for bhuvi , I should get 2 rows, bhuvi and bhuvi Kumar

I used re.findall() for this, but it just returns the substrings, not the whole lines.

Can someone please suggest what I can do here?

Answer 1

If your input file is huge, you cannot use readlines , but nothing prevents you from reading one line in a loop.

As the file object is iterable, you can write the loop as:

for line in fh:

and process the content of the input line inside the loop.

The file size is not important, as you do not attempt to read all lines at once.

To check for presence of your string ( bhuvi ) in the line use re.search , not re.findall . Actually you don't need any list of matches, it is enough to find a single match (it works quicker).

Below you have an example program ( Python 3.7 ), writing the lines contaning your string, along with the line number:

import re

cnt = 0
with open('input.txt') as fh:
    for line in fh:
        line = line.rstrip()
        cnt += 1
        if re.search('bhuvi', line):
            print(f'{cnt}: {line}')

Note that I used rstrip() to remove the trailing newline, if any.

Edit after your comment:

You wrote that the file to check is huge . So there is a risk that if you try to read it whole into the computer memory, the program runs out of memory.

In such a case you would have to read the file chunk by chunk and perform search in each chunk separately.

There is also a risk that a row with the text you are looking for will be partially read in one chunk and the rest in the next, so you have to take some measure to avoid this in your program.

On the other hand, if there is no other way but using mmap , try something like re.finditer(r'[^\\n]*bhuvi[^\\n]*', map) , ie create an iterator looking for:

1. A sequence of chars other than \\n .
2. Your string.
3. Another sequence of chars other than \\n .

This way the match object returned by the iterator will match the whole line , not your string alone.