通过python从大文件中读取零件文本作为页面

Question

in python, file.readlines() get all line, it's waste if file size is large(several Mb). is there a efficient way to get parts of files as page? usually the part text display as a page in webapp, consider text are to be further decorated.

Answer 1

currently i though of a rough way by byte size:

import os
def getpage(fname, pageindex, pagesize=100, ahead=20):
    """read page roughly by byte size"""
    size = os.path.getsize(fname)
    pagenum = size/pagesize
    f=open(fname,'r')
    pos=pageindex * pagesize
    #ahead some line
    pos -= ahead
    if pos <0: 
        pos = 0
    f.seek(pos)
    f.readline()
    txt = f.read(pagesize)
    txt += f.readline()    
    return txt

it's not fixed lines, some text are loose, some tight. but for moderate pagesize, it's ok for user's view.

Answer 2

You can do this in a very pythonic and efficent way using Generators:

def getPage(fileName, numberOfLinesInAPage):

    f = open(fileName)

    lines = (line.strip() for line in f)
    pageBuffer = []
    for lineNum, eachLine in enumerate(lines,1):
        pageBuffer.append(eachLine)
        if lineNum % numberOfLinesInAPage == 0:
            yield pageBuffer
            pageBuffer = []
    if pageBuffer:
        yield pageBuffer    
    f.close()   

for i in getPage('test.txt',100):
    print i