Python file.read（）方法

Question

I'm reading a text file, line by line, using Python. Each line is of a variable length. The first line could be 10 characters, the next one could be 100; there's no way of telling. Presently, I issue a file.readline() method for each line, process it, and then save it to a database. This method guarantees me one full line of input. I'd like to do this faster however. Is there a way to do a bulk read using the Python file.read() method such that I can guarantee an end-of-line read character when the buffer stops in the middle of a line? What's the best way to handle this?

Answer 1

The typical way to handle these are just:

for line in fileobj:
    process(line)

There's no way to actually specify that read ends at the end of a line. You could do something kludgy with fileobj.seek . Basically, you'd read N bytes, find the last newline in the returned string (using string.rfind ) you could seek backward that many bytes.

---

Of course, if you have sufficient memory, you can read the entire file in at once:

list_of_lines = fileobj.readlines()

---

However, I'm really not positive that you'll see any noticeable speedup here. Are you sure you're not optimizing before you need to?

Answer 2

You can use ..

lines = file_handle.read().split('\n')
# Or 
lines = file_handle.readlines()

Check their documentation for accurate behavior with '\\n' .

Answer 3

The way the encoders I've messed with have done this, is to read whatever's there, or a particular chunk size, note the position of the last newline (.rfind('\\n')), process the data up to that newline, and then store from the newline to the end of the chunk in a list. When reading the next block, you read from the same position as you stopped reading before, and you append the leftover string from before onto it. The performance was reasonable, and it's stable, of course this was for network sockets, where you can't seek backwards, I'm not sure which method would actually perform better on files.