Make the readline method of Python recognize both end-of-line variations?

Question

I am writing a Python file that needs to read in several files of different types. I am reading the files in line by line with the traditional for line in f after using f = open("file.txt", "r") .

This doesn't seem to be working for all files. My guess is some files end with different encodings (such as \\r\\n versus just \\r). I can read the whole file in and do a string split on \\r, but that is hugely costly and I'd rather not. Is there a way to make the readline method of Python recognize both end-of-line variations?

Answer 1

Use the universal newline support -- see http://docs.python.org/library/functions.html#open

In addition to the standard fopen() values mode may be 'U' or 'rU'. Python is usually built with universal newline support; supplying 'U' opens the file as a text file, but lines may be terminated by any of the following: the Unix end-of-line convention '\\n', the Macintosh convention '\\r', or the Windows convention '\\r\\n'. All of these external representations are seen as '\\n' by the Python program. If Python is built without universal newline support a mode with 'U' is the same as normal text mode. Note that file objects so opened also have an attribute called newlines which has a value of None (if no newlines have yet been seen), '\\n', '\\r', '\\r\\n', or a tuple containing all the newline types seen.

Answer 2

You can try to use a generator approach to read the lines by yourself and ignore any EOL characters:

def readlines(f):
    line = []
    while True:
        s = f.read(1)
        if len(s) == 0:
            if len(line) > 0:
                yield line
            return
        if s in ('\r','\n'):
            if len(line) > 0:
                yield line
            line = []
        else:
            line.append(s)

for line in readlines(yourfile):
    # ...