Python无法正确读取日志文件，除非我将其内容粘贴到新的文本文件中

Question

I am using open() to read a log file, but I got strange content. If I open the log file by Notepad++, copy the content and paste it in new file, save it as .txt file, open() can read the correct content. Code is:

with open(a_file, 'r') as my_file:
    for line in my_file:
        line_number += 1
        if line_number == 1060 or line_number == 1061:
            print(line)

I tried many methods:

1. cat the log file and redirect to a new text file, no help
2. Open the log file in Notepad++, save as... a new text file, no help
3. Use linux tail command, redirect the output to a new text file, no help
4. Use python codecs to read it as utf-8, error happen "python codec can't decode byte 0xff in position 0: invalid start byte"
5. Open the log file in Notepad++, copy its content, paste in a new file, save to a new text file, it works.

Answer 1

You can't read the file because the it's encoded in UTF-16 , you can tell by the first characters which are the BOM . 0xff is part of the BOM for UTF-16 . So when reading just add encoding='utf16' (or use codecs.open with utf16 in python2)