正则表达式的Python 3.6.3与2.7.3：同一脚本的结果不同

Question

I am running the same script with Python versions 3.6.3 and 2.7.3. The script works fine in 2.7.3, but not in 3.6.3. It seems the difference is in the regular expression portion of my code.

I am searching for some strings in the same external file for both script versions, saving the hits in lists. The len() of the resulting lists are different for the two versions.

I tried to make a MWE that reproduces the error by creating a small file to use for the regexes, but then both versions of Python produce the same output. The only solution I have is to provide the original file. But this is quite a long text file, so you can download it from here: https://ufile.io/jjc56 This file is available for 30 days. I thought perhaps this was better than pasting everything into the question.

This piece of code reproduces the error.

import re

inputfile = "opt-guess-firsttetint-r-h2o.out"
with open(inputfile,"r") as input_file:
    input_string = input_file.read()
    input_file.close()

match_geometry = list(re.findall('CARTESIAN COORDINATES \(ANGSTROEM\)(.*?)CARTESIAN COORDINATES \(A\.U\.\)', input_string, re.DOTALL))

match_energy = list(re.findall('FINAL SINGLE POINT ENERGY(.*?)-------------------------', input_string, re.DOTALL))

print(len(match_geometry))
print(len(match_energy))

Output with Python 3.6.3:

78
77

Output with Python 2.7.3:

188
188

For comparison:

$ grep "CARTESIAN COORDINATES (ANGSTROEM)" externalfile | wc -l
> 188

$ grep "FINAL SINGLE POINT ENERGY" externalfile | wc -l
> 188

If you need more information, please say so!

Answer 1

The main difference between Python 2 and Python 3 is text handling: while in Python 2 text is treated like in bare C, ie a sequence of bytes which happen to match ASCII characters in the range 32-128, that is not true for Python 3 - where the bytes in your file are assumed to be in some text encoding, and decoded to proper unicode character points before being treated in the program.

Likewise, in Python2, regexps operate by default on "byte strings", and on Python 3 on text strings (in Python 2 you can work with text as well if both the expression and the text are 'unicode' objects, rather than 'str')

We'd need more context there, but your problem likely lies on Python 3 reading your text file assuming an incorrect encoding - like, your data is utf-8, but Python is assuming it as Latin 1 - that would read characters out of the ASCII range as incorrect, without giving you an error, since all bytes from 0-255 are valid Latin-1 - but the resulting mojibake would fail the regexp.

Just force a proper encoding="..." to match your file when reading your data and you should be fine.

FYI, one character that would trigger the behavior I described above is "Å" - which I don't find unlikely to occur in this particular case.