Python readline() method caused UnicodeDecodeError

Question

I`m trying to read and extract information of a large txt and to write it in another document, and I get this error: Here is my code:

#Create list with PLZ, city and state
cepfinal = open("cepfinal.txt", "w")    #file to be written

with open("ceptest2.txt", "r") as fp:   #read file
    while True:
        line = fp.readline()
#   print(str(line))
        x = line.split("\t")            #separate all that have double space
        plz = x[0]                      #extract PLZ
#   print(plz)

        y = x[1]
        mun = y.split("/")              #separe city from state
#   print(mun)
        plzmun = [plz] + mun
#   print(plzmun)
        final = plzmun.pop(2)           #remove state
        plzmun = " ".join(plzmun)       #create string
        print(plzmun)
        cepfinal.write(plzmun + "\n")

fp.close()

It is a 45 Gb file, so I suppose I have a memory issue. Can someone help me to make a lean code?

Answer 1

your problem is with encoding, you can try this to solve your problem

with  open("ceptest2.txt", "r", encoding="utf8") as fp: