Unicodedata.normalize : TypeError: normalize() 参数 2 必须是 str，而不是列表

Question

I am trying to load a file in python. If you run the below code and load a file that contains only English words, it will load just fine.

Listado.txt is a spanish language file that contains the following words : abacá, abadí, abadía, abajeño, abaniquería

Spanish Language contains accented letters (eg é) or special characters ( diacritics ), and here is where the problem lies, when I try to load this file into Python it complains. I would like to be able to normalize the list, or remove accented characters and load the list.

I have tried normalizing using :

unicodedata.normalize('NFD', line).encode('ascii', 'ignore')

and I get the below error :

TypeError: normalize() argument 2 must be str, not list

Code so far :

import random
import string
import unicodedata

#WORDLIST_FILENAME = "words_alpha.txt"
WORDLIST_FILENAME = "listado.txt"

def loadWords():
    print("Loading word list from file...")
    # inFile: file
    inFile = open(WORDLIST_FILENAME, 'r')
    wordlist =[]
    for line in inFile:
        line = line.split()
        wordlist.extend(line)
#        unicodedata.normalize('NFD', line).encode('ascii', 'ignore')
        print(" "), len(wordlist), ("words loaded.")

    return wordlist

Answer 1

As the error says, you are trying to normalize line , which is a list, as you've done line = line.split() earlier. Just normalize the line before you split it into words, as follows:

for line in inFile:
    unicodedata.normalize('NFD', line).encode('ascii', 'ignore')
    line = line.split()
    wordlist.extend(line)
    print(" "), len(wordlist), ("words loaded.")

Alternatively, if you want to extend your wordlist before normalizing, you can do the following using a list comprehension to normalize each word individually.

for line in inFile:
    line = line.split()
    wordlist.extend(line)
    line = [unicodedata.normalize('NFD', x).encode('ascii', 'ignore') for x in line]
    print(" "), len(wordlist), ("words loaded.")