使用 spacy 时如何解决属性错误？

Question

I am using spacy for natural language processing in german. But I am running into this error:

AttributeError: 'str' object has no attribute 'text'

This is the text data I am working with:

tex = ['Wir waren z.B. früher auf\'m Fahrrad unterwegs in München (immer nach 11 Uhr).',
        'Nun fahren wir öfter mit der S-Bahn in München herum. Tja. So ist das eben.',
        'So bleibt mir nichts anderes übrig als zu sagen, vielen Dank für alles.',
        'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.']

My code:

data = [re.sub(r"\"", "", i) for i in tex]
data1 = [re.sub(r"\“", "", i) for i in data]
data2 = [re.sub(r"\„", "", i) for i in data1]

nlp = spacy.load('de')
spacy_doc1 = []
for line in data2:
    spac = nlp(line)
    lem = [tok.lemma_ for tok in spac]
    no_punct = [tok.text for tok in lem if re.match('\w+', tok.text)]
    no_numbers = [tok for tok in no_punct if not re.match('\d+', tok)]

I am writing every string in a seperate list, because I need to assign the result of the processing to the original specific string.

I also understand that the result that is written into lem is not in a format anymore that spacy can process.

So how can I do this correctly?

Answer 1

The problem here lies in the fact that SpaCy's token.lemma_ returns a string, and that strings have no text attribute (as the error states).

I suggest doing the same as you did when you wrote:

no_numbers = [tok for tok in no_punct if not re.match('\\d+', tok)]

The only difference with this line in your code would be that you'd have to include the special string "-PRON-" in case you encounter English pronouns:

import re
import spacy

# using the web English model for practicality here
nlp = spacy.load('en_core_web_sm')

tex = ['I\'m going to get a cat tomorrow',
        'I don\'t know if I\'ll be able to get him a cat house though!']

data = [re.sub(r"\"", "", i) for i in tex]
data1 = [re.sub(r"\“", "", i) for i in data]
data2 = [re.sub(r"\„", "", i) for i in data1]

spacy_doc1 = []

for line in data2:
    spac = nlp(line)
    lem = [tok.lemma_ for tok in spac]
    no_punct = [tok for tok in lem if re.match('\w+', tok) or tok in ["-PRON-"]]
    no_numbers = [tok for tok in no_punct if not re.match('\d+', tok)]
    print(no_numbers)

# > ['-PRON-', 'be', 'go', 'to', 'get', 'a', 'cat', 'tomorrow']
# > ['-PRON-', 'do', 'not', 'know', 'if', '-PRON-', 'will', 'be', 'able', 'to', 'get', '-PRON-', 'a', 'cat', 'house', 'though']

Please tell me if this solved your problem as I may have misunderstood your issue.