从 spacy 对象中删除命名实体

Question

I am trying to remove Named Entities from the document using Spacy. I didn't find any troubles to recognize the named entities. used this code:

ne = [(ent.text, ent.label_) for ent in doc.ents]
print(ne)
persons = [ent.text for ent in doc.ents if ent.label_ == 'PERSON']
print(persons)

Output:

'Timothy D. Cook',
 'Peter',
 'Peter',
 'Benjamin A. Reitzes',
 'Timothy D. Cook',
 'Steve Milunovich',
 'Steven Mark Milunovich',
 'Peter',
 'Luca Maestri'

But then I am trying to use this chunk to actually remove them from the document:

text_no_namedentities = []

ents = [e.text for e in doc.ents]
for item in doc:
    if item.text in ents:
        pass
    else:
        text_no_namedentities.append(item.text)
print(" ".join(text_no_namedentities))

It does not work, since the NE are n-grams. And if I just check the contents of a little chunk of spacy object it is as follows:

for item in doc:
    print(item.text)

iPad
has
a
78
%
Steve
Milunovich
share
of
the
U.S.
commercial
tablet
market

So the spacy object is tokenized. Hence I can't remove the NEs with my code above. Any ideas on how I can remove all the named entities from the object?

Answer 1

The condition you want to check on is

if item.ent_type:

This will evaluate to True if the item ("token") is part of a named entity. token.ent_type will be a hash ID of the actual type of the entity, which you can query with token.ent_type_ (note the _).

This would be the code I'd use:

    text_no_namedentities = ""
    for token in doc:
        if not token.ent_type:
            text_no_namedentities += token.text
            if token.whitespace_:
                text_no_namedentities += " "

Note that you can use token.whitespace_ to determine whether or not the original token in the original sentence was followed by a space or not.

For more information, see the docs on Token here .

FYI - for the future, it would be more convenient to include a working minimal snippet of your code, instead of just parts of it.