[英]Extracting names from a text file using Spacy
我有一个文本文件,其中包含如下所示的行:
Electronically signed : Wes Scott, M.D.; Jun 26 2010 11:10AM CST
The patient was referred by Dr. Jacob Austin.
Electronically signed by Robert Clowson, M.D.; Janury 15 2015 11:13AM CST
Electronically signed by Dr. John Douglas, M.D.; Jun 16 2017 11:13AM CST
The patient was referred by
Dr. Jayden Green Olivia.
我想使用 Spacy 提取所有名称。 我正在使用 Spacy 的词性标注和实体识别,但无法获得成功。 我可以知道它是怎么做到的吗? 任何帮助将不胜感激
我以这种方式使用一些代码:
import spacy
nlp = spacy.load('en')
document_string= " Electronically signed by stupid: Dr. John Douglas, M.D.;
Jun 13 2018 11:13AM CST"
doc = nlp(document_string)
for sentence in doc.ents:
print(sentence, sentence.label_)
所有模型的问题在于它们没有 100% 的准确性,即使使用更大的 model 也无助于识别日期。 以下是 NER 模型的准确度值(F 分数、精确度、召回率)——它们都在 86% 左右。
document_string = """
Electronically signed : Wes Scott, M.D.; Jun 26 2010 11:10AM CST
The patient was referred by Dr. Jacob Austin.
Electronically signed by Robert Clowson, M.D.; Janury 15 2015 11:13AM CST
Electronically signed by Dr. John Douglas, M.D.; Jun 16 2017 11:13AM CST
The patient was referred by
Dr. Jayden Green Olivia.
"""
小 model 两个日期项目被标记为“人”:
import spacy
nlp = spacy.load('en')
sents = nlp(document_string)
[ee for ee in sents.ents if ee.label_ == 'PERSON']
# Out:
# [Wes Scott,
# Jun 26,
# Jacob Austin,
# Robert Clowson,
# John Douglas,
# Jun 16 2017,
# Jayden Green Olivia]
对于更大的 model en_core_web_md
,结果在精度方面甚至更差,因为存在三个错误分类的实体。
nlp = spacy.load('en_core_web_md')
sents = nlp(document_string)
# Out:
#[Wes Scott,
# Jun 26,
# Jacob Austin,
# Robert Clowson,
# Janury,
# John Douglas,
# Jun 16 2017,
# Jayden Green Olivia]
我还尝试了其他模型( xx_ent_wiki_sm
、 en_core_web_md
),它们也没有带来任何改进。
在这个小例子中,不仅文档看起来结构清晰,而且错误分类的实体都是日期。 那么为什么不将初始的 model 与基于规则的组件结合起来呢?
好消息是在 Spacy 中:
可以通过多种方式组合统计和基于规则的组件。 基于规则的组件可用于提高统计模型的准确性
(来自https://spacy.io/usage/rule-based-matching#models-rules )
因此,通过遵循该示例并使用dateparser库(人类可读日期的解析器),我将一个基于规则的组件放在一起,该组件在该示例中运行良好:
from spacy.tokens import Span
import dateparser
def expand_person_entities(doc):
new_ents = []
for ent in doc.ents:
# Only check for title if it's a person and not the first token
if ent.label_ == "PERSON":
if ent.start != 0:
# if person preceded by title, include title in entity
prev_token = doc[ent.start - 1]
if prev_token.text in ("Dr", "Dr.", "Mr", "Mr.", "Ms", "Ms."):
new_ent = Span(doc, ent.start - 1, ent.end, label=ent.label)
new_ents.append(new_ent)
else:
# if entity can be parsed as a date, it's not a person
if dateparser.parse(ent.text) is None:
new_ents.append(ent)
else:
new_ents.append(ent)
doc.ents = new_ents
return doc
# Add the component after the named entity recognizer
# nlp.remove_pipe('expand_person_entities')
nlp.add_pipe(expand_person_entities, after='ner')
doc = nlp(document_string)
[(ent.text, ent.label_) for ent in doc.ents if ent.label_=='PERSON']
# Out:
# [(‘Wes Scott', 'PERSON'),
# ('Dr. Jacob Austin', 'PERSON'),
# ('Robert Clowson', 'PERSON'),
# ('Dr. John Douglas', 'PERSON'),
# ('Dr. Jayden Green Olivia', 'PERSON')]
尝试这个:
import spacy
en = spacy.load('en')
sents = en(open('input.txt').read())
people = [ee for ee in sents.ents if ee.label_ == 'PERSON']
试试这个,它工作正常,我正在使用Jupyter
text=open('input.txt').read()
nlp = spacy.load("en_core_web_lg")
sents = nlp(open('input.txt').read()).to_json()
people=[ee for ee in sents['ents'] if ee['label'] == 'PERSON']
print(people)
for pps in people:
print(text[pps['start']:pps['end']])
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.