[英]Result Difference in Stanford NER tagger NLTK (python) vs JAVA
我使用python和java來運行斯坦福NER標記器,但我看到結果的差異。
例如,當我輸入句子“參與使用ERwin作為主要軟件的數據建模的所有方面。”,
JAVA結果:
"ERwin": "PERSON"
Python結果:
In [6]: NERTagger.tag("Involved in all aspects of data modeling using ERwin as the primary software for this.".split())
Out [6]:[(u'Involved', u'O'),
(u'in', u'O'),
(u'all', u'O'),
(u'aspects', u'O'),
(u'of', u'O'),
(u'data', u'O'),
(u'modeling', u'O'),
(u'using', u'O'),
(u'ERwin', u'O'),
(u'as', u'O'),
(u'the', u'O'),
(u'primary', u'O'),
(u'software', u'O'),
(u'for', u'O'),
(u'this.', u'O')]
Python nltk包裝器無法將“ERwin”作為PERSON捕獲。
這里有趣的是Python和Java使用2015-04-20發布的相同訓練數據(english.all.3class.caseless.distsim.crf.ser.gz)。
我的最終目標是讓python以與Java相同的方式工作。
我在nltk.tag中查看StanfordNERTagger,看看有什么我可以修改的。 下面是包裝代碼:
class StanfordNERTagger(StanfordTagger):
"""
A class for Named-Entity Tagging with Stanford Tagger. The input is the paths to:
- a model trained on training data
- (optionally) the path to the stanford tagger jar file. If not specified here,
then this jar file must be specified in the CLASSPATH envinroment variable.
- (optionally) the encoding of the training data (default: UTF-8)
Example:
>>> from nltk.tag import StanfordNERTagger
>>> st = StanfordNERTagger('english.all.3class.distsim.crf.ser.gz') # doctest: +SKIP
>>> st.tag('Rami Eid is studying at Stony Brook University in NY'.split()) # doctest: +SKIP
[('Rami', 'PERSON'), ('Eid', 'PERSON'), ('is', 'O'), ('studying', 'O'),
('at', 'O'), ('Stony', 'ORGANIZATION'), ('Brook', 'ORGANIZATION'),
('University', 'ORGANIZATION'), ('in', 'O'), ('NY', 'LOCATION')]
"""
_SEPARATOR = '/'
_JAR = 'stanford-ner.jar'
_FORMAT = 'slashTags'
def __init__(self, *args, **kwargs):
super(StanfordNERTagger, self).__init__(*args, **kwargs)
@property
def _cmd(self):
# Adding -tokenizerFactory edu.stanford.nlp.process.WhitespaceTokenizer -tokenizerOptions tokenizeNLs=false for not using stanford Tokenizer
return ['edu.stanford.nlp.ie.crf.CRFClassifier',
'-loadClassifier', self._stanford_model, '-textFile',
self._input_file_path, '-outputFormat', self._FORMAT, '-tokenizerFactory', 'edu.stanford.nlp.process.WhitespaceTokenizer', '-tokenizerOptions','\"tokenizeNLs=false\"']
def parse_output(self, text, sentences):
if self._FORMAT == 'slashTags':
# Joint together to a big list
tagged_sentences = []
for tagged_sentence in text.strip().split("\n"):
for tagged_word in tagged_sentence.strip().split():
word_tags = tagged_word.strip().split(self._SEPARATOR)
tagged_sentences.append((''.join(word_tags[:-1]), word_tags[-1]))
# Separate it according to the input
result = []
start = 0
for sent in sentences:
result.append(tagged_sentences[start:start + len(sent)])
start += len(sent);
return result
raise NotImplementedError
或者,如果是因為使用了不同的分類器(在java代碼中,它似乎使用AbstractSequenceClassifier,另一方面,python nltk包裝器使用CRFClassifier。)有沒有一種方法可以在python包裝器中使用AbstractSequenceClassifier?
嘗試在maxAdditionalKnownLCWords
的屬性文件(或命令行)中將maxAdditionalKnownLCWords設置為0,如果可能,也為NLTK設置。 這會禁用一個選項,允許NER系統稍微從測試時間數據中學習,這可能會導致偶爾出現略微不同的結果。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.