Stanford CoreNLP：-nthreads标志使所有ner值均为O

Question

I'm trying to figure out why adding the -nthreads {int} argument to the Stanford CoreNLP (version stanford-corenlp-full-2015-12-09 ) causes all tokens to have an output NER value of O (= not a named entity).

As a simple example, create a file called sample-file.txt and make its contents say Samuel Huntington . Then run:

java -Xmx6g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner -file sample-file.txt -outputFormat json

This will generate the expected output, with "Samuel" recognized as a person:

{
  "sentences": [
    {
      "index": 0,
      "parse": "SENTENCE_SKIPPED_OR_UNPARSABLE",
      "tokens": [
        {
          "index": 1,
          "word": "Samuel",
          "originalText": "Samuel",
          "lemma": "Samuel",
          "characterOffsetBegin": 0,
          "characterOffsetEnd": 6,
          "pos": "NNP",
          "ner": "PERSON",
          "before": "",
          "after": " "
        }, ...

If you add -nthreads 8 to the command above, however, the output does not indicate Samuel is a person. Full command:

java -Xmx6g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner -file sample-file.txt -outputFormat json -nthreads 8 

Which generates:

{
  "sentences": [
    {
      "index": 0,
      "parse": "SENTENCE_SKIPPED_OR_UNPARSABLE",
      "tokens": [
        {
          "index": 1,
          "word": "Samuel",
          "originalText": "Samuel",
          "lemma": "Samuel",
          "characterOffsetBegin": 0,
          "characterOffsetEnd": 6,
          "pos": "NNP",
          "ner": "O",
          "before": "",
          "after": " "
        },

For what it's worth, -nthread {int} (that is, thread with no s) resolves the problem, so I can just use this command. I'll leave this question here in case others try using the -nthreads flag however.

PS Here's my CLASSPATH (obtained from echo $CLASSPATH ), which contains only the Stanford CoreNLP distribution I downloaded last week: /Users/dduhaime/Desktop/everett/wiki_facts/stanford-corenlp-full-2015-12-09/*:

Answer 1

I ran this command:

java -Xmx6g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner -file sample-file.txt -outputFormat json

with sample-file.txt just containing the text "Samuel Huntington" and it found both tokens to be PERSON.

My CLASSPATH contains all the jars in the distribution.

Two things you should start with are re-download the 2015-12-09 distribution (we sometimes make small changes to this before it its finalized) just to make sure you have the latest version and investigate in your Python code what the CLASSPATH is. Perhaps before running the java command add an "echo $CLASSPATH ; " Because looking at your java command you aren't setting the CLASSPATH, so it is not totally clear what the CLASSPATH that is being used is.

Answer 2

使用-nthread {int}代替-nthreads {int}生成预期的NER输出。