斯坦福 NER 和 POS，大数据的多线程

Question

I am trying to use Stanford NER and Stanford POS Tagger to parse about 23000 documents. I have implemented it using the following pseudocode -

`for each in document:
  eachSentences = PunktTokenize(each)
  #code to generate NER Tagger
  #code to generate POS Taggers on the above output`

For a 4 core machine, with 15 GB RAM, the run time just for NER is approximately, 945 hours. I have tried to step up things by using the "threading" library, but I get the following error-

`Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "removeStopWords.py", line 75, in partofspeechRecognition
    listOfRes_new = namedEntityRecognition(listRes[min:max])
  File "removeStopWords.py", line 63, in namedEntityRecognition
    listRes_ner.append(namedEntityRecognitionResume(eachResSentence))
  File "removeStopWords.py", line 50, in namedEntityRecognitionResume
    ner2Tags = ner2.tag(each.title().split())
  File "/home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/tag/stanford.py", line 71, in tag
    return sum(self.tag_sents([tokens]), [])
  File "/home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/tag/stanford.py", line 98, in tag_sents
    os.unlink(self._input_file_path)
OSError: [Errno 2] No such file or directory: '/tmp/tmpvMNqwB'`

I am using NLTK version - 3.2.1, Stanford NER,POS - 3.7.0 jar file , along with the threading module. As far as I can see, this might be due to a thread lock on /tmp. Please correct me if I am wrong, also what is the best way to run the above using threads or a better way to implement it.

I am using the 3 Class Classifier for NER and Maxent POS Tagger

PS - Please ignore the name of the Python file, I still haven't removed the stopwords or the punctuations from the original text.

Edit - Using cProfile, and sorting on cumulative time, I got the following top 20 calls

600792 function calls (595912 primitive calls) in 60.795 seconds

Ordered by: cumulative time
List reduced from 3357 to 20 due to restriction <20>

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.000    0.000   60.811   60.811 removeStopWords.py:1(<module>)
    1    0.000    0.000   58.923   58.923 removeStopWords.py:76(partofspeechRecognition)
   28    0.001    0.000   58.883    2.103 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/tag/stanford.py:69(tag)
   28    0.004    0.000   58.883    2.103 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/tag/stanford.py:73(tag_sents)
   28    0.001    0.000   56.927    2.033 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/internals.py:63(java)
  141    0.001    0.000   56.532    0.401 /usr/lib/python2.7/subprocess.py:769(communicate)
  140    0.002    0.000   56.530    0.404 /usr/lib/python2.7/subprocess.py:1408(_communicate)
  140    0.008    0.000   56.492    0.404 /usr/lib/python2.7/subprocess.py:1441(_communicate_with_poll)
  400   56.474    0.141   56.474    0.141 {built-in method poll}
    1    0.001    0.001   43.522   43.522 removeStopWords.py:69(partofspeechRecognitionRes)
    1    0.000    0.000   15.401   15.401 removeStopWords.py:62(namedEntityRecognition)
    1    0.001    0.001   15.367   15.367 removeStopWords.py:46(namedEntityRecognitionRes)
  141    0.004    0.000    2.302    0.016 /usr/lib/python2.7/subprocess.py:651(__init__)
  141    0.020    0.000    2.287    0.016 /usr/lib/python2.7/subprocess.py:1199(_execute_child)
   56    0.002    0.000    1.933    0.035 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/internals.py:38(config_java)
   56    0.001    0.000    1.931    0.034 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/internals.py:599(find_binary)
  112    0.002    0.000    1.930    0.017 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/internals.py:582(find_binary_iter)
  118    0.009    0.000    1.928    0.016 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/internals.py:453(find_file_iter)
    1    0.001    0.001    1.318    1.318 /usr/lib/python2.7/pickle.py:1383(load)
    1    0.046    0.046    1.317    1.317 /usr/lib/python2.7/pickle.py:851(load) 

Answer 1

It seems like the Python wrapper is the culprit here. Java implementation is not taking as much time. It's takes approximately what @Gabor Angeli mentioned. Try it.

Hope it helps!

Answer 2

May be this is solved already, but still for the people who are trying to speed up Stanford NLP in Python, here is the tried and tested answer.. How to speedup Stanford NLP in Python?

Basically it asks you to run NER server in the backend and call sner library and further to do all Stanford NLP related task.

Found the answer.

Initiate the Stanford NLP Server in background in the folder where Stanford NLP is unzipped.

Portion of the answer given below..

java -Djava.ext.dirs=./lib -cp stanford-ner.jar edu.stanford.nlp.ie.NERServer -port 9199 -loadClassifier ./classifiers/english.all.3class.distsim.crf.ser.gz
Then initiate Stanford NLP Server tagger in Python using sner library.

from sner import Ner
tagger = Ner(host='localhost',port=9199)

Then run the tagger.

%%time
classified_text=tagger.get_entities(text)
print (classified_text)
Output:

    [('My', 'O'), ('name', 'O'), ('is', 'O'), ('John', 'PERSON'), ('Doe', 'PERSON')]
CPU times: user 4 ms, sys: 0 ns, total: 4 ms
Wall time: 18.2 ms