Does PyPy work with NLTK?

Question

Does PyPy work with NLTK, and if so, is there an appreciable performance improvement, say for the bayesian classifier?

While we're at it, do any of the other python environments (shedskin, etc) offer better nlkt performance than cpython?

Answer 1

At least some of NLTK does work with PyPy and there is some performance gain, according to someone on #pypy on freenode. Have you run any tests? Just download PyPy from pypy.org/download.html and instead of "time python yourscript.py data.txt" type "time pypy yourscript.py data.txt".

Answer 2

I got a response via email (Seo, please feel free to respond here) that said:

The main issues are:

PyPy implements Python 2.5. This means adding "from future import with_statement" here and there, rewriting usages of property.setter, and fixing up new in 2.6 library calls like os.walk.

NLTK needs PyYAML. Simply symlinking (or copying) stuffs to pypy-1.4/site-packages work.

And:

Do you have NLTK running with PyPy, and if so are you seeing performance improvements?

Yes, and yes.

So apparently NLTK does run with PyPy and there are performance improvements.

Answer 3

You can run nltk with pypy now. There's a benchmark under pypy 1.8 , although later releases (currently pypy 2.0 is the latest) will perform better still. nltk runs its unit tests under pypy these days, so the nltk developers are ensuring it works.