Lucene 4.1中的简单代码抛出NullPointerException

Question

Tokenizer tokenizer 
           = new org.apache.lucene.analysis.standard.StandardTokenizer
                                    (Version.LUCENE_41,new StringReader("!!!"));
tokenizer.incrementToken();

Throws

java.lang.NullPointerException
    at org.apache.lucene.analysis.standard.std34.StandardTokenizerImpl34.zzRefill(StandardTokenizerImpl34.java:812)
    at org.apache.lucene.analysis.standard.std34.StandardTokenizerImpl34.getNextToken(StandardTokenizerImpl34.java:1019)
    at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:180)

This code was working okay with Lucene 36, why is it thorweing an exception

Answer 1

I had the exact same problem as you (ie NullPointerException after upgrading to 4.1 from 3.6), although I am using a org.apache.lucene.analysis.TokenStream . After digging around in the Lucene source, I found that calling the reset method before calling incrementToken addresses the issue.

You can have a look at the documentation for the previous version here:

https://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/api/all/org/apache/lucene/analysis/standard/StandardTokenizer.html#reset(java.io.Reader)

Answer 2

Perhaps you replaced the text passed to the StringReader in your production code when you posted it here, but "!!!" seems like something that could lead to odd behavior to me. Since the string is entirely punctuation, I don't believe there are any tokens there for it to consume. I'm honestly not sure what .incrementToken() should do when called with an input that has no tokens in it.

Have you tried it using an input with valid tokens?