Lucene TokenStream异常

Question

I've got exactly the same problem of this thread , so I'm opening a new question. Sorry to all for answered to the linked thread, BTW.

So: I'm trying to avoid the java.lang.IllegalStateException: TokenStream contract violation.

I had a code very similar to the above linked:

protected TokenStreamComponents createComponents( String fieldName, Reader reader ) {

String token;
CharArraySet stopWords = new CharArraySet( Version.LUCENE_48, 0, false );
stopWords.addAll( StopAnalyzer.ENGLISH_STOP_WORDS_SET );
keepWords.addAll( getKeepWordList() );

Tokenizer source = new StandardTokenizer( Version.LUCENE_48, reader );
TokenStream filter = new StandardFilter( Version.LUCENE_48, source );
filter = new StopFilter( Version.LUCENE_48, filter, stopWords );
ShingleFilter shiFilter = new ShingleFilter( filter, 2, 3 );
CharTermAttribute cta = shiFilter.addAttribute( CharTermAttribute.class );

try {
    shiFilter.reset();
    while( shiFilter.incrementToken() ) {

        token = cta.toString();
        System.out.println( token );
    }
    shiFilter.end();
    shiFilter.close();
} 
catch ( IOException ioe ) {

    ioe.printStackTrace();
}
return new TokenStreamComponents( source, filter );
}

I don't understand the proposed solution: what does it means "simply construct a new TokenStream" or "Resetting the reader"? I've tried both solution, like adding:

source.setReader( reader );

Or change to:

filter = new StopFilter( Version.LUCENE_48, filter, stopWords );
ShingleFilter shiFilter = new ShingleFilter( filter, 2, 3 );

But the error last. Any suggestion?

Answer 1

I didn't understand exactly what you are trying to do. I believe that you would want to get the get the bigrams and trigrams from the token stream in addition to the unigrams. The following code fragment (which I developed after a bit of cleaning of your code) runs for me and is the standard way of doing this.

import java.io.*;
import org.apache.lucene.analysis.core.*;
import org.apache.lucene.analysis.*;
import org.apache.lucene.analysis.ngram.NGramTokenFilter;
import org.apache.lucene.analysis.shingle.ShingleFilter;
import org.apache.lucene.analysis.standard.*;
import org.apache.lucene.util.*;
import org.apache.lucene.analysis.util.*;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;

class TestAnalyzer extends Analyzer {

TestAnalyzer() {
    super();
}

protected TokenStreamComponents createComponents( String fieldName, Reader reader ) {
    String token;
    TokenStream result = null;

    Tokenizer source = new StandardTokenizer( Version.LUCENE_CURRENT, reader );
    result = new ShingleFilter(source, 2, 3);

    return new TokenStreamComponents( source, result );
}
}

public class LuceneTest {

public static void main(String[] args) {

    TestAnalyzer analyzer = new TestAnalyzer();

    try {
        TokenStream stream = analyzer.tokenStream("field", new StringReader("This is a damn test."));
        CharTermAttribute termAtt = stream.addAttribute(CharTermAttribute.class);

        stream.reset();

        // print all tokens until stream is exhausted
        while (stream.incrementToken()) {
            System.out.println(termAtt.toString());
        }

        stream.end();
        stream.close();
     }
     catch (Exception ex) {
         ex.printStackTrace();
     }
}

}

Answer 2

Your problem is that you are consuming everything in the filter and closing it before passing it back from the createComponents call.

You are, I assume, trying to debug how analysis works with this:

try {
    shiFilter.reset();
    while( shiFilter.incrementToken() ) {
        token = cta.toString();
        System.out.println( token );
    }
    shiFilter.end();
    shiFilter.close();
} 
catch ( IOException ioe ) {
    ioe.printStackTrace();
}

Note, though, that when you are done with it, shiFilter is at the end of the stream, and has been closed. You now pass it back from the method, in the TokenStreamComponents, when Lucene will now attempt to use it to index the documents. It will call reset() first, and will throw the indicated exception for trying to use an already closed resource.

If you want to debug this, I'd recommend just creating an instance of your custom analyzer, and call analyzer.tokenStream to get the stream for debugging output. If you really need to debug by iterating through the filter instance, rather than the analyzer, you'll need to build a separate one, rather than consuming the stream in createComponents .