Lucene Search for japanese characters

Question

I have implemented lucene for my application and it works very well unless you have introduced something like japanese characters.

The problem is that if I have japanese string こんにちは、このバイネイです and I search with こ that is the first character than it works well whereas if I use more than one japanese character(こんにち)in search token search fails and there is no document found.

Are japanese characters supported in lucene? what are the settings to be done to get it working?

Answer 1

Built-in analyzer of lucene does not support japanese.

You need to install some analyzer like sen , which is java port of mecab , quite popular japanese analyzer, and its fast.

There is 2 sub types called

1. CJKAnalyzer, which support chinese, and korean too, and using bi-gram method
2. JapaneseAnalyzer, which only support japanese, using Morphological Analyzer and supposed to be very fast.

Answer 2

I don't think there can be an analyzer that will work for all languages. The problem is that different languages have different rules about word boundaries and stemming (for example, the Thai language doesn't use spaces at all to separate words). Or if there is, I certainly wouldn't want to be the maintainer!

What you will need to do is "tag" blocks of text as one language or another and use the correct analyzer for that particular language. You can attempt to detect the language "automatically" by doing character analysis (ie text using predominantly Japanese Katakana is likely Japanese)

Answer 3

You should use the new Japanese analyzers recently released in Lucene 3.6.0. They are based on the excellent Kuromoji morphological analyzer recently donated to Lucene in LUCENE-3305 .

Docs are a bit sparse as of this writing, so here are a few more links…

• If you use Solr, here's a sample schema that will work on Websolr .
• Slides from my presentation at the 20 Apr 2012 herokujp meetup, on full-text search with an emphasis on analyzing Japanese.

(This is all for the Java version of Lucene.)