Searching names with Apache Solr

Question

I've just ventured into the seemingly simple but extremely complex world of searching. For an application, I am required to build a search mechanism for searching users by their names.

After reading numerous posts and articles including:

How can I use Lucene for personal name (first name, last name) search?
http://dublincore.org/documents/1998/02/03/name-representation/
what's the best way to search a social network by prioritizing a users relationships first?
http://www.gossamer-threads.com/lists/lucene/java-user/120417
Lucene Index and Query Design Question - Searching People
Lucene Fuzzy Search for customer names and partial address

... and a few others I cannot find at-the-moment. And getting at-least indexing and basic search working in my machine I have devised the following scheme for user searching:

1) Have a first, second and third name field and index those with Solr
2) Use edismax as the requestParser for multi column searching
3) Use a combination of normalization filters such as: transliteration, latin-to-ascii convesrion, etc.
4) Finally use fuzzy search

Evidently, being very new to this I am unsure if the above is the best way to do it and would like to hear from experienced users who have a better idea than me in this field.

I need to be able to match names in the following ways:

1) Accent folding: Jorn matches Jörn and vise versa
2) Alternative spellings: Karl matches Carl and vice versa
3) Shortened representations (I believe I do this with the SynonymFilterFactory): Sue matches Susanne, etc.
4) Levenstein matching: Jonn matches John, etc.
5) Soundex matching: Elin and Ellen

Any guidance, criticisms or comments are very welcome. Please let me know if this is possible ... or perhaps I'm just day-dreaming. :)

---

EDIT

I must also add that I also have a fullname field in case some people have long names, as an example from one of the posts: Jon Paul or Del Carmen should also match Jon Paul Del Carmen

And since this is a new project, I can modify the schema and architecture any way I see fit so there are very limited restrictions.

Answer 1

It sounds like you are catering for a corpus with searches that you need to match very loosely?

If you are doing that you will want to choose your fields and set different boosts to rank your results.

So have separate "copied" fields in solr:

• one field for exact full name (with filters)
• multivalued field with filters ASCIIFolding, Lowercase...
• multivalued field with the SynonymFilterFactory ASCIIFolding, Lowercase...
• PhoneticFilterFactory (with Caverphone or Double-Metaphone )

See Also: more non-english Soundex discussion

Synonyms for names, I don't know if there is a public synonym db available.

Fuzzy searching, I've not found it useful, it uses Levenshtein Distance.

Other filters and indexing get more superior "search relevant" results.

Unicode characters in names can be handled with the ASCIIFoldingFilterFactory

You are describing solutions up front for expected use cases.

If you want quality results, plan on tuning your Search Relevance

This tuning will be especially valuable, when attempting to match on synonyms, like MacDonald and McDonald (which has a larger Levenshtein distance than Carl and Karl).

Answer 2

Found a nickname db, not sure how good: http://www.peacockdata2.com/products/pdnickname/

Note that it's not free.

Answer 3

The answer in another post is pretty good: Training solr to recognize nicknames or name variants

 <fieldType name="name_en" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.ASCIIFoldingFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.ASCIIFoldingFilterFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="english_names.txt" ignoreCase="true" expand="true"/> </analyzer> </fieldType> 

Answer 4

For phonetic name search you might also try the Beider-Morse Filter which works pretty well if you have a mixture of names from different countries.

If you want to use it with a typeahead feature, combine it with an EdgeNGramFilter:

<fieldType name="phoneticNames" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
  <analyzer type="index">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.BeiderMorseFilterFactory" nameType="GENERIC" ruleType="APPROX" concat="true" languageSet="auto"/>
    <filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="15"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.BeiderMorseFilterFactory" nameType="GENERIC" ruleType="APPROX" concat="true" languageSet="auto"/>
  </analyzer>
</fieldType>

Answer 5

We created a simple 'name' field type that allows mixing both 'key' (eg, SOUNDEX) and 'pairwise' portions of the answers above.

Here's the overview:

1. at index time, fields of the custom type are indexed into a set of (sub) fields with respective values used for high-recall matching different kinds of variations

Here's the core of its implementation...

List<IndexableField> createFields(SchemaField field, String name) {
        Collection<FieldSpec> nameFields = deriveFieldsForName(name);
        List<IndexableField> docFields = new ArrayList<>();
        for (FieldSpec fs : nameFields) {
            docFields.add(new Field(fs.getName(), fs.getStringValue(),
                         fs.getLuceneField()));
        }
        docFields.add(createDocValues(field.getName(), new Name(name)));
        return docFields;
}

The heart of this is deriveFieldsForName(name) in which you can include 'keys' from PhoneticFilters, LowerCaseFolding, etc.

2. at query time, first a custom Lucene query is produced that has been tuned for recall and that uses the same fields as index time

Here's the core of its implementation...

public Query getFieldQuery(QParser parser, SchemaField field, String val) {
        Name name = parseNameString(externalVal, parser.getParams());
        QuerySpec querySpec = buildQuery(name);
        return querySpec.accept(new SolrQueryVisitor(field.getName())); 
}

The heart of this is the buildQuery(name) method which should produce a query that is aware of deriveFieldsForName(name) above so for a given query name it will find good candidate names.

3. then second, Solr's Rerank feature is used to apply a high-precision re-scoring algorithm to reorder the results

Here's what this looks like in your query...

&rq={!myRerank reRankQuery=$rrq} &rrq={!func}myMatch(fieldName, "John Doe")

The content of myMatch could have a pairwise Levenstein or Jaro-Winkler implementation.

NB Our own full implementation uses proprietary code for deriveFieldsForName, buildQuery, and myMatch (see http://www.basistech.com/text-analytics/rosette/name-indexer/ ) to handle more kinds of variations that the ones mentioned above (eg, missing spaces, cross-language).