elasticsearch 中的 uax_url_email tokenizer 为具有特殊字符的电子邮件生成多个令牌

Question

I use uax_url_email tokenizer for email fields in our index. It works perfect and generates single token for normal emails like johndoe@yahoo.com. However it generates multiple tokens when the email has foreign or special characters. Is there a solution for this? I don't want multiple tokens generated

PUT email-test-index
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "email_analyzer": {
            "filter": ["lowercase"],
            "tokenizer": "email_tokenizer"
          }
        },
        "tokenizer": {
          "email_tokenizer": {
            "type": "uax_url_email"
          }
        }
      }
    }
  },
  "mappings": {
    "date_detection": false,
    "numeric_detection": false,
    "properties": {
      "EMAIL": {
        "type": "text",
        "store": true,
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        },
        "analyzer": "email_analyzer"
      }
    }
  }
}

When it work:

GET email-test-index/_analyze
{
  "field": "EMAIL",
  "text": "johndoe@yahoo.com"
}

{
  "tokens" : [
    {
      "token" : "johndoe@yahoo.com",
      "start_offset" : 0,
      "end_offset" : 17,
      "type" : "<EMAIL>",
      "position" : 0
    }
  ]
}

When it does not work:

GET email-test-index/_analyze
{
  "field": "EMAIL",
  "text": "johndoeó8@yahoo.com"
}

{
  "tokens" : [
    {
      "token" : "johndoeó8",
      "start_offset" : 0,
      "end_offset" : 9,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "yahoo.com",
      "start_offset" : 10,
      "end_offset" : 19,
      "type" : "<URL>",
      "position" : 1
    }
  ]
}

Answer 1

Tldr;

You can not without getting rid of the special char. I might be wrong but I don't think such char are even allowed by the email standard.

Solution

You could be using the mapping char filter and catch all non ascii char at map them to ascii.

POST _analyze
{
  "tokenizer": "uax_url_email",
  "char_filter": [
    {
      "type": "mapping",
      "mappings": [
        "ó => o"
      ]
    }
  ],
  "text": "Email me at johndóe8@yahoo.com"
}

{
  "tokens": [
    {
      "token": "Email",
      "start_offset": 0,
      "end_offset": 5,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "me",
      "start_offset": 6,
      "end_offset": 8,
      "type": "<ALPHANUM>",
      "position": 1
    },
    {
      "token": "at",
      "start_offset": 9,
      "end_offset": 11,
      "type": "<ALPHANUM>",
      "position": 2
    },
    {
      "token": "johndoe8@yahoo.com",
      "start_offset": 12,
      "end_offset": 30,
      "type": "<EMAIL>",
      "position": 3
    }
  ]
}

Notice that the ó has been replaced by 'o'