删除所有非字母数字字符但允许多字词

Question

For a MapReduce job I'm trying to remove all non-alphanumerical characters, stem the token and lowerCase it if it's not an acronym but I want to allow multi-word terms like "life-changing". This is what I did so far, how should I change it?

 String line = value.toString();
        
         StringTokenizer itr = new StringTokenizer(line);
         
         while (itr.hasMoreTokens())
         { 
            String token = stem(caseFold(itr.nextToken())); 
            token=token.replaceAll("^[^a-zA-Z0-9]*|[^a-zA-Z0-9]*$", "");


             ....
         }

Answer 1

您可以使用公开可用的字典 API，例如 dictionaryapi.com