在Java中将unicode字符串转换为ASCII，可在unix / linux中使用

Question

I have already tried using Normalizer

String s = "口水雞 hello Ä";

String s1 = Normalizer.normalize(s, Normalizer.Form.NFKD);
String regex = Pattern.quote("[\\p{InCombiningDiacriticalMarks}\\p{IsLm}\\p{IsSk}]+");

String s2 = new String(s1.replaceAll(regex, "").getBytes("ascii"), "ascii");

System.out.println(s2);
System.out.println(s.length() == s2.length());

i want it to work in Unix/Linux ,

Answer 1

There is an ASCII character class for matching code points in the ASCII set:

String s = "口水雞 hello Ä";

String s1 = Normalizer.normalize(s, Normalizer.Form.NFKD);
String nonAscii = "[^\\p{ASCII}]+";
String s2 = s1.replaceAll(nonAscii, "");

System.out.println(s2);
System.out.println(s.length() == s2.length());

As Joop Eggan notes , Java string and char types are always UTF-16. You can only have ASCII-encoded data in byte form:

byte[] ascii = s2.getBytes(StandardCharsets.US_ASCII);

Answer 2

Explanation

First in java text (String/Reader/Writer) is already Unicode. For the java source code (String literals) the editor and the javac compiler should use the same encoding. Ideally UTF-8.

The normalizer splits into base letter and combining diacritical mark(s) and regular expression removes those marks. Converting text with accents like ä é fi fl ĉ œ to ae fi fl c oe to ASCII.

Hence you would get - I think - "??? hello A" .

Charset ascii = StandardCharsets.US_ASCII;
String s2 = new String(s1.replaceAll(regex, "").getBytes(ascii), ascii);

To prevent receiving the question marks (and distinguishing between a ? in the original string), you can use a Charset.newDecoder() .

For ASCII you would still need some transliteration to latin script.

Answer

As most Linux operating systems of newer origin already use UTF-8 as operating system default, you probably can simply do:

System.out.println("We are using encoding: " + System.getProperty("file.encoding"));
System.out.println(s);

Here s is converted to the operating system encoding.