奇怪的Java Unicode正则表达式StringIndexOutOfBoundsException

Question

My question is quite simple yet puzzling. It could be that there is a simple switch which fixes this but I'm not much experienced in Java regexes...

String line = "💕💕💕";
line.replaceAll("(?i)(.)\\1{2,}", "$1");

This crashes. If I remove the (?i) switch, it works. The three unicode characters are not random, they were found amidst a big Korean text, but I don't know they are valid or not.

Strange thing is that the regex works for all the other text but this. Why do I get the error?

This is the exception I get

Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: 6
    at java.lang.String.charAt(String.java:658)
    at java.lang.Character.codePointAt(Character.java:4668)
    at java.util.regex.Pattern$CIBackRef.match(Pattern.java:4846)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4125)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Start.match(Pattern.java:3408)
    at java.util.regex.Matcher.search(Matcher.java:1199)
    at java.util.regex.Matcher.find(Matcher.java:592)
    at java.util.regex.Matcher.replaceAll(Matcher.java:902)
    at java.lang.String.replaceAll(String.java:2162)
    at tokenizer.Test.main(Test.java:51)

Answer 1

The characters you mentioned are actually " Double byte characters ". Which means that two bytes form one character. But for Java to interpret this, the encoding information ( when it is different from the default platform encoding ) needs to be passed explicitly ( or else default platform encoding will be used ).

To prove this, consider following

String line = "💕💕💕";
System.out.println(line.length());

this prints the length as 6 ! Whereas we only have three characters,

now the following code

String line1 = new String("💕💕💕".getBytes(),"UTF-8");
System.out.println(line1.length());

prints length as 3 which intended.

if you replace the line

String line = "💕💕💕";

with

 String line1 = new String("💕💕💕".getBytes(),"UTF-8");

it works and regex does not fail . I have used UTF-8 here. Please use the appropriate encoding of your intended platform.

Java regex libraries depend heavily on Character Sequence which in turn depends on the encoding scheme. For the strings having character encoding different from the default encoding, characters cannot be decoded correctly (it showed 6 chars instead of 3 !) and hence regex fails.

Answer 2

What's explained by Santosh in this answer is incorrect. This can be demonstrated by running

String str = "💕💕💕";
System.out.println("code point: " + .codePointAt(0));

which will output (at least for me) the value 128149, which is confirmed by this page as correct. So Java does not interpret the string in a wrong way. It did interpret it wrong when using the getBytes() method.

However, as explained by OP, it seems the regular expression crashes on that. I have no other explanation for it as it being a bug in java. Either that, or then it doesn't support UTF-16 fully by design.

Edit:

based on this answer :

the regex compiler screws up on the UTF-16. Again, this can never be fixed or it will change old programs. You cannot even get around the bug by using the normal workaround to Java's Unicode-in-source-code troubles by compiling with java -encoding UTF-8, because the stupid thing stores the strings as nasty UTF-16, which necessarily breaks them in character classes. OOPS!

It would seem that this is a limitation of regular expressions in java.

---

Since you commented that

it would be best if I could simply ignore the UTF-16 characters and apply the regex rather than throw an exception.

This can certainly be done. A straightforward way is to only apply your regex to a certain range. Filtering unicode character ranges has been explained in this answer . Based on that answer, example that doesn't seem to choke but just leaves the problem characters alone:

line.replaceAll("(?Ui)([\\u0000-\\uffff])\\1{2,}", "$1")    

// "💕💕💕" -> "💕💕💕"
// "foo 💕💕💕 foo" -> "foo 💕💕💕 foo"
// "foo aAa foo" -> "foo a foo"

Answer 3

Actually, it's just a bug.

This is what stack traces and open source are for.

When CIBackRef (for case-insensitive back reference) compares with the group, it doesn't bump the loop index correctly. This shows the fix:

        // Check each new char to make sure it matches what the group
        // referenced matched last time around
        int x = i;
        for (int index=0; index<groupSize; ) {
            int c1 = Character.codePointAt(seq, x);
            int c2 = Character.codePointAt(seq, j);
            if (c1 != c2) {
                if (doUnicodeCase) {
                    int cc1 = Character.toUpperCase(c1);
                    int cc2 = Character.toUpperCase(c2);
                    if (cc1 != cc2 &&
                        Character.toLowerCase(cc1) !=
                        Character.toLowerCase(cc2))
                        return false;
                } else {
                    if (ASCII.toLower(c1) != ASCII.toLower(c2))
                        return false;
                }
            }
            int n = Character.charCount(c1);
            x += n;
            index += n;  // was index++
            j += Character.charCount(c2);
        }

groupSize is the total charCount of the group. j is the index for the referenced group.

The test

  //9ff0 9592 9ff0 9592 9ff0 9592
  val line = "\ud83d\udc95\ud83d\udc95\ud83d\udc95"
  Console println Try(line.replaceAll("(?ui)(.)\\1{2,}", "$1"))

fails normally

apm@mara:~/tmp$ skalac kcharex.scala ; skala kcharex.Test
Failure(java.lang.StringIndexOutOfBoundsException: String index out of range: 6)

but succeeds with the fix

apm@mara:~/tmp$ skala -J-Xbootclasspath/p:../bootfix kcharex.Test
Success(💕)

The other bug in the original sample code is that the inline flags should include ?ui . The javadoc on Pattern.CASE_INSENSITIVE says:

By default, case-insensitive matching assumes that only characters in the US-ASCII charset are being matched. Unicode-aware case-insensitive matching can be enabled by specifying the UNICODE_CASE flag in conjunction with this flag.

As you can see from the code snippet, without u , it will fail only if ASCII.toLower doesn't compare equal, which is not intended. I'm not sophisticated enough to know of a supplementary character that would fail that test without writing code to figure it out.