多字节字符拆分在保存到数据库时生成垃圾符号

Question

In my application, dynamic long strings are generated. these values I am saving in a database with a maximum length. when the maximum length is crossed, the string is split using a custom code and a new line gets inserted in database.

The problem here occurs when multi-byte characters are used. At the split of the string if a word is getting split at a Vowel signs (matra), then it generates a junk symbols like a diamond with question mark in between .

    int blockSize = 12;
    String str1 = "<SOME STRING>";

    byte[] b = str1.getBytes("UTF-8");    

    int loopCount = x; // in actual code dynamically generated
    String outString = "";
    for (int i = 0; i <= loopCount; i++) {
        if (i != loopCount) {
            outString = new String(b, i * blockSize, blockSize, "UTF-8");
        } else {
            outString =
                    new String(b, i * blockSize, (b.length - loopCount * blockSize));
        }
    }

1. How can I avoid splitting of string when in between a word and instead take to complete word to the next time.? 2.Or is there any other way for stopping generation of junk symbols.

Answer 1

Text as conceived in Unicode has its problems on several levels.

As pure text composed from Unicode code points . ĉ can be represented as one code point U+109, in UTF-16 (binary format) as one char '\ĉ' , or as c plus a zero-width so called combining diacritical mark for ^ . So splitting between code points already is problematic. . So splitting between code points already is problematic. java.text.Normalizer` can normalize to either composed or decomposed form. Then there are the Left-To-Right and Right-To-Left markers to consider when using a part of a text.

On the UTF-16 level, java char , some code points need 2 chars, a so called surrogate pair. This is testable in java using Character . The Character class and also regular expression Pattern has a rather good Unicode support. One can find categories like combining diacritical marks.

On the UTF-8 level some (non-ASCII) chars or code points need multibyte sequences, so splitting a byte array causes UTF-8 illegal garbage at the split point.

The solution?

1. Maybe sensible to normalize text; mind file names.
2. Do not consider byte sub-arrays as valid text
3. Treat boundaries of byte arrays careful, even a c at the end might be ĉ , consider a shifting boundary buffer.