将ByteArray转换为字符串然后返回会产生不同的字符串

Question

I have to store huge list of booleans and I chose to store them as byte array as string. But I can't understand, why converting to string and back produces different string values:

Support methods:

  fun ByteArray.string(): String {

    var str = ""

    this.reversed().forEach {
      str += intToString(it, 4)
    }

    return str
  }

  fun intToString(number: Byte, groupSize: Int): String {
    val result = StringBuilder()

    for (i in 7 downTo 0) {
      val mask = 1 shl i
      result.append(if (number.toInt() and mask != 0) "1" else "0")

      if (i % groupSize == 0)
        result.append(" ")
    }
    result.replace(result.length - 1, result.length, "")

    return result.toString()
  }

First example:

Given selected indices [0, 14] my code converts to: as bytes: [1, 64]. .string() produces:

0100 0000 0000 0001

Convert it to string and back:

array.toString(Charsets.UTF_8).toByteArray(Charsets.UTF_8)

Result: [1, 64], .string() produces:

0100 0000 0000 0001

Second example:

Given selected indices [0, 15] my code converts to: as bytes: [1,-128]. .string() produces:

1000 0000 0000 0001

Which seems pretty legal. Now convert it to the string and back

It produces an array of 4 bytes: [1, -17, -65, -67], .string() produces:

1011 1101 1011 1111 1110 1111 0000 0001

Which doesn't look like [0, 15] indices or [1,-128] for me :)

How can this happen? I suspect this last "1" in "1000 0000 0000 0001", probably it may cause this issue, but still, I don't know the answer.

Thanks.

PS Added java tag to the question, because I think the answer is the same for both kotlin and java.

Answer 1

Here's a MCVE for your problem (in Java):

import java.nio.charset.*;

class Test {
  public static void main(String[] args) {
    byte[] array = { -128 };
    byte[] convertedArray = new String(array, StandardCharsets.UTF_8).getBytes(StandardCharsets.UTF_8);
    for(int i=0; i<convertedArray.length; i++) {
      System.out.println(convertedArray[i]);
    }
  }
}

Expected output:

-128

Actual output:

-17
-65
-67

This happens because the byte -128 is not a valid UTF-8 character, so it gets replaced with the Unicode replacement character U+FFFD " ".

You can instead encode and decode the string as ISO-8859-1 aka Latin1, since all byte strings are valid in the ISO-8859 family of encodings. ISO-8859-1 has the convenient property that each byte value corresponds directly to the same unicode code point, so that 0x80 is encoded as U+0080, 0xFF as U+00FF etc.