How to read a NSInputStream with UTF-8?

Question

I try to read a large file in iOS using NSInputStream to separate the files line by newlines (I don't want to use componentsSeparatedByCharactersInSet as it uses too much memory).

But as not all lines seem to be UTF-8 encoded (as they can appear just as ASCII, same bytes) I often get the Incorrect NSStringEncoding value 0x0000 detected. Assuming NSASCIIStringEncoding. Will stop this compatiblity mapping behavior in the near future. Incorrect NSStringEncoding value 0x0000 detected. Assuming NSASCIIStringEncoding. Will stop this compatiblity mapping behavior in the near future. warning.

My question is: Is there a way to surpress this warning by eg setting a compiler flag?

Furthermore: Is it save to append/concatenate two buffer reads, as reading from the byte stream, then converting the buffer to string and then appending the string could make the string corrupted?

Below an example method that demonstrates that the byte to string conversion will discard the first and second half of the UTF-8 character, as being invalid.

- (void)NSInputStreamTest {
  uint8_t testString[] = {0xd0, 0x91}; // @"Б"

  // Test 1: Read max 1 byte at a time of UTF-8 string
  uint8_t buf1[1], buf2[1];
  NSString *s1, *s2, *s3;
  NSInteger c1, c2;
  NSInputStream *inStream = [[NSInputStream alloc] initWithData:[[NSData alloc] initWithBytes:testString length:2]];

  [inStream open];
  c1 = [inStream read:buf1 maxLength:1];
  s1 = [[NSString alloc] initWithBytes:buf1 length:1 encoding:NSUTF8StringEncoding];
  NSLog(@"Test 1: Read %d byte(s): %@", c1, s1);
  c2 = [inStream read:buf2 maxLength:1];
  s2 = [[NSString alloc] initWithBytes:buf2 length:1 encoding:NSUTF8StringEncoding];
  NSLog(@"Test 1: Read %d byte(s): %@", c2, s2);
  s3 = [s1 stringByAppendingString:s2];
  NSLog(@"Test 1: Concatenated: %@", s3);
  [inStream close];

  // Test 2: Read max 2 bytes at a time of UTF-8 string
  uint8_t buf4[2];
  NSString *s4;
  NSInteger c4;
  NSInputStream *inStream2 = [[NSInputStream alloc] initWithData:[[NSData alloc] initWithBytes:testString length:2]];

  [inStream2 open];
  c4 = [inStream2 read:buf4 maxLength:2];
  s4 = [[NSString alloc] initWithBytes:buf4 length:2 encoding:NSUTF8StringEncoding];
  NSLog(@"Test 2: Read %d byte(s): %@", c4, s4);
  [inStream2 close];
}

Output:

2013-02-10 21:16:23.412 Test[11144:c07] Test 1: Read 1 byte(s): (null)
2013-02-10 21:16:23.413 Test[11144:c07] Test 1: Read 1 byte(s): (null)
2013-02-10 21:16:23.413 Test[11144:c07] Test 1: Concatenated: (null)
2013-02-10 21:16:23.413 Test[11144:c07] Test 2: Read 2 byte(s): Б

Answer 1

First of all, in line: s3 = [s1 stringByAppendingString:s2]; you are trying to concatenate to 'nil' values. The result would be 'nil' also. So, you may want to concatenate bytes instead of strings:

uint8_t buf3[2];
buf3[0] = buf1[0];
buf3[1] = buf2[0];
s3 = [[NSString alloc] initWithBytes:buf3 length:2 encoding:NSUTF8StringEncoding];

Output:

2015-11-06 12:57:40.304 Test[10803:883182] Test 1: Read 1 byte(s): (null)
2015-11-06 12:57:40.305 Test[10803:883182] Test 1: Read 1 byte(s): (null)
2015-11-06 12:57:40.305 Test[10803:883182] Test 1: Concatenated: Б

Secondary, length of UTF-8 character may lay in [1..6] bytes.

(1 byte)   0aaa aaaa         //if symbol lays in 0x00 .. 0x7F (ASCII)
(2 bytes)  110x xxxx 10xx xxxx
(3 bytes)  1110 xxxx 10xx xxxx 10xx xxxx
(4 bytes)  1111 0xxx 10xx xxxx 10xx xxxx 10xx xxxx
(5 bytes)  1111 10xx 10xx xxxx 10xx xxxx 10xx xxxx 10xx xxxx
(6 bytes)  1111 110x 10xx xxxx 10xx xxxx 10xx xxxx 10xx xxxx 10xx xxxx

So, if you are intended to read from NSInputStream raw bytes and then translate them into UTF-8 NSString, you probably want to read byte by byte from NSInputStream until you will get valid string:

#define MAX_UTF8_BYTES 6
NSString *utf8String;
NSMutableData *_data = [[NSMutableData alloc] init]; //for easy 'appending' bytes

int bytes_read = 0;
while (!utf8String) {
    if (bytes_read > MAX_UTF8_BYTES) {
        NSLog(@"Can't decode input byte array into UTF8.");
        return;
    }
    else {
        uint8_t byte[1];
        [_inputStream read:byte maxLength:1];
        [_data appendBytes:byte length:1];
        utf8String = [NSString stringWithUTF8String:[_data bytes]];
        bytes_read++;
    }
}

Answer 2

ASCII (and hence the newline character) is a subset of UTF-8, so there should not be any conflict.

It should be possible to divide your stream at the newline characters, as you would in a simple ASCII stream. Then you can convert each chunk ("line") into an NSString using UTF-8.

Are you sure the encoding errors are not real, ie, that your stream may actually contain erroneous characters with respect to a UTF-8 encoding?

Edited to add from the comments:

This presumes that the lines consist of sufficiently few characters to keep a whole line in memory before converting from UTF-8.