'???' Showing up in string on iOS Device, but not Simulator

Question

I have found that the same cell with the same text and the same font is changing ' characters into ??? on a device, but the simulator displays the character fine. This is also happening with some other special characters.

Here are some screenshots to help explain what I mean:

On the iOS Simulator

On my iPhone 6 Plus (or any iOS device)

Thanks in advance!

EDIT: Code and Logs

JSON response from server (in Simulator)

{"title":"The science behind Netflix's first major redesign in four years"}`

JSON response from server (on Device)

{"title":"The science behind Netflix???s first major redesign in four years"}`

CODE (I am using Xamarin so it is in C#)

var obj = JObject.Parse(response);
cell.TextLabel.Title = obj["title"].ToString();

Obviously the response from the server is different for the simulator and the device. How could this be though? They are both going to the same place, getting the same info...

Thanks again!

Some more details

The data is being downloaded using C#'s System.Net.WebClient's DownloadString() method.

The data is served up by our own servers (running node.js and expressjs)

The node endpoint gets the data from our database, encodes it to JSON and then uses res.send to display it.

Hope this helps some more! Thanks for bearing with me!

Answer 1

You are downloading JSON data. JSON is typically (but not required to be) encoded in UTF-8.

The JSON data in question contains Unicode codepoint U+2019 RIGHT SINGLE QUOTATION MARK , rather than U+0027 APOSTROPHE . Codepoint U+2019 is encoded in UTF-8 as bytes 0xE2 0x80 0x99 .

When downloading the JSON, the raw bytes are not being interpreted as UTF-8 when converted to a String . The bytes are being interpreted in a charset that does not support bytes 0xE2 , 0x80 , or 0x99 , so they are each being converted to either codepoint U+003F QUESTION MARK or U+FFFD REPLACEMENT CHARACTER (depending on how the decoder is implemented).

According to Xamarin's WebClient.DownloadString() documentation (which matches MSDN's documentation ):

This method retrieves the specified resource. After it downloads the resource, the method uses the encoding specified in the WebClient.Encoding property to convert the resource to a String .

And according to Xamarin's WebClient.Encoding documentation (which is similar to MSDN's documentation ):

When a string is downloaded using the System.Net.WebClient.DownloadString or System.Net.WebClient.DownloadStringAsync methods, System.Net.WebClient uses the System.Text.Encoding returned by this to convert the downloaded byte array into a string.

Xamarin's documentation does not state what the default value of the WebClient.Encoding property is, but the MSDN documentation does:

The default value of this property is the encoding returned by Default .

Where Default is referring to the System.Text.Encoding.Default property:

Gets an encoding for the operating system's current ANSI code page .

Note:
Different computers can use different encodings as the default, and the default encoding can even change on a single computer. Therefore, data streamed from one computer to another or even retrieved at different times on the same computer might be translated incorrectly. In addition, the encoding returned by the Default property uses best-fit fallback to map unsupported characters to characters supported by the code page. For these two reasons, using the default encoding is generally not recommended. To ensure that encoded bytes are decoded properly, you should use a Unicode encoding, such as UTF8Encoding or UnicodeEncoding, with a preamble. Another option is to use a higher-level protocol to ensure that the same format is used for encoding and decoding.

So, the charset used by System.Text.Encoding.Default in the iOS Simulator (which is the underlying OS's charset) is different than the charset used by System.Text.Encoding.Default in the iOS device. That would account for the difference in decoding that you are seeing.

Assuming that the JSON really is encoded as UTF-8 during transmission, you need to set the WebClient.Encoding property to System.Text.Encoding.UTF8 before calling WebClient.DowloadString() . Then it will decode correctly on both device and simulator.