Character Encoding causes my ASP Web API request to fail with HTTP 400: Bad Request

Question

I have a json object which I serialize and post. I found that if one of the string members of my json object has a few unicode characters such as é, ó, 6̄ (I have some other unicode characters that 犞ݮ do not return an HTTP 400 error) that the request returns HTTP 400 Bad Request.

Here is the simplified code for how the call is being made:

WebClient client;
Foo myObject = new Foo();
myObject.StringField1 = "é";
string serializedObjects = Newtonsoft.Json.JsonConvert.SerializeObject(myObject)
client.Headers[HttpRequestHeader.ContentType] = "application/json;charset=utf-8"; 
var response = client.UploadString(url, serializedObjects);

Here is the code for how the calls is being received on server side:

public async Task<IHttpActionResult> Post([FromBody] IList<Foo> myObjects){}

Here are some things that I have researched/tried and some assumptions I have made:

1) Get the string object as UTF8 bytes, then transform it back to a string

Encoding.UTF8.GetString(Encoding.UTF8.GetBytes(serializedObjects))

2) When I make the request and capture the traffic using Fiddler and inspect the request in Raw. For the unicode characters that don't return a 400 error they are replaced with a '?' however for the ones that do cause a 400 they are displayed as a blank. However when I look at JSON view it shows as a null character ' '

3) HTTP 400 is returned is ASP Web API can't deserialize the object correctly, so I tried using Newtonsoft library to serialize and deserialize the object and this works just fine. (The default serializer for ASP Web API is JSON.NET so it should be performing the same serialization http://www.asp.net/web-api/overview/formats-and-model-binding/json-and-xml-serialization )

string b = JsonConvert.SerializeObject(a);
Foo c = (Foo) JsonConvert.DeserializeObject(b, typeof(Foo));
// And I tried deserializing using the generic method
Foo d = JsonConvert.DeserializeObject<Foo>(b);

4) I was convinced that this was an issue with my server rejecting these special characters, but when I used Fiddler to retry the request (inserting the é back in the place where Fiddler had removed it from in the original request) the call succeeds. 5) I validated my Content-Type on http://www.iana.org/assignments/media-types/media-types.xhtml and also specify the utf8 charset explicitly.

So this seems to me to be an encoding issue, but é should be a valid UTF-8 character. I validated it in Jon Skeet's Unicode Explorer widget on his page. So I'm out of ideas for why this is causing issues...

Answer 1

As @roeland and @dbc pointed out there is an Encoding property on the WebClient . Setting this to Encoding.UTF8 resolved the issue.

client.Encoding = Encoding.UTF8;