Data modified on AWS API Gateway Response body

Question

I am trying to return hexadecimal string as response from my AWS Lambda function. When it reaches to the client the data seems to be modified.

• Data :
  47 49 46 38 39 61 01 00 01 00 80 00 00 00 00 00
  ff ff ff 21 f9 04 01 00 00 01 00 2c 00 00 00 00
  01 00 01 00 00 08 04 00 03 04 04 00 3b

• Hexadecimal Excaped Data ( Sent Data ):

  \\x47\\x49\\x46\\x38\\x39\\x61\\x01\\x00\\x01\\x00\\x80\\x00\\x00\\x00\\x00\\x00" "\\xff\\xff\\xff\\x21\\xf9\\x04\\x01\\x00\\x00\\x01\\x00\\x2c\\x00\\x00\\x00\\x00" "\\x01\\x00\\x01\\x00\\x00\\x08\\x04\\x00\\x03\\x04\\x04\\x00\\x3b

• Received Data
  47 49 46 38 39 61 01 00 01 00 c2 80 00 00 00 00
  00 c3 bf c3 bf c3 bf 21 c3 b9 04 01 00 00 01 00
  2c 00 00 00 00 01 00 01 00 00 08 04 00 03 04 04
  00 3b

  How to fix this?

Answer 1

Last time I checked it was not very explicit in the doc, but API Gateway is really made for json (or similar) and support for binary is 'on the roadmap' but clearly doesn't seem to be a priority. It converts everything it sends to utf-8.

Comparing precisely your original data with the received one you can see it :

47 49 46 38 39 61 01 00 01 00 80    00 00 00 00 00 ff    ff    ff    21 f9    04 01 00 00 01 00 2c 00 00 00 00 01 00 01 00 00 08 04 00 03 04 04 00 3b
47 49 46 38 39 61 01 00 01 00 c2 80 00 00 00 00 00 c3 bf c3 bf c3 bf 21 c3 b9 04 01 00 00 01 00 2c 00 00 00 00 01 00 01 00 00 08 04 00 03 04 04 00 3b 

Everything under 0x7f is OK because the unicode code point is the same as the encoded byte (U+0047 -> 47), but for 0x80 or more the problem arises : U+0080 -> c2 80, U+00FF -> c3 bf and so on.

We had a similar problem recently : binary data was corrupted and bigger when sent through Gateway than with direct access to our backend. It was because a lot of bytes get replaced by Unicode special 'replacement character' aka 'U+FFFD' aka '0xEF 0xBF 0xBD'.

How to fix ? We just stopped using Gateway but if you can afford your data to be bigger, you can base64 encode it.