Azure Data Factory V2 - Parsing JSON HTTP GET Dataset

Question

I am trying to consume the API response below using the HTTP GET connector in Azure Data Factory V2. I am unable to get the Dataset properly parsed and I keep getting an error that the activity failed during deserialization as the data is not in proper JSON object nor array format.

I have tried to leverage JSONPath to isolate specific key-value pairs but I keep getting the same error. I have successfully done this with many other APIs but can't seem to figure this one out. Ultimately, I just want to create a table with a column date, and another one for Revenue, another one for Cost of Revenue, and so on.

Any ideas on the best way to parse this API response using Azure Data Factory V2? Thanks in advance.

 {
     "AAPL": {
         "Revenue": {
             "2013-09": "170910",
             "2014-09": "182795",
             "2015-09": "233715",
             "2016-09": "215639",
             "2017-09": "229234",
             "TTM": "239176"
         },
         "Cost of revenue": {
            "2013-09": "106606",
            "2014-09": "112258",
            "2015-09": "140089",
            "2016-09": "131376",
            "2017-09": "141048",
            "TTM": "147254"
         }
     } 
}

Answer 1

I think your issue is more of a data structure nature. The JSON format used when importing can be used with the jsonPathDefinition parameter to map paths as columns, the thing is, the formatter cannot perform a correlation between an item under Revenue and an item under Cost of revenue (meaning take item index 0 from both locations and create a single row for them).

I think you'll need to due some custom pre-processing on the JSON file before importing.

Hope it helps!