如何从 Azure 数据工厂中的 JSON 获取数组？

Question

My actual (not properly working) setup has two pipelines:

1. Get API data to lake: for each row in metadata table in SQL calling the REST API and copy the reply (json-files) to the Blob datalake.
2. Copy data from the lake to SQL: For Each file auto create table in SQL.

The result is the correct number of tables in SQL. Only the content of the tables is not what I hoped for. They all contain 1 column named odata.metadata and 1 entry, the link to the metadata. If I manually remove the metadata from the JSON in the datalake and then run the second pipeline, the SQL table is what I want to have.

Have:

{  "odata.metadata":"https://test.com", 
"value":[
{
  "Key":"12345",
"Title":"Name",
"Status":"Test"
}]}

Want:

[{
"Key":"12345",
"Title":"Name",
"Status":"Test"
}]

I tried to add $.['value'] in the API call. The result then was no odata.metadata line, but the array started with {value: which resulted in an error copying to SQL

I also tried to use mapping (in sink) to SQL. That gives the wanted result for the dataset I manually specified the mapping for, but only goes well for the dataset with the same number of column in the array. I don't want to manually do the mapping for 170 calls...

Does anyone know how handle this in ADF? For now I feel like the only solution is to add a Python step in the pipeline, but I hope for a somewhat standard ADF way to do this!

Answer 1

You can add another pipeline with dataflow to remove the content from JSON file before copying data to SQL, using flatten formatters.

Before flattening the JSON file :

This is what I see when JSON data copied to SQL database without flattening:

After flattening the JSON file :

Added a pipeline with dataflow to flatten the JSON file to remove 'odata.metadata' content from the array.

Source preview :

Flatten formatter :

Select the required object from the Input array

After selecting value object from input array, you can see only the values under value in Flatten formatter preview.

Sink preview :

File generated after flattening.

Copy the generated file as Input to SQL.

Note : If your Input file schema is not constant, you can enable Allow schema drift to allow schema changes

Reference: Schema drift in mapping data flow