数据工厂中的 Azure cosmos db 错误 - 数据流接收器 -> 作业失败，原因是：从 StructType 转换

Question

I'm building a simple data flow in azure data factory to get some specific data from a content hub location. This information is in json format.

Transformations:

1. Source: rest API get method to retrieve the data from the URL
2. Transformation 1: flatten to put into rows an item list that contains all the articles.
3. Transformation 2: select to chose specific attributes from each items in the list.
4. Transformation 3: alter row to upsert data if condition true()
5. Sink: using cosmosdb dataset to load the selected data into a collection.

The problem is with the last item elements since this is a StructType {} :

elements: {
  headline: {
    title: "Title",
    dataType: "string",
    name: "headline",
    variations: { },
    multiValue: false,
    :type: "string",
  },
  alternativeHeadline: {
    title: "Subtitle",
    dataType: "string",
    name: "alternativeHeadline",
    variations: { },
    multiValue: false,
    :type: "string",
  },
  author: {
    title: "Author",
    dataType: "string",
    name: "author",
    variations: { },
    multiValue: false,
    :type: "string",
  },
...
}

When I run the job I'm getting this error:

{"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Conversion from StructType(StructField(headline,StructType(StructField(:type,StringType,true), StructField(dataType,StringType,true), StructField(multiValue,BooleanType,true), StructField(name,StringType,true), StructField(title,StringType,true), StructField(value,StringType,true), StructField(variations,StructType(StructField(mobile,StructType(StructField(:type,StringType,true), StructField(dataType,StringType,true), StructField(multiValue,BooleanType,true), StructField(name,StringType,true), StructField(title,StringType,true), StructField(value,StringType,true)),true), StructField(spanish,StructType(StructField(:type,StringType,true), StructField(dataType,StringType,true), StructField(multiValue,BooleanType,true), StructField(name,StringType,true), StructField(title,StringType,true), StructField(value,StringType,true)),true)),true)),true), StructField(icon,StructType(StructField(:type,StringType,true), StructField(dataType,StringType,true), StructField(multiValue,BooleanType,true),","Details":""}

It seems one ore more data struct types are wrong from the source vs the target but I would like this to be dynamic. Since it is some metadata, the attribute can or can't be there.

I tried using a copy activity directly in a pipeline using the same dataset and it worked fine the problem is I'll need to transform the data later and the copy activity is limited in this aspect. Any thoughts?

Answer 1

I found the root of the problem. In the dataset, I imported the schema, once I cleared it, the struct data type verification stopped in the data-flow. The load ran successfully.