Azure Data Factory DataFlow Error: Key partitioning does not allow computed columns

Question

We have a generic dataflow that works for many tables, the schema is detected at runtime. We are trying to add a Partition Column for the Ingestion or Sink portion of the delta.

We are getting error: Azure Data Factory DataFlow Error: Key partitioning does not allow computed columns Job failed due to reason: at Source 'Ingestion'(Line 7/Col 0): Key partitioning does not allow computed columns

Can we pass the partition column as a parameter to a generic dataflow?

Answer 1

Can we pass the partition column as a parameter to a generic dataflow?

I tried your scenario and got similar error.

There is a limitation of key partition method is we cannot apply any calculation to the partition column while declaring it. Instead, this must be created in advanced, either using derived column or read in from source.

To resolve this, you can try following steps -

• First, I created a pipeline parameter with datatype string and gave column name as value.

• Click on Dataflow >> Go to Parameter >> In value of parameter select Pipeline expression >> and pass the above created parameter.

OUTPUT:

It is taking it as partition key column and partitioning data accordingly.

Reference : How To Use Data Flow Partitions To Optimize Spark Performance In Data Factor