Facing error in azure data factory ("copy activity pipeline") multiple concurrent copy activities runs

Question

Iam facing issue in the copy activity pipeline

Operation on target... : Operation on target.....failed: Failure happened on 'Sink' side. ErrorCode=AdlsGen2OperationFailedConcurrentWrite,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when trying to upload a file. It's possible because you have multiple concurrent copy activities runs writing to the same file '...folder path...'. Check your ADF configuration.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'PreconditionFailed',Source=Microsoft.DataTransfer.ClientLibrary,'

Please help me to rectify this error

Answer 1

The message in the above error is,

Message=Error occurred when trying to upload a file. It's possible because you have multiple concurrent copy activities runs writing to the same file '...folder path...'. Check your ADF configuration

As per the Official Documentation , the cause of this is,

You have multiple concurrent copy activity runs or applications writing to the same file.

So, there is a possibility that you are having multiple copy activities running in your pipeline on the same destination file which causing this error.
Please check the sink datasets of all the copy activities in your pipeline and try to change the destination according to your requirement.

If you have only one copy activity in your pipeline, this error may occur if you set multiple triggers on this in which any two triggers running the pipeline at the same time. In that scenario, please set the concurrency limit of the pipeline to 1 which can control concurrent execution. You can find it in the pipeline settings.

Also, please check if any of your other pipelines having copy activity running on same time with same file as destination.