Azure Data Factory CICD error: The document creation or update failed because of invalid reference

Question

All, when running a build pipeline using Azure Devops with ARM template, the process is consistently failing when trying to deploy a dataset or a reference to a dataset with this error:

ARM Template deployment: Resource Group scope (AzureResourceManagerTemplateDeployment) BadRequest: The document creation or update failed because of invalid reference 'dataset_1'.

I've tried renaming the dataset and also recreating it to see if that would help.

I then deleted the dataset_1.json file from the repo and still get the same message so it's some reference to this dataset and not the dataset itself I think. I've looked through all the other files for references to this but they all look fine.

Any ideas on how to troubleshoot this?

thanks

Answer 1

try this

Looks like you have created 'myTestLinkedService' linked service, tested connection but haven't published it yet and trying to reference that linked service in the new dataset that you are trying to create using Powershell. In order to reference any data factory entity from Powershell, please make sure those entities are published first. Please try publishing the linked service first from the portal and then try to run your Powershell script to create the new dataset/actvitiy.

Answer 2

I think I found the issue. When I went into the detailed logs I found that in addition to this error there was an error message about an invalid SQL connection string, so I though it may be related since the dataset in question uses Azure SQL database linked service.

I adjusted the connection string and this seems to have solved the issue.