Azure Data Factory V2 Copy data issue - Error code: 2200

Question

I'm trying to pull data from ServiceNow tables using the REST API connector in Azure Data Factory V2 . While pulling the data, I'm getting the following error sometimes and sometimes I'm not getting any error and the pipeline is successfully running.

{
"errorCode": "2200",
"message": "Failure happened on 'Source' side. ErrorCode=UserErrorInvalidJsonDataFormat,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when deserializing source JSON data. Please check if the data is in valid JSON object format.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Newtonsoft.Json.JsonReaderException,Message=Invalid character after parsing property name. Expected ':' but got: ,. Path 'result', line 1, position 15740073.,Source=Newtonsoft.Json,'",
"failureType": "UserError",
"target": "REST-API-ServiceNow" 
}

Can someone please help me here?

Thanks in advance!

Answer 1

1.Error occurred when deserializing source JSON data. Please check if the data is in valid JSON object format. 2.Message=Invalid character after parsing property name. Expected ':' but got: ,.

I think the error details indicates that your source data can't be deserialized by ADF because it's not standard JSON format.Illegal JSON data can't pass through ADF copy activity.

I can't reproduce your issue on my side because i can't touch your source data.However, i suggest you using WEB Activity to call your REST API before the execution of Copy Activity. And collect the output of Web Activity (Response from REST API) to store them in other residence. So that you could check whether your source data is legal every time.

---

My idea is as below:

1.Configure Web Activity to call your REST API, then you could get the response from your source data.

2.Configure a Function App Activity to log the output of above Web Activity.

Body should be set the output of Web Activity: @activity('Web1').output , then log it in the function app. Some sample function code as below:

using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;

namespace FunctionAppStoreCSV
{
    public static class Function2
    {
        [FunctionName("Function2")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest req,
            ILogger log)
        {
            log.LogInformation("C# HTTP trigger function processed a request.");

            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();


            dynamic data = JsonConvert.DeserializeObject(requestBody);

            log.LogInformation(requestBody);

            return requestBody != null
                ? (ActionResult)new OkObjectResult($"Log Successfully")
                : new BadRequestObjectResult("Please pass output in the request body");
        }
    }
}

I did a test locally and you may see the log data something like:

If it is in the portal, you could view the log on the KUDU url: D:\home\LogFiles\Application\Functions\function\Function2>

3.Connect both of them in ADF before your Copy Activity.

Tips:

My way could save the money for the azure storage,i just log the data so that you could check whether the data is followed strict format. Surely,you could store them in the blob storage. Just write code to store the Body data into Blob storage as you want inside Azure Function app.