Azure Data Factory, Passing REST GET response to stored procedure in Azure SQL Database

Question

I'm trying to build a (I think) very simple pipeline:

1. Get the textual body of a GET operation.
2. Pass the (json) output as-is (= no transformations needed in ADF) to a "Json" parameter of a stored procedure in an Azure SQL Server database. The stored procedure handles the (complex) parsing/mapping.

I thought that this can be done with just 1 Copy activity, but now I think I'm wrong.

In de Copy activity the Sink configuration looks like this:

"sink": {
  "type": "AzureSqlSink",
  "sqlWriterStoredProcedureName": "[dbo].[spParseJson]",
  "sqlWriterTableType": "What to enter here?",
  "storedProcedureTableTypeParameterName": "What to enter here?",
  "storedProcedureParameters": {
  "Json": {
     "type": "String",
     "value": "<output of Source>"
     }
   }
 }

I really tried to read and understand the documentation, but imho the documentation doesn't explain much or in a bad vague way.

The "output of Source" should be the output from Source. But what function or variable to use for that?

What should I enter for "sqlWriterTableType" / "storedProcedureTableTypeParameterName"? After some digging I understand that ADF will create temp tables and such, but that isn't what I want.

I've also tried an other approach:

1. Use the Web activity to just download the Json.
2. Execute SP with the input: @Activity("WebactivityName").output.

But then I found out that the Web activity is limited to 1MB. The Json is about 1,5 MB. If the limit wouldn't be there, then I would have a solution. Argh.

FYI: The content of the Json has a dynamically changing schema and is not well structured, so there's really no way that I can use the standard mapping capabilities in ADF.

Any help or guidance is appreciated. If you know of some documentation that is informative then that would also help.

Answer 1

I have updated this answer to split it into 2 parts. The first part deals with a simple implementation, which limits the Json response size to ~1MB. The second part deals with more complex implementation that does not impose this limit on Json response size.

Part 1

What you want to do is chain Web activity with Stored procedure Activity.

This will allow you to pass the output from GetJson Web Activity onto the Stored procedure Activity.

Next you will want to add a parameter to your Stored procedure Activity so it can dynamically receive the chained output from the first step.

This should enable you to pass the information successfully.

Here is a Json representation of the Pipeline in question:

{
"name": "get-request-output-to-mssql-stored-procedure",
"properties": {
    "activities": [
        {
            "name": "GetJson",
            "type": "WebActivity",
            "dependsOn": [],
            "policy": {
                "timeout": "7.00:00:00",
                "retry": 0,
                "retryIntervalInSeconds": 30,
                "secureOutput": false,
                "secureInput": false
            },
            "userProperties": [],
            "typeProperties": {
                "url": "https://jsonplaceholder.typicode.com/posts/1",
                "method": "GET"
            }
        },
        {
            "name": "Exec stored proc",
            "type": "SqlServerStoredProcedure",
            "dependsOn": [
                {
                    "activity": "GetJson",
                    "dependencyConditions": [
                        "Succeeded"
                    ]
                }
            ],
            "policy": {
                "timeout": "7.00:00:00",
                "retry": 0,
                "retryIntervalInSeconds": 30,
                "secureOutput": false,
                "secureInput": false
            },
            "userProperties": [],
            "typeProperties": {
                "storedProcedureParameters": {
                    "Json": {
                        "value": {
                            "value": "@activity('GetJson').output",
                            "type": "Expression"
                        },
                        "type": "String"
                    }
                }
            },
            "linkedServiceName": {
                "referenceName": "your-server-def",
                "type": "LinkedServiceReference"
            }
        }
    ],
    "annotations": []
}

}

Part 2

Instead of creating a Web Activity, we will use Lookup activity in combination with a Linked Service Definition and a Json Dataset definition.

You will need to create a Linked Service of type HTTP.

And configure it to use the URL you would like to get the Json response from:

You can then create a new Dataset (of type HTTP) which will use this Linked Service to get data.

And choose Json as its Format type:

You can then set the request URL, and set the schema (unless you need it) to None.

You can then create a Lookup activity which uses the Json dataset as the Source dataset, and set the Request method to GET.

Here is a Json representation of the Pipeline in question:

{
"name": "get-request-output-to-mssql-stored-procedure-2",
"properties": {
    "activities": [
        {
            "name": "Exec stored proc",
            "type": "SqlServerStoredProcedure",
            "dependsOn": [
                {
                    "activity": "RetrieveJson",
                    "dependencyConditions": [
                        "Succeeded"
                    ]
                }
            ],
            "policy": {
                "timeout": "7.00:00:00",
                "retry": 0,
                "retryIntervalInSeconds": 30,
                "secureOutput": false,
                "secureInput": false
            },
            "userProperties": [],
            "typeProperties": {
                "storedProcedureParameters": {
                    "Json": {
                        "value": {
                            "value": "@activity('RetrieveJson').output",
                            "type": "Expression"
                        },
                        "type": "String"
                    }
                }
            },
            "linkedServiceName": {
                "referenceName": "ASD_SINGLE",
                "type": "LinkedServiceReference"
            }
        },
        {
            "name": "RetrieveJson",
            "type": "Lookup",
            "dependsOn": [],
            "policy": {
                "timeout": "7.00:00:00",
                "retry": 0,
                "retryIntervalInSeconds": 30,
                "secureOutput": false,
                "secureInput": false
            },
            "userProperties": [],
            "typeProperties": {
                "source": {
                    "type": "JsonSource",
                    "storeSettings": {
                        "type": "HttpReadSettings",
                        "requestMethod": "GET"
                    },
                    "formatSettings": {
                        "type": "JsonReadSettings"
                    }
                },
                "dataset": {
                    "referenceName": "JsonDataset1",
                    "type": "DatasetReference"
                },
                "firstRowOnly": false
            }
        }
    ],
    "annotations": []
}

}