Azure Data Factory v2 Data Transformation

Question

I am new to Azure Data Factory. And my question is, I have a requirement to move the data from an on-premise Oracle and on-premise SQL Server to a Blob storage. The data need to be transformed into JSON format. Each row as one JSON file. This will be moved to an Event Hub. How can I achieve this. Any suggestions.

Answer 1

The Data copy tool as part of the azure data factory is an option to copy on premises data to azure. the data copy tool comes with a configuration wizard where you do all the required steps like configuring the source, sink, integration pipeline etc. In the source you need to write a custom query to fetch data from the tables you require in json format.

In case of SQL server to select json you would use the options OPENJSON, FOR JSON AUTO to convert the rows to json. Supported in SQL 2016. For older versions you need to explore the options available. Worst case you can write a simple console app in C#/java to fetch the rows and then convert them to json file. And then you can upload the file to azure blob storage. If this is an one time activity this option should work and you may not require a data factory.

In case of ORACLE you can use the JSON_OBJECT function.

Answer 2

You could use lookup activity + foreach activity. And inside the foreach, there is a copy activity. Please reference this post. How to copy СosmosDb docs to Blob storage (each doc in single json file) with Azure Data Factory