appending to a file in azure data lake using data factory

Question

I am running into issues while appending data to a file in azure data lake, using data factory. I am trying to get data from MS Graph API and I am able to get the data and copy it for Single API calls where I push the response to the data lake, using "Copy Data" functionality, but if I want to do multiple calls where I get a response, and then append these responses to a single file, I am not sure how to do that. I don't think the "Copy data" activity is the right action for it.

One Example: API to get all groups in tenant:

https://graph.microsoft.com/v1.0/Groups/

API to get all members associated with a group:

https://graph.microsoft.com/v1.0/groups/"GroupID"/owners

"Group ID" comes from the top API call. **

I am able to build a loop and do the calls correctly. It is just appending the results of the second call where I am at a loss. I don't think creating a new file for each group would be the right approach.

Answer 1

I think you have a couple issues to contend with. First, standard Blobs don't support append operations. For that you'll need an AppendBlob. The second problem is that ADF doesn't support AppendBlob.

Here is a question where I discuss Copy activity and AppendBlob.

Here is another answer (not mine) with an interesting approach that uses the native REST API to append the blob.

Another option would be to let the process create a new file per run. After they've all been created, you could use DataFlow to collapse them into a single file.