如何使用 VM 上的文件使用 Azure 数据工厂运行 python 脚本？

Question

I have a VM that contains files that I run a python script against. How could I make a data factory pipeline on my local machine that connects to the VM and runs the python script?

I have looked into using a custom batch activity, but I'm not sure how this can connect to the VM.

Thanks a lot

Answer 1

Yes, You can Upload the python script into Azure blob storage , First connect Blob storage to VM and also we can use AzCopy to upload files into Azure Blob Storage.

Follow these steps for Custom batch activity and use Below Reference in which we have detailed information about:

1. Create the Azure Batch Account

2. Create the Azure Pool. You can follow this link for Azure Pool creation https://learn.microsoft.com/en-us/azure/batch/quick-create-portal

3. Upload the python script in the Azure blob storage

4. Add the custom activity in the Azure Data factory Pipeline and configure to use the Azure batch pool and run the python script Default output of any batch activity is stored in storage account under output/stdout.txt and if any program failure happens, it will get stored in output/stderr.txt

Reference:

https://azurelib.com/custom-activity-azure-data-factory/

https://medium.com/@ashish.kats/custom-batch-activity-in-azure-data-factory-110d8c1c957b