将数据从 SAP BW Open Hub 导入 Azure 数据湖时，如何在数据工厂中使用文件名前缀？

Question

I have a source of SAP BW Open Hub in data factory and a sink of Azure data lake gen2 and am using a copy activity to move the data.

I am attempting to transfer the data to the lake and split into numerous files, with 200000 rows per file. I would also like to be able to prefix all of the filenames eg 'cust_', so the files would be something along the lines of cust_1, cust_2, cust_3 etc.

This method only seems to be an issue when using SAP BW Open Hub as a source (it works fine when using SQL Server as a source. Please see the warning message below. After checking with out internal SAP BW team, they assure me that the data is in a tabular format, and no explicit partition is enabled, so there shouldn't be an issue.

When executing the copy activity, the files are transferred to the lake but the file name prefix setting is ignored, and the filenames instead are set automatically, as below (the name seems to be automatically made up of the SAP BW Open Hub table and the request ID):

Here is the source config:

All other properties on the other tabs are set to default and have been unchanged.

QUESTION : without using a data flow, is there any way to split the files when pulling from SAP BW Open Hub and also be able to dictate the filenames in the lake?

Answer 1

I tried to reproduce the issue and it works fine with a work around. Instead of splitting the data while copying from SAP BW to Azure data lake storage, you can just simply copy the entire exact data (without partition) into the Azure SQL Database. Please follow copy data from SAP Business warehouse by using azure data factory (make sure to use Azure SQL Database as sink).

Now the data is in you Azure SQL Database, you can now simply use the copy activity to copy the data to Azure data lake storage.

In source configuration, keep “Partition option” as None.

Source Config:

Sink config:

Output: