Copying data from on premises FTP to Azure storage with gateway/VPN and Logic apps/Functions

Question

We have data on premises ftp server that I need to copy on schedule to Azure database. The ftp servers are behind NAT and can be reached from our on premises network. I have tried several things to copy the data:

1) Logic apps have ftp transfer option, but they don't support Data Management Gateway? How could I integrate VPN from onprem<->Azure with Logic Apps/Functions?

2) Azure Data Factory. This is not optimal, because I want to add ftp sources programmatically, but I tried this option to see if it works. I wasn't able to reach local ftp servers with Data Management Gateway. Probably because the NAT.

All suggestions how I should proceed with this would be greatly appreciated. Let me know if you need more info.

The scenario is:

-----------------------------------------
| Tosibox VPN     |   Azure VPN/Gateway |
-----------------------------------------
| ftp server |   On prem    |   Azure   |
-----------------------------------------
| 10.10.10.* | 192.168.75.* | 10.10.0.* |
-----------------------------------------

EDIT:

I still claim that Logic apps ftp-plugin doesn't work with on premises Data management gateway, so that doesn't go for an answer.

I tried to mount the ftp-server to my local windows machine and share that with file connector + data management gateway. However ftp-server mounts as a network drive and doesn't give local drive letter where file connector could connect out of the box. There is some hacks how one gets the drive letter and I was able to do that, but it didn't work other than the root ftp-folder.

I any case this would feel like unscalable hack, because I have several ftp-sources and in future might have alot more. I don't think there is easy way to get Logic apps/functions work with on premises ftp at the moment.

I think I am going to try to make VPN-gateway to on premises and copy the files using virtual machine/Web job.

I understand that FTP is boring and legacy, but still should be better supported from Microsoft imo. One more thing. In case you wonder why FTP and not something else, these are building automation controllers which support only FTP as output. So this is sort of IoT-case...

Answer 1

Why not use the File-connector which is supported by the Data Management Gateway and read from the underlying FTP-folders instead of using the FTP-connector?

Or are the FTP root folders not accessible from the machine on which you installed the gateway (by for example a fileshare) ?

FTP is not supported by Data Management Gateway, so if you want to connect to the FTP, it will need to be publicly available for your application to be able to make the FTP-connection.

Answer 2

With Azure Logic Apps, you can use on-premises data gateway to connect to on-premises systems, it supports FTP, or even just plain file share. Depending on which database you want to insert the data into, Logic Apps most likely also supports it - Cosmos Db, SQL, etc., and all are possible via designer without you having to write any code.

Answer 3

Here's more details about connecting to on-premises data sources from logic apps:

• Access data sources on premises from logic apps
• Install the on-premises data gateway for Azure Logic Apps

Answer 4

We used Function and VPN gateway in the end to fetch the FTP data. To use Function with VNET integration one needs to create the function with App Service Plan.