How to host long running process into Azure Cloud?

Question

I have a C# console application which extracts 15GB FireBird database file on a server location to multiple files and loads the data from files to SQLServer database. The console application uses System.Threading.Tasks.Parallel class to perform parallel execution of the dataload from files to sqlserver database.

It is a weekly process and it takes 6 hours to complete.

What is best option to move this (console application) process to azure cloud - WebJob or WorkerRole or Any other cloud service ?

How to reduce the execution time (6 hrs) after moving to cloud ?

How to implement the suggested option ? Please provide pointers or code samples etc.

Your help in detail comments is very much appreciated.

Thanks

Bhanu.

Answer 1

let me give some thought on this question of yours

"What is best option to move this (console application) process to azure cloud - WebJob or WorkerRole or Any other cloud service ?"

First you can achieve the task with both WebJob and WorkerRole, but i would suggest you to go with WebJob.

PROS about WebJob is:

• Deployment time is quicker, you can turn your console app without any change into a continues running webjob within mintues ( https://azure.microsoft.com/en-us/documentation/articles/web-sites-create-web-jobs/ )

• Build in timer support, where WorkerRole you will need to handle on your own

• Fault tolerant, when your WebJob fail, there is built-in resume logic

Answer 2

You might want to check out Azure Functions. You pay only for the processing time you use and there doesn't appear to be a maximum run time (unlike AWS Lambda).

They can be set up on a schedule or kicked off from other events.

If you are already doing work in parallel you could break out some of the parallel tasks into separate azure functions. Aside from that, how to speed things up would require specific knowledge of what you are trying to accomplish.

In the past when I've tried to speed up work like this, I would start by spitting out log messages during the processing that contain the current time or that calculate the duration (using the StopWatch class). Then find out which areas can be improved. The slowness may also be due to slowdown on the SQL Server side. More investigation would be needed on your part. But the first step is always capturing metrics.

Since Azure Functions can scale out horizontally, you might want to first break out the data from the files into smaller chunks and let the functions handle each chunk. Then spin up multiple parallel processing of those chunks. Be sure not to spin up more than your SQL Server can handle.