Deploying MPI application on Windows Azure

Question

There is an existing MPI scientific application written in C (for linux) that I would like to run on Windows Azure. Is that possible?

1. If possible, how to go about deploying the application?

2. Is it necessary to convert it to Microsoft MPI?

3. Is there a specific kind of Azure service I need to buy for this?

4. Is it necessary to write a managed wrapper to make it work?

Any suggestions/views/references would be very helpful.

PS I am new to Azure.

Answer 1

So you don't have to do anything too special to run MPI in Azure. Note thought that Azure nodes (unlike say Amazon's cluster compute instances) aren't really set up for this well; you don't necessarily have a high-speed network, and the nodes aren't necessarily network-ily near each other. So a tightly-coupled code probably doesn't work all that well, whereas a lot of people use Amazon or other services successfully for that sort of work.

At any rate, the "Microsoft" MPI that comes with HPC Server is just a re-branded and possibly slightly tweaked MPICH2 ; so it's a bog-standard MPI and that should cause no problems. Any MPI program that adheres to the MPI standard should be fine.

As to running a single MPI job, it's no harder than running an MPI job for the first time on a handful of (say) desktops; you have to make sure that your hostfile is set up correctly, and that MPI is set up to talk to the appropriate IP block. On Technet, they have an example of running the linpack benchmark on Azure nodes (including a link to what it takes to make sure your Azure workers are running HPC server ). They skip a couple steps because linpack is a pre-built binary you can download, but compiling with MPI is pretty straightforward.

As to deploying an application so that you can run it routinely (or have others run it), I'll defer to others about how best to do that.

Answer 2

Microsoft will soon be launching support for Linux VM's to be hosted in Azure . So you could conceivably build a Linux VM, install and configure your MPI app and push it up into Azure.

However, you should probably first consider what it is that you want to achieve. If your goal is to take advantage of some of Azure's features (eg multi-instance availability & scale, for example), then you'll need to augment your app with some additional code. Thanks to the open-source community and Microsoft's rapidly increasing support for open-source, you could build additional functionality in, for example:

• Node.js running on Linux or Windows
• Perl , Python , Ruby and/or PHP all running on Linux or Windows
• .NET code running in a Windows worker role or on Mono in your Linux VM

If you're most interested in taking advantage of your Azure service config, dynamically adding/removing worker roles, using azure storage, etc., then you're in luck - all of the above can be controlled via HTML/JSON REST interfaces which means anything that can talk HTTP can (with the correct credentials) configure, manage, monitor your services and store/retrieve data to/from storage.

HTH.

Answer 3

I know this thread is rather old! But in case anybody is still ending up here:

Azure has been meanwhile extended by special high performance instances, which support high throughput connections between the instances.

So if you look onto the A8 and A9 instances , they have, besides the default 10Gbps network interface, an additional network interface optimized for MPI applications.

32-Gbps back end, RDMA capable: Enables low-latency, high-throughput application communication between instances within a single cloud service or availability set. Reserved for MPI traffic only.

There is also quite some documentation for the setup.