Amazon EC2 Windows AMI with shared S3 storage

Question

I've currently got a base Windows 2008 Server AMI that I created on Amazon EC2. I use it to create 20-30 EBS-based EC2 instances at a time for processing large amounts of data into PDFs for a client. However, once the data processing is complete, I have to manually connect to each machine and copy off the files. This takes a lot of time and effort, and so I'm trying to figure out the best way to use S3 as a centralised storage for the outputted PDF files.

I've seen a number of third party (commercial) utilities that can map S3 buckets to drives within Windows, but is there a better, more sensible way to achieve what I want? Having not used S3 before, only EC2, I'm not sure of what options are available, and I've not been able to find anything online addressing the issue of using S3 as centralised storage for multiple EC2 Windows instances.

Update : Thanks for suggestions of command line tools for using S3. Was hoping for something a little more integrated and less ad-hoc. Seeing as EC2 is closely related to S3 (S3 used to be the default storage mechanism for AMIs, etc), that there might be something neater/easier I could do. Perhaps even around Private Cloud Networks and EC2 backed S3 servers, etc, or something (an area I know nothing about). No other ideas?

Answer 1

I'd probably look for a command line tool. A quick search on Google lead me to a .Net tool:

http://s3.codeplex.com/

And a Java one:

http://www.beaconhill.com/opensource/s3cp.html

I'm sure there are others out there as well.

Answer 2

您可以将EC2实例与通过samba导出的EBS一起使用，它可以充当Windows实例可以映射的集中存储？

Answer 3

this sounds very much like a hadoop/Amazon MapReduce job to me. Unfortunately, hadoop is best deployed on Linux:

Hadoop on windows server

I assume the software you use for pdf-processing is Windows only? If this is not the case, I'd seriously consider porting your solution to Linux.