使用Apache Camel基于S3事件通知传输S3文件

Question

I have an application (running on an EC2 instance) that parses new log files arriving in an S3 bucket. I have setup S3 notifications, on this bucket, to put a message in SQS queue on PUT request. The aim is to transfer a newly arrived log file from S3 to the EC2 instance.

I think Apache Camel would be an ideal candidate for this problem. I have following questions/observations on the solution and would greatly appreciate some feedback on that:

1. We can have a Camel route to read messages from SQS and then extract the object key from the JSON message. Is it possible to use this object key in another Camel route (using S3 component) that copies this object from S3 to local FS? From the looks of it seems that the Camel S3 component is only capable of processing new objects in a bucket since it takes only bucket name in the URI. Thus it might not be possible to copy a specific key from a S3 bucket. Also I am not sure if its possible to have dynamic URIs in Camel since object name would be different each time.

2. If Camel S3 component is indeed capable of processing new objects arriving in a S3 bucket then we dont need S3 notifications altogether. But this has a problem (since its stateless) that if for some reason this application goes down then it will miss the objects that arrived during downtime.

3. Another possibility is to use Camel for reading messages from SQS, process those messages and copy object from S3 using a normal Java class. But this kind of doesnt fully utilize the strengths of Apache Camel.

Answer 1

Having a SNS to decouple the file processing and file transfer is good. But do you really need this? It will add the complexity of your application. If you just want to processing the incoming files to the S3 bucket without miss, the simplest way is to create a bucket say "incoming" and another call "processed". Each time you application can scan the incoming bucket and process it, after done put the file to the processed bucket.

One thing you need to take care of is to flag the file under processing to avoid conflict. You can put special suffix to it.

Also AWS provide third way to do "Event Notifications" call " Lambda function". You can also take a look of that. It can enable some simple handler which do not need writing extra application yourself.