使用AWS Lambda函数从SNS主题触发器读取和复制S3库存数据

Question

I am a data analyst and new to AWS lambda functions. I have an s3 bucket where I store the Inventory data from our data-lake which is generated using Inventory feature under S3 Management tab.

So lets say the inventory data (reports) looks like this:

s3://my-bucket/allobjects/data/report-1.csv.gz
s3://my-bucket/allobjects/data/report-2.csv.gz
s3://my-bucket/allobjects/data/report-3.csv.gz

Regardless of the file contents, I have an Event setup for s3://my-bucket/allobjects/data/ which notifies an SNS topic during any event like GET or PUT. (I cant change this workflow due to strict governance)

Now, I am trying to create a Lambda Function with this SNS topic as a trigger and simply move the inventory-report files generated by the S3 Inventory feature under

s3://my-bucket/allobjects/data/ 

and repartition it as follows:

s3://my-object/allobjects/partitiondata/year=2019/month=01/day=29/report-1.csv.gz
s3://my-object/allobjects/partitiondata/year=2019/month=01/day=29/report-2.csv.gz
s3://my-object/allobjects/partitiondata/year=2019/month=01/day=29/report-3.csv.gz

How can I achieve this using the lambda function (node.js or python is fine) reading an SNS topic? Any help is appreciated.

I tried something like this based on some smaple code i found online but it didnt help.

console.log('Loading function');

var AWS = require('aws-sdk');  
AWS.config.region = 'us-east-1';

exports.handler = function(event, context) {  
console.log("\n\nLoading handler\n\n");
var sns = new AWS.SNS();

sns.publish({
    Message: 'File(s) uploaded successfully',
    TopicArn: 'arn:aws:sns:_my_ARN'
}, function(err, data) {
    if (err) {
        console.log(err.stack);
        return;
    }
    console.log('push sent');
    console.log(data);
    context.done(null, 'Function Finished!');  
});
};

Answer 1

The preferred method would be for the Amazon S3 Event to trigger the AWS Lambda function directly. But since you cannot alter this port, the flow would be:

• The Amazon S3 Event will send a message to an Amazon SNS topic.
• The AWS Lambda function is subscribed to the SNS topic, so it is triggered and receives the message from S3.
• The Lambda function extracts the Bucket and Key, then calls S3 to copy_object() to another location. (There is no move command. You will need to copy the object to a new bucket/key.)

The content of the event field is something like:

{
    "Records": [
        {
            "EventSource": "aws:sns",
            "EventVersion": "1.0",
            "EventSubscriptionArn": "...",
            "Sns": {
                "Type": "Notification",
                "MessageId": "1c3189f0-ffd3-53fb-b60b-dd3beeecf151",
                "TopicArn": "...",
                "Subject": "Amazon S3 Notification",
                "Message": "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"ap-southeast-2\",\"eventTime\":\"2019-01-30T02:42:07.129Z\",\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"AWS:AIDAIZCFQCOMZZZDASS6Q\"},\"requestParameters\":{\"sourceIPAddress\":\"54.1.1.1\"},\"responseElements\":{\"x-amz-request-id\":\"...",\"x-amz-id-2\":\"..."},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"...\",\"bucket\":{\"name\":\"stack-lake\",\"ownerIdentity\":{\"principalId\":\"...\"},\"arn\":\"arn:aws:s3:::stack-lake\"},\"object\":{\"key\":\"index.html\",\"size\":4378,\"eTag\":\"...\",\"sequencer\":\"...\"}}}]}",
                "Timestamp": "2019-01-30T02:42:07.212Z",
                "SignatureVersion": "1",
                "Signature": "...",
                "SigningCertUrl": "...",
                "UnsubscribeUrl": "...",
                "MessageAttributes": {}
            }
        }
    ]
}

Thus, the name of the uploaded Object needs to be extracted from the Message .

You could use code like this:

import json

def lambda_handler(event, context):

    for record1 in event['Records']:
        message = json.loads(record1['Sns']['Message'])

        for record2 in message['Records']:

            bucket = record2['s3']['bucket']['name'])
            key = record2['s3']['object']['key'])

            # Do something here with bucket and key

    return {
        'statusCode': 200,
        'body': json.dumps(event)
    }