简体   繁体   中英

AWS - Moving data from one S3 bucket to another with CloudFormation

I'm trying to create a stack with CloudFormation. The stack needs to take some data files from a central S3 bucket and copy them to it's own "local" bucket.

I've written a lambda function to do this, and it works when I run it in the Lambda console with a test event (the test event uses the real central repository and successfully copies the file to a specified repo).

My current CloudFormation script does the following things:

  1. Creates the "local" S3 bucket
  2. Creates a role that the Lambda function can use to access the buckets
  3. Defines the Lambda function to move the specified file to the "local" bucket
  4. Defines some Custom resources to invoke the Lambda function.

It's at step 4 where it starts to go wrong - the Cloudformation execution seems to freeze here ( CREATE_IN_PROGESS ). Also, when I try to delete the stack, it seems to just get stuck on DELETE_IN_PROGRESS instead.

Here's how I'm invoking the Lambda function in the CloudFormation script:

"DataSync": {
    "Type": "Custom::S3DataSync",
    "Properties": {
        "ServiceToken": { "Fn::GetAtt" : [ "S3DataSync", "Arn" ] },
        "InputFile": "data/provided-as-ip-v6.json",
        "OutputFile": "data/data.json"
    }
},
"KeySync1": {
    "Type": "Custom::S3DataSync",
    "Properties": {
        "ServiceToken": { "Fn::GetAtt" : [ "S3DataSync", "Arn" ] },
        "InputFile": "keys/1/public_key.pem"
    }
},
"KeySync2": {
    "Type": "Custom::S3DataSync",
    "Properties": {
        "ServiceToken": { "Fn::GetAtt" : [ "S3DataSync", "Arn" ] },
        "InputFile": "keys/2/public_key.pem"
    }
}

And the Lambda function itself:

exports.handler = function(event, context) {
    var buckets = {};
    buckets.in = {
        "Bucket":"central-data-repository",
        "Key":"sandbox" + "/" + event.ResourceProperties.InputFile
    };   
    buckets.out = {       
        "Bucket":"sandbox-data",       
        "Key":event.ResourceProperties.OutputFile || event.ResourceProperties.InputFile   
    };   

    var AWS = require('aws-sdk');   
    var S3 = new AWS.S3();   

    S3.getObject(buckets.in, function(err, data) {       
        if (err) {           
            console.log("Couldn't get file " + buckets.in.Key);           
            context.fail("Error getting file: " + err)       
        }       
        else {           
            buckets.out.Body = data.Body;           
            S3.putObject(buckets.out, function(err, data) {               
                if (err) {                   
                    console.log("Couln't write to S3 bucket " + buckets.out.Bucket);                   
                    context.fail("Error writing file: " + err);               
                }               
                else {                   
                    console.log("Successfully copied " + buckets.in.Key + " to " + buckets.out.Bucket + " at " + buckets.out.Key);                   
                    context.succeed();               
                }           
            });       
        }   
    });
}

Your Custom Resource function needs to send signals back to CloudFormation to indicate completion, status, and any returned values. You will see CREATE_IN_PROGRESS as the status in CloudFormation until you notify it that your function is complete.

The generic way of signaling CloudFormation is to post a response to a pre-signed S3 URL. But there is a cfn-response module to make this easier in Lambda functions. Interestingly, the two examples provided for Lambda-backed Custom Resources use different methods:

Yup, i did the same thing. We need to upload(PUT request) the status of our request.(Need to send the status as SUCCESS)

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM