AWS s3 object upload to google cloud storage

Question

we are trying to migrate data from aws s3 to gcp storage. we tried transfer job in gcp and its working fine. So we wanted to achieve that programmatically with aws lambda since we have dependencies on aws. When i tried importing google.cloud module I am getting this error lambda cloudwatch logs Here is my code:

import os
import logging
import boto3
#from StringIO import StringIO
from google.cloud import storage
#import google-cloud-storage

# Setup logging
LOG = logging.getLogger(__name__)
LOG.setLevel(os.environ.get('LOG_LEVEL', 'INFO'))

GCS_BUCKET_NAME=os.environ['GCS_BUCKET_NAME']
S3 = boto3.client('s3')


def lambda_handler(event, context):
    try:
        l_t_bucketKey = _getKeys(event)

        # Create google client
        storage_client = storage.Client()
        gcs_bucket = storage_client.get_bucket(os.environ['GCS_BUCKET_NAME'])

        LOG.debug('About to copy %d files', len(l_t_bucketKey))
        for bucket, key in l_t_bucketKey:
            try:
                inFileObj = StringIO()
                S3.download_fileobj(
                    Bucket=bucket,
                    Key=key,
                    Fileobj=inFileObj
                )
                blob = gcs_bucket.blob(key)
                blob.upload_from_file(inFileObj, rewind=True)  # seek(0) before reading file obj

                LOG.info('Copied s3://%s/%s to gcs://%s/%s', bucket, key, GCS_BUCKET_NAME, key)
            except:
                LOG.exception('Error copying file: {k}'.format(k=key))
        return 'SUCCESS'
    except Exception as e:
        LOG.exception("Lambda function failed:")
        return 'ERROR'


def _getKeys(d_event):
    """
    Extracts (bucket, key) from event
    :param d_event: Event dict
    :return: List of tuples (bucket, key)
    """
    l_t_bucketKey = []
    if d_event:
        if 'Records' in d_event and d_event['Records']:
            for d_record in d_event['Records']:
                try:
                    bucket = d_record['s3']['bucket']['name']
                    key = d_record['s3']['object']['key']
                    l_t_bucketKey.append((bucket, key))
                except:
                    LOG.warn('Error extracting bucket and key from event')
    return l_t_bucketKey

And I downloaded google-cloud-storage module from pypi website and imported that in aws lambda layer. Please help in providing me the best link for downloading this module.

Answer 1

Google Storage Bucket can be used with S3 APIs, so you can just use it in your Lambda functions without any extra GCP libraries.

    source_client = boto3.client(
        's3',
        endpoint_url='https://storage.googleapis.com',
        aws_access_key_id=os.environ['GCP_KEY'],
        aws_secret_access_key=os.environ['GCP_SECRET']

To get access_key and secret - go to the GS bucket settings -> Interoperability -> Access keys for your user account -> Create a key