AWS ClientError when using Lambda and S3 to insert data to bucket

Question

I am trying to put a json blob into an S3 bucket using lambda and I am getting the following error when looking at the cloudwatch logs

[ERROR] ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
Traceback (most recent call last):
  File "/var/task/main.py", line 147, in lambda_handler
    save_articles_and_comments(sub, submissions)
  File "/var/task/main.py", line 125, in save_articles_and_comments
    object.put(Body=json.dumps(articles))
  File "/var/task/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "/var/task/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/var/task/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/task/botocore/client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)

All of the block public access settings are set to "off" and the bucket name in the code is the same as in S3. This is the code that is putting the json blob into their respeectable folders in my S3 bucket and the lambda handler

def save_articles_and_comments(sub, submissions):
    """
    """
    s3 = boto3.resource('s3')
    now = dt.datetime.utcnow()
    formatted_date = now.strftime("%Y-%m-%d-%H-%M-%S")

    articles, comments = data_for_subreddit(submissions)
    print("Number of articles, comments {}, {}".format(len(articles), len(comments)))
    articles_name = 'articles/' + formatted_date + '_' + sub + '_articles.json'
    comments_name = 'comments/' + formatted_date + '_' + sub + '_comments.json'
    object = s3.Object('diegos-reddit-bucket', articles_name)
    object.put(Body=json.dumps(articles))
    print("Finished writing articles to {}".format(articles_name))

    object = s3.Object('diegos-reddit-bucket', comments_name)
    object.put(Body=json.dumps(comments))
    print("Finished writing comments to {}".format(comments_name))


def lambda_handler(x, y):
    """
    """
    import time
    import random
    idx = random.randint(0, len(SUBREDDITS)-1)
    start = time.time()
    assert PRAW_KEY is not None
    sub = SUBREDDITS[idx]
    red = reddit_instance()
    subreddit = red.subreddit(sub)

    print("Pulling posts from {}, {}.".format(sub, "hot"))
    submissions = subreddit.hot()
    save_articles_and_comments(sub, submissions)
    print("="*50)

    print("Pulling posts from {}, {}.".format(sub, "new"))
    submissions = subreddit.new()
    save_articles_and_comments(sub, submissions)
    print("="*50)

    print("Pulling posts from {}, {}.".format(sub, "top"))
    submissions = subreddit.top()
    save_articles_and_comments(sub, submissions)
    print("="*50)

    print("Pulling posts from {}, {}.".format(sub, "rising"))
    submissions = subreddit.rising()
    save_articles_and_comments(sub, submissions)
    end = time.time()
    print("Elapsed time {}".format(end - start))

I do not see what is the problem in the code for me to getting said error. swapped out my lambda_handler function with a main to test locally. With the main it works, and writes to the S3 bucket and its respected folders. When I try and run via AWS Lambda, I get said error after the function is done pulling posts from the first subreddit and trying to put the json blob into the folder in the S3 bucket. This is what my output is supposed to look like

Pulling posts from StockMarket, hot.
Number of articles, comments 101, 909
Finished writing articles to articles/2020-06-03-02-48-44_StockMarket_articles.json
Finished writing comments to comments/2020-06-03-02-48-44_StockMarket_comments.json
==================================================
Pulling posts from StockMarket, new.
Number of articles, comments 101, 778
Finished writing articles to articles/2020-06-03-02-49-10_StockMarket_articles.json
Finished writing comments to comments/2020-06-03-02-49-10_StockMarket_comments.json
==================================================
Pulling posts from StockMarket, top.
Number of articles, comments 101, 5116
Finished writing articles to articles/2020-06-03-02-49-36_StockMarket_articles.json
Finished writing comments to comments/2020-06-03-02-49-36_StockMarket_comments.json
==================================================
Pulling posts from StockMarket, rising.
Number of articles, comments 24, 170
Finished writing articles to articles/2020-06-03-02-52-10_StockMarket_articles.json
Finished writing comments to comments/2020-06-03-02-52-10_StockMarket_comments.json
Elapsed time 215.6588649749756

Is there a problem in my code or is this a problem on the AWS side?

Answer 1

The problem occurs because you have no permissions to write objects to the bucket:

PutObject operation: Access Denied

To rectify the issue, have to look at lambda execution role : does it have permissions to write to S3? Also can inspect bucket policy.

With the main it works, and writes to the S3 bucket and its respected folders. When I try and run via AWS Lambda, I get said error

When you test locally, your code is using your own permissions (your IAM user) to write to S3. Thus it works. When you execute the code on lambda, your function does not use your permissions. Instead it uses permissions defined in lambda execution role .