Amazon S3 batch job through boto3 - request invalid

Question

I am trying to create an Amazon S3 Batch (not AWS Batch, this is S3 Batch operation) job via boto3 using S3Control, and gets the request invalid. I tired it through Amazon S3 batch operation through console which worked and I am trying to do it through boto3 to create the batch job. Attached is the code and error message.

import boto3
from datetime import datetime
from botocore.exceptions import ClientError


client = boto3.client('s3')
client_control = boto3.client('s3control')

client_control.create_job(
    AccountId='71652345619605',
    ConfirmationRequired=True,
    Operation={
        'S3PutObjectCopy': {
            'TargetResource': 'arn:aws:s3:::worm-all-bucket-inventory-embark',
            'StorageClass': 'DEEP_ARCHIVE',
            'ObjectLockLegalHoldStatus': 'OFF',
            'ObjectLockMode': 'COMPLIANCE',
            'ObjectLockRetainUntilDate': datetime(2115, 1, 1)
        }
    },
    Report={
        'Bucket': 's3-batch-ops-report',
        'Enabled': True,
        'ReportScope': 'AllTasks'
    },
    Manifest={
        'Spec': {
            'Format': 'S3BatchOperations_CSV_20180820',
            'Fields': [
                    'Bucket', 'Key', 'VersionId', 'TaskStatus', 'ErrorCode', 'HTTPStatusCode', 'ResultMessage'
            ]
        },
        'Location': {
            'ObjectArn': 'arn:aws:s3:::all-bucket-inventory-embark/agentwidget/2/2021-08-26T00-00Z/manifest.json',
            'ETag': 'fdc154f55282ef570cc9023068a72fc2'
        }
    },
    Priority=10,
    RoleArn='arn:aws:iam::716293619605:role/s3BatchOps'
)

Response:

Traceback (most recent call last):
  File "/Users/davefogo/Documents/Workspace/sprinters/AWS/AWS_CLI_Scripts/python-scripts/create_job.py", line 10, in <module>
    client_control.create_job(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CreateJob operation: Request invalid

Answer 1

Based on the documentation of boto3 , the field ObjectLockRetainUntilDate should be a datetime object, yours currently is an str object. You could try changing it from '2115, 01, 01' to datetime.datetime(2115, 1, 1)

Answer 2

You need to add the format in the report variable.

Report={
    'Bucket': 's3-batch-ops-report',
    'Enabled': True,
    'ReportScope': 'AllTasks',
    'Format': 'Report_CSV_20180820' <== this needs to be added. 
},

Additionally, ClientRequestToken is missing. According to docs ,

ClientRequestToken (string) -- [REQUIRED]

An idempotency token to ensure that you don't accidentally submit the same request twice. You can use any string up to the maximum length.

This field is autopopulated if not provided.

ClientRequestToken can be added in the following way -

import uuid
...

def lambda_handler(event, context):
    ...
    RequestToken = str(uuid.uuid4())
    ...
    response = clientControl.create_job(
      ...
      ClientRequestToken=RequestToken,
      ...
    )
    ....

Also you need to specify the region when you are creating the boto3 client as best practice.

client_control=boto3.client('s3control', region_name=[Region_Name])

I have documented my syntax of using create job in S3 Control Create Job Invalid Request Error