AWS Lambda 在 S3 存储桶上触发 Neptune“无法为源启动新负载”

Question

I wrote a lambda function which will trigger python code when a create event happens in S3 and Python script is supposed to read files from S3 and post them to Neptune server.

When I test that, I am getting the following error.

{
 "requestId":"xxxxxxxx-1234-5678-9012-xxxxxxxxxxxxx",  
 "code":"ThrottlingException",  
 "detailedMessage":"Failed to start new load for the source s3://my-s3-url/file.ttl. 
    Max concurrent load limit breached. Limit is 1"
}

Code:

def lambda_handler(event, context):
    file_names = ["a.ttl", "b.ttl", "c.ttl"]
    source_url = "s3://my-s3.aws.com/"
    role = "my-role"
    neptune_url = "https://my-neptune-server.aws.com/loader"
    headers = {"Content-Type": "application/json"}

    for name in file_names:
        file = source_url+name
        data = {"source": file, "iamRoleArn": role, "region": "region-1", "failOnError": "FALSE", "format": "turtle"}
        loop = asyncio.get_event_loop()
        task = loop.create_task(post_async(neptune_url, json.dumps(data), headers))
        resp = loop.run_until_complete(task)
        print(resp)

async def post_async(neptune_url, data, headers):
    async with aiohttp.ClientSession() as session:
        async with session.post(neptune_url, data=data, headers=headers) as response:
            result = await response.text()
            return result

I tried both Synchronous and Asynchronous ways. I am getting limited documentation in the web. Can some one point me right direction?

Answer 1

According to documentation: Max concurrent load limit is one

So probably you need to introduce some queue in your upload process. Or it may be

• SQS
• preserve loading queue with file names in parameter store and recursively execute lambda function for deploying files one by one
• or you own idea... :)

Answer 2

According to documentation https://docs.aws.amazon.com/neptune/latest/userguide/load-api-reference-load.html ,

Max concurrent load limit breached  (HTTP 400)

If a load request is submitted without "queueRequest" : "TRUE", 
and a load job is currently running, the request will fail with this error

you can add the following field to your payload,

data = {
  "source": file, 
  "iamRoleArn": role, 
  "region": "region-1", 
  "failOnError": "FALSE", 
  "format": "turtle", 

  "queueRequest" : "TRUE"
}