AWS。 通过Glue将数据从S3写入Elasticsearch

Question

I have a question related to Glue. My current task it read data from S3 and write it to Elastisearch (on AWS). And I have to use Glue. Glue is supported read from S3 as source, but cannot use Elasticsearch as target. My question how can I write data from Glue to Elasticsearch with the least effort?

Answer 1

If you must use glue , You can simply leverage glue to write python code without leveraging spark. Configure a new job with the "type" as "python shell". Writing python code, you can now leverage boto3 library to access your files in S3:

s3 = boto3.client('s3')

After retrieving the you want to upload to elastic search, you can then use a post request to upload the document to elastic search:

r = requests.post(url, auth=awsauth, json=document, headers=headers)

Please see AWS documentation below in the Loading streaming Data into Amazon ES from Amazon S3 Section. "Loading Streaming Data into Amazon Elasticsearch Service"