使用 Glue 向 AWS Elastic Search 输入数据

Question

I'm looking for a solution to insert data to AWS Elastic Search using AWS Glue python or pyspark. I have seen Boto3 SDK for Elastic Search but could not find any function to insert data into Elastic Search. Can anyone help me to find solution? Any useful links or code?

Answer 1

For aws glue you need to add an additional jar to the job.

1. Download the jar from https://repo1.maven.org/maven2/org/elasticsearch/elasticsearch-hadoop/7.8.0/elasticsearch-hadoop-7.8.0.jar
2. Save the jar on s3 and pass it to the glue job.
3. Now while saving the dataframe use following

df.write.format("org.elasticsearch.spark.sql").\
         option("es.resource", "index/document").\
         option("es.nodes", host).\
         option("es.port", port).\
         save()

If you are using aws managed elastic search, try setting this to true

option("es.nodes.wan.only", "true")

For more properties check https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html

NOTE The elasticsearch-spark connector is compatible with spark 2.3 only as it is prebuilt on scala 2.11 while spark 2.4 and spark 3.0 is prebuilt on scala 2.12