Connect Spark 2.X to ElasticSearch 2.X

Question

I'm working with Spark and ElasticSearch and can't find how to use Spark 2.X with ElasticSearch 2.x. ElasticSearch Spark libs support only Spark 1.6 for ES 2.X and supports Spark 2 for ES 5.alpha What do you use to connect Spark to ElasticSearch?

Answer 1

Please have a look at this.

Apache Spark support

Apache Spark is computing framework that is not tied to Map/Reduce itself however it does integrate with Hadoop, mainly to HDFS. elasticsearch-hadoop allows Elasticsearch to be used in Spark in two ways: through the dedicated support available since 2.1 or through the Map/Reduce bridge since 2.0. Spark 2.0 is supported in elasticsearch-hadoop since version 5.0

Hope that helps!

Also have a loot at ElasticSearch Spark Error Where user is using spark 2.0 with lower version(Spark 2.0.0 ElasticSearch-Spark 2.3.4) of Elastic search and @Crackerman was able to resolve his issue(other issue not version related stuff)

Answer 2

i am using spark 2.2 read elasticsearch 5.5.2 is work

scalaVersion := "2.11.10"
val spark = "2.2.0"
val es = "5.5.2"
libraryDependencies ++= Seq(
  "org.elasticsearch" % "elasticsearch-spark-20_2.11" % es,
  "org.apache.spark" % "spark-core_2.11" % spark % "provided" ,
  "org.apache.spark" % "spark-sql_2.11" % spark % "provided",
  "org.apache.spark" % "spark-hive_2.11" % spark % "provided"
)
 val sparkSession = SparkSession.builder()
    .config("es.nodes",locahost)
    .config("es.port","9200")
    .appName("ES")
    .master("local[*]")
    .enableHiveSupport()
    .getOrCreate()

   sparkSession.read.format("org.elasticsearch.spark.sql")
  .option("es.query", "?q=id:(123)") 
  .load("es_index/es_type")

Answer 3

Yes it supports it.

You can find org.elasticsearch libraries here: https://mvnrepository.com/artifact/org.elasticsearch

https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-spark-20

Also, take care to match the same scala version (2.11 or 2.10). For instance, Ambari provides Spark2 - scala 2.11.