[英]elasticsearch-spark indexing error : Cannot handle type Map within Map using ScalaValueWriter
我正在尝试使用带有spark-1.3.1的elasticsearch-spark-2.1.0在Elasticsearch中建立数据索引,但是出现以下错误:
org.elasticsearch.hadoop.serialization.EsHadoopSerializationException: Cannot handle type [class scala.collection.immutable.Map$Map3] within type [class scala.collection.immutable.Map$Map4], instance [Map(word -> ..., pos -> ...)] within instance [Map(page_title -> ..., full -> ..., tokens -> [Lscala.collection.immutable.Map;@1efb3e9)] using writer [org.elasticsearch.spark.serialization.ScalaValueWriter@200c86fd]
这是我为Spark RDD编制索引的代码。
val spark = new SparkContext(...)
val filesRDD = spark.wholeTextFiles("hdfs://" + source_dir + "/*", 200)
// val sentenceList: RDD[Map[String, Object with Serializable { .. }]]
val sentenceList = filesRDD.flatMap(file => ...)
.flatMap { page =>
page.sentences.map { sentence =>
Map("page_title" -> page.title,
"full" -> sentence.map(_.word).mkString(" "),
"tokens" -> sentence.map { t =>
Map("word" -> t.word, "pos" -> t.pos)
}.toArray)
}
}
EsSpark.saveToEs(sentenceList, ES_RESOURCE)
为什么我不能在地图中索引地图,如何解决? 谢谢。
我终于解决了问题。
我只是删除了Map中的.toArray调用。 似乎库无法解析它。
结果图为:
Map("page_title" -> page.title,
"full" -> sentence.map(_.word).mkString(" "),
"tokens" -> sentence.map { t =>
Map("word" -> t.word, "pos" -> t.pos)
})
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.