![](/img/trans.png)
[英]Scala - How to pass a string value to a data frame filter (Spark-Shell)
[英]How to filter the data in spark-shell using scala?
我有以下数据,需要使用spark(scala)进行排序,这样,我只需要访问“ Walmart”而不是“ Bestbuy”的人员的ID。 商店可能是重复性的,因为一个人可以多次访问该商店。
输入数据:
ID,存储
1,沃尔玛
1,沃尔玛
1,百思买
2,目标
3,沃尔玛
4,百思买
预期产量:3,沃尔玛
我已经获得了使用dataFrames的输出,并在spark上下文上运行了SQL查询。 但是有没有办法在没有dataFrames的情况下使用groupByKey
/ reduceByKey
等来做到这一点。 有人可以帮我提供代码groupByKey
map-> groupByKey
,已经形成了ShuffleRDD
,我在过滤CompactBuffer
时遇到了困难!
我使用sqlContext
获得的代码如下:
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.createSchemaRDD
case class Person(id: Int, store: String)
val people = sc.textFile("examples/src/main/resources/people.txt")
.map(_.split(","))
.map(p => Person(p(1)trim.toInt, p(1)))
people.registerTempTable("people")
val result = sqlContext.sql("select id, store from people left semi join (select id from people where store in('Walmart','Bestbuy') group by id having count(distinct store)=1) sample on people.id=sample.id and people.url='Walmart'")
我现在正在尝试的代码是这样,但是在执行第三步后我被打断了:
val data = sc.textFile("examples/src/main/resources/people.txt")
.map(x=> (x.split(",")(0),x.split(",")(1)))
.filter(!_.filter("id"))
val dataGroup = data.groupByKey()
val dataFiltered = dataGroup.map{case (x,y) =>
val url = y.flatMap(x=> x.split(",")).toList
if (!url.contains("Bestbuy") && url.contains("Walmart")){
x.map(x=> (x,y))}}
如果我做dataFiltered.collect(),我得到Array [Any] = Array(Vector((3,Walmart)),(),())
请帮助我完成此步骤后如何提取输出
要过滤RDD,只需使用RDD.filter
:
val dataGroup = data.groupByKey()
val dataFiltered = dataGroup.filter {
// keep only lists that contain Walmart but do not contain Bestbuy:
case (x, y) => val l = y.toList; l.contains("Walmart") && !l.contains("Bestbuy")
}
dataFiltered.foreach(println) // prints: (3,CompactBuffer(Walmart))
// if you want to flatten this back to tuples of (id, store):
val result = dataFiltered.flatMap { case (id, stores) => stores.map(store => (id, store)) }
result.foreach(println) // prints: (3, Walmart)
我也尝试了另一种方法,它解决了
val data = sc.textFile("examples/src/main/resources/people.txt")
.filter(!_.filter("id"))
.map(x=> (x.split(",")(0),x.split(",")(1)))
data.cache()
val dataWalmart = data.filter{case (x,y) => y.contains("Walmart")}.distinct()
val dataBestbuy = data.filter{case (x,y) => y.contains("Bestbuy")}.distinct()
val result = dataWalmart.subtractByKey(dataBestbuy)
data.uncache()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.