将一个数据框列值传递给另一数据框过滤条件表达式+ Spark 1.5

Question

I have two input datasets First input dataset like as below :

year,make,model,comment,blank
"2012","Tesla","S","No comment",
1997,Ford,E350,"Go get one now they are going fast",
2015,Chevy,Volt

Second Input dataset :

TagId,condition
1997_cars,year = 1997 and model = 'E350'
2012_cars,year=2012 and model ='S'
2015_cars ,year=2015 and model = 'Volt'

Now my requirement is read first data set and based on the filtering condition in second dataset need to tag rows of first input dataset by introducing a new column TagId to first input data set so the expected should look like :

year,make,model,comment,blank,TagId
"2012","Tesla","S","No comment",2012_cars
1997,Ford,E350,"Go get one now they are going fast",1997_cars
2015,Chevy,Volt, ,2015_cars

I tried like :

val sqlContext = new SQLContext(sc)
val carsSchema = StructType(Seq(
    StructField("year", IntegerType, true),
    StructField("make", StringType, true),
    StructField("model", StringType, true),
    StructField("comment", StringType, true),
    StructField("blank", StringType, true)))

val carTagsSchema = StructType(Seq(
    StructField("TagId", StringType, true),
    StructField("condition", StringType, true)))


val dfcars = sqlContext.read.format("com.databricks.spark.csv").option("header", "true") .schema(carsSchema).load("/TestDivya/Spark/cars.csv")
val dftags = sqlContext.read.format("com.databricks.spark.csv").option("header", "true") .schema(carTagsSchema).load("/TestDivya/Spark/CarTags.csv")

val Amendeddf = dfcars.withColumn("TagId", dfcars("blank"))
val cdtnval = dftags.select("condition")
val df2=dfcars.filter(cdtnval)
<console>:35: error: overloaded method value filter with alternatives:
  (conditionExpr: String)org.apache.spark.sql.DataFrame <and>
  (condition: org.apache.spark.sql.Column)org.apache.spark.sql.DataFrame
 cannot be applied to (org.apache.spark.sql.DataFrame)
       val df2=dfcars.filter(cdtnval)

another way :

val col = dftags.col("TagId")
val finaldf = dfcars.withColumn("TagId", col)
org.apache.spark.sql.AnalysisException: resolved attribute(s) TagId#5 missing from comment#3,blank#4,model#2,make#1,year#0 in operator !Project [year#0,make#1,model#2,comment#3,blank#4,TagId#5 AS TagId#8];

finaldf.write.format("com.databricks.spark.csv").option("header", "true").save("/TestDivya/Spark/carswithtags.csv")

Would really appreciate if somebody give me pointers how can I pass the filter condition to filter function of dataframe. Or another solution . My apppologies for such a naive question as I am new to scala and Spark

Thanks

Answer 1

There is no simple solution to this. I think there are two general directions you can go with it:

• Collect the conditions ( dftags ) to a local list. Then go through it one by one, executing each on the cars ( dfcars ) as a filter . Use the results to get the desired output.

• Collect the conditions ( dftags ) to a local list. Implement the parsing and evaluation code for them yourself. Go through the cars ( dfcars ) once, evaluating the ruleset on each line in a map .

If you have a high number of conditions (so you cannot collect them) and a high number of cars, then the situation is very bad. You need to check every car against every condition, so this will be very inefficient. In this case you need to optimize the ruleset first, so it can be evaluated more efficiently. (A decision tree may be a nice solution.)