pysparkintersection() 函数来比较数据帧

Question

Below is the code I have written to compare two dataframes and impose intersection function on them.

import os
from pyspark import SparkContext
sc = SparkContext("local", "Simple App")
from pyspark.sql import SQLContext, Row
sqlContext = SQLContext(sc)
from pyspark.sql import HiveContext
sqlContext = HiveContext(sc)

df = sqlContext.read.format("jdbc").option("url","jdbc:sqlserver://xxx:xxx").option("databaseName","xxx").option("driver","com.microsoft.sqlserver.jdbc.SQLServerDriver").option("dbtable","xxx").option("user","xxxx").option("password","xxxx").load()

df.registerTempTable("test")

df1= sqlContext.sql("select * from test where amitesh<= 300")
df2= sqlContext.sql("select * from test where amitesh <= 400")

df3= df1.intersection(df2)
df3.show()

I am getting below error:

AttributeError: 'DataFrame' object has no attribute 'intersection'

If my understanding is correct, intersection() is an inbuilt sub-function derived from python set function. So,

1) if I am trying to use it inside pyspark, do I need to import any special module inside my code, or it should work as in-built for pyspark as well?

2) To use this intersection() function, do we first need to convert df to rdd?

Please correct me wherever I am wrong. Can somebody give me a working example?

My motive is to get the common record from SQL server and move to HIVE. As of now, I am first trying to get my intersection function work and then start with the HIVE requirement that I can take care off if intersection() is working.

Answer 1

我让它为我工作，而不是intersect()，我使用了intersect()，它起作用了。