如何在Spark中比较两个rdds？

Question

I have loaded 2 csv files into two spark RDD's, one containing country codes and second containing tweet data. I am trying find the following:

1. how many different countries are mentioned in the tweets?
2. compute the total number of times any country is mentioned.

Code:

country_lines = sc.textFile('country-data.csv')
words = country_lines.flatMap( lambda country_lines: country_lines.split(" )")
country_tuples = words.map(lambda word : (word, 1))
countryDF = sqlContext.createDataFrame(country_tuples, ["country" , "code"])

tweets = sc.textFile("tweet_data.csv")

I am trying to find how many time each country in the CountryDF occurs in the tweets csv (there is only column with the tweet text).

country_DF looks like this:

Afghanistan  AFG
Albania  ALB
Algeria  ALG
American Samoa   ASA
Andorra  AND

How do I count the occurrence of each country in the tweets pyspark rdd using python?

Answer 1

您可以在tweet.csv中将tweetDF分组以获取每个国家的数量，然后与countryDF一起获得该数量。

df =  tweetDF.groupby("CountryName").count().join(countryDF,["CountryName"])