使用pyspark将两个csv文件加入一个键值rdd

Question

I am trying to combine two csv files with nothing in common (no key is common) into a key-value paired rdd using pyspark

Lets say A.csv has

a
b
c

and B.csv has

1
2
3

is there an option in pyspark to get an rdd by joining these two, like this

a:1
b:2
c:3

of course the number of rows in both the csv files should match. Is this something that is easy in pyspark or should this be done in regular python first. That is, do a nested loop of both the files and then make a tuple of tuples like ((a,1),(b,2)...) and then pass this to parallelize.

Answer 1

Just a tool solution, showing the general principle, but not focusing on your specific data structures:

with file('A.csv','r') as f:
    a = f.read().split('\n')
with file('B.csv','r') as f:
    b = f.read().split('\n')
dic = dict(zip(a,b))

If you got more complex data structures, your should add a CSV parser (eg the csv module from the standard Python library)

Answer 2

I am writing this for people who may need this in the future. I am just modifying the code from @sciroccorics slightly

import os

with open("/dbfs/FileStore/tables/a.csv",'r') as f:
    a = f.read().split('\n')
with open("/dbfs/FileStore/tables/b.csv",'r') as f:
    b = f.read().split('\n')
tup = tuple(zip(a,b))
key_rdd = spark.sparkContext.parallelize(tup)

Notice the use of tuple(zip(a,b))