如何将 pyspark 数据框列转换为字典

Question

I have a dataframe with 2 columns.

Col1: String, Col2:String.

I want to create a dict like {'col1':'col2'} .

For example, the below csv data:

var1,InternalCampaignCode
var2,DownloadFileName
var3,ExternalCampaignCode

has to become:

{'var1':'InternalCampaignCode','var2':'DownloadFileName', ...}

The dataframe is having around 200 records.

Please let me know how to achieve this.

Answer 1

The following should do the trick:

df_as_dict = map(lambda row: row.asDict(), df.collect())

---

Note that this is going to generate a list of dictionaries, where each dictionary represents a single record of your pyspark dataframe:

[
  {'Col1': 'var1', 'Col2': 'InternalCampaignCode'},
  {'Col1': 'var2', 'Col2': 'DownloadFileName'},
  {'Col1': 'var3', 'Col3': 'ExternalCampaignCode'},
]

Answer 2

You can do a dict comprehension:

result = {r[0]: r[1] for r in df.collect()}

which gives

{'var1': 'InternalCampaignCode', 'var2': 'DownloadFileName', 'var3': 'ExternalCampaignCode'}