How do I transform the data set into a dictionary inside the repo. I am using pyspark within foundry

Question

I created a fusion sheet data to be synced to the data set. now, I want to use that data set for creating a dictionary in the repo. I am using pyspark in the repo. later I want to use that dictionary to be passed so that it populates descriptions as it is in Is there a tool available within Foundry that can automatically populate column descriptions? If so, what is it called? .

it would great if anyone can help me creating the dictionary from data set using pyspark in the repo.

Answer 1

The following code would convert your pyspark dataframe into a list of dictionaries:

fusion_rows = map(lambda row: row.asDict(), fusion_df.collect())

However, in your particular case, you can use the following snippet:

col_descriptions = {row["column_name"]: row["description"] for row in fusion_df.collect()}
my_output.write_dataframe(
    my_input.dataframe(),
    column_descriptions=col_descriptions
)

Assuming your Fusion sheet would look like this:

+------------+------------------+
| column_name|       description|
+------------+------------------+
|       col_A| description for A|
|       col_B| description for B|
+------------+------------------+