将字典另存为 pyspark Dataframe 并加载它 - Python，Databricks

Question

I have a dictionary as follows:

my_dict = {'a':[12,15.2,52.1],'b':[2.5,2.4,5.2],'c':[1.2,5.3,12]}

I want to save this dictionary in Databricks in order for me not to obtain it every time I want to start working with it. Furthermore, I would like to know how to retrieve it and have it in its original form again.

I have tried doing the following:

from itertools import zip_longest 

column_names, data = zip(*my_dict.items())
spark.createDataFrame(zip_longest(*data), column_names).show()

and

column_names, data = zip(*dict_brands.items())

spark.createDataFrame(zip(*data), column_names).show()

However, I get the following error:

zip_longest argument #10342 must support iteration

I also do not know how to reload it or upload it. I tried with a sample dataframe (not the same one), as follows:

df.write.format("tfrecords").mode("overwrite").save('/data/tmp/my_df')

And the error is:

Attribute name "my_column" contains invalid character(s) among ",;{}()\n\t=". Please use alias to rename it.

Finally, in order to obtain it, I thought about:

my_df = spark.table("my_df") # Get table
df = my_df.toPandas() # Make pd dataframe

and then make it a dictionary, but maybe there is an easier way than making it a dataframe and then retrieving as dataframe and converting into dictionary back again.

I would also like to know the computational cost for the solutions, since the actual dataset is very large.

Answer 1

Here is my sample code for realizing your needs step by step.

1. Convert a dictionary to a Pandas dataframe

    my_dict = {'a':[12,15.2,52.1],'b':[2.5,2.4,5.2],'c':[1.2,5.3,12]} import pandas as pd pdf = pd.DataFrame(my_dict)

   

2. Convert a Pandas dataframe to a PySpark dataframe

    df = spark.createDataFrame(pdf)

   

3. To save a PySpark dataframe to a file using parquet format. Format tfrecords is not supported at here.

    df.write.format("parquet").mode("overwrite").save('/data/tmp/my_df')

   

4. To load the saved file above as a PySpark dataframe.

    df2 = spark.read.format("parquet").load('/data/tmp/my_df')

   

5. To convet a PySpark dataframe to a dictionary.

    my_dict2 = df2.toPandas().to_dict()

   

The computational cost of these code above is depended on the memory usage for your actual dataset.