I have a pandas dataframe containing both numpy arrays and dictionaries:
results_df.head(1)
best_params cv_results
{'max_depth': 3, 'min_impurity_decrease': 0.2} {'mean_fit_time': [0.6320801575978597, 1.08473]}
I would like to be able to create a Spark Dataframe containing similar nested structures (they can be Spark objects if needed) and I tried:
spark.createDataFrame(results_df)
TypeError: not supported type: <class 'numpy.ndarray'>
One solution is to use a databricks supported module called koalas. The performance is also pretty good. For more info on koalas: https://koalas.readthedocs.io/en/latest/
import koalas as ks
spark_df = ks.from_pandas(pandas_df)
It's as simple as this in koalas!
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.